Most of the momentos, when shown to The Stranger or other characters, will trigger some specific dialogue revealing more about the backstory of the game. Some of them, however, create monster pets that you can interact with at the Bastion. These monster pets are useful much later on in the game, when...
...The Bastion comes under attack. There, the monster pets will help you to take out some of the enemies and make the fight much easier.
Other than these, I believe the only other noteworthy momento is the Ceramic Pot, which allows you to access one of the special fragment-farming dungeons (Who Knows Where).