I know that pets and mounts won't be available in Sanctuary, but I was thinking about something like a worthless item related to the lore, or a physical modification that has no impact on stats. I could even be happy with a stone that says "Hi!" with Deckard Cain's voice every time it's clicked!
Yes, there are achievements that reward you with shapes, patterns, sigils and accents (images that appears on your banner).
Here is an example: The Art of Conversation achievement.
Or this one: Slice of Life achievement.
You can tell if an achievement has a reward with just a glance in the achievement window. The small star icon indicates the number of rewards the achievement has.
In the example below, the Scavenger achievement has 6 rewards because there are 8 levels of the achievement. All but the first and seventh level have rewards.