Each character has their own "Favorite Genuses" of familiars. If paired properly, the familiar will gain a small (I believe 10%) boost to their stats. This stat boost is only dependent on which character has the familiar. You don't have to level the familiar with that character, whenever they're paired up you'll get the bonus.
You can check the Genus and Favorite Genuses in the "Friends and Familiars" screen on the main menu.
Once you're on the "Friends and Familiars" screen, highlight a familiar and press square. In the upper right of each character's familiar lineup, you'll see three icons that represent the "Favorite Genuses" of each character.
The genus of a particular familiar is shown using the same icon system just below the type of familiar, in the upper right of the screen.
Not all familiars are created equal, and there are some real winners when it comes to the familiars within each genus. I've got my suggested lineup (and where to catch them) for the main party members in my Ni No Kuni quickstart guide.