I notice that if I don't have a companion with me, I don't tend to gain any affection for them. For example, I've never gained/lost any affection from my ship droid, because I've never had him with me outside the ship.
But will the same apply if I send my current companion away on a crew task and perform some action they would then like or dislike?