You get an option to let someone into the Inquisition but can also deny them, which gives the flavor text that they will not return. Why would you not accept everyone offering?
- You may hate particular characters yourself. I hate Cassandra, for example, but alas, can't make her leave.
- Some characters, like Cole, are hated by many others, and it will make people happy if they go away.
- Or maybe you just don't enjoy having a poltergeist in your castle. You are an inquisitor goddamnit, you get to choose who you work with!
- Given how the story is unfolding so far, I would not be surprised if some of the characters betray you at some point, or just make things complicated.