Peaceful enterance depends on sneaking I'm afraid. It is however achievable quite easily and possibly doesn't require any skill in it. You do need some mechanics though. Also, due to how the game conversations work, you only need a single character to get through, the rest waiting safely by the exit.
Start with the south - there is an unguarded locked door there, ready for you to enter (key to which is findable somewhere in the accessible zone).
Immediately turn left - on the right the door leads to multiple hostile priests. They won't bother you or try to notice you, unless you approach it.
Again two choices, this time not so important - south has some items (and a key to the other door), north has the path you need to take.
There you enter a big room with a throne-table thing, and little stars on the floor pattern. On a star on the right there is a hidden switch to open the wall. At the far side of the room there is the only enemy that can possibly notice you if you're not careful - but it's nothing to really worry about, and you can hide behind the table waiting for opportune moment to move.
Once you're through the secret door, you encounter a group of pale elves - they are your target, and are not hostile.
It is worth noting, that to fulfill Rymrgard's will you will need to pass a resolve 17 check, or kill them anyway.