You have to get there from level four. There's an alcove in the corridor that leads from the area with the pressure-plate puzzles (the "Trials of Thought") to the four-way intersection where you find the scrolls after each test.
In this alcove there's a hidden switch, hit it and bolt into the intersection where some of the holes will be temporarily covered with trapdoors. In the south-east corner of the room (the only corner you can reach) there's yet another hidden switch that will shut of the center (return) portal, allowing you to drop down into the locked room below.
The timing on all of this is pretty tight, so you might want to start by looking around the intersection to make sure you know where you're going. Mess it up, and you'll fall down and will have to redo the Trials of Thought to get back.
If you're ok with spoilers, the alcove is number 7 on this map.