I want to do this because the mac client sucks and I haven't been able to find any other clients (except botting clients), which support filling the whole window with the applet, but not the whole screen.
Yes, it is still against the rules.
Any game-specific, third-party software that encourages contradiction of the previous statement, bypasses the normal navigation of our website, automatically requests pages from our website or which endangers user accounts are also regarded as detrimental to RuneScape and should not be used.
And under What type of third-party software is NOT allowed?:
Modified or replacement versions of our game applets. Reverse-engineering our game applets is strictly prohibited. Using a modified version of our game applets is not permitted for any reason. Third-party software that displays pages/content from our website(s), and that has specific features (i.e. toolkits) must not be used to access our website unless that software conforms with all of the other rules, and the following additional rules:
It must not hide or encourage the hiding of any of the adverts on our website. To avoid encouraging users to hide the advert above the game, the pane displaying pages from our website must default to at least 630 pixels high (if viewed on a screen with a resolution higher than 800x600), and it must not be possible to resize it to less height than that, i.e. the page must be displayed in such a way that both the game and advert are fully visible at the same time.
Based on their guidelines here in their Terms & Conditions, even if you're using said applet only for the fullscreen properties, it still violates their T&C because their terms state that their advertisements must remain visible.
Additionally, if the program you have does anything else that's prohibited, and falls under
Software that monitors, modifies or decrypts the communications between our game applets and our game worlds, your account may be at risk if the applet tries to make hooks that you're unaware of.