When I maximize Minecraft on my Mac it just changes the size of the window, not the actual game. How can I get a higher resolution?
I believe that is 1280x800px.
You can't increase the resolution of the Minecraft world, but you can make sure the interface elements don't scale upward.
As of Minecraft Beta 1.5, you can set the scale of the UI by going into Options → Video settings... and clicking on GUI scale.
If you leave GUI scale set to auto, it'll continue to use the old GUI scaling: if your window exceeds 1280x720, Minecraft scales the interface elements up with the window.
There doesn't seem to be a configuration option for this (on Mac or otherwise), but this mod prevents the interface elements from scaling, even at large resolutions.
The textures are set to a certain resolution, so even though you're scaling your screen resolution, it will still look the same. Notch designed Minecraft to look like an old 320 x 200 pixelated game. Because the game is currently just played in windowed mode (even when maximixed), you're just wasting processing power cranking up your screen resolution. You can crank it down to 800 x 600 to increase your performance.
If you want nicer graphics/resolution, you can get a mod that uses higher res graphics. Using one of those then cranking up your screen res will make a difference.
Notch is eventually going to make the game like other commercial games, IE: when it starts, it goes into fullscreen mode and takes control of your computer's screen resolution. This will allow you to set the res within the game itself. But for now, we're all stuck with windowed mode, and have to mess with the screen res / apply graphic mods ourselves.