I fell down to the base and now I can't get back to where the Goron Elder was or anything
Its been a while since I played through this part of Oracle of Ages.
You may have to use Gale Seeds to get back up to the trees on the west side of Rolling Ridge.
However, remember that geography changes between past and present. Just because there isn't a path up in one time period doesn't mean there's no path up in the other time period.