The castle is NNW of Richard's Villa, Richard being the person who sent you on this fetch quest.
Follow the red line from Richard's Villa to reach it. The first time you visit, the castle gate is up; a button inside will let the gate down, so you can take the blue path to enter/exit instead.
Note for Link's Awakening DX players: you can get a photo with Richard by standing in front of the gate. If you lower it before you do this, you can't get it later!
"Woah," you say to yourself -- "there's no bridge there!" That is correct! You also need to have progressed the main trading quest far enough to have dropped by Sales' House o' Bananas on the beach. (For those who haven't, yet: Yoshi Doll (Trendy Game), Bow (house north of Rooster statue), Dog Food (small kennel next to Madam MeowMeow's house)). You'll recognize Sales' house because the trees out front actually have bananas on them.
After you've done this, you can give the bananas to Kiki the monkey, which will cause him to effectively build a bridge for you, letting you cross. (You'll also get the next trade quest item.)
This can only be done after completing Level 2 and returning Bow-Wow to Madam Meowmeow, though you can't talk to Richard in the first place unless you've ditched the dog. If you try to do it before, Kiki and Bow-Wow will get in a fight and Kiki will run off until you return. You can't otherwise enter Kanalet Castle, because you need the flippers to enter water, which you don't get until later.