The short answer is find the maps from blacksmiths/armorers and follow the quests.
The long answer:
Go to Hattori, the Elven master weapon smith in Novigrad, and buy the new items that look like maps to the Skellige islands. They will be "The adjective notes of Hieronymus on the Witcher Elgar." Reading these notes should not only get you the quest associated with that particular map, but the base level quest as well. We're off to Kaer Morhen.
You may have noticed a small ruin south of the fast travel point at the fortress (if you look south, it's at the top of the giant cliff in the center of the screen). Make your way there to find a ruined watch tower. Geralt should comment on the tower. To the left just inside of the entrance, you'll see some scaffolding sticking out of the tower. Above head height on the external wall of the tower is a portal crystal. Take it and head inside the tower.
Climb up through the ruins to find a fancy chest. Inside you'll find a journal by this Hieronymus fellow, which will add the locations of the sword diagrams to your map. Go back to the bottom level of the tower, and insert the portal crystal into the device on the left. Use Aard on it and it's partner behind you and jump into the portal (and you will have to jump, take a leap of faith)
You'll find yourself in a cave, but don't bother breaking out the Cat. Immediately to your left you'll see a glowing corpse on the floor and an exit beyond. Get ready to fight the wraith who spawns from the body and collect the wolf armor diagrams.
The steel sword diagram is in the old bastion. If you haven't already been here, it's the large ruin visible on the map to the west of the main Kaer Morhen fortress. Wraiths haunt the area unless you've already taken care of them. The plans are on a mostly buried corpse surrounded by rats in a burned out building in the lower part of the courtyard.
Finally, the silver sword diagram is at the ruined watch tower near the north shore of the lake north of the fortress. A wraith will spawn from a body leaning near a door way, you really can't miss it.
That's just the base diagrams. The rest must be found from maps sold by various merchants around the world. Hopefully this helps.
I can now confirm the following merchants have maps:
Blacksmith in Lindenvale, Armorer at Kaer Trolde (main port in Skellige, but in the fortress), Blacksmith in Kaer Muire (home of crazy Jarl the Mad, again in the fortress), the armorer in Hierarch Square (due north of the bank) and Hattori, as previously mentioned.
