If you are on PC, you can use the console to add and remove perks.
Finding a perk id:
help [name of perk] 4
help Armsman 4
Removing a perk
player.removeperk [perk id]
player.removeperk 00079342
Adding a perk
player.addperk [perk id]
player.addperk 00079342
Notes:
- You can use the PageUp/PageDown keys to scroll the console.
- Use double quotes when searching for a perk with a space: "Steel Smithing" -yx
- The UESP has the perk id for all the skills. -Mark Trapp
- Console commands brand you as a cheater, preventing achievements. -Mark Trapp