Ok i ran into this solution, in a few ways, some seem to have stumbled across it and came to a wrong conclusion, but what actually happens is this:
When equiping a faction specific armor if you do not remove it first(clicking on it) BEFORE storing it or putting on ANY other armor you can glitch the "faction assosication" even while not wearing it. so if you for instance wear Legion armor, then click on the spacesuit armor you can get "stuck" on wearing legion armor while not wearing it.
Solution: Equip the faction armor again and unequip it. If you do not know which faction armor you did this "bug" with, then equip all faction specific armors and unequip them one at a time(khan,legion,ncr, etc) Only one piece of each is needed to turn the setting to 0.
Secondary bug: This also can really screw up your faction standings when killing other things(several times Legion has been neutral to me, despite me killing them every chance i got).