In Skyrim, does heavy armour still penalize Sneak when Muffle is cast, or when wearing boots with the Muffle property?
"It is unclear whether or not the magnitude of the Muffle effect is significant. Based on the description of the Muffled Movement perk (Noise from armor is reduced 50%), a magnitude of 0.5 is equivalent to a 50% reduction. However, the magnitude of enchantments and spells is never displayed in-game, suggesting that the magnitude does not matter. If the magnitude is significant, most enchanted items have a magnitude of 0.5, implying they are only 50% effective"
According to this from the UESP, yes Heavy Armour does still penalize sneak when muffle is cast. Muffle only reduces detection by 50%
According to this answer from the 'Muffle Magnitude Difference' question, casting the Muffle spell gives 100% armor noise reduction. If you're sneaking, it won't matter if you're using Heavy or Light armor with the Muffle spell.
As to boots with the Muffle spell enchantment, the Nightingale Boots (level 1-18 version) gets 25% noise reduction, while the Nightingale Boots (level 32+ version) and Predator's Grace provides 100% noise reduction. Everything else gets 50% noise reduction. If you are using boots with < 100% noise reduction, Heavy Armor will still penalize sneaking.
Also (from this answer):