I think you're really asking two separate questions here. The first is, "What order do the various MR reduction / penetration effects follow?"
The answer to that is relatively simple.
% Based Armor/MR Reduction
% Based Armor/MR Penetration
Flat/Linear Armor/MR Penetration
So to answer the question as per the above, first Fiddlesticks' passive is applied to Ashe, reducing her Magic Resistance to 20. Next, the Abyssal Scepter effects is calculated, reducing Ashe's MR by 20, putting her at 0. Finally, the flat penetration from the Sorcerer's Shoes and the Haunting Guise are considered -- but Ashe is already at 0 MR, so there is no further MR to "penetrate".
Penetration can never bring a champion's defenses below zero.
So in the above scenario, spells would do 100% damage, so the full 100 damage in your example case.
To answer your second question, about the general case of negative magic resistances (possible on jungle creeps, and in some games featuring Karthas / Soraka / Kayle, etc), I bid you to look no further than this question.
The effects of negative defenses are exactly the same as the effects of positive defenses* - linear. In a nutshell, this means that an Abyssal Scepter will increase your spell DPS by the same amount regardless of whether the champion is starting with 100 MR or -100.
*the formula itself is different, because the armor calculation doesn't handle negative values well, but the end result is a continuation of Armor's linear scaling.