I found an answer at this reddit link:
Here's the relevant part:
Damage = Power * Proficiency * Defence * Protego
Power = BasicEnemyPower * (1 + 0.0816 * Level) * (1 - WeakeningHex)
Proficiency = 1 + (BasicEnemyProficiencyPower * (1 + 0.01 * Level) - DeficiencyDefence)
ProficiencyPower is also scaled with level. That's why DeficiencyDefence > 0.5 is still useful.
Defence = 1 - (Defence + ProfessorShield - (EnemyDefenceBreach - ConfusionHex))
Protego = 1 - ProtegoPower
No comment in the article about how much EnemyDefenseBreach might be, but assuming that number is zero, a character with 44% defense and 45% protego power would take 100*(1-.44)*(1-.45)=30.8 damage from a 100-damage monster.
With a 30-point professor shield, that goes to 100*(1-.74)*(1-.45)=14.3 damage from a 100-damage monster.
The reddit link seems to imply that monsters start at 0.5 proficiency power at level 0, and they scale to 1.5 proficiency power at level 100. So with 50% deficiency defense, damage would be unchanged for a level-0 monster and would be doubled for a level-100 monster.
Although taking less damage is nice, it won't actually get you much farther in the tower. The tower is a race to kill all the monsters before the time runs out. In a large group, if you go down a magizoologist will just bring you back. To make progress in the tower, focus on damage output.