Rivalry is applied as a modifier, not specifically to the Attack Stat or Base Damage. The effects of Rivalry and Swords Dance do essentially multiply together.
Consider the Pokemon Damage Formula
From Bulbapedia (emphasis mine)
other counts for things like held items, Abilities, field advantages, and whether the battle is a Double Battle or Triple Battle or not.
Rivalry falls under other as part of Modifier. It multiplies into the modifier value as either 1.25 or 0.75 based on gender.
With 3 three Swords Dance, you will have +6 Attack, which multiplies the attack term by 8/2 (so 4 times) per the table below.
Consider Level 50 Haxorus with 160 Attack with Giga Impact (Base Power 150), attacking something with 160 defense for simplicity.
I chose a move without STAB bonus for simplicity, assume there is no change due to type effectiveness, and we will assume no critical, and again for simplicity take the maximum value from the random term. This means that our Modifier is just other. And the only current other effect is Rivalry
The damage formula will be (if you didn't use Sword's Dance)
((2 x 50 + 10)/250 x (160/160) x 150 + 2) x Rivalry
(0.44 x 1 x 150 + 2) x Modifier = (66 + 2) x Modifier = 68 x Rivalry
Compare with the Max Attack (from Swords Dance)
((2 x 50 + 10)/250 x (8/2 x 160/160) x 150 + 2) x Rivalry
(0.44 x 4 x 150 + 2) x Modifier = (264 + 2) x Modifier = 266 x Rivalry
So max attack will give 266 x 1.25 = 332.5 against same gender, and 266 x 0.75 = 199.5 against opposite.