As noted by Arcane, "speed" in this context means the total number of turns it takes to complete all of the battles. As noted on the wiki, the way to get an A is to beat all of the battles in 28 turns or less:
Speed Rank Multiplier Total Turns
A 1.5x Turns ≤ 28
B 1.4x 29-31
C 1.3x 32-35
D 1.2x 36-39
E 1.1x Turns ≥ 40
This number does not appear to be affected by the number of maps, so it's much harder to get an A rank on the harder difficulties. With only 3 maps, that's an average of ~9 turns/map, while with 7 maps you only get 4 turns/map.
A suggestion that may help in this category is to use units with more mobility and range. Using cavalry units will help you get into position easier, and bow/dagger/magic users will be able to hit enemy units from a farther distance. A team of armored units are unlikely to clear these battles efficiently enough to get an A for speed.
Note that each team you lose loses you 5% bonus from the survival category, which stacks multiplicatively. An
A for survival with an
E rank for speed is 1.2*1.1 = 1.32 which is slightly worse than a
D survival and
C speed (1.05*1.3 = 1.365), so it'll be worth losing a few weaker teams for speed as long as you can at least get a