To cast spells in Dungeon Master you have to combine runes.
The spells are found on scrolls throughout the game, but they use the names of the symbols rather than the symbols themselves. This is actually a copy protection feature, as the manual gives the names of each symbol in a chart. There's a comprehensive list here on the most comprehensive Dungeon Master website around.
The first scroll you find says cast FUL for a magic torch, if you look at the table on the link (or directly at the rune chart from the manual) then you'll see the symbol for FUL. The first glyph is the power, so initially you want to be using level 1 of the power or your spells will fail.
So you'd click the first rune, which looks like <<
, followed by the rune that looks like (\)
, and then click the box where both runes appear to cast it.
If you grab a copy of the manual you can read what each glyph signifies, which allows you to sometimes infer spells rather than using blind trial and error.
Note that there's also a windows port of the atari ST version of the game which you might prefer to using the DOS emulated version (since it will run natively). There's also a very active forum for the game if you get stuck or want to ask questions.