Short answer, you can quickly check compatibility by dragging the ammo/mag over the gun and it will turn red if incompatible (in a raid or not). And you can check the type quickly by inspecting items.
I'm not really sure how you would have loaded them wrong as you can only load bullets into the mag if it's compatible with the mag, and then you can only load compatible mags into a gun it's compatible with.
Note: as was mentioned in the comments by TimmyJim;
"Some mags, like the Izhmash AK magazine, can hold two types of ammo, in this case, 7.62x39mm, or .366 TKM ammo. However, there are no guns that can fire both. Thus if you load a mag with .366 TKM ammo into a gun that fires 7.62x39mm ammo, the mag will go into the gun, but you won't be able to chamber a round, and thus won't be able to fire."
Just be sure you have compatible gear before going into the raid, and when picking up things during the raid, double-check it before attempting to use it in a firefight. You'll generally just pick up on what's what and where things go the more you play the game.
So you shouldn't ever be allowed to load a 7.62x39 bullet into a 5.45x39 mag and then into a 7.62x39 gun.
Example: AK-105 (5.45x39mm gun) can take a magazine such as the 'Pmag 30 AK74 GEN M3 5.45x39' magazine. That magazine can be loaded with any 5.45 ammo such as BP or BT. Neither that ammo nor that gun could be loaded into an AK-103 (7.62x39mm gun) which would take a magazine such as the 'AK-103 30-round 7.62x39 magazine'.
In raid and in your stash it would not let you attach those incorrect magazines to those guns. You can check compatibility by inspecting the guns and magazine, or by quickly looking at the icons on guns.
Example: (From https://escapefromtarkov.gamepedia.com/Kalashnikov_AKS-74U_5.45x39)
The gun says the ammo type on it. If you inspected a magazine such as this
you would see it takes the same ammo type, and you wouldn't be able to load in a different type.
You can also look through the wiki to get ideas on items and their uses, as well as compatibility.