In the Video settings for the video-game Smite, in the right half, above ANTI ALIASING and below SCREEN TYPE, there is a checkbox to "Use D3D11". What does it do exactly?
Enabling that option will cause the game to use Direct3D 11 as it's rendering API. By default, the game should use Direct3D 9.
Direct3D is a library used to access the graphics card and perform the hardware-accelerated rendering that basically every modern game requires.
D3D9 is an older version of the API, but up until recently was almost always the default because it would run on many more machines.
D3D11 is a more modern version of the library with newer features, and can provide improved visual fidelity. It may also affect performance (positively or negatively, depending on how the game uses it and what hardware you have).
To use D3D11 features you need Vista or better and a D3D11-capable graphics card. If you only have Vista or better, a game may still allow you to use D3D11 the API, but it won't be able to actually use any of the features of D3D11 (since you'd be using D3D11 with one of the 9.x feature levels).