"execute" command not working in command block?

Question

Here is what is in my command block:

execute @p[x=~5,y=~2,z=~17,r=5] ~ ~ ~ /spawnpoint @p[r=1] ~3 ~3 ~-5

I want it to find the nearest player (within 5 blocks) that is standing 5 blocks east, 2 blocks above and 17 blocks south of the command block, and I want it to set the spawnpoint of that player to 3 blocks east, 3 blocks up and 5 blocks north of himself.

The output from the command block is:

[13:50:02] Entity '@p[x=~5,y=~2,z=~17,r=5]' cannot be found

When I am standing directly on the block it should execute on. Any reason for this? Is my command wrong?

EDIT: Just replaced the command block with another command block with the command:

setblock ~5 ~2 ~17 wool

And it placed the block at my feet

EDIT2: Tried replacing the "@p" in the execute with "@a" - did nothing. Also tried replacing the "@p" in the spawnpoint with "@s" - did nothing.

Answer 1

You cannot use tilde (~) as part of a selector argument, unless you're using Bedrock Edition. Since you didn't state that you were using BE, I'm going to assume you're using the standard Java edition.

Your only real option in 1.12 is to use the actual coordinates that you're trying to point to, such as

execute @p[x=40,y=80,z=120,r=5] ~ ~ ~ spawnpoint @s ~3 ~3 ~-5

Or you can shift the execution point with execute, then you can even simplify the rest of the command:

execute @s ~5 ~2 ~17 spawnpoint @p[r=5] ~3 ~3 ~-5

Also notice that I'm using the @s selector for the first spawnpoint command. This will target the player found in the execute portion.
The same applies to the second execute: You want to execute relative to the player who executes the command, so you can use @s.

Answer 2

Instead of using [x=~5,y=~2,z=~17,r=5] use ~5 ~2 ~17 r=5. See if that works. other than that I don't know.