Seeds are afraid of the dark, they don´t like the dark and cold farmland when the light level of the block they are in is under 8. They will want to stay with you, so you can keep them warm and protected.
Seeds are so afraid of the dark that they may "pop off""pop off" the ground in an attempt to find a player who can take care of them.
Give them a night light if you want your seeds to feel safe in a cave: