There are some FEZ puzzles which I can't solve. That led me to think that, maybe those require a controller which can provide vibration feedback (I am currently playing with keyboard). Is my guess correct?
Yes and no. There are certain puzzles, generally referred to as tuning fork rooms, which ask you to solve a puzzle based on vibration feedback on the console version.
However, if you are playing on the PC version (which seems most likely, if you are using m+kb), then the rumble requirement is replaced by a low tone being played out of your left or right speaker or earphone.
So: if you're on console with a keyboard, then yes, this is probably your problem. If you're on PC with a keyboard, the problem is instead most likely that you have your volume set too low, or you are using an audio setup that makes it difficult or impossible to tell which 'direction' the audio clues are supposed to be coming from. (If you're using speakers instead of headphones, for example, you almost definitely won't be able to hear it - it's an incredibly subtle noise.)