You used the wrong entity name. A hanging itemframe is not called ItemFrame but item_frameitem_frame.
So, the following command would work:
/testfor @e[type=item_frame,r=10] {Item:{id:minecraft:writable_book,tag:{display:{Name:"Keys"}}}}
Please note: this command checks the name given to an item. You are essentially looking for a writable book that has been renamed using an anvilrenamed using an anvil. If your goal is however to check for a closed book with a title you should use the following command:
/testfor @e[type=item_frame,r=10] {Item:{id:minecraft:written_book,tag:[{title:Keys}][2]}}