Homeless are created when low-wealth sims do not have any source of income. This does apply only to low wealth sims as medium and high will simply move out and leave the city when they run out of money. Due to the random nature of resources being moved around your city (including the sims) it is possible that you can get homeless even with out an unemployment rate. Certain buildings may just simply not get a return worker to them as they found closer houses to return to of their wealth level.
Homeless will despawn over time but it can take a significant amount of time and they do not help a city in any manner to keep them around. Abandoned buildings will keep them in the city longer. They will also use up slots for visitors to your parks thus reducing their effectiveness.
To actively remove homeless from your town you need to keep your city rid of abandoned buildings and provide a bus system. It is the bus that the homeless will take to leave your town.
It is honestly better however to try and keep ahead of the curve. If you have unemployed within your city you have to remember that building more places for them to work is not the only solution to that situation. You can rezone some of the residential area for a park or the like reducing the population you have while boosting the overall happiness (not land value directly if you use the right parks) of your sims in the area.
Hope this helps.