After each battle in Pokemon sun and moon you get the opportunity to 'care' for your pokemon.
So far I can't tell if this makes any difference to just leaving them.
Are there any down sides to neglecting my pokemon.
No, there are no downsides, per se. There are just no upsides.
If you do choose to care for them, you will get more buffs as Pokemon have more Affection towards you. A full breakdown of these can be seen in this Q&A.
If you care for your Pokemon after battle, they will have a chance to do things like survive a hit that would have otherwise knocked them out, be more likely to get a critical hit, or dodge an attack. Thus, it's recommended to spend the ~30 seconds brushing your Pokemon after a battle if you receive the prompt.
The after-battle treatment screen is actually Pokemon Refresh. Everything you can do in Refresh, you can do in that screen, with the addition that if your pokemon have taken a certain level of damage or have been hit with a status condition, you can clean them up, dry them off, apply medicine, ect. to give them additional affection.
In Refresh, caring for your pokemon increases its affection for you (note that this a different stat than friendship). A pokemon starts with 0 hearts of affection for you, and this can increase up to a maximum of 5, with each heart of affection after the first giving you additional buffs (such as additional experience, better critical hit ratio, a chance of a focus sash-like effect, ect). After maxing out a pokemon's affection, subsequent attempts to pet or feed the pokemon do not have any additional effect (besides looking adorable); this applies to the after-battle care tools as well. The only exception is the medicine (which looks like a pair of tweezers holding a cotton ball). If your pokemon has a status ailment that persists outside of battle, such as sleep, paralysis, or poison, using the medicine on the pokemon will remove that ailment, saving you money on full heals/antidotes/ect.
Also important to note; if you plan on playing in matches against human opponents or participating in the Battle Tower, note that you can't use the buffs from Refresh.
One downside, depending on how you look at it, is that certain pokemon like golbat, Type: Null, and Pichu can't evolve without having high friendship. Sure, there may be other ways to evolve them, like getting massages every once or a while, or putting on a soothe bell and leveling them up, but the most efficient way to evolve them is to just pet them once in a while or give them rainbow pokebeans.
Another downside is if your pokemon has a status problem and you dont cure it, you have to use one of your items. And that's wasteful, considering that curing them in Pokemon Refresh is free and doesnt use any of your items.