Is it possible for the auto-buy to use a custom order of items/abilities?
Yes, you can set custom orders of items and abilities independently of each other.
I'm not sure what you mean by
mix the God builder with auto-buy, but if you mean "mixing the ability-builder with the item-builder's auto-buy process", yes, you can set a custom order for your abilities to be upgraded using the Ability Builder in the Gods menu.
There's no real way to intermingle your item builds and your ability builds though, if that's what you mean, because these two purchasing systems aren't really related: The item build reacts to how much gold you have gained throughout the match, and the ability build reacts to how much XP you've gained. While it's true that most XP actions also give you gold, they may not always be accrued at the same rate in every match.
I am trying to make the transition between only using auto-buy and using some manual selections.
What it sounds like you want to do is have the game auto-buy some items, and you purchase the others. AFAIK, this is only possible in three ways: By interrupting an auto-build mid-game and finishing the build manually, by building a partial build and then buying late-game items manually, or by finishing an auto-build and then selling items to buy different ones.
Interrupting a build
You can only enable auto-buy at the character select screen before each match. However, you can start a match with auto-buy enabled, and when you feel the need to change it, you can enter the marketplace and disable auto-buy. You now have to buy the rest of your items manually, but with a lot of gods, the late-game items are the ones you use to adapt to the other team's strategies, so that can be a useful time to be using manual-buy.
Building a partial build
You can use a Custom build that is only a partial build, e.g. only the first 2 or 3 items. If your Bellona always gets boots, wards, Sprint, and a Jotunn, that can be her whole build. Auto-buy will get those items, and only those items, as fast as it can whenever you return to base. After they're bought, you can choose how to finish your build, either adapting to the way the other team has been playing (e.g. add CC reduction), or maybe to take advantage of how your team has leveled itself up (grab some late-game healing with the Gauntlet of Thebes).
Some characters, like the magnificent Khepri, don't get much gold early-game, or at all. I have often seen players use Starter Items, such as a Watcher's Gift, to help with early-game gold. After they finish buying a build of cheap, defensive items, they sell the Watcher's Gift, and replace it with a Tahuti (if they're manly), or more defense to become an ultra-tank. The same can be said for a lot of glass-cannon finishers who want to get early-game kills: grab a cheap starter item so you can survive the fray / get a kill, then when you're doing a lot more damage, sell that cheap low-power item for some more power or attack speed.
If so, can I specify a sub-set of items and allow the auto-buy to "fill the blanks"? For example, I want sentry wards but will be happy with other items being auto-picked.
Not really. If you use a Custom build with auto-buy, and you picked Sentry Wards, auto-buy will have bought you a Ward at the first chance it could, and it won't buy anything you didn't add to your build.
Keep in mind that this game is using slightly different versions from platform to platform, and receives constant updates and patches, some of which drastically change the UI, characters, and items. My advice for specific items may become irrelevant or backwards at any point, depending on the state of the game. The buying/selling system hasn't been changed much though, despite the many changes to the game itself.