It seems indeed that there is a bug related to thieves guild leadership:
If the player completes the main story arc before doing the side and improvement quests for Vex and Delvin, Brynjolf may not interact with the player and instead repeat that he is busy and that he will talk to you later. Consequently, becoming the Guildmaster will be rendered impossible. This may be a part of a bug where Karliah disappears and does not appear in either the Nightingale Hall or the Twilight Sepulcher. She does, however, return to the Snow Veil Sanctum where you first encounter her.
Run outside when finishing your first interaction with Brynjolf. You need to run outside as quick as possible by going to the ladder and exiting. You should see the quest update pop up reading: "Started Under New Management". Afterward, go back inside and proceed to Brynjolf. To talk to Brynjolf, let him go to the middle of the Cistern first, and wait. He then should start talking to you, finishing the quest. If this does not resolve the issue, it may be worth it to try attacking them, paying the reparations, and trying again.
Another possible fix is to get the quest about fixing debt from the guy at the Riften stables. The player will be told to talk to Sapphire. Instead, Sapphire should be sitting on the bridge just inside Riften and when approached will say "I don't have any business with you." or "We'll talk after the ceremony." Attack her 3 times, and this should cause her to start attacking the player notifying them that "Under New Management" has been completed and that the player should talk to Brynjolf. Upon entering The Ragged Flagon, the player will be required to pay 1000 gold to Vex as reparations for killing a fellow Thieves Guild member. After doing so, speak with Brynjolf and the questline SHOULD be completed.
As a last resort, you can use the console to fix this issue:
setStage TGLeadership 0