It's not entirely clear what he means by this. The Elder Scrolls Wiki article on Mallory has this to say:
After the scene involving Mercer angering the Guild, Delvin will have a line of dialogue about betrayal, and that it's "like the Dark Brotherhood all over again." This could indicate he was in the Brotherhood and betrayed by them, or even by Astrid herself, though it leaves much to speculation because there will be no other discussion on the matter. Astrid will only smugly reveal that they have a "history." It could even indicate that he, like Astrid herself, felt betrayed when the Night Mother did not choose a new Listener and let the Brotherhood decline.
That same article elaborates a bit on Mallory's relationship with the Brotherhood:
When Delvin was a new member of the Thieves Guild, he attempted a robbery in an unspecified tavern that resulted in him accidentally committing murder. Gallus, caring for Delvin, sent him to live with the Dark Brotherhood for several years until the heat caused from the murder died down. It was here he developed a relationship with Astrid, which ended when he left the sanctuary.
I did some additional research on the people and organizations involved, and if we just limit ourselves to Skyrim, then it's really not clear. It could be the theory from the wiki, it could be a specific betrayal we don't know about, or it could be a general reference to murder being more the Dark Brotherhood's domain.
Possible Oblivion Reference
However, as santyclause pointed out, a somewhat similar situation occurred in the Dark Brotherhood plotline in Oblivion. Since Mallory was a member of the Brotherhood, it's possible that he's aware of this history.
In Oblivion, Mathieu Bellamont infiltrates the Dark Brotherhood with the express purpose of destroying it as revenge for the murder of his mother. He not only lives among the "family", but becomes one of their brightest stars. He ascends to a position of leadership, as Frey did in the Thieves' Guild, and uses this position to cause the deaths of other members of the Brotherhood. This has parallels to Frey framing Karliah as a traitor.
These events precipitate the fall of the Dark Brotherhood, leading to the sorry state it is in at the start of Skyrim. Mallory may be worried that the same fate awaits the Thieves' Guild.