"If anyone here can save our race, we will give you a key to half the Space Garden," Violet declared once again. He was keen on making promises, it seemed. "Half of its entirety will be shared. That is our pledge."
The group of elites had all reached the Space Garden via the Flower God's battleship. They weren't aware of the route they took to get there or even how they had made it through the border of the xenogeneic space. Because of that, they still didn't know if entry to the Space Garden required some special password or technique. So, even if they took down the Space Garden now, there was no point in doing so if they wouldn't be able to find or enter it again. As a result, no one tried anything suspicious or hostile.
Based on what Violet had said, it seemed that something special was required to enter the Space Garden. And that made everyone think.