The 4 Families and the government itself knew exactly what they did to the Overlord
They cruelly betrayed him even after he did everything he can to ensure humanity is safe and also secure its future
Thus, they were afraid that they would get their karma. Their wrongdoings came back knocking at their doorsteps, threatening to destroy them
As people in power, they absolutely cannot let anyone threaten their position. They did a lot of effort to make sure that humanity will progress but not to the point that the government's power and authority would be threatened
If they want to, they can colonize the wild realms. Humanity definitely can expand its reach and in return expanding its power and influence in the universe
Even without the special barrier that the 4 Human Realms have, it was enough to station Grand Masters or Lords depending on the importance of the realm