While Grey and the leader of the Gnome group were fighting, the other Gnomes had moved out. The Seventh Prince, who Alice was going to handle, was the first to move out. After he left, the others left as well. The Second Prince didn't see the need in doing all this, but the fear of Grey coming to attack him while he was alone made him follow the other Peak Venerables out of the castle.
For the first time since Gnomes were in this realm, their castle for those in the Elemental Venerable Plane was empty, and it was all because of the same person that made something like this happen in the lower realm, Grey.