The masters led by Sloth saw William advancing like a flash, and he'd be soon away from their spirit sense coverage range. They didn't want the enemy to send a detachment to stop William once he distanced himself from them far enough.
Not to mention they all wanted to know what William was trying to do here. According to Sloth, he wanted to make a formation inside the Black Holes area, one that would help the other one built outside the zone.
They knew about the formation William told them to make, but they felt like the one he was going to build inside that zone was quite different. And so they started to pick up pace as well, crushing any monster they faced.
At first, they advanced smoothly, even getting closer to William. But after ten hours, more scary monsters appeared, and they had to exert their full attention and power to kill them.