The fact that he had escaped death so many times before wasn't due to his ability to hide in plain sight; rather, it was due to the attackers' apparent underestimation of him. He would have been killed long ago if he had fought them head on. He required some sort of advantage. He had to take advantage of every opportunity that came his way.
When they finally started attacking in earnest, Zavier easily dodged and avoided every blow. He almost seemed to be dancing around them, which irritated them because he refused to focus on them. When he was about to charge forward again, he heard a loud cracking sound and a chunk of stone fell to the ground beside him.
It turned out that his opponent was still casting a spell to slow him down, but the spell had now failed.