In the Thousand Meal House
Ezra still ended up rejecting Ash's invitation, because he knew that though there was power in numbers, it also came with more disputes and troubles and he has to take care of his own force.
Ezra's three main objectives in the Spirit Devouring Mountain were to reach the Foundation Establishment stage as soon as possible. Second was to rule this entire sect with his little army and the third was to find anything related to fight between the sect's patriarch and Bjorn's 1st disciple.
He also has the thought of his kingdom in mind. Though the Brownmane Empire was currently doing all it could to stabilize the situation at its borders, once Emperor Bruce Brownmane got his hands free, he would surely turn his eyes to the Bloodfang Kingdom.
Therefore, Ezra needed to step into the Foundation Establishment stage before that happened. And there was an inevitable war with Duo Mountain Peak.