Kyoya would rather take down the twin tail woman, but she used some technique to seal the entrance to the tower. And it was also clear she took two of these already. So Kyoya's party could only clear the rest faster!
The tower where Kyoya got attacked was the last available tower excluding the one where the other 'absurd' woman was currently having a lot of fun. It was impossible to enter due to the seal blocking the entrance, but it wasn't like Kyoya's party was that limited.
The husband himself had powerful sealing abilities thanks to his diamond card, but he didn't know how strong the sealing powers were on the other side.
That was another reason why Kyoya's party resolved to just give up on the tower, but now that they completed their run, Kyoya took out a diamond card with the highest value he could produce together with Xaya.