The conversation between the two wasn't done covertly, so their exchange was open for all to hear and a few of them like Gong Jie, and Duan Hu had visible expressions of shock and surprise when they heard Yao De say he wasn't sorry, especially the former emperor.
Being sworn brothers with him even though it was now evident that Yao De had only used that relationship to exploit him, he still knew well enough about Yao De, especially the things that he revered, and one of those things was the leadership of the pavilion.
He would have never expected Yao De to be emboldened enough to be that direct and honest, even Lin Duyi spared enough attention to look over in clear surprise before he hid his head back in his mother's embrace.
When it came to Xiao Gun and Yao Chang, they too reacted to his words, and it was one of anger.