It was no secret that neither Ser Ballmer nor Anoon spared an ounce of love for each other, being frequently at loggerheads with each other.
And this mostly had to do with their personal characters.
Ser Ballmer was an open and upright man who blindly followed his given orders without ever questioning or second guessing them.
While Anoon was the complete opposite, an openly amoral and opportunistic man who cared about his skin much more than anyone else's.
And thus the two had come to butt heads from an early point on, where Anoon had challenged and even defied some of Ser Ballmer's orders, after finding them not conducive to his own safety.
It was from these interactions that Ser Ballmer came to understand Anoon's true nature and determined that the other would betray the family at the drop of a hat if it ever suited him in the slightest.
Thus the veteran man looked to find faults in every little thing that Anoon did.