Leonard knew that Serena was no angel. He has witnessed firsthand how she chose and keep on choosing to commit actions which were morally wrong. He knew she cheated on Geoffrey from the start, then she left Charlton hanging and even abandoned him while he was at his worst. He wondered what the reason was and then began doubting the paternity of the first child she carried. When it ended in a miscarriage, he left his thoughts at that. After that, he learned of what she did to some of the ladies who dared covet her position, though he must say that they just got what they deserve.
In any case, all those things could be said to be petty in comparison to what she did lately. Adultery. Gods. Why has she been so stupid? Why did she not wait? If she really wanted to be with Charlton, she should have divorced from Geoffrey first.