During that time, when she did what she did, she was pregnant with Leia.
True, to some people, it may not be a big deal, but to him, it was.
Hazel was his wife, and even though he may not have given her the attention she wanted or shown her that he loved her, he didn't expect her to go be with another man while pregnant with their child.
It enraged him and made him so angry that he couldn't look at her the same way after she gave birth to Leia.
She made him mad whenever he set his eyes on her, and as for Leia, he couldn't find it in his heart to accept and love her as his daughter the way he should.
She kept reminding him of what happened, and even now, he has not found it in his heart to forgive Hazel or look at Leia differently.
Whether he loved Hazel or not, she had no right to hurt him like that. She was his wife!
She belonged to him!