People have always said to him as the son of those two; he has nothing to worry regarding his strength. Indeed, ever since he was a child, nobody dared to pick on him. He lived a smooth childhood minus the situation with his father missing and all that.
Yuhi-san treated him like his very own son, during the time they were apart from Mother he didn't neglect him once. Even after returning, nothing changed. It wasn't until that incidence with Haruka and the others did Huan realize, 'Ah, this whole time he's been lonely.'
Though nobody neglected him, he's always felt like something was missing. The missing part is his father. Even when his father returned, something still felt wrong. It felt like he didn't have a place he truly belonged.