Baiyi was starting to get really frustrated with Lord Nolan’s way of speaking.
To be frank, he was not seeking to assume the parental role in Mia’s life, if every sentimentality was stripped off.
He needed to be someone even more important than just a father figure, because Little Mia was the most important key in releasing the Voidwalkers from their agonizing limbo.
He wouldn’t mind Mia not calling him 'daddy', but he did mind how close she rated their relationship. Without any compromise, Baiyi reasoned that he had to be the one who mattered the most to the girl. She had to be completely dependent on him, no matter emotionally or when it comes to problem solving.
He could not allow anyone else to influence Mia. That was why he forbade the girl to have a relationship and why he was initially so hostile to her father. To a girl this age, a lover or a father’s words might change the girl’s world view, which would in turn intervened with his plans.