According to Hitomi’s recounting of events that led her to come and pick me up, she messed up in front of her grandfather.
Apparently, given the old man Hirokage’s knowledge of my complex relationship with other girls, he hinted to the old butler that his granddaughter could also be one of those girls.
To confirm, the old butler waited upon Hitomi's return to interrogate and get a clear answer from her.
Perhaps with the combination of her deference to her grandfather as well as the positive mood brought about by the time we spent earlier, Hitomi easily admitted to all of it.
Although a little off with my guess that she was told not to meddle with Otoha’s affairs, it’s actually in the same essence.
The old butler convinced her that her relationship with me would go nowhere. Or at least, from the way he worded it, he made her recall that their family or clan – the whole Mihara Family – will always be a family in servitude towards the whole Kaneko lineage.