Harrison left the family estate, the castle, through a less conspicuous exit that could be found near the wall that was put around the entire estate. There were multiple hidden entrances and exits, all well-guarded in order to make sure nobody that was not a part of the family could get in.
Harrison, of course, had chosen an exit that was guarded by two of his own men. Even though he didn't dare put his spies near the family, he had planted a couple of guards of his own inside the estate.
He thought that his actions back then were not noticed, but he was certain that his father did in fact know about it, but simply didn't do anything to stop him. As long as whatever he or his siblings did was not found out by others, or was not against the family's interests, he would turn a blind eye to them.