Asher was about to go back to his place like other servants since it was already the early hours of the night. The servants had prepared dinner and had gone back to their places. As usual, Mary stayed to serve and take care of other things. But that day, Asher was late because he went to check on Anne and feed her dinner since she woke up late from the previous sleep. Asher had to stay there until she fell asleep again so it took some time as her usual routine had been messed up with the previous sleep.
So he was just going back now.
However, just as he was about to step out of the mansion, he was faced with duke.