Off the top of my head, there's only three groups of people that I can think of that might be watching us.
The first one was obvious, which would be the maids from my family watching over me of course. But the fact that I felt that chill meant that it wasn't exactly a friendly type of look I felt so it could not be them.
The next possible group would be Emilia's people watching her and though they are against her, I doubt they would send such a killing intent at her since they aren't aware that Emilia knows about their plans yet.
So with those two options out, that leaves the third option… Which was that ninja family that Crystal is part of.
I guess either she didn't or couldn't tell them about us in time or they were watching us for some other reason.
Though… I suppose there might be another completely unrelated group of people watching us as well. But I'm not sure who else might be interested in us.