No one stopped Anfey from entering Blackwater City after he walked with Catherine. South of Blackwater City, the houses in a large residential area had been torn down to make it into the station for the elves. Anfey couldn't care less about whether or not those residents wanted to move and where they moved to because those were problems for Mourtta and Hotchbini.
The elf station was tranquil and peaceful. In fact, it was highly guarded. There were four ancient trees at each corner of the station and they were connected with bushes.