Even though the three of them have gotten stronger thanks to their bloodline evolving into a higher-ranked one, they are still not strong enough to dwell inside the Humming Damned Forest freely as if it was any other forest.
Most of the mutated animals and cursed creatures in there can kill them easily.
From small snakes that can inject poison strong enough to topple over a powerful Awakened in a couple of minutes, cursed creatures that have different horrors to them, to the Mindless Vampires that although has been reduced, there are still some dwelling inside the thick forest.
In order to freely roam the forest, one needs to be at least in the seventh-rank realm.
Anything below that would be a complete suicide act as there's no chance they can survive inside the Humming Damned Forest for even a day, and this is why Gelmar was confused as to what Rex means by the castle and beyond.