The potential to leave the human dictated wilderness territories was not something any of them had thought about. They knew this as the wilderness full of danger and worry for most people. There were some smaller villages along the borders, but it was very rare for anyone to go to these places unless need be.
There were many spaces that were off limits because of dangerous dungeons that had broken out with beasts that were too hard to deal with. Many beasts and dungeons like that would claim large areas of the wilderness that only the strongest of human tamers could wander.
The same was said for the spaces where dungeons connected other worlds. The elves, dwarves, and a few other races had appeared in the wilderness. As a way of making peace, the humans had left that land for those races to have a city for themselves and maintain relations. It was very dangerous to approach certain places because of the risk that came with disturbing relationships with these races.