A few months late, but in earlier chapters it was explained that many don't actually have the token yet. It was only before the princess mission did he learn that the fate token was already with that guy, and that he didn't aquire it in the hunt or silver city. Also the reason he couldn't just go kill that wang guy before he started cultivating was that it would incur punishment from heaven, this I don't fully understand yet, but something about heavens plans and fate.

See this! I just gifted the story: Balloon