Since another elf was involved, everyone knew of the urgency and seriousness of the issue at hand.
Nobody knew what kind of elf is was or what kind of abilities it had. All they knew was that it could quietly cross the physical barrier and cause people to fall asleep. This was the time when everyone needed to work together, but now Famous abruptly said that he wanted to go to bed...
This... this was unreasonable!
Had the rest not known that Famous's concern for Zhang Zian wouldn't affect one of the elves, they might've started to fight over it.
However, everyone needed him to give a reasonable explanation.
Famous also knew that it wasn't a good time to be sleeping. He could only vaguely explain, "There is an elf who may know the clue. I don't know the specifics, sot I have to go to my imagined world and ask them."