Relocation
Since the Temple Master has only brought the top-level management from his temple, their Life Stones will be placed in the Temple Master's Abode, unlike the Gatekeeper's Life Stone.
According to Uncle Grey, their life stones won't be checked daily; instead, they will be watched only when they enter the wilderness on their missions.
However, if the Temple Master, the Temple Elders and the Temple Guardians were not returned to the Temple, the watchers of the temples would know that something was wrong and check on the Life Stones.
Nevertheless, it would be only after three to four hours before they check as it would be midnight in four hours.
So, they have time for another few hours before the members that were left in the Temple notice the broken life stones of the Temple Master, Elders and Guardians.
'Uncle Grey is more knowledgeable than he looks.'
Of course, Leo learned this information from Uncle Grey before Uncle Grey rushed to gather the family members to move out.