After Shiori's death, Ken and the other two heroes grieve deeply. The person who was most devastated by Shiori's death was Ken. She has the closest relationship with him. Moreover, Shiori died trying to protect him. The incident made Ken sad and afraid of the fate that awaited them in the future.
Ken and the other two heroes have lost one of their friends who were summoned together to that world, and it was their first experience. In addition, they are afraid because the enemy they are fighting will become stronger in the future. They didn't know if they would be able to endure to the end and return to their homeworld.
Ken, Sayuki, and Kazuki decided to return to the Adrienne Empire to calm themselves down. They are down and need time to be able to forget the incident. It was natural considering their young age.