The Druids didn't have any intentions to fight in the first place. With someone finally giving a command to retreat, everyone of them immediately turned around and ran off. They didn't even bother caring if the one giving the command was a Druid of higher rankings or just a commoner—it wasn't even important to them now.
It wasn't wrong to think of the Druids as bandits, because other than their usual activities which resembled the bandits, their behaviours were also lawless. The difference in levels between the Druids themselves weren't huge. Their hierarchy was maintained based on levels of prestige; however, such terms were unclear, unlike soldiers in the military.