Sunrise Sword Emperor was a myth in the Green Fire World, a bloody one.
Though he became famous later than Empyrean Patriarch, he was far more powerful than him.
Most important of all, he was a capricious loner. He killed others to his liking without any scruples at all.
Like Samsara Temple Master and Void Temple Master, Empyrean Patriarch admired their strength but did not fear them too much.
After all, they were detached and had quite a lot of things to worry about. Thus, they would never kill him unless he crossed the bottom lines of the Green Fire World. But Sunrise Sword Emperor was different.
If he destroyed the Rock Demon Clan today and annoyed him, it was perfectly possible for him to kill him and leave him no chance to escape.