Just like all immortal's path, the sword immortals were usually the embarrassment in a room — the butt of the joke, per se. The source of this disparity is that they've lost one of the core aspects of a swordsman — the Dao part. They must rely on their Immortal Qi to replicate their former abilities and swordsmanship realm.
Despite this flaw, they still considered themselves swordsmen and kept that mindset. They trained their Qi and their mind with such a mindset, thus still giving them a slight edge over regular immortals. However, the ones who truly shine in this banquet are the swordsmen with unique Sword Qis.