The old man meant more to Damien than even he realized.
After all, the guidance this old man gave him during his time in the Abyss, and the semi-frequent visits the old man made over the decade Damien spent traveling, were both integral in different ways.
The visits allowed him to keep his sanity. He wouldn't have gone insane in the Abyss by himself, but his mentality would've taken a hit.
The company he gained through the old man's presence when he was in a place more dreary than anywhere else was something he appreciated heavily.
The advice allowed him to reach his current heights. If it wasn't for the old man, he wouldn't have been able to understand the secrets of tethering so quickly.
Yes, he would have still died on Al'Katra regardless, but what about what came after?
Would he have been able to gain anything from his stay in Nonexistence?