Ping Pong
I've played a lot of ping pong in the recent few months. Almost everyday for a few hours. Thus I say from personal experience, playing outdoors in freezing temperatures is definitely alright. The only thing that'll bother you is cold hands.
If you're hands are cold for an extended period of time, they basically lose all of their pain tolerance. If your finger gets hit by a ping pong ball, it'll hurt a lot. (And if they stop hurting you might be in a lot of trouble)
The trick is to just wear very thick gloves and it's surprisingly easy to adjust to.