Though I cannot locate it right now, I do remember looking this up many years ago.
If I’m not mistaken, aside from the ball issue, which would only be a sha’alo for Sefaradim, there is also the issue of psik reisha that the ball eventually breaks. If it is a kli and is not muktzeh, thi would be a problem of soser kli. I cannot remember who said what. But I do know that since that Shabbos that I looked it up and saw the discussion I stopped playing ping pong on Shabbos.