Consider the machlokis tana’im on the matter of why a Nazir brings a korbon for being choteh al ha-nefish. Is it because he withheld permissible pleasure from himself and it is sufficient what the Torah forbade? Is it because he is bringing himself from a higher level of kedusha to a lower level by ending his term of prishus? Is it both?
Also, consider the Yerushalmi that a person will have to answer for what he had an opportunity to derive pleasure from and doesn’t.
Obviously, making pleasure seeking ones goal in life is contrary to Yidishkeit. Intent matters.But it cuts both ways. You can use the Yerushalmi to become a baal taiva and you can use prishus to become a baal gaiva.
Also, maintaining awareness of our purpose for being in this world and working on our midos are not matters that are lefnai meshuras hadin left to chassidim and baalai musser. They are incumbent on every single yid.