Joseph,
[My point is hats and jackets are the cloth of the Jew. Ties are not in the same category]
I highly doubt the the Mechaber, the Ben Ish chai, Rav Ovadya Yosef etc and even the Chofetz Chaim would agree with you. The Chofetz Chaim did not wear a borsalino or a black hat for that matter. And please don’t try to tell me that by hat you meant any type of hat.