No one holds you aren’t yotzei sukah if you don’t have in mind lemaan yedu. It’s that he didn’t fulfill the mitzva kara’uy. But no one holds he wasn’t yotzeh.
My point was that we don’t love gerim because we too were gerim. That would be kind of childish. The Torah is reminding us that we should know better than to give a hard time to gerim because we too experienced the same thing. That’s hardly the main focus of why it’s important to love gerim. I think anyone with common sense can figure out why Hashem wants us to love gerim without having to say we were gerim too. It’s the same as our obligation to not be harsh to a yasom and almanah. Loving a ger needs a bigger reminder because people tend to exclude those who are mufka.