The people who make attacks of “The Gedolim aren’t infallible” are plain anti-semites who happen to be Jewish. Some wear yarmulkas and claim to be frum.
The fact is NO ONE EVER claimed or claims the Gedolim are infallible. It is a strawman. They hate the Torah and Judaism but are embarrassed to say so. So instead they attack the Gedolm by effectively saying the Gedolim aren’t infallible, so therefore anytime they don’t like what the Gedolim say they can say the Gedolim are wrong, so they can do whatever they want to do.