This isn’t all that complicated an issue. Milk is a product sold through interstate commerce, and as such, the Constitution explicitly gives the government the right to regulate it. If you believe that the Constitution means nothing more than it says (and most people on this site do feel that way, I believe), that’s really the end of the story.
You can argue if you’d like that there’s some right not written in the Constitution to eat whatever you’d like, but then I’d expect you to be much more sympathetic to those who argue for similar freedom to engage in medical procedures or marital relationships without government interference.
In any event, it’s not surprising that the FDA is arguing against the creation of that right in this instance, since regardless of what you think about raw milk, a court deciding that people have the Constitutional right to eat whatever they want would effectively put the FDA out of business.