NLH – if you are a newlywed, you are right, it’s expected. But if not, simply tell the families on both sides that you made other plans this year. Unless they are very elderly, have no other kids or friends and will be utterly alone, there is no reason why you cannot spend yom tov in your own home. Is there room for compromise, like coming the first days, but not the last, or vice versa?
