Maybe you can suggest to him that he keep some toys or something to keep his kids entertained with when they come in his own desk, because they seem to wonder over to your desk looking for something to play with. Also let him know that you realize he wasn’t aware this was happening, but you are particular about your things and really don’t like anyone touching your desk. Hatzlocha.
