Sometimes one has to be assertive, especially in matters that affect one’s entire future life. To the shadchan in question, it’s just another number, to you it’s your life.
Why don’t you try what I probably would; ask the shadchan how to say no to him/her without sounding so bad.