If you are going to step up and wear nice clothes, regardless of color, then grief will surely follow. Congrats to you for doing what you did.
I too have worn the white pants with white shoes and gotten the whole "ice cream man" stuff. But at the end of the day, I was collecting cash and NOT handing out any treats!
I also wear Ian Poulter products, with a lot of tartan shorts and some of his fun trousers, like light blue. And of course I have shoes to coordinate with all of it, which just makes people crazy. I get the
"Hey, I have drapes that look like those shorts!" Really? you must have a nicely accessorized home... Just give it right back to them Fred!