I heard that wearing a glove increases the swing weight of your club by two points. Imagine how much a watch changes things. :-)
I'm not sure if I am on board with the glove or a watch increasing your swingweight as it is based on the balance of the head weight in comparison with the shaft, but I am certainly no expert. I'm sure there are some guys with much more technical knowledge that can chime in on this one. If so, I am doomed as I have a 130g shaft in my irons and a 145g watch and maybe more doomed on my driver where I only have a 74g shaft!
In all seriousness, I think it comes down to personal preference and what you are used to. For a couple of years, I didn't wear a watch (as I currently do) and I didn't really see any impact on my golf game either way. That being said, my watch stays in the same place on my arm and does not slide around or move. If it did, I could see where this would be a serious issue.