When I first started I never took the glove off, but thanks to a similar thread about a year ago, I decided to take it off. I did notice a difference in the feel and an improvement in my putting.
Now, I am looking forward to trying to chip and pitch without it to see if that is going to make a difference. As I never take off the glove until I hit the green.