Smokers have a lot of problems with an ash falling on their sweaters. Cigars are particularly problematic. The other issue could be moths, a couple of good pieces of cedar in your closet will take care of that. Finally if you are washing your sweater other zippers in the wash will snag a sweater faster than anything.
Fortunately I quit smoking years ago, have a moth free closet and, and the little lady washes the sweaters with couple of towels to protect them. No problems.