I wipe the stick clean with my fingers, and wipe my fingers off on a rag (most paper towels are too linty for my comfort). I read an article once that claimed this was the factory procedure for checking tranny fluid level in a Benz, and it works just as well for engine oil. It just lessens the chance of particulate contamination unless you were just in the sand box and didn't wash your hands
To answer your question, I don't imagine you're doing much damage, lint fibers are likely filtered out pretty quickly, but why take the chance? I would say that engines are only slightly more forgiving that trannys in regards to damage via particulate contamination, and any extra care that can be given, should be. These are afterall, the things that make the car go!
Probably overkill, but hey, when it doesn't take any extra effort...
Regards,
- Ryan