You don't have to take much material off, and it takes a lot less time to dress down the washers a few thou than it takes to drive to the dealer or find some place that sells them at a reasonable price. It's the path of least resistance for me, and I see no technical reason why taking a few thou off the washer thickness will affect it's sealing ability.

Duke