These explanations (housing) sound very plausible (likely, actually). I never leave the rubber seal out! I think my mounts are going, and even if not, vibration between the cap and the threads, causing abrasion, would explain it, especially since oil pressure would push it upward toward where I found it.
I feel better if it's from the housing. I was just extremely concerned something internal (to the motor) was wearing