The lobe is part of the cam, so no, it doesn't just fly off. What happens is that the friction slowly reduces the cam lobe to dust. As for the chances of it happening, I really don't know. As I said, I've gotten away with this in the past. I also know of one instance of a brand new lifter taking out a brand new cam (but that may have been due to improper prelubricating).

The chance is reduced to nearly zero if you use new parts or matched used parts and proper camshaft assembly lubricant. With new parts, you should break in the surfaces if possible, following the manufacturer's procedure. Typically this involves running at 2K RPM (and no less) for 20 minutes, then changing the oil.