I don't know how M104 cams are held in place but for other engines you're supposed to tighten the cam towers a little at the time. In some cases, there's even a tightening sequence. The risk of not following the procedure is breaking the cam.

Sixto
95 S420
87 300SDL