Two trains of thought on mixing used rockers with a used cam:
1) they wear together, so miniscule differences where peaks now rub peaks versus valleys can cause wear,
2) the cam only rotates half as fast as the engine, is it a big deal?

Ultimately, while it's bad to constantly get these "Rotated", if you do mix them up, they'll have a little longer break in period, and then likely be just fine after. If you're wearing your hardened faces off via a swap... well, to be honest , probably not much hardened face was lefty anyway. I say just don't rev too hard for the first 100 miles, change the filter after 300, and then you should be fine (no need for a full oil change unless the filter was worse than normal).

Now. If your rocker arms and cam lobes actually have grooves visible, or ridges you can feel... replacement, not keeping them together, is the proper solution from a mechanical perspective as the hardened faces are shot.

__________________
Current:
2021 Charger Scat Pack Widebody "Sinabee"
2018 Durango R/T

Previous:
1972 280SE 4.5
2014 Jeep Grand Cherokee Limited "Hefe", 1992 Jeep Cherokee Laredo "Jeepy", 2006 Charger R/T "Hemi"
1999 Chrysler 300M - RIP @ 221k