Probably not what you want to hear, but yes it does matter. The bearings are interchangeable when new, but yours will have worn in to match the races they were installed with.

__________________
1984 300 D runs, interior
1984 300 D light hit in front, RUST, mint/perfect dash, decent interior (see above), parts car
(hate to kill one)
1998 Dodge diesel