The reason I suggested the shifter bushing is because I had the same problems (with the exception of the rough handling) and that is what it turned out to be. Evidently the vibrations from the road (which did get worse with speed and decreased when slowing down) caused a resonating effect when the transmission and shifter "rubbed" - causing a resonating effect (sound like marbles or tin can being dragged under the car).
I would think a worn ball joint or wheel bearing sound would eminate from either the front or rear of the car, not from the transmission.