Check for vacuum leaks on all the lines you can find. It doesn't take much to upset the transmission.
The reason it is smoother with A/C on may be that the fuel system is feeding more air/fuel into the engine to compensate for the compressor load, however it is able to cut the air down, whereas at normal idle the airflow is so small it can't, hence the roughness.
'08 Chrysler 300CRD (MB OM 642 engine)
'95 E220 estate
'89 230TE (R.I.P.)