I suspect a vaccum problem, what is the condition of your servo? Has it been replaced recently? If not and in order to eliminate a vaccum leak in the climate control system as the cause:
1) Look near your break booster for the green/black vaccum hose that goes into the car. That is the main Climate Control vaccum line.
2) Unplug it from the rubber splitter and cap the splitter, by doing this there is no vaccum going into the Climate Control system and no vaccum loss from the engine.
3) Start the car and see what happens to your idle.
4) If idles is ok then there is definetely a vaccum leak afecting the idle.
5) You will then have to check where the leak is. Let's try the above first.
One more question: You mentioned the compressor switch does not do anything. Is your cpmpressor cycling at all? There is an idle compensation switch that kicks in when the compressor is on. It is located at the firewall near the servo next to the outside air temp sensor plug and it also may be defective.
1987 Jaguar XJ6