You have a high resistance somewhere. Without a load, the voltage is there, but when there is a load, the resistance doesn't allow the voltage to come through.
Think of electricity like water- Voltage is pressure, amps is flow. If you have a garden hose with a nozzle on the end and close the nozzle, you have pressure in the line without flow. This is like potential voltage. Now, if you kink the hose, it still has the same pressure in it until you open the nozzle. Flow (amps) can't get past the kink (resistance). Clear as mud?