There is a vacuum diverter valve buried in the dash which controls the flaps. That may be bad. But it's expensive enough that you don't want to swap it out to test. If it's good, your climate-control pushbutton unit may be at fault.

Are you sure that your fan is blowing faster when you increase the control past half-way? If not, you could have a bad blower regulator, which is a very common problem on your car.