There is often a way around those rules. In my town, any owner in the historic district who is denied a demolition permit can then put the property up for sale. If the property doesn't sell within a year, then the owner can demolish the supposedly historic building. The trick might be trying to prove to the town that the owner made a good faith effort to sell the property.