the trim is more than decoration, it protects the sides from other careless doors, i would say leave it just for that factor alone.