Unless your Oil is really scummy to begin with leaving the Oil Filter in an changing the Oil should be OK.
But, with your Oil Filter Housing issue I can't imagine why Changing the Oil is more important the Fixing the Housing.
If your setup comes loose on the Oil Filter Housing you could loose your Engine.

__________________
84 300D, 82 Volvo 244Gl Diesel