It looks like the sponge was actually coming from a hard line. I finally just replaced them to rule that out and now they feel ok with the car off and I probably just have more bleeding to do.