Wheel "ticks" when going back to forward

[1983/300CD]

It sounds as if the caliper weren't tightened down. When I roll backwards, then apply the brakes, I hear a "tick". Then when rolling forwards, the first time applying the brakes in that direction will give another "tick". It isn't the caliper. What else I should check?