As long as the inverter keeps working, you are right; you still get some (reduced) benefit for just letting them run.

At some point the inverter (a separate box, usually inside the house) will fail and need to be replaced. At that point you'd have to balance the price of a new inverter vs. the expected return of energy from the panels.