Even if there was a leap second every year... does it matter? It would still take several hundred years for the difference to be noticeable, at which point you can just change UTC by +-1h.
What it means is that the discrepancy between clocks and time of day is growing quadratically. It won't matter to us in our lifetimes. But eventually it will be a problem.
It will take on the order of a thousand years for the discrepancy to add up to an hour. In 5000 years, it will be around a day. The fact that it is currently growing about a minute per century is true, but not a good predictor of what will happen.
That said, I prefer if we just lose the astronomical basis for time keeping, and let our distant descendants figure out that they should use time zones, and not modify UTC. Hopefully by the time it becomes obvious that clocks are drifting too fast for time zones to change all of the time, we're a multi-planetary species and paying attention to what is happening on the Earth seems quaint.
