Let's Encrypt has to be down for days before people begin to feel the pain. DNS is very different, it breaks stuff immediately everywhere.

No it doesn't. DNS breaks as soon as TTLs run out. It's your choice to set them so low that stuff breaks immediately.

What do you recommend then? DNS doesn't usually change that often, but if you mess it up when it does, you're in for some pain if TTLs are high!

Not the one you're replying to, but I'd keep TTL high normally and lower it one TTL ahead of a planned change.

I would define high as "double time needed to fix a dns issue" and account for weekends

This is the way.

Unfortunately you can't set DNS TTL arbitrarily high (or low) without some resolvers ignoring your suggestion and using arbitrary values.

Most historical outages lasted minutes or hours. One arguably lasted much longer, when someone lost control of their servers due to civil war.

I haven't followed this closely, but have there been any... shall we say plain outages longer than six hours? That's not an outrageous TTL. Or a day.

This assumes that the host name you want has been recently queried. If it's not cached, good luck...

TL;DR: If it's not cached, does it really matter if it's offline for some time?

Long version:

If you're so popular all around that you really really want a very very short TTL, people will query all the time from all the places that "count", won't they? So it's gonna be cached.

If you're not so popular or not all around, what does it matter even if you had a very very short TTL? You're not loosing much.