I think a good compromise is having metrics that are always on, with the ability to enable more verbose debugging logs as needed.

It doesn't need to happen on device, just upstream of storage (and as close to the source as possible to minimise transport overheads). Most of the OTel collectors are good at this, but IMO Grafana Alloy is particularly neat.

This works for when you cannot change the log source too (e.g. third party component or even legacy hardware that may be syslog only).

I agree. I much prefer Prometheus and Sentry over logs.

Metrics are useless in my experience for figuring out a problem. For the most part they only tell you that you have a problem. Being able to slice and dice logs that you have faith in is critical in my experience.

problem_counter: 1

days_since_problem compresses better if you have long streams of '0'.

Yeah you still need logs, but as a last line.

Metrics tell you when and what went wrong, but unless you've got lots of coverage, its less likely to tell you _why_