Announcing a new SRE utility I wrote for processing linux syslog Out of Memory (OOM) entries, oom-report.
It parses logfiles for OOMs and reports on how much RAM (vm + rss) each process used and sorts from smallest to largest. (By default, the linux OOM-killer kills the largest process.)
It’s a typical Unix-style filter program, so you can run it manually on the host, or copy it to each cluster host and use ssh or a configuration management tool to run it remotely.
perl oom_report.pl < /var/log/syslog | tail -10 salt-minion = 264,981 dockerd = 297,009 docker-containe = 442,017 agent = 584,720 java3 = 877,946 (dd-agent) java1 = 1,205,023 (logstash) nginx = 2,926,052 java2 = 5,608,659 (app server) total = 13,427,264
Process names are squashed if identical, except java process names are uniqified with a numeric suffix.