Sentry v2: Fix the Pressure Before the OOM Killer Has To

Overview

It’s 3am and a service has died for the fourth time. The journal says “permission denied”. ausearch hands you two hundred SELinux denials from the last hour, and on the same box a batch job has been eating memory since midnight. Which one killed it? You find out by hand, one terminal at a time, while the alerts keep coming.

Sentry v2 does that legwork for you. It runs on Rocky Linux 10 and the rest of the EL10 family, watches three things the kernel and systemd already report, and works out how they connect. Then, if you let it, it does what your runbook would have done, and undoes it when the time’s up.

Think of it as the operational sibling of Aegis v2. Aegis watches what processes do, through eBPF. Sentry watches whether the box is coping: memory, CPU and I/O pressure, units that fail, denials that land. Neither one changes anything on day one. Why I built both before I needed either is a post of its own.

The problem

The OOM killer is a last resort, and a blunt one. By the time it fires, everything on the host has spent a long while stalled waiting for memory, and it picks its victim by its own arithmetic, not yours. That isn’t always the job that caused the trouble.

Unit failures have the opposite problem: too much evidence. The denial that killed your service sits in the audit log beside a hundred that didn’t matter, and nothing on the box tells you which is which.

What I built

One Python daemon with three collectors. Pressure comes from the kernel’s own PSI triggers on cgroup v2, so Sentry is told the moment tasks start stalling instead of polling for it. Unit failures come from the journal, and only messages from systemd itself are trusted. Denials come straight off the audit netlink stream, with /var/log/audit/audit.log as a fallback.

When a unit fails, Sentry looks at the denials from five seconds before to one second after, and scores each one with its reasons. Was the denied process in the unit’s cgroup? Does its SELinux domain match the unit’s? Does its name match ExecStart? In the demo on a Rocky Linux 10.2 VM, two services are denied the same file a second apart and one of them dies. The denial that killed it scores 0.976. The other scores nothing and stays out of the report, even though ausearch shows both.

Every incident becomes one redacted JSON bundle, written to disk and, if you want, sent on to a signed webhook, syslog, Splunk or Elasticsearch.

Fixes with an expiry date

This is the part I care about most. A policy can act, and every action it takes has a clock on it.

Say a batch job is hogging memory. Sentry sets its MemoryHigh to 80% of what it’s using and halves its CPUWeight, for five minutes, so the kernel reclaims from that job first instead of from everyone. A job thrashing the disk can be frozen for a fixed time, and thawed early if pressure drops first. Before systemd is touched, each change is written to disk along with the values it’ll restore. They go back at expiry, on shutdown, or straight after a crash, because systemd runs a rollback the moment the daemon exits. Changed a value by hand in the meantime? Sentry leaves yours alone.

What it won’t do matters just as much. It never restarts, stops or kills a service. It never changes SELinux policy; it shows you the denial, and a person decides. SSH, journald, D-Bus, auditd and Sentry itself are protected. And a unit that needs three actions in half an hour gets put on hold, because by then it needs a person, not a fourth throttle.

Out of the box, though, it does none of this. The daemon and every shipped policy start in dry-run, running the whole pipeline and recording what they would have done. Nothing is enforced until both the daemon and the policy say enforce.

Where it is now

Version 0.2.6, the eighth release in six days since the first on 15 October. It ships as RPMs: the daemon with a hardened systemd unit, an SELinux module that confines it in its own domain, and a Cockpit page showing pressure, incidents and the actions in force, each with a way to roll it back. There’s an optional Prometheus endpoint on localhost too. Every runtime dependency comes from EL10’s own BaseOS and AppStream repositories.

The integration suite runs on Rocky Linux 10.2 with SELinux enforcing. AlmaLinux, RHEL and CentOS Stream 10 use the same packages but haven’t had test runs of their own yet. EL9, Debian, Ubuntu and cgroup v1 aren’t supported, and aren’t planned.

Try it

Grab the RPMs from the latest release, check them against SHA256SUMS, and install:

sudo dnf install ./sentry-v2-0.2.6-*.noarch.rpm ./sentry-v2-selinux-0.2.6-*.noarch.rpm
sudo systemctl enable --now sentry-v2

EL kernels ship with PSI switched off, so you may need psi=1 on the kernel command line and a reboot first; the README has the one-liner. Only want to check policies in CI? pip install sentry-v2 gets you sentry-v2ctl policy validate. Then break something on a throwaway VM and read the bundle. The code and the docs are on GitHub.