Log Sentinel

How it works

The original Jev experiment replayed a synthetic firehose from seven services (api-gateway JSON, payments JSON and Java stack traces, auth logfmt, Kubernetes events, postgres, nginx access logs, cron) and asked Jev four questions about every line: is it actionable, how severe is it, what is the root-cause category, and is it security-relevant. The fixtures, their hand-written ground truth, the seeded generator, the multi-line stack-trace grouper, the incident tracker and the regex baseline are copied here unchanged. Only the transport changed: the Node server is gone and the whole pipeline runs in this page.

One batch of up to 32 lines costs four requests, sent in parallel. Each line is prefixed with its service, and a stack trace is cut to its head plus two frames:

• /yes-no with texts and one statement: “An on-call engineer should investigate this log line right now.” Both hints open with “judge what the line means, not its level”. The when_true hint names a 5xx on checkout, a provider timeout or an open breaker, a 200 with an empty body on a payment path, a crash in live traffic, a backup of 0 bytes, replication lag, a disk filling up, a certificate expiring soon and credential stuffing; the when_false hint names the shapes that make up almost all of the stream — a request that completed, a charge captured or declined, a login that worked, a scheduled pod, a checkpoint — plus a 404, an applied rate limit, an expected canary failure, a retry that worked and a client that hung up. Naming those everyday shapes is what holds the healthy traffic down; an earlier hint that simply pardoned anything “even at ERROR level” also silenced the provider timeouts and the open breaker at the end of the storm. A line is actionable at probability ≥ 0.55: the live stream is almost all healthy traffic, the everyday “request completed” lines land at 0.34–0.48, and 0.5 let a few of them flip into the incident pane every minute.
• /rate with the same texts and a five-level scale, low to high: noise, informational, degraded, customer-impacting, outage. Each level describes its consequence, not the log level, so a successful retry stays noise even when it says ERROR and a backup that wrote nothing is at least degraded.
• /classify with the same texts and six described labels: deploy, capacity, dependency_failure, security, data_integrity, config. The original had eight, but noise and transient swallowed real problems, so the root cause is asked only as a root cause and the yes/no answer decides whether a line matters at all. The one fixture the original labelled transient is relabelled noise here, because a line that is not actionable never gets a root cause.
• A second /yes-no, the fourth question: “This log line shows credential abuse, scanning, or an over-broad privilege grant.” This one is asked, not derived. Deriving it from the security label made it unreachable: the root cause is gated on the actionable verdict, and a security line on its own scores 0.41–0.45 there, under the 0.55 gate — so the flag was never once true. It has its own threshold, 0.75, and a security line is actionable whatever the first question said. The hints matter more here than anywhere else: a looser statement, “this log line describes a security event”, read almost every outage as security-relevant and put nginx 499s and provider timeouts above 0.8. With the wording above the three security fixtures land at 0.83–0.95 and everything else at 0.41–0.67.

A batch leaves when it is full or after a 600 ms linger, and at most three batches are in flight. The shared demo key allows about two requests a second for every visitor at once, so four requests per batch means roughly 16 lines a second get a verdict. The rate slider runs from 4 to 20 lines a second; anything the budget cannot absorb shows up as backlog rather than as a hidden queue. Every run stops itself after 60 seconds, and the page stops the stream when the tab goes to the background.

Grouping is code, not inference: actionable lines from the same service with the same root cause inside a 60-second window are one incident, and the incident keeps the highest severity it has seen. The scoreboard compares the model to the regex and severity rules against the fixture labels, and lists both sides' mistakes. Inject storm adds a 14-step payments-provider degradation whose first seconds of trouble are INFO-level anomalies: the timeline shows when the model first flagged it and when the regex rules first paged, which is only after the first ERROR at +5.4 s.

Numbers on this page are measured, not quoted. Run src/demos/log-sentinel/smoke.mjs over the 44 fixture templates and the 14 storm steps: one rendered line per template, one line per storm step, scored against the hand-written labels. On the 44 templates the model scores about 0.73 precision at 0.84 recall, against 0.47 and 0.37 for the regex rules, which are deterministic; on the storm alone it flags 13 of 14 lines where the rules catch 6, and it never false-pages there. Those figures give every template equal weight. The live stream does not: only 2.3% of its lines are actionable, because gw.ok, nginx.ok, pay.ok, auth.ok and pg.fast are most of it. Weighted the same way the stream is generated, the same verdicts give about 0.25 precision at 0.84 recall for the model against 0.12 and 0.37 for the rules, so that is roughly what the scoreboard will read after a full run. Both numbers are the same model; the difference is the base rate, and precision is what a base rate of 2.3% does to any classifier. The scoreboard counts what the run actually produced, so it will not match either figure exactly. It still trips on a few: a startup probe that failed once, a duplicate-key retry, a declined card and a debug dump read as worth a look, one single failed login reads as credential abuse at 0.92 and nothing in the wording moved it, and the severity lands one level off about a third of the time. Add ?mock=1 to the URL to replay the fixture labels without calling the API.