Alerts
Where alerts go, what each one means, and how to verify webhook deliveries.
shellz raises an alert when a server goes silent, a recording has a gap, someone tampers with the recorder, or a session can't be recorded. Alerts appear on the server or fleet page and are sent to your alert destinations within a minute.
Alert destinations
Owners and admins configure destinations in ~/security, under Alert destinations. Click Add destination, choose the type, enter the target and click Save.
| Type | Target |
|---|---|
| Any email address | |
| Slack | A Slack incoming webhook URL (https://hooks.slack.com/...) |
| webhook | Any public HTTPS URL |
Each destination can be switched on or off. Until you add one, alerts are emailed to the org's owners.
Webhook payload
shellz sends a POST with a JSON body:
{
"type": "alert",
"alert": {
"id": "8f0c...",
"orgId": "1b2e...",
"serverId": "6f1c...",
"fleetId": null,
"serverName": "prod-api-01",
"kind": "tamper-repaired",
"message": "the recorder unit was modified and has been restored",
"severity": "critical",
"at": "2026-11-04T10:12:03.000Z"
},
"url": "https://app.shellz.dev/servers/6f1c..."
}Verifying the signature
When you add a webhook, shellz shows a signing secret once. Every delivery then carries two headers:
X-Shellz-Timestamp: the send time in milliseconds since the Unix epoch;X-Shellz-Signature: the hex HMAC-SHA256 of<timestamp>.<raw body>, keyed with the secret.
import { createHmac, timingSafeEqual } from "node:crypto";
export function verify(secret, timestamp, rawBody, signature) {
const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
const fresh = Math.abs(Date.now() - Number(timestamp)) < 5 * 60_000;
return fresh && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}Compute the HMAC over the body exactly as received, before parsing it. Rejecting old timestamps protects against replayed deliveries.
Alert reference
Severity is critical, warning or info. While a server or fleet member has an open critical alert, the dashboard shows it as tampered.
Server health
| Alert | Severity | Meaning |
|---|---|---|
heartbeat-missing | critical | No heartbeat for 90 seconds. The server is down, cut off, or shellz was stopped. |
heartbeat-recovered | info | The server is sending heartbeats again. |
recorder-started | info | The recorder started. |
policy-updated | warning | The server applied a new signed policy. The message gives its revision. |
update-rolled-back | warning | A new version failed to start and the previous one was restored. |
storage-quota-warning | warning | The org is close to its storage quota. |
storage-quota-exceeded | critical | The org is over its storage quota. Servers keep recordings in their spool until there's room. |
Tampering
| Alert | Severity | Meaning |
|---|---|---|
tamper | critical | The recorder was killed or crashed and was restarted. |
tamper-repaired | critical | A shellz file, setting, login shell or socket was changed and has been put back. The message says what. |
uninstall-refused | critical | Someone ran shellz uninstall without an approved removal. The message names the account. |
enforcement-failed | critical | shellz couldn't put its configuration back. |
policy-rejected | critical | The server received a policy that wasn't validly signed or was older than its current one. |
org-key-rejected | critical | The server received an org key that doesn't chain to the key it trusts. |
immutable-unsupported | warning | The immutable flag can't be set here (unprivileged LXC). Repairs still run. |
Recording
| Alert | Severity | Meaning |
|---|---|---|
unrecorded-session | critical | A login avoided the recorder and was allowed to continue (fail-open). |
unrecorded-session-terminated | critical | A login avoided the recorder and was ended. |
session-refused | warning | A session was refused because recording wasn't possible. |
recording-failed | critical | Recording of a running session failed. |
segment-signature-invalid | critical | An uploaded segment failed its signature check and was rejected. |
segment-conflict | critical | An uploaded segment conflicts with one already stored. |
Containers
| Alert | Severity | Meaning |
|---|---|---|
docker-tcp-listener | critical | Docker accepts API calls over TCP, which bypass recording. |
unsupported-container-runtime | warning | A container runtime or Docker setup that shellz can't record is in use. |
unrecorded-container-exec | critical | An exec into a container bypassed the shell shim. |
unrecorded-noninteractive-exec | warning | A command without a terminal ran while recording was unavailable. It was logged once recording was back. |
member-lost | warning | A fleet member stopped sending heartbeats without a clean stop. |
member-disabled | warning | A member is running with a revoked fleet token and refuses sessions. |
fleet-full | warning | A new container was refused because the fleet reached its member limit. |
fleet-token-rejected | warning | A container tried to join with a rotated, revoked or invalid token. |
Repeated alerts of the same kind are grouped, with a count.