~/docs

Alerts

Where alerts go, what each one means, and how to verify webhook deliveries.

↗ .mdmcp

shellz raises an alert when a server goes silent, a recording has a gap, someone tampers with the recorder, or a session can't be recorded. Alerts appear on the server or fleet page and are sent to your alert destinations within a minute.

Alert destinations

Owners and admins configure destinations in ~/security, under Alert destinations. Click Add destination, choose the type, enter the target and click Save.

TypeTarget
emailAny email address
SlackA Slack incoming webhook URL (https://hooks.slack.com/...)
webhookAny public HTTPS URL

Each destination can be switched on or off. Until you add one, alerts are emailed to the org's owners.

Webhook payload

shellz sends a POST with a JSON body:

{
  "type": "alert",
  "alert": {
    "id": "8f0c...",
    "orgId": "1b2e...",
    "serverId": "6f1c...",
    "fleetId": null,
    "serverName": "prod-api-01",
    "kind": "tamper-repaired",
    "message": "the recorder unit was modified and has been restored",
    "severity": "critical",
    "at": "2026-11-04T10:12:03.000Z"
  },
  "url": "https://app.shellz.dev/servers/6f1c..."
}

Verifying the signature

When you add a webhook, shellz shows a signing secret once. Every delivery then carries two headers:

  • X-Shellz-Timestamp: the send time in milliseconds since the Unix epoch;
  • X-Shellz-Signature: the hex HMAC-SHA256 of <timestamp>.<raw body>, keyed with the secret.
verify.js
import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(secret, timestamp, rawBody, signature) {
  const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
  const fresh = Math.abs(Date.now() - Number(timestamp)) < 5 * 60_000;
  return fresh && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}

Compute the HMAC over the body exactly as received, before parsing it. Rejecting old timestamps protects against replayed deliveries.

Alert reference

Severity is critical, warning or info. While a server or fleet member has an open critical alert, the dashboard shows it as tampered.

Server health

AlertSeverityMeaning
heartbeat-missingcriticalNo heartbeat for 90 seconds. The server is down, cut off, or shellz was stopped.
heartbeat-recoveredinfoThe server is sending heartbeats again.
recorder-startedinfoThe recorder started.
policy-updatedwarningThe server applied a new signed policy. The message gives its revision.
update-rolled-backwarningA new version failed to start and the previous one was restored.
storage-quota-warningwarningThe org is close to its storage quota.
storage-quota-exceededcriticalThe org is over its storage quota. Servers keep recordings in their spool until there's room.

Tampering

AlertSeverityMeaning
tampercriticalThe recorder was killed or crashed and was restarted.
tamper-repairedcriticalA shellz file, setting, login shell or socket was changed and has been put back. The message says what.
uninstall-refusedcriticalSomeone ran shellz uninstall without an approved removal. The message names the account.
enforcement-failedcriticalshellz couldn't put its configuration back.
policy-rejectedcriticalThe server received a policy that wasn't validly signed or was older than its current one.
org-key-rejectedcriticalThe server received an org key that doesn't chain to the key it trusts.
immutable-unsupportedwarningThe immutable flag can't be set here (unprivileged LXC). Repairs still run.

Recording

AlertSeverityMeaning
unrecorded-sessioncriticalA login avoided the recorder and was allowed to continue (fail-open).
unrecorded-session-terminatedcriticalA login avoided the recorder and was ended.
session-refusedwarningA session was refused because recording wasn't possible.
recording-failedcriticalRecording of a running session failed.
segment-signature-invalidcriticalAn uploaded segment failed its signature check and was rejected.
segment-conflictcriticalAn uploaded segment conflicts with one already stored.

Containers

AlertSeverityMeaning
docker-tcp-listenercriticalDocker accepts API calls over TCP, which bypass recording.
unsupported-container-runtimewarningA container runtime or Docker setup that shellz can't record is in use.
unrecorded-container-execcriticalAn exec into a container bypassed the shell shim.
unrecorded-noninteractive-execwarningA command without a terminal ran while recording was unavailable. It was logged once recording was back.
member-lostwarningA fleet member stopped sending heartbeats without a clean stop.
member-disabledwarningA member is running with a revoked fleet token and refuses sessions.
fleet-fullwarningA new container was refused because the fleet reached its member limit.
fleet-token-rejectedwarningA container tried to join with a rotated, revoked or invalid token.

Repeated alerts of the same kind are grouped, with a count.