# Alerts

URL: https://shellz.dev/docs/guides/alerts

> Where alerts go, what each one means, and how to verify webhook deliveries.

shellz raises an alert when a server goes silent, a recording has a gap, someone tampers with the recorder, or a session can't be recorded. Alerts appear on the server or fleet page and are sent to your alert destinations within a minute.

## Alert destinations

Owners and admins configure destinations in **~/security**, under **Alert destinations**. Click **Add destination**, choose the type, enter the target and click **Save**.

| Type | Target |
|---|---|
| email | Any email address |
| Slack | A Slack incoming webhook URL (`https://hooks.slack.com/...`) |
| webhook | Any public HTTPS URL |

Each destination can be switched on or off. Until you add one, alerts are emailed to the org's owners.

## Webhook payload

shellz sends a `POST` with a JSON body:

```json
{
  "type": "alert",
  "alert": {
    "id": "8f0c...",
    "orgId": "1b2e...",
    "serverId": "6f1c...",
    "fleetId": null,
    "serverName": "prod-api-01",
    "kind": "tamper-repaired",
    "message": "the recorder unit was modified and has been restored",
    "severity": "critical",
    "at": "2026-11-04T10:12:03.000Z"
  },
  "url": "https://app.shellz.dev/servers/6f1c..."
}
```

### Verifying the signature

When you add a webhook, shellz shows a signing secret once. Every delivery then carries two headers:

- `X-Shellz-Timestamp`: the send time in milliseconds since the Unix epoch;
- `X-Shellz-Signature`: the hex HMAC-SHA256 of `<timestamp>.<raw body>`, keyed with the secret.

```js title="verify.js"
import { createHmac, timingSafeEqual } from "node:crypto";

export function verify(secret, timestamp, rawBody, signature) {
  const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex");
  const fresh = Math.abs(Date.now() - Number(timestamp)) < 5 * 60_000;
  return fresh && timingSafeEqual(Buffer.from(expected), Buffer.from(signature));
}
```

Compute the HMAC over the body exactly as received, before parsing it. Rejecting old timestamps protects against replayed deliveries.

## Alert reference

Severity is **critical**, **warning** or **info**. While a server or fleet member has an open critical alert, the dashboard shows it as **tampered**.

### Server health

| Alert | Severity | Meaning |
|---|---|---|
| `heartbeat-missing` | critical | No heartbeat for 90 seconds. The server is down, cut off, or shellz was stopped. |
| `heartbeat-recovered` | info | The server is sending heartbeats again. |
| `recorder-started` | info | The recorder started. |
| `policy-updated` | warning | The server applied a new signed policy. The message gives its revision. |
| `update-rolled-back` | warning | A new version failed to start and the previous one was restored. |
| `storage-quota-warning` | warning | The org is close to its storage quota. |
| `storage-quota-exceeded` | critical | The org is over its storage quota. Servers keep recordings in their spool until there's room. |

### Tampering

| Alert | Severity | Meaning |
|---|---|---|
| `tamper` | critical | The recorder was killed or crashed and was restarted. |
| `tamper-repaired` | critical | A shellz file, setting, login shell or socket was changed and has been put back. The message says what. |
| `uninstall-refused` | critical | Someone ran `shellz uninstall` without an approved removal. The message names the account. |
| `enforcement-failed` | critical | shellz couldn't put its configuration back. |
| `policy-rejected` | critical | The server received a policy that wasn't validly signed or was older than its current one. |
| `org-key-rejected` | critical | The server received an org key that doesn't chain to the key it trusts. |
| `immutable-unsupported` | warning | The immutable flag can't be set here (unprivileged LXC). Repairs still run. |

### Recording

| Alert | Severity | Meaning |
|---|---|---|
| `unrecorded-session` | critical | A login avoided the recorder and was allowed to continue (fail-open). |
| `unrecorded-session-terminated` | critical | A login avoided the recorder and was ended. |
| `session-refused` | warning | A session was refused because recording wasn't possible. |
| `recording-failed` | critical | Recording of a running session failed. |
| `segment-signature-invalid` | critical | An uploaded segment failed its signature check and was rejected. |
| `segment-conflict` | critical | An uploaded segment conflicts with one already stored. |

### Containers

| Alert | Severity | Meaning |
|---|---|---|
| `docker-tcp-listener` | critical | Docker accepts API calls over TCP, which bypass recording. |
| `unsupported-container-runtime` | warning | A container runtime or Docker setup that shellz can't record is in use. |
| `unrecorded-container-exec` | critical | An exec into a container bypassed the shell shim. |
| `unrecorded-noninteractive-exec` | warning | A command without a terminal ran while recording was unavailable. It was logged once recording was back. |
| `member-lost` | warning | A fleet member stopped sending heartbeats without a clean stop. |
| `member-disabled` | warning | A member is running with a revoked fleet token and refuses sessions. |
| `fleet-full` | warning | A new container was refused because the fleet reached its member limit. |
| `fleet-token-rejected` | warning | A container tried to join with a rotated, revoked or invalid token. |

Repeated alerts of the same kind are grouped, with a count.