# shellz documentation > shellz records every terminal session on Linux servers and in containers, from people and AI agents: SSH logins, remote commands, SFTP, consoles, docker exec and kubectl exec. Recordings are encrypted on the server before they leave it and decrypted only in the viewer's browser. With Perfect Secrecy, only the team's passkeys can decrypt them, not shellz. # Introduction URL: https://shellz.dev/docs > What shellz records, how it protects the recordings, and where to start. shellz records every terminal session on your Linux servers and in your containers, whether a person or an AI agent is typing. Recordings are encrypted on the server before they leave it, stream to the dashboard about a second behind, and are decrypted only in your browser. ## What gets recorded | Session | Recorded as | |---|---| | Interactive SSH logins | `ssh` | | Remote commands (`ssh host cmd`) | `command` | | SFTP transfers | `sftp` | | Physical, serial and hypervisor consoles (`tty1`, `ttyS0`, `hvc0`, `/dev/console`) | `console` | | Other local logins, `su -`, desktop terminals | `local` | | `docker exec`, `docker attach`, `docker run -it`, `docker compose exec` on a Docker host | `container` | | `kubectl exec`, ECS Exec and dev-container terminals, with shellz in the image | `container`, or `command` for one-off commands without a terminal | Each session is kept twice: as a full terminal replay and as a clean text transcript. Input typed at hidden prompts, such as `sudo` and `passwd` passwords or `read -s`, is never recorded. tmux and screen started inside a recorded session are captured by that session's recording. > [!NOTE] > shellz records agents that log in or start a shell: SSH, remote commands, SFTP, `su -`, `docker exec` and `kubectl exec`. A process running as a service on the host that never starts a login session isn't a session, so it isn't recorded. ## How it's protected - Every session gets its own AES-256-GCM key, which is wrapped to your org's X25519 key on the server. shellz stores ciphertext only. - Every piece of a recording is signed by the server's own Ed25519 key and chained to the previous one, so missing or edited pieces show up as a gap or as tampering. - Uploaded recordings are stored write-once with a 365-day retention lock. - The recorder can't be stopped through systemd and restarts if it's killed. Changes made with root access are reverted and reported, and a server that goes silent raises an alert. - With [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md) turned on, the keys that open recordings exist only in your team's browsers, unlocked by their passkeys. shellz can't decrypt your recordings. ## Where to start - [Getting started](https://shellz.dev/docs/getting-started.md): sign up and record your first session in about two minutes. - [How it works](https://shellz.dev/docs/how-it-works.md): the path a keystroke takes from your server to your screen. - [Choose an install method](https://shellz.dev/docs/install.md): Linux servers, Docker hosts, Kubernetes, ECS, Cloud Run and dev containers. - [Security overview](https://shellz.dev/docs/security.md): the threat model, the encryption and what shellz can and can't see. ## Docs for AI agents These docs are available as plain Markdown and through an MCP server, so coding agents can read them directly. See [Use the docs with AI](https://shellz.dev/docs/reference/ai.md). # Getting started URL: https://shellz.dev/docs/getting-started > Sign up, install shellz on a Linux server with one command, and watch your first session live. This guide takes you from a new account to watching a live session in about two minutes. You need a Linux server with systemd where you can run `sudo`. For containers on Kubernetes, ECS or similar platforms, see [Choose an install method](https://shellz.dev/docs/install.md). ## 1. Create your account Go to [app.shellz.dev/signup](https://app.shellz.dev/signup) and sign up with Google, GitHub or a passkey. - **Google or GitHub**: you're signed in straight away. You'll be offered a passkey afterwards; it's optional until you turn on Perfect Secrecy. - **Passkey**: enter your email address, open the link shellz sends you (it works once and expires after 15 minutes), then create the passkey on that device. shellz tests every passkey before accepting it. It must be discoverable, require user verification (a fingerprint, face or PIN) and support the WebAuthn PRF extension, which [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md) uses to derive encryption keys. If yours fails, shellz says why and lists authenticators known to work: iCloud Keychain, Google Password Manager, 1Password and YubiKey 5 (firmware 5.7 or later). Your org (your team's shared workspace) is created automatically. Owners and admins can rename it from the onboarding screen. ## 2. Install shellz on your server The onboarding screen, **Record your first session**, shows an install command with a fresh join token already filled in. Copy it and run it on your server: ```bash title="On your server" curl -fsSL https://get.shellz.dev | sudo sh -s -- szt_ ``` The token works once and expires after 10 minutes. To add more servers later, open **~/servers** and click **Add server** for a new command. The installer: 1. Checks that it's running as root on a systemd Linux server (amd64 or arm64) with curl and OpenSSL 3, and that the token isn't damaged. 2. Downloads the shellz binary for your architecture and verifies its Ed25519 release signature and SHA-256 checksum against the key built into the script. 3. Enrolls the server: it creates the server's own signing key, redeems the token and pins your org's public key. 4. Starts the recorder and routes every login on the server through it. ```text title="Output" → Fetching the latest release for linux/amd64 ✓ Release 0.1.2 signature verified → Downloading shellz 0.1.2 ✓ Checksum verified ... This server is now recording. Watch sessions live at https://app.shellz.dev ``` > [!TIP] > On Rocky, Alma and RHEL, sudo's `secure_path` doesn't include `/usr/local/bin`, so call shellz by its full path: `sudo /usr/local/bin/shellz status`. Installing somewhere else? See [Linux server](https://shellz.dev/docs/install/linux.md) for supported distributions and exactly what the installer changes, and [Docker host](https://shellz.dev/docs/install/docker-host.md) if the server runs Docker. ## 3. Watch your first session As soon as the server connects, the onboarding screen switches to **Your server is recording**. Open a terminal and SSH into the server as you normally would. The session appears in the dashboard about a second behind what you type. Before showing any recording, shellz asks you to confirm it's you if your last sign-in was more than 10 minutes ago. The recording is then decrypted in your browser, and the view is written to the audit log. In the player you can: - follow the session live, or replay it at 0.5×, 1×, 2× or 4× speed; - switch between the terminal replay and the text transcript, and click a transcript line to jump to it; - download the session as `.cast` (replay) or `.txt` (transcript); - check the integrity badge: **verified** means every piece is signed and chained, **gap** means a piece is missing, and **tampered** means a signature or the chain failed. ## 4. Check the server On the server, `shellz status` shows the enrollment, policy and recorder state: ```bash sudo /usr/local/bin/shellz status ``` ```text title="Output" shellz 0.1.2 Server prod-api-01 (6f1c…) Org key 3b9e…, standard mode Policy revision 1, recording on Recorder running, pid 812, version 0.1.2 Sessions 1 active ssh alice since 2026-11-04 10:12:03 Upload up to date ``` ## Next steps - Invite teammates from **~/team**. See [Team and access](https://shellz.dev/docs/guides/team.md). - Choose where alerts go: email, Slack or a webhook. See [Alerts](https://shellz.dev/docs/guides/alerts.md). - Decide what happens when recording can't keep up. See [Recording policy](https://shellz.dev/docs/guides/recording-policy.md). - Turn on [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md) so only your team can decrypt recordings. # How it works URL: https://shellz.dev/docs/how-it-works > The path a session takes from a login on your server to a replay in your browser, and the pieces in between. shellz has three parts: an agent on each server or in each container, a cloud service that stores and relays encrypted recordings, and the dashboard in your browser, which is the only place recordings are decrypted. ```text your server shellz your browser ┌──────────────────────┐ ┌───────────────────────┐ ┌──────────────────────┐ │ login → session shell│ │ ingest: checks │ │ re-auth, unwrap the │ │ → recorder │ signed, │ signature + chain │ live │ session key, verify, │ │ encrypt + sign each │─────────▶│ write-once storage │──────▶│ decrypt, play │ │ segment, then upload │ encrypted│ (365-day lock) │ │ │ └──────────────────────┘ └───────────────────────┘ └──────────────────────┘ ``` ## 1. Every login goes through the recorder On a Linux server, the installer changes the login shell of root and of every regular account (UID from `UID_MIN` in `/etc/login.defs` up to 59999) to the shellz session shell, `/usr/local/libexec/shellz-shell`. The original shells are saved and restored if shellz is removed. Service accounts with `nologin` or `false` keep their shells. When someone logs in by SSH, at a console, with `su -` or by any other route that starts a login shell, the session shell asks the recorder (the `shellz-recorder` systemd service) to take over. The recorder: 1. Checks who is asking. The request travels over a root-only Unix socket, and the recorder reads the caller's identity from the kernel, not from anything the caller says. 2. Opens a new pseudo-terminal, starts the user's real shell inside it, and copies everything between the user and that shell. 3. Records the output and what was typed, plus terminal resizes. Input typed while the terminal has echo turned off, such as password prompts, is left out. A session auditor watches for logins that avoided the session shell. If it finds one, it ends that session and raises an `unrecorded-session-terminated` alert. Under the fail-open policy it leaves the session running and raises `unrecorded-session` instead. On a Docker host, the recorder also sits in front of the Docker socket so that `docker exec` and similar commands are recorded. Inside containers, a container entrypoint does the same job without systemd. See [Docker host](https://shellz.dev/docs/install/docker-host.md) and [Container image](https://shellz.dev/docs/install/container-image.md). ## 2. Recordings are encrypted on the server Each session gets a new random 32-byte data key. The recorder: - writes the session in two forms, a terminal replay (asciinema-compatible `.cast`) and a plain-text transcript; - cuts them into segments of at most 1 MiB, compresses each with gzip and encrypts it with AES-256-GCM under the session's data key; - wraps the data key to your org's X25519 public key, so only the org's private key can open it; - signs each segment header with the server's own Ed25519 key and links it to the previous segment with a SHA-256 hash chain. The server never holds a key that can decrypt a finished recording. Its own key can only sign. See [Encryption](https://shellz.dev/docs/security/encryption.md) for the formats and algorithms. ## 3. Segments stream up about once a second The recorder flushes a segment every second, and every 250 ms while someone is watching live. Segments go first to a spool on disk (`/var/lib/shellz/spool`), then to `ingest.shellz.dev`. If the network is down, segments wait in the spool until it comes back. The [recording policy](https://shellz.dev/docs/guides/recording-policy.md) decides what happens if the spool fills up. Every 30 seconds the agent sends a signed heartbeat to `api.shellz.dev`. The heartbeat reports the recorder's health, active sessions and any local alerts, and returns the current policy and org keys. A server that misses heartbeats for 90 seconds is marked offline and raises a `heartbeat-missing` alert. The agent talks only to three hosts, all over HTTPS: `api.shellz.dev`, `ingest.shellz.dev` and `releases.shellz.dev`. ## 4. shellz checks and stores ciphertext When a segment arrives, shellz checks that the server's signature is valid, that the segment's sequence number and hash link follow the previous segment, and that the server belongs to your org. It then writes the segment once to object storage. Stored segments can't be overwritten or deleted for 365 days, by anyone, including shellz. shellz can read the segment header: which server, which user, when, from which address, the terminal size and, in standard mode, the command line. It can't read what was typed or shown. Live viewers receive the same ciphertext as it arrives. ## 5. Your browser decrypts When you open a recording: 1. shellz asks you to confirm it's you if your last sign-in was more than 10 minutes ago, and writes the view to the audit log. 2. Your browser gets the session's wrapped data key unwrapped: - In **standard mode**, shellz unwraps it with the org private key it holds in a sealed form and sends the data key to your browser over TLS. - In **Perfect Secrecy** mode, shellz doesn't have the org private key. Your browser unlocks it with your passkey and unwraps the data key itself. See [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md). 3. Your browser checks every signature and hash link, decrypts and decompresses the segments, and plays them. The integrity badge shows **verified**, **gap** or **tampered**. You can also decrypt downloaded segments offline with the `shellz decrypt` command, without the dashboard. See the [CLI reference](https://shellz.dev/docs/reference/cli.md#shellz-decrypt). ## 6. The agent protects itself A user with root access could try to stop recording. shellz makes that loud rather than quiet: - The recorder unit refuses manual stops and restarts automatically. - Every 15 seconds, and immediately when `/etc/passwd` changes, the agent checks its files, the login shells and the SSH settings. Anything changed is put back and reported as a `tamper` alert. - Removing shellz requires approval from an owner or admin in the dashboard. Otherwise `shellz uninstall` is refused and reported. - If the agent stops sending heartbeats, the dashboard raises an alert after 90 seconds. See [Tamper resistance](https://shellz.dev/docs/security/tamper-resistance.md) for the full list and its limits. ## 7. Updates are signed The agent checks for a new release about every 6 hours. Releases are signed with the shellz release key, which is pinned in the agent and the installer, and the agent refuses older versions. New versions run a self-test before switching, and the agent rolls back automatically if the new version fails to start. See [Updates and removal](https://shellz.dev/docs/guides/updates-and-removal.md). # Choose an install method URL: https://shellz.dev/docs/install > Which install fits your infrastructure, from plain Linux servers to Kubernetes pods. shellz runs in one of two modes. Pick the one that matches where you can install software. ## On a host Use this when you control the machine: a VM, a bare-metal server, an LXC container or a Proxmox guest. shellz installs as a systemd service and records every login on the machine. If Docker runs on that machine, `docker exec` into any of its containers is recorded too, with nothing added to your images. - [Linux server](https://shellz.dev/docs/install/linux.md): any systemd Linux server. - [Docker host](https://shellz.dev/docs/install/docker-host.md): a Linux server that runs Docker. Same install, plus container sessions. ## Inside the container Use this when you can't install anything on the host: Kubernetes, ECS and Fargate, Cloud Run, or dev containers. shellz is added to your image and runs as PID 1 in front of your app. Every container that starts with a fleet token joins a **fleet** and records every shell opened in it. - [Container image](https://shellz.dev/docs/install/container-image.md): add shellz to any image. Start here for all container platforms. - [Kubernetes](https://shellz.dev/docs/install/kubernetes.md): `kubectl exec` into pods. - [ECS, Fargate and Cloud Run](https://shellz.dev/docs/install/ecs-cloud-run.md): ECS Exec and other managed container platforms. - [Dev containers](https://shellz.dev/docs/install/dev-containers.md): VS Code and Codespaces-style development containers. ## Which one? | Your setup | Use | |---|---| | Linux VMs or servers you SSH into | [Linux server](https://shellz.dev/docs/install/linux.md) | | Servers running Docker or Docker Compose | [Docker host](https://shellz.dev/docs/install/docker-host.md) | | Kubernetes (EKS, GKE, AKS or self-managed) | [Kubernetes](https://shellz.dev/docs/install/kubernetes.md) | | ECS on Fargate, or Cloud Run | [ECS, Fargate and Cloud Run](https://shellz.dev/docs/install/ecs-cloud-run.md) | | ECS on EC2 instances you manage | Either: [Docker host](https://shellz.dev/docs/install/docker-host.md) on the instances, or [in the image](https://shellz.dev/docs/install/ecs-cloud-run.md) | | Dev containers | [Dev containers](https://shellz.dev/docs/install/dev-containers.md) | | Podman, rootless Docker, containerd without Docker | [Container image](https://shellz.dev/docs/install/container-image.md) | > [!NOTE] > Don't install both on the same container. A host install already records `docker exec` into every container on that host, so the in-container install is only needed where you can't reach the host. ## What both modes have in common - The same encryption, signing and storage. A container session and an SSH session look the same in the dashboard. - The same [recording policy](https://shellz.dev/docs/guides/recording-policy.md) options: fail closed, buffer or fail open. - Outbound HTTPS only, to `api.shellz.dev`, `ingest.shellz.dev` and `releases.shellz.dev`. Nothing listens for inbound connections. - Linux on amd64 (x86_64) or arm64 (aarch64). # Linux server URL: https://shellz.dev/docs/install/linux > Install shellz on a systemd Linux server, what the installer changes, and supported distributions. One command installs shellz on a Linux server and starts recording every login on it. This page covers requirements, what changes on the server, and distribution-specific notes. ## Requirements - Linux with systemd, on amd64 (x86_64) or arm64 (aarch64). - Root access through `sudo`. - `curl` and OpenSSL 3, used by the installer to download and verify the release. - If the server runs OpenSSH, its `sshd_config` must include `/etc/ssh/sshd_config.d/*.conf`. Current versions of every supported distribution do. - Outbound HTTPS (port 443) to `api.shellz.dev`, `ingest.shellz.dev` and `releases.shellz.dev`. ## Supported distributions | Distribution | Status | |---|---| | Ubuntu 24.04 | Tested | | Debian 13 | Tested | | Rocky Linux 9, AlmaLinux 9 | Tested, including with SELinux enforcing | | Fedora 44 | Tested | | Arch Linux | Tested | | Ubuntu 22.04, Debian 12, RHEL 9, Amazon Linux 2023 | Expected to work (same packages and OpenSSL 3) | | Amazon Linux 2, RHEL, Rocky and Alma 8, anything with OpenSSL 1.x | Not supported. The installer refuses and says why. | LXC containers, systemd-nspawn machines and Proxmox guests count as servers, because they boot their own systemd. To record inside application containers (Docker, Kubernetes and so on), see [Choose an install method](https://shellz.dev/docs/install.md). ## Install In the dashboard, open **~/servers** and click **Add server**. Copy the command and run it on the server: ```bash curl -fsSL https://get.shellz.dev | sudo sh -s -- szt_ ``` Each join token works once and expires after 10 minutes. It also carries the fingerprint of your org's public key, so the server only trusts keys from your org. The server appears in **~/servers** under its hostname, without the domain part. If that name is taken, shellz adds a number (`web-2`). The installer refuses to run, without changing anything, if: - it isn't running as root, on systemd, on amd64 or arm64; - `curl` or OpenSSL 3 is missing (it names the package to install for your package manager); - it's running inside an application container; - sshd doesn't read `/etc/ssh/sshd_config.d` (it tells you the `Include` line to add); - the server is already enrolled. If the release signature or checksum doesn't match, it stops before installing anything. ## What the installer changes | What | Where | |---|---| | The shellz binary | `/usr/local/bin/shellz` | | The session shell | `/usr/local/libexec/shellz-shell`, also added to `/etc/shells` | | The recorder service | `/etc/systemd/system/shellz-recorder.service`, enabled at boot | | Configuration (no secrets) | `/etc/shellz` | | State: server key, pinned org keys, policy, spool | `/var/lib/shellz` (root only) | | Recorder socket | `/run/shellz/broker.sock` | | SSH setting | `/etc/ssh/sshd_config.d/00-shellz.conf`, containing `PermitUserRC no` | | Login shells | root and regular accounts (UID from `UID_MIN` up to 59999) are switched to the session shell. Original shells are saved and restored on removal. | On SELinux systems, shellz also installs a one-rule policy module so recorded sessions keep the login's own SELinux context. SELinux is never set to permissive. Nothing else on the server changes. shellz doesn't install packages, open ports, or modify PAM. ### Why `PermitUserRC no` OpenSSH runs `~/.ssh/rc` for each login before the shell starts, which would let a user run commands outside the recording. shellz turns it off. ### Service accounts Accounts below `UID_MIN` (usually 1000) other than root, and accounts whose shell is `nologin` or `false`, keep their shells. If any of them does start a login session, the session auditor notices and, depending on the [recording policy](https://shellz.dev/docs/guides/recording-policy.md), ends it and raises an alert. ## Check that it's working ```bash sudo /usr/local/bin/shellz status ``` `Recorder running` and `Sessions N active` mean the server is recording. Then SSH in from another terminal and watch the session appear in the dashboard. `--json` prints the same information as JSON for monitoring tools. ## Distribution notes ### Rocky, Alma, RHEL and Fedora sudo's `secure_path` on these distributions doesn't include `/usr/local/bin`, so `sudo shellz status` reports "command not found". Use the full path, `sudo /usr/local/bin/shellz status`. Every hint shellz prints already does. With SELinux enforcing, the installer needs `semodule` (package `policycoreutils`, installed by default). Logins mapped to confined SELinux users (`user_u`, `staff_u`, `sysadm_u`, `guest_u`, `xguest_u`) aren't supported: they can't reach the recorder, so their logins are refused, or let through unrecorded under fail-open. ### Arch Linux Check that `/etc/ssh/sshd_config` contains `Include /etc/ssh/sshd_config.d/*.conf`. Current Arch packages ship it; older configurations may not. ### Unprivileged LXC containers shellz can't set the immutable flag (`chattr +i`) on its files in an unprivileged LXC container. It still installs and records, still repairs changes every 15 seconds, and raises one `immutable-unsupported` warning. ## Docker on the same server If Docker runs on the server, the same install records container sessions too. See [Docker host](https://shellz.dev/docs/install/docker-host.md). ## Troubleshooting **"This server is already enrolled."** shellz is already installed. Run `sudo /usr/local/bin/shellz status`. To move the server to another org, [remove it](https://shellz.dev/docs/guides/updates-and-removal.md#removing-shellz) first. **"This join token is no longer valid."** Tokens work once and expire after 10 minutes. Click **Add server** for a new command. **"The join token checksum does not match."** The token was cut off or mistyped while copying. Copy the whole command again. **"sshd does not read sshd_config.d."** Add `Include /etc/ssh/sshd_config.d/*.conf` near the top of `/etc/ssh/sshd_config`, then run the install command again. Nothing was changed. **OpenSSL 1.x.** Amazon Linux 2 and EL8 ship OpenSSL 1.1, which can't verify the Ed25519 release signature. Upgrade to a distribution with OpenSSL 3. # Docker host URL: https://shellz.dev/docs/install/docker-host > Record docker exec, docker attach and docker compose exec on a Linux server that runs Docker, without changing your images. On a Linux server that runs Docker, the standard install also records every interactive session into a container: `docker exec`, `docker attach`, `docker run -it` and `docker compose exec`. Your images and containers don't change. ## Install Install shellz on the server exactly as described in [Linux server](https://shellz.dev/docs/install/linux.md): ```bash curl -fsSL https://get.shellz.dev | sudo sh -s -- szt_ ``` shellz finds Docker automatically. Docker isn't restarted and running containers keep running. If you install Docker after shellz, it's picked up within 15 seconds. Check it: ```bash sudo /usr/local/bin/shellz status ``` ```text title="Output" ... Containers recording Docker sessions on /run/docker.sock ``` Right after install, the line may end with `(recording; dockerd moves to /run/shellz/dockerd.sock at its next restart)`. That's normal: sessions are already recorded, and Docker switches sockets the next time it restarts. A line starting with `NOT RECORDED` explains what's wrong; see [What isn't covered](https://shellz.dev/docs/install/docker-host.md#what-isnt-covered). ## How it works Every Docker command talks to the Docker daemon through the API socket, `/run/docker.sock`. shellz moves the daemon to a private socket, `/run/shellz/dockerd.sock`, readable only by root, and serves the original socket path itself with the same owner and permissions. Members of the `docker` group keep working as before. shellz forwards every Docker API call to the daemon unchanged. When a call opens a session into a container (exec or attach), shellz records the stream in both directions before passing it on. The change is made with systemd drop-in files (`zz-shellz.conf`) for `docker.socket` and `docker.service`, which are removed when shellz is removed. If someone deletes or replaces the socket, shellz puts it back within a second and raises a `tamper-repaired` alert. ## What gets recorded | Command | Recorded | |---|---| | `docker exec -it bash` | Yes, as a full session | | `docker exec ` (no terminal) | Yes, as a command session | | `docker attach ` | Yes | | `docker run -it ` | Yes, as an attach session | | `docker compose exec sh` | Yes | | Docker API calls from tools and SDKs using the socket | Exec and attach calls, yes | | `docker logs`, `docker ps`, image builds and pulls | No, these aren't sessions | Each container session shows: - the container's ID, name and image, and whether it was exec or attach; - the user inside the container; - the server account that called Docker, read from the kernel, not from anything the client says; - a link to the parent session when the `docker` command was run from a recorded SSH session. As with SSH sessions, input typed at hidden prompts isn't recorded. ## When recording isn't possible If shellz can't record a new container session, for example because the spool is full while the network is down, the [recording policy](https://shellz.dev/docs/guides/recording-policy.md) decides what happens: - **Fail closed** and **buffer**: the Docker API returns an error, the client shows `shellz: recording unavailable, container session refused`, and a `session-refused` alert is raised. - **Fail open**: the session runs. ## What isn't covered shellz raises an alert when it finds Docker set up in a way it can't record: | Alert | Cause | Fix | |---|---|---| | `docker-tcp-listener` (critical) | The Docker daemon also accepts API calls on a TCP port. Calls over TCP bypass recording. | Remove the `-H tcp://...` flag or the TCP `hosts` entry. | | `unsupported-container-runtime` (warning) | `hosts` is set in `/etc/docker/daemon.json`, so shellz can't move the socket. | Move the socket settings into the systemd unit and remove `hosts` from `daemon.json`. | | `unsupported-container-runtime` (warning) | Docker runs without a `docker.service` unit, for example from a snap. | Use Docker's packages for your distribution. | | `unsupported-container-runtime` (warning) | Rootless Docker, Podman, or containerd tools (`ctr`, `nerdctl`, `crictl`) are in use. | Add shellz [inside the image](https://shellz.dev/docs/install/container-image.md) for those containers. | > [!WARNING] > Root on the server can reach containers without Docker, for example with `nsenter` into a container's namespaces. That's still a login on the server, so the outer SSH session records it, but the container session itself isn't recorded separately. ## Removing shellz from a Docker host When shellz is [removed](https://shellz.dev/docs/guides/updates-and-removal.md#removing-shellz), it deletes its drop-ins and gives the original socket back to Docker. If containers are running, shellz doesn't restart Docker. Instead it prints a note asking you to run this when convenient, so systemd manages the socket again: ```bash sudo systemctl restart docker.socket docker.service ``` # Container image URL: https://shellz.dev/docs/install/container-image > Add shellz to a container image so every exec into the container is recorded, on any platform. When you can't install software on the host, add shellz to your container image instead. shellz runs as PID 1 in front of your app, and records every shell opened in the container by `kubectl exec`, `docker exec`, ECS Exec or a dev-container terminal. Each container that starts joins a **fleet** using a fleet token, and shows up in the dashboard as a fleet member. This page covers the image and the fleet token. The platform guides add the deployment details: [Kubernetes](https://shellz.dev/docs/install/kubernetes.md), [ECS, Fargate and Cloud Run](https://shellz.dev/docs/install/ecs-cloud-run.md), [Dev containers](https://shellz.dev/docs/install/dev-containers.md). ## 1. Create a fleet In the dashboard, open **~/servers**, find **Container fleets** and click **Add fleet**. Give it a name (for example `api-prod`), choose the platform, and optionally set a maximum number of members. shellz shows the **fleet token** once, starting with `szf_`, along with setup snippets for your platform. Copy the token and store it in your secret manager. Unlike a server join token, a fleet token is reusable: every container started with it joins the fleet. ## 2. Add shellz to the image Add two lines to your Dockerfile: ```dockerfile title="Dockerfile" RUN curl -fsSL https://get.shellz.dev/container | sh ENTRYPOINT ["/usr/local/bin/shellz", "run", "--"] CMD ["/app/server", "--port", "8080"] # your original entrypoint and args ``` The install step runs as root, needs `curl` (or `wget`) and OpenSSL 3, and: 1. downloads the shellz release for the image's architecture and verifies its Ed25519 signature and SHA-256 checksum; 2. installs `/usr/local/bin/shellz`; 3. runs `shellz image-install`, which puts a small recording shim in front of the image's shells (`/bin/sh`, `/bin/bash`, `ash`, `dash`, `zsh`, `fish`) and keeps the real shells alongside. It never enrolls and never needs a token at build time. It refuses to run outside an image build or container, and refuses if you pass it a token. Then make `shellz run --` the entrypoint, with your original entrypoint and arguments as the command. If your image already has an `ENTRYPOINT`, move it into `CMD` after `shellz run --`. ### Without curl, sh or OpenSSL in the image For minimal and distroless images, use BuildKit's checksum-verified `ADD`. The dashboard's setup snippets fill in the current version and checksum for each architecture; you can also read them from `https://releases.shellz.dev/stable/linux-amd64.json` and `linux-arm64.json`. ```dockerfile tab="amd64" ADD --chmod=0755 --checksum=sha256: https://releases.shellz.dev/agent//shellz-linux-amd64 /usr/local/bin/shellz RUN ["/usr/local/bin/shellz", "image-install"] ENTRYPOINT ["/usr/local/bin/shellz", "run", "--"] CMD ["/app/server", "--port", "8080"] ``` ```dockerfile tab="arm64" ADD --chmod=0755 --checksum=sha256: https://releases.shellz.dev/agent//shellz-linux-arm64 /usr/local/bin/shellz RUN ["/usr/local/bin/shellz", "image-install"] ENTRYPOINT ["/usr/local/bin/shellz", "run", "--"] CMD ["/app/server", "--port", "8080"] ``` Images with no shell at all (distroless) have nothing for the shim to wrap. shellz still runs as PID 1 and watches for exec sessions, as described in [How sessions are caught](https://shellz.dev/docs/install/container-image.md#how-sessions-are-caught). ## 3. Pass the fleet token at runtime Each container reads the token when it starts, from either: - the file `/run/secrets/shellz-fleet-token` (recommended), or - the environment variable `SHELLZ_FLEET_TOKEN`. If both are set, the variable wins. > [!IMPORTANT] > Prefer the file. An environment variable set on the container is visible to anyone who can exec into it, for example with `docker exec env`. shellz removes the variable from your app's environment and from recorded sessions, but not from other execs. With plain Docker: ```bash tab="docker run" printf '%s' 'szf_' > shellz-fleet-token && chmod 0444 shellz-fleet-token docker run -d --stop-timeout 30 \ -v "$PWD/shellz-fleet-token:/run/secrets/shellz-fleet-token:ro" \ ghcr.io/acme/app:1.4 ``` ```yaml tab="docker compose" services: app: image: ghcr.io/acme/app:1.4 stop_grace_period: 30s secrets: [shellz-fleet-token] secrets: shellz-fleet-token: file: ./shellz-fleet-token # contains szf_ ``` ## 4. Check it's recording Start a container and open a shell in it: ```bash docker exec -it bash ``` The fleet page lists the container as a member with the status **recording**, and the session appears in the dashboard. ## Stopping gracefully When the container is stopped, shellz forwards the signal to your app, stops accepting new sessions, waits for your app to exit, and uploads the rest of the recordings before exiting. This drain takes up to 20 seconds (`SHELLZ_DRAIN_SECONDS`). Give containers a stop timeout of at least 30 seconds so the drain isn't cut short: `--stop-timeout 30` for `docker run`, `stop_grace_period: 30s` in Compose. Docker's default is 10 seconds. Kubernetes' default of 30 seconds is fine. ## How shellz runs your app As PID 1, `shellz run --` behaves like a minimal init: - It starts your command, forwards `SIGTERM`, `SIGINT`, `SIGHUP`, `SIGQUIT`, `SIGUSR1`, `SIGUSR2` and `SIGWINCH` to it, reaps zombie processes, and exits with your app's exit code. - Your app's own shell use (`sh -c` from scripts, process managers and so on) is never recorded and adds about 1 ms per shell started. - If the token is missing or rejected, or shellz can't reach the API, **your app still starts and runs**. Interactive sessions are refused until the container is enrolled, and shellz retries in the background. If something else is already PID 1, such as `docker run --init` or tini, shellz runs as its child and still works. ## How sessions are caught shellz uses two layers inside the container: 1. **The shell shim.** When a shell starts, the shim checks whether it was started by an exec into the container rather than by your app. Exec sessions go to the recorder; everything else goes straight to the real shell. 2. **The exec watcher.** Every 500 ms, PID 1 looks for processes that came from an exec and aren't being recorded, for example someone running the moved real shell or another interpreter directly. Under fail-closed, an unrecorded session with a terminal is killed and raises `unrecorded-container-exec`. Otherwise it's reported. ### Commands without a terminal Health checks and probes (`kubectl exec pod -- cat /tmp/ready`, `sh -c 'test -f /ready'`) run without a terminal. shellz never blocks them, even when recording is unavailable, so a recording problem can't fail your probes or restart your pods: - They're recorded as `command` sessions when recording is available. - When the same command produces the same output and exit status as a run in the last 10 minutes, shellz drops the duplicate recording and counts it. The dashboard shows one row with a `×count` badge instead of thousands of identical probe sessions. - If recording is unavailable, the command runs anyway, and shellz logs its command line, user, timing and exit status once it's back, with an `unrecorded-noninteractive-exec` warning. - Very short non-shell commands (under 500 ms) may be missed by the watcher. ## Fleet members Each running container is a member of its fleet. The fleet page shows each member's status: | Status | Meaning | |---|---| | **recording** | Enrolled and sending heartbeats. | | **waiting** | Enrolled, waiting for the first heartbeat. | | **stopped** | The container stopped cleanly and reported its exit. | | **lost** | No heartbeat for 3 minutes, and no clean stop. Raises a `member-lost` warning. | | **tampered** | The member has an open critical alert, or isn't accepting sessions. | Members are named after the pod, task or container. On Kubernetes, the pod's namespace and node are shown too. ## Managing the fleet token From the fleet page, owners and admins can: - **Rotate token**: issue a new token. The old one stops working for new containers straight away; running members keep recording. Update your secret and roll your deployment. - **Revoke**: turn the fleet off. Running members stop accepting sessions and raise a `member-disabled` warning. - **Delete fleet**: revoke the fleet and remove it from the dashboard. Its recordings are kept. Fleet tokens can't be used to read recordings; they only let a container join the fleet. A leaked token lets someone add members to the fleet, so rotate it if that happens. A maximum member count limits how many can be active at once; containers beyond it are refused and raise `fleet-full`. ## Settings See [Container settings](https://shellz.dev/docs/reference/container-settings.md) for every environment variable `shellz run` reads. # Kubernetes URL: https://shellz.dev/docs/install/kubernetes > Record kubectl exec into pods on any Kubernetes cluster, including EKS, GKE and AKS. On Kubernetes, shellz runs inside your pods' containers. Every `kubectl exec` into a container with shellz is recorded, and each pod shows up as a member of a fleet, named after the pod with its namespace and node. This works on any cluster, managed or not, because nothing is installed on the nodes. ## Before you start 1. [Create a fleet](https://shellz.dev/docs/install/container-image.md#1-create-a-fleet) with the platform **Kubernetes** and copy its token. 2. [Add shellz to your image](https://shellz.dev/docs/install/container-image.md#2-add-shellz-to-the-image). ## 1. Store the token as a Secret ```bash kubectl create secret generic shellz-fleet --from-literal=token=szf_ ``` Create it in each namespace where the workload runs, or sync it from your secret manager (External Secrets, Sealed Secrets and so on) as usual. ## 2. Mount it into the container ```yaml title="deployment.yaml" spec: template: spec: terminationGracePeriodSeconds: 30 containers: - name: app image: ghcr.io/acme/app:1.4 command: ["/usr/local/bin/shellz", "run", "--"] args: ["/app/server", "--port", "8080"] env: - name: SHELLZ_POD_NAME valueFrom: { fieldRef: { fieldPath: metadata.name } } - name: SHELLZ_POD_NAMESPACE valueFrom: { fieldRef: { fieldPath: metadata.namespace } } - name: SHELLZ_NODE_NAME valueFrom: { fieldRef: { fieldPath: spec.nodeName } } volumeMounts: - name: shellz-fleet mountPath: /run/secrets/shellz-fleet-token subPath: token readOnly: true volumes: - name: shellz-fleet secret: secretName: shellz-fleet defaultMode: 0444 ``` - **The token is mounted as a file**, not passed as `SHELLZ_FLEET_TOKEN`, because environment variables are visible to every `kubectl exec`. - **The downward-API variables** name the member after its pod, namespace and node. Without them, shellz falls back to the pod's hostname and service-account namespace. - **`terminationGracePeriodSeconds: 30`** (the Kubernetes default) leaves time for shellz to finish uploading recordings, which takes up to 20 seconds, before the kill. - **If your image's `ENTRYPOINT` is already shellz**, drop `command` and `args`. Optionally, set `SHELLZ_IMAGE` to the image reference so it's shown in the dashboard. ## 3. Roll out and test ```bash kubectl rollout status deploy/app kubectl exec -it deploy/app -- bash ``` The fleet page lists each pod as a member, and the session appears in the dashboard within a second or two. ## Probes Exec liveness and readiness probes run without a terminal, so shellz never blocks them, even if recording is unavailable. A recording problem can't fail your probes or restart your pods. Identical probe runs are grouped: the first run is recorded in full, later runs with the same output and exit status are counted on one row (`×count`) instead of being recorded again. A probe whose output changes is recorded each time. See [Commands without a terminal](https://shellz.dev/docs/install/container-image.md#commands-without-a-terminal). ## Sidecars, init containers and multi-container pods shellz records execs into the container it runs in. In a pod with several containers, add shellz to each container you want recorded. Init containers and sidecars that nobody execs into don't need it. ## Ephemeral debug containers `kubectl debug` adds a new ephemeral container from a separate image, which doesn't contain shellz, so it isn't recorded. If people use `kubectl debug` in production, restrict it with RBAC (the `pods/ephemeralcontainers` subresource) or an admission policy. > [!WARNING] > Similarly, someone with access to the node itself can enter a container's namespaces without `kubectl exec`. Install shellz on nodes you manage with the [Linux server](https://shellz.dev/docs/install/linux.md) install to record node logins. ## Removing shellz Remove the shellz entrypoint from the deployment (or roll back to an image without it), then revoke or delete the fleet on its page. # ECS, Fargate and Cloud Run URL: https://shellz.dev/docs/install/ecs-cloud-run > Record ECS Exec sessions on ECS and Fargate, and run shellz in Cloud Run and other managed container platforms. On ECS (including Fargate) and Cloud Run, shellz runs inside your containers. Each task or instance joins a fleet. shellz detects the platform on its own and reports task and container details to the dashboard. ## Before you start 1. [Create a fleet](https://shellz.dev/docs/install/container-image.md#1-create-a-fleet) with the platform **ECS / Fargate** or **Cloud Run**, and copy its token. 2. [Add shellz to your image](https://shellz.dev/docs/install/container-image.md#2-add-shellz-to-the-image). ## ECS and Fargate ### 1. Store the token in Secrets Manager ```bash aws secretsmanager create-secret --name shellz-fleet-token --secret-string szf_ ``` The task execution role needs `secretsmanager:GetSecretValue` on this secret. ### 2. Update the task definition ```json title="task-definition.json" { "containerDefinitions": [ { "name": "app", "image": "123456789012.dkr.ecr.eu-west-1.amazonaws.com/app:1.4", "entryPoint": ["/usr/local/bin/shellz", "run", "--"], "command": ["/app/server", "--port", "8080"], "stopTimeout": 30, "secrets": [ { "name": "SHELLZ_FLEET_TOKEN", "valueFrom": "arn:aws:secretsmanager:eu-west-1:123456789012:secret:shellz-fleet-token" } ] } ] } ``` - `stopTimeout: 30` gives shellz time to finish uploading recordings (up to 20 seconds) before the container is killed. - shellz reads the task, container name and image from the ECS task metadata endpoint, so you don't need to set them. > [!IMPORTANT] > ECS can only inject secrets as environment variables, and anyone with ECS Exec access to the task can read those. Either limit `ecs:ExecuteCommand` to the people who should have it, or have an init container write the secret to a shared volume mounted at `/run/secrets/shellz-fleet-token` and drop the `secrets` entry. ### 3. Test with ECS Exec ```bash aws ecs execute-command --cluster prod --task --container app --interactive --command "/bin/bash" ``` ECS Exec sessions are recorded like any other exec. The fleet page lists each task as a member. ### ECS on EC2 If you run ECS on EC2 instances you manage, you can install shellz on the instances instead, with the [Docker host](https://shellz.dev/docs/install/docker-host.md) install. That records every exec into every container on the instance without changing images, and records logins to the instance itself. ## Cloud Run Cloud Run has no built-in exec command. If you open shells in Cloud Run containers through other tooling, shellz records them like any other exec. Add shellz to the image, then mount the token from Secret Manager as a file: ```bash printf '%s' 'szf_' | gcloud secrets create shellz-fleet-token --data-file=- gcloud run deploy app \ --image=europe-docker.pkg.dev/acme/app/app:1.4 \ --set-secrets=/run/secrets/shellz-fleet-token=shellz-fleet-token:latest ``` The member is named after the Cloud Run service. ## Other platforms Nomad, Fly.io, Railway, Render and similar platforms work the same way: add shellz to the image, make `shellz run --` the entrypoint and give the container the token as a file at `/run/secrets/shellz-fleet-token` or as `SHELLZ_FLEET_TOKEN`. Set `SHELLZ_CONTAINER_NAME` and `SHELLZ_IMAGE` to label members, and give the container a stop timeout of 30 seconds or more. See [Container settings](https://shellz.dev/docs/reference/container-settings.md). # Dev containers URL: https://shellz.dev/docs/install/dev-containers > Record terminals opened in development containers, for people and for coding agents. Development containers are where coding agents often get a shell. With shellz in the dev container's image, every terminal opened in it is recorded, whether a person opened it in VS Code or an agent started it. ## 1. Create a fleet and save the token [Create a fleet](https://shellz.dev/docs/install/container-image.md#1-create-a-fleet) with the platform **Dev container**. Save the token on your machine: ```bash mkdir -p ~/.config/shellz printf '%s' 'szf_' > ~/.config/shellz/fleet-token ``` ## 2. Add shellz to the dev container's Dockerfile ```dockerfile title=".devcontainer/Dockerfile" FROM mcr.microsoft.com/devcontainers/base:ubuntu RUN curl -fsSL https://get.shellz.dev/container | sh ENTRYPOINT ["/usr/local/bin/shellz", "run", "--"] CMD ["sleep", "infinity"] ``` ## 3. Update devcontainer.json ```json title=".devcontainer/devcontainer.json" { "build": { "dockerfile": "Dockerfile" }, "overrideCommand": false, "mounts": [ "source=${localEnv:HOME}/.config/shellz/fleet-token,target=/run/secrets/shellz-fleet-token,type=bind,readonly" ], "runArgs": ["--stop-timeout", "30"], "containerEnv": { "SHELLZ_FLEET_PLATFORM": "devcontainer" } } ``` - `overrideCommand: false` keeps the image's shellz entrypoint. Without it, the dev container tooling replaces the command and shellz never starts. - The token is bind-mounted read-only from your machine instead of being set as an environment variable, which every terminal could read. - `--stop-timeout 30` lets shellz finish uploading when the container stops. Rebuild the container. Each new VS Code terminal is a recorded session. ## What's recorded - Terminals opened in the editor, and shells started by coding agents through the editor's terminal. - `docker exec` into the dev container from your machine. The editor's own background processes (language servers, extensions, file watchers) aren't sessions and aren't recorded. > [!NOTE] > A [Docker host](https://shellz.dev/docs/install/docker-host.md) install on the machine running your dev containers records `docker exec` into them, but not editor terminals: those are started by the editor's server inside the container. To record editor terminals, use the in-image install on this page. # Security overview URL: https://shellz.dev/docs/security > What shellz protects, who can read recordings in each mode, and the limits of the design. Session recordings contain the most sensitive thing on a server: what people and agents typed and saw. shellz is designed so that a recording is encrypted before it leaves the server, can't be changed without detection once it's written, and can only be read by your team. ## Summary - **Encrypted on the server.** Each session is encrypted with its own AES-256-GCM key before it's written to disk or sent anywhere. See [Encryption](https://shellz.dev/docs/security/encryption.md). - **Signed and chained.** Each server signs every piece of every recording with its own Ed25519 key, and links each piece to the previous one by hash. Missing or altered pieces are detected and shown. - **Write-once storage.** Stored recordings can't be overwritten or deleted for 365 days, by anyone, including shellz. - **Decrypted only in your browser**, after a recent sign-in, with every view written to the audit log. - **Perfect Secrecy** (optional): the org key lives only in your team's browsers, unlocked by their passkeys. shellz can't decrypt your recordings. See [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md). - **Hard to turn off quietly.** Changes to the recorder are reverted and reported, removal needs approval in the dashboard, and silent servers raise alerts. See [Tamper resistance](https://shellz.dev/docs/security/tamper-resistance.md). ## Who can read recordings | | Standard mode | Perfect Secrecy | |---|---|---| | Your team, after signing in | Yes | Yes, with a passkey that holds the org key | | Someone with a copy of shellz's database or storage | No | No | | shellz, as the operator of the service | Technically able to, by using the org key it holds | No. It never has the org key. | | An attacker who takes over the shellz service | Could release session keys | Gets ciphertext and metadata only | | The servers that produce recordings | No. A server can encrypt, but can't decrypt even its own past recordings. | No | In **standard mode**, the org's private key is stored by shellz, sealed with a separate encryption key, and shellz uses it to release a session's key to a signed-in teammate. That makes setup instant and recovery easy, but it means you trust shellz to protect the key and only release it to your team. In **Perfect Secrecy** mode, the private key is generated in an owner's browser and only ever stored encrypted to your teammates' passkeys and to a recovery kit you keep. shellz stores ciphertext and public keys. ## What shellz can see In both modes, shellz can read session metadata, because it's needed to list, search and alert on sessions: - server, fleet and container names, container images; - the account name and UID, the source IP address of SSH sessions, and the terminal device; - start and end time, size, exit status and integrity; - in standard mode only, the command line of non-interactive sessions (`ssh host cmd`, `docker exec ... cmd`). In Perfect Secrecy mode it's encrypted with the session. shellz can't see what was typed or shown in a session, in either mode, without using a decryption key. ## Threat model shellz is built to keep recordings trustworthy against: | Threat | Protection | |---|---| | A user on the server wants to avoid being recorded | Login shells point at the recorder; sessions that bypass it are ended or alerted on; changes to shellz's files and settings are reverted within 15 seconds and reported | | A user with root stops or removes the recorder | The service refuses manual stops and restarts if killed; removal needs a signed approval from your org; missing heartbeats alert within 90 seconds | | Someone edits or deletes a recording after the fact | Signed, hash-chained segments; write-once storage with a 365-day lock; gaps and tampering are shown on the session | | Someone reads recordings from storage or a database backup | Everything is encrypted with per-session keys; storage holds no usable keys | | A stolen join or fleet token | Join tokens are single-use and expire after 10 minutes. Fleet tokens can be rotated and revoked. Neither can read recordings. | | A forged server sends fake recordings | Every segment is signed by an enrolled server's key and checked on arrival | | shellz itself, or an attacker inside it (Perfect Secrecy) | The org key never reaches shellz; servers pin your org key and only accept signed rotations; policies must be signed in your browser | ## Limits These are outside what shellz can protect against: - **Root on a server sees that server's sessions as they happen.** Root can read process memory and terminals directly. shellz makes interfering with recording visible, not impossible. - **A compromised server can lie about future sessions.** If an attacker takes over a server, recordings from that point on depend on that server. Earlier recordings stay intact and signed. - **The dashboard's code is trusted.** In Perfect Secrecy mode, recordings are decrypted by JavaScript served by shellz. Malicious code served to your browser could leak a key you unlock. `shellz decrypt` lets you verify and decrypt recordings offline, without the web app. - **Processes that don't start a session aren't recorded.** shellz records every agent that logs into your servers or opens a shell in a container. An agent running as a service or cron job never logs in, so use your platform's audit tools for those. ## Compliance Recording privileged sessions, keeping tamper-evident records for a fixed period and reviewing access is part of many control frameworks. shellz helps you meet SOC 2, ISO 27001, PCI DSS and HIPAA requirements for monitoring privileged access, alongside your own policies and other controls. ## Reporting a vulnerability Email [hello@shellz.dev](mailto:hello@shellz.dev) with "security" in the subject, and include steps to reproduce. Please don't open public issues for vulnerabilities. # Encryption URL: https://shellz.dev/docs/security/encryption > The keys, algorithms and formats shellz uses to encrypt, sign and verify recordings. This page describes exactly how a recording is protected, from the server to your browser. The same design applies to servers and to containers. ## Keys at a glance | Key | Type | Where it lives | Used for | |---|---|---|---| | Session data key | 32 random bytes | Only in the recorder's memory during the session, and wrapped in each segment header | Encrypting one session's content (AES-256-GCM) | | Org encryption key | X25519 key pair | Public half on every server. Private half: sealed by shellz (standard) or only in your team's browsers (Perfect Secrecy) | Wrapping session data keys | | Org signing key | Ed25519 key pair | Same as the org encryption key | Signing policies and org key rotations | | Server key | Ed25519 key pair | Created on the server at enrollment; the private half never leaves it (containers keep it in memory) | Signing segments and requests to shellz | | Release key | Ed25519 | Pinned in the agent and installers | Verifying agent downloads and updates | All signatures use a domain-separation prefix (for example `shellz:segment:v1:`), so a signature made for one purpose can never be valid for another. ## Encrypting a session When a session starts, the recorder: 1. Generates a 32-byte random **data key** for the session. 2. **Wraps** the data key to the org's current X25519 public key: - generates an ephemeral X25519 key pair and computes the shared secret with the org public key; - derives a wrapping key with HKDF-SHA256, using the ephemeral and org public keys as salt and `shellz:session-key:v1:` as info; - encrypts the data key with AES-256-GCM under the wrapping key, with the org key ID as associated data. 3. Records the session as two streams: an asciicast v2 event stream (output, input, resizes) and a plain-text transcript. About once a second, the recorder closes a **segment**: 1. The new events are serialized as JSON and compressed with gzip. 2. The result is encrypted with AES-256-GCM under the data key, with a fresh 12-byte nonce, and with `shellz:segment-content:v1::` as associated data. A segment can't be moved to another session or position. 3. A **header** is built: org, server and session IDs, the sequence number, the SHA-256 hash of the ciphertext, the hash of the previous segment's header, the wrapped data key, the org key ID, a timestamp, and on the first segment the session details (kind, user, source address, terminal, container). 4. The header is signed with the server's Ed25519 key. Segments are at most 1 MiB. The recorder discards the data key when the session ends. Only the wrapped copy in the headers remains, so the server can't decrypt a session once it's over. ### Hidden input Keystrokes are only recorded while the terminal echoes them. Passwords typed at `sudo`, `passwd`, `ssh` or `read -s` prompts turn echo off, so they're never written to the recording. ## The hash chain Each header contains `prevHash`, the SHA-256 hash of the previous segment's header. The first segment of a session references a fixed genesis value, and the last one is marked `final`. Together they form a chain: ```text seq 0 seq 1 seq 2 (final) ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ prevHash: 0… │◀─────│ prevHash: h0 │◀─────│ prevHash: h1 │ │ cipherHash │ │ cipherHash │ │ cipherHash │ │ signature │ │ signature │ │ signature │ └──────────────┘ └──────────────┘ └──────────────┘ ``` When a recording is played, every segment is checked: - the server's signature over the header; - that the ciphertext matches `cipherHash`; - that sequence numbers are consecutive and every `prevHash` matches; - that nothing follows the final segment. The result is shown as the integrity badge: **verified**, **gap** (a segment is missing, for example because a server lost its disk before uploading) or **tampered** (a signature or hash didn't match). ## Upload and storage The server uploads segments over HTTPS. Before accepting a segment, shellz checks that it comes from an enrolled server of your org, that the server's signature is valid, and that it extends the session's chain: - a segment with a bad signature is rejected and raises `segment-signature-invalid`; - a segment that conflicts with one already stored (same position, different content) is rejected, the session is shown as **tampered**, and a `segment-conflict` alert is raised; - a segment that skips ahead is stored, and the session is marked as having a gap. Accepted segments are written once to object storage under a retention lock: for 365 days, they can't be overwritten or deleted by anyone, including shellz. Servers also sign every request to the shellz API (heartbeats, alerts, policy and key sync) with their server key, so a stolen network position can't impersonate a server. ## Decrypting in the browser To play a recording, the browser: 1. gets the session's data key, either from shellz (standard mode) or by unwrapping it itself with the org private key unlocked by your passkey ([Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md)); 2. downloads the segments as ciphertext; 3. verifies signatures and the chain as above; 4. decrypts and decompresses each segment with the Web Crypto API. Opening a recording requires a sign-in within the last 10 minutes, and each view is written to the audit log. ## Policies and key rotation Each server follows a **policy**: whether recording is on, what happens when the spool fills up, and whether removal is allowed. Policies are signed by the org's Ed25519 key and carry a revision number. A server only applies a policy that is signed by your org, names that server (or its fleet), and has a higher revision than the one it has. A replayed or downgraded policy is rejected with a `policy-rejected` alert. A new org key is signed by the previous one, forming a chain back to the key the server pinned at enrollment. A server only switches to a new key whose chain it can verify, and raises `org-key-rejected` otherwise. ## Releases The installers and the agent verify every release against the pinned release key before installing it: an Ed25519 signature over the version, architecture, URL, SHA-256 hash and size. The agent refuses to install an older version than it runs. ## Offline verification `shellz decrypt` verifies and decrypts recordings without the dashboard or network access: given a directory of segment files and either the org private key or one session's data key, it checks the chain (and server signatures with `--host-key`) and writes `.cast` and `.txt` files. See the [CLI reference](https://shellz.dev/docs/reference/cli.md#shellz-decrypt). ## Algorithms | Purpose | Algorithm | |---|---| | Content encryption | AES-256-GCM, 96-bit random nonces | | Key agreement | X25519 | | Key derivation | HKDF-SHA256 | | Signatures | Ed25519 | | Hashes | SHA-256 | | Compression | gzip | | Transport | TLS 1.2 or later | # Perfect Secrecy URL: https://shellz.dev/docs/security/perfect-secrecy > How passkeys, the WebAuthn PRF extension and browser-held org keys make sure only your team can decrypt recordings. With Perfect Secrecy turned on, the private key that opens your recordings exists only in your team's browsers, unlocked by their passkeys. shellz stores ciphertext, public keys and keys encrypted to your team, and can't decrypt your recordings. This page explains how that works, step by step, and what it does and doesn't protect against. ## Standard mode and Perfect Secrecy Every org has an **org key**: an X25519 key pair that servers encrypt session keys to, and an Ed25519 key pair that signs policies. The difference between the two modes is who holds the private halves. | | Standard mode | Perfect Secrecy | |---|---|---| | Org private keys generated | By shellz | In an owner's browser | | Org private keys stored | By shellz, sealed with a separate service key | Only encrypted to each teammate's passkey-protected key, and to your recovery kit | | Who unwraps a session key | shellz, then sends it to your browser over TLS | Your browser | | Who signs server policies | shellz | Your browser | | Command line of `ssh host cmd` sessions | Visible to shellz, for search | Encrypted with the session | | If you lose access | shellz can restore it | Your recovery kit restores it | Recordings are encrypted on your servers in both modes. Standard mode makes setup instant; Perfect Secrecy removes shellz from the list of parties you have to trust with recordings. ## The key hierarchy ```text your passkey ──(WebAuthn PRF)──▶ 32-byte secret │ HKDF-SHA256 ▼ key-encryption key (stays in the browser) │ AES-256-GCM ▼ your member key (X25519), wrapped │ X25519 + HKDF + AES-256-GCM ▼ org private keys (X25519 + Ed25519), one envelope per teammate │ ▼ session data keys, wrapped by your servers to the org public key │ AES-256-GCM ▼ recording segments ``` Each layer only exists in decrypted form in the browser tab of a teammate who just used their passkey. ### 1. Passkeys with the PRF extension A passkey is a key pair held by an authenticator: your phone or laptop's platform authenticator, a password manager, or a security key. Signing in proves you hold it. shellz also uses the WebAuthn **PRF extension** (built on CTAP `hmac-secret`): given a salt, the authenticator returns a 32-byte secret computed from a key that never leaves it, and only after user verification (fingerprint, face or PIN). The same passkey and salt always give the same secret, and a different passkey gives an unrelated one. shellz uses the fixed salt `SHA-256("shellz-prf-v1")`. The secret is returned only to the browser, for `shellz.dev` only; it's never sent to shellz. When you add a passkey, shellz checks that it's discoverable, requires user verification and returns a stable PRF secret. Authenticators known to pass: - iCloud Keychain (Safari or Chrome on macOS 15, iOS 18 or later) - Google Password Manager (Chrome on desktop or Android) - 1Password (recent browser extension) - YubiKey 5 series (firmware 5.7 or later) ### 2. Your member key The first time you set up a passkey **for encryption** (in **~/security**, **Your passkeys**, **Set up for encryption**), your browser: 1. generates an X25519 key pair, your **member key**; 2. asks your passkey for its PRF secret; 3. derives a key-encryption key: `HKDF-SHA256(PRF secret, info = "shellz:user-key-wrap:v1:" + credential ID)`. It's imported into Web Crypto as non-extractable; 4. encrypts the member private key with AES-256-GCM under that key, bound to your user ID; 5. sends shellz the member public key, the credential ID and the encrypted private key. You can set up several passkeys for encryption. Each one wraps the same member key, so any of them unlocks it. shellz stores only your member public key and encrypted copies of the private key. Without one of your passkeys and your fingerprint, face or PIN, they're useless. #### Several devices A synced passkey, for example in iCloud Keychain, Google Password Manager or 1Password, has the same credential ID and gives the same PRF secret on every device it syncs to. So once it's set up for encryption, it unlocks your member key on all of those devices with no extra setup: the browser fetches your encrypted member key from shellz, and the passkey unlocks it. Your password manager never receives your member key, and shellz never receives the passkey's secret. Neither can open recordings alone. Protect the password manager account accordingly: anyone who takes it over and can unlock one of your devices can use the passkey. A passkey that doesn't sync, such as one on a security key or stored only on one device, works where it is. To use another device, set up a second passkey for encryption, or use the security key there. Using a passkey from another device, for example scanning a QR code with your phone to sign in on a computer, depends on the computer's browser passing the PRF secret back from the phone. Not every browser does yet. See [Troubleshooting](https://shellz.dev/docs/security/perfect-secrecy.md#troubleshooting). ### 3. Org keys and envelopes When an owner turns on Perfect Secrecy, the owner's browser generates the new org keys: an X25519 pair for encryption and an Ed25519 pair for signing. They're never sent to shellz unencrypted. Instead, the browser seals an **envelope** of the two private keys to each recipient's X25519 public key: 1. generate an ephemeral X25519 key pair and compute the shared secret with the recipient's key; 2. derive a key with HKDF-SHA256, using both public keys as salt and `shellz:org-key-envelope:v1::` as info; 3. encrypt the private keys with AES-256-GCM, with `:` as associated data, so an envelope can't be passed off as another key's or another person's. Recipients are the teammates who have been approved, plus your recovery kit. ### 4. Opening a recording When you open a recording: 1. If your last sign-in is older than 10 minutes, shellz asks you to sign in again with your passkey. 2. Your browser fetches your encrypted member key and your envelope for the org key the recording uses. 3. Your passkey's PRF secret unlocks your member key, which opens the envelope. 4. The org X25519 private key unwraps the session's data key from the segment header. 5. Your browser verifies signatures and the hash chain, then decrypts the segments. Unlocked keys are kept in memory in that tab only, never written to storage, and cleared when you leave the page. If shellz is asked for a session key in Perfect Secrecy mode, it can only answer that it doesn't hold the key. ## Turning on Perfect Secrecy Only an org owner can turn it on, in **~/security** under **Perfect Secrecy**, with **Set up Perfect Secrecy**. The setup takes about a minute and must happen in one browser tab: 1. **How Perfect Secrecy works**: a summary of what changes. 2. **Protect your key with a passkey**: your member key is created and wrapped with your passkey, as above. Your passkey must support PRF. 3. **Generate org keys in this browser**: the new X25519 and Ed25519 key pairs. 4. **Save the recovery kit**: see [Recovery kit](https://shellz.dev/docs/security/perfect-secrecy.md#recovery-kit). You must save it to continue. 5. **Switch on Perfect Secrecy**: the browser publishes the new org public keys, your envelope and the recovery envelope. The new key record is signed by the previous (standard) org key, so your servers accept the change. Fleet policies are re-signed by the new key. 6. **Re-encrypt existing recordings**: your browser receives each existing session's data key once, wraps it to the new org key and uploads the result. Recordings themselves aren't modified; they're signed and write-once. When every session has a new wrapped key, shellz destroys the standard-mode private keys. Servers pick up the new org key on their next heartbeat, within about 30 seconds, and encrypt every new session to it. > [!IMPORTANT] > Perfect Secrecy can't be turned off. Once the standard keys are destroyed, there's nothing left for shellz to decrypt with. > [!NOTE] > Recordings made before you turned on Perfect Secrecy were protected by standard mode at the time they were made. After the switch, they're readable only with the new org key. ## Adding a teammate 1. An owner or admin invites them from **~/team**. 2. The teammate signs up, then sets up a passkey for encryption in **~/security**. Their status shows as "awaiting approval", along with a short fingerprint of their member key (for example `3f9a 0c12 7be4 51d8`). 3. An owner or admin opens **~/team**, finds them under **Pending access approvals**, compares the fingerprint with the teammate (in person, or over a channel you trust), and clicks **Approve**. 4. The approver's browser unlocks the org keys with their passkey and seals new envelopes to the teammate's member key, for the current org key and any older Perfect Secrecy keys the approver holds. The fingerprint check matters: it's what stops a substituted public key from being approved. shellz shows the same fingerprint to both people, computed in each browser. ## Removing a teammate When an owner or admin clicks **Remove** for a teammate in Perfect Secrecy mode: 1. Their membership ends immediately, and with it their access to the dashboard. 2. The admin's browser then **rotates the org key**: it generates new org keys, signs the new key record with the old signing key, seals envelopes to every remaining teammate and the recovery kit, and re-signs fleet policies. 3. shellz revokes the removed person's envelopes. Servers move to the new key on their next heartbeat. Anything recorded after the rotation is encrypted to a key the removed person never had. If the rotation can't finish (for example, the passkey prompt is cancelled), the dashboard asks you to finish it in **~/security** with **Rotate org key**. > [!WARNING] > Rotation protects future recordings. A former teammate who kept copies of the older keys and could still obtain the ciphertext could decrypt older recordings. shellz no longer serves them anything, but it can't erase what they already had. Owners and admins can also rotate at any time from **~/security**. ## Recovery kit The recovery kit is a 56-character code, printed in 8 groups of 7, for example: ```text K7QX2MA-4RT9WZP-... ``` It encodes 32 random bytes plus a 3-byte checksum, so typos are caught. From it, the browser derives an X25519 key pair: `HKDF-SHA256(secret, info = "shellz:recovery:v1:" + orgId)`. shellz stores only the public half, so every key rotation can include an envelope for the recovery kit without anyone entering the code. If every passkey that held the org key is lost, an owner or admin can recover from **~/security**: they set up a new passkey for encryption and enter the code. The code is checked in the browser and opens the recovery envelope, and the org keys are sealed to their new member key. The code is never sent to shellz. They can then approve the rest of the team again. > [!CAUTION] > shellz keeps no copy of the recovery kit and can't reset it. Without the kit, losing every passkey that holds the org key means losing access to your recordings. Store it offline, for example printed in a safe, and replace it with **Create a new recovery kit** if it might be exposed. ## How servers stay on your key A malicious service could try to make your servers encrypt to a key it controls. Perfect Secrecy prevents that: - **Pinned at enrollment.** Every join and fleet token contains a fingerprint of your org's signing key, generated in the dashboard. A server only accepts org keys that match it. - **Chained rotations.** Each new org key must be signed by the previous one. A server only switches to a key whose whole chain it can verify, and raises `org-key-rejected` otherwise. - **Signed policies.** In Perfect Secrecy mode, only your browser can sign policies, so shellz can't turn recording off, weaken the failure mode, or authorize removal of a server. - **Pinned in your browser.** The dashboard pins your org's first signing key on first use and verifies the key chain before approving teammates or decrypting. ## What Perfect Secrecy doesn't protect - **The dashboard's JavaScript is trusted.** Recordings are decrypted by code that shellz serves. Code altered to leak keys would get the keys of anyone who unlocked them while it was served. For high-assurance review, use `shellz decrypt` offline. See the [CLI reference](https://shellz.dev/docs/reference/cli.md#shellz-decrypt). - **Metadata is visible to shellz.** Server names, account names, source addresses, timing and sizes are needed to list sessions and raise alerts. See [What shellz can see](https://shellz.dev/docs/security.md#what-shellz-can-see). - **A teammate's device.** Anyone who controls an approved teammate's unlocked device and passkey can open recordings, just like the teammate. - **The server itself.** Root on a server sees sessions on that server as they happen. ## Troubleshooting When a passkey can't unlock your key, shellz explains which of these cases it ran into, with steps for the device you're on. ### Passkey from your phone or a security key doesn't work on this computer You signed in on a computer by scanning a QR code with your phone, or with a security key, and shellz says the browser didn't pass the encryption result back. The passkey is fine; this browser didn't relay its PRF secret from the other device. - **Use the device that holds the passkey.** Open shellz on your phone and view the recording, approve the teammate or finish the task there. For a security key, use a computer where it has worked before. - **Update the browser and operating system.** Support for PRF across devices is still arriving in browsers. - **Make this computer work directly.** Add a passkey that lives on, or syncs to, this computer (see the table below), then set it up for encryption from a device where your current passkey works: setting up a new passkey needs an existing one to unlock your member key first. ### Passkey that worked before doesn't work in this browser shellz says your passkey is set up for encryption but the browser didn't return its encryption result. Usually the browser or operating system is too old for PRF, or the browser can reach the passkey but not its PRF secret. - Update the browser and operating system, then try again. - Try a browser from the table below for your platform. - Or use a device where the passkey has worked before. ### This passkey can't unlock encryption keys The passkey, or the password manager it's saved in, doesn't support PRF. It can still sign you in. Add a passkey from the table below for your platform, then set it up for encryption in **~/security**. | Your device | Passkeys known to work | |---|---| | Mac | iCloud Keychain (Safari or Chrome, macOS 15 or later), Google Password Manager (Chrome), 1Password extension, YubiKey 5 series | | iPhone or iPad | iCloud Keychain (iOS 18 or later) | | Android | Google Password Manager (Chrome) | | Windows, Linux or ChromeOS | Google Password Manager (Chrome), 1Password extension, YubiKey 5 series (firmware 5.7 or later) | ### Every passkey that held your key is lost - **Teammates:** in **~/security**, click **Set up for encryption** on a passkey that works. When shellz can't unlock your old key, choose **Replace my encryption key**. An owner or admin then approves you again in **~/team**, comparing the new fingerprint. - **Owners and admins:** use the [recovery kit](https://shellz.dev/docs/security/perfect-secrecy.md#recovery-kit) to get the org key back, then approve the rest of the team again. ## Algorithms | Step | Algorithm | |---|---| | Passkey secret | WebAuthn PRF (CTAP2 `hmac-secret`), salt `SHA-256("shellz-prf-v1")` | | Key-encryption key | HKDF-SHA256 | | Member key wrap | AES-256-GCM, associated data: user ID | | Org key envelope | X25519, HKDF-SHA256, AES-256-GCM | | Org key chain and policies | Ed25519 | | Recovery key | HKDF-SHA256 to X25519 | | Member key fingerprint | First 8 bytes of SHA-256 of the public key | # Tamper resistance URL: https://shellz.dev/docs/security/tamper-resistance > How shellz keeps recording when someone with access to a server tries to stop it, and how it reports attempts. Anyone with root access can interfere with any software on their own machine, shellz included. What shellz guarantees is that interference is undone where possible, and always reported: shellz can't be disabled quietly. ## On a server ### The recorder service The `shellz-recorder` systemd unit is set up so that: - `systemctl stop shellz-recorder` is refused (`RefuseManualStop=yes`); - if the process is killed or crashes, systemd restarts it after one second, with no limit on restarts, and shellz raises a `tamper` alert on the next start; - the kernel's out-of-memory killer avoids it, and it doesn't produce core dumps; - the process can't be inspected with `ptrace` by unprivileged users. ### Continuous repair Every 15 seconds, and immediately when `/etc/passwd` changes, the agent checks and repairs: - the shellz binary and session shell; - the recorder unit file, and whether it's enabled and running; - the login shells of recorded accounts; - the SSH drop-in that turns off `~/.ssh/rc`; - the SELinux policy module and file labels, where SELinux is enabled; - on Docker hosts, the Docker socket and drop-ins (checked every second). Each repair raises a `tamper-repaired` alert describing what was changed. shellz files are also marked immutable (`chattr +i`), so changing them first requires removing the flag. ### Sessions that avoid the recorder A session auditor looks for login sessions on the server that didn't go through the session shell, for example a login using a service account with a shell set by hand. Unless the recording policy is fail-open, it ends them and raises `unrecorded-session-terminated`. Under fail-open it raises `unrecorded-session` and lets them continue. ### Removal `shellz uninstall` only works after an owner or admin has approved removal of that server in the dashboard, which sends the server a signed policy that allows it. Without that, the command is refused and an `uninstall-refused` alert names the account that tried. See [Updates and removal](https://shellz.dev/docs/guides/updates-and-removal.md#removing-shellz). ### Going silent A server that stops sending heartbeats, for example because it was shut down, cut off from the network, or shellz was forcibly removed, is marked offline after 90 seconds and raises a `heartbeat-missing` alert. When it comes back, `heartbeat-recovered` is raised. ## Inside containers In containers, shellz runs as PID 1: - `kill -9 1` from inside the container has no effect, because the kernel doesn't deliver unhandled signals to PID 1 from inside its own namespace. - The exec watcher finds sessions that bypass the shell shim within 500 ms. Under fail-closed, unrecorded sessions with a terminal are killed and raise `unrecorded-container-exec`. - Stopping the container stops the workload too, and is reported as **stopped** with its reason and exit code. A container that disappears without a clean stop is shown as **lost** after 3 minutes. ## Recordings Once uploaded, recordings are protected by signatures, the hash chain and write-once storage with a 365-day retention lock. See [Encryption](https://shellz.dev/docs/security/encryption.md#the-hash-chain). A server also can't rewrite its own history: shellz refuses a segment that conflicts with one it already stored, and marks the session as tampered. ## Policies and keys Servers only apply policies and org keys signed by your org, with a revision higher than the current one. Attempts to replay an older policy or substitute a key are refused and raise `policy-rejected` or `org-key-rejected`. With [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md), only your team's browsers can sign them. ## Limits - Root can still interfere, for example by editing memory, loading a kernel module, or cutting the network. shellz is designed so that these leave a trace: a gap in a recording, a tamper alert, or a silent server. - In an unprivileged LXC container, `chattr +i` isn't available. shellz still repairs changes every 15 seconds and raises a single `immutable-unsupported` warning. - In containers, someone with access to the node can enter a container's namespaces without going through `kubectl exec` or `docker exec`. Record node logins with a [host install](https://shellz.dev/docs/install/linux.md) where you manage the nodes. See [Alerts](https://shellz.dev/docs/guides/alerts.md) for every alert and its severity. # Recording policy URL: https://shellz.dev/docs/guides/recording-policy > Choose what happens when recordings can't be uploaded, with fail closed, buffer or fail open. Recordings are written to a spool on the server or in the container first, then uploaded. Usually the spool is nearly empty. If the network or shellz is unreachable for a while, it fills up. The **recording policy** decides what happens then. ## Failure modes | Mode | When the spool reaches its limit | Use it when | |---|---|---| | **Fail closed** | New interactive sessions are refused. Once the spool is full, running sessions are ended. | Every session must be recorded, even at the cost of access. | | **Buffer** | New sessions are refused. Running sessions continue, up to twice the limit, so nobody is cut off mid-task. | You want strong guarantees without interrupting work. | | **Fail open** | Sessions continue unrecorded, with an alert. | Access matters more than the recording, for example on break-glass machines. | In every mode, a problem is reported as an alert: `session-refused` when a login is refused, `recording-failed` when a running session can't be recorded. ### Per-account fairness Outside fail-open, one account can use at most a quarter of the spool limit. If one account produces a huge amount of output while uploads are stalled, its new sessions are refused (and, under fail-closed, its running sessions ended) before it can fill the spool for everyone else. ### Commands without a terminal in containers In containers, commands run without a terminal, such as health checks and probes, are never refused in any mode, so a recording problem can't restart your pods. See [Commands without a terminal](https://shellz.dev/docs/install/container-image.md#commands-without-a-terminal). ## Defaults | | Servers | Container fleets | |---|---|---| | Failure mode | Buffer | Fail closed | | Spool limit | 512 MiB | 64 MiB | | Record consoles | On | Off | **Record consoles** controls whether logins on physical, serial and hypervisor consoles (`tty1`, `ttyS0`, `hvc0`) are recorded. It's on by default for servers. Turning it off keeps a way in through the console if recording ever refuses logins. ## Changing the policy ### Container fleets Open the fleet from **~/servers**, then use the **Recording policy** card: choose **Fail closed**, **Buffer** or **Fail open**, set the **Spool limit (MiB)**, and save. Owners and admins can change it, after confirming it's them. Members pick up the change on their next heartbeat, within about 30 seconds. ### Servers Servers use the defaults above. Per-server policy changes are made by owners and admins through the shellz API; a policy editor for servers isn't in the dashboard yet. ## How a policy reaches a server Policies are signed by your org's key and carry a revision number. A server applies a policy only if the signature is valid, it names that server (or its fleet), and its revision is higher than the one it has. In [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md) mode, policies are signed in your browser, so shellz can't change them on its own. Every applied policy raises a `policy-updated` alert on that server, and every change made in the dashboard or API is written to the audit log. A policy can also turn recording off for a server, which restores the original login shells; like any policy, it has to be signed by your org. # Alerts URL: https://shellz.dev/docs/guides/alerts > Where alerts go, what each one means, and how to verify webhook deliveries. shellz raises an alert when a server goes silent, a recording has a gap, someone tampers with the recorder, or a session can't be recorded. Alerts appear on the server or fleet page and are sent to your alert destinations within a minute. ## Alert destinations Owners and admins configure destinations in **~/security**, under **Alert destinations**. Click **Add destination**, choose the type, enter the target and click **Save**. | Type | Target | |---|---| | email | Any email address | | Slack | A Slack incoming webhook URL (`https://hooks.slack.com/...`) | | webhook | Any public HTTPS URL | Each destination can be switched on or off. Until you add one, alerts are emailed to the org's owners. ## Webhook payload shellz sends a `POST` with a JSON body: ```json { "type": "alert", "alert": { "id": "8f0c...", "orgId": "1b2e...", "serverId": "6f1c...", "fleetId": null, "serverName": "prod-api-01", "kind": "tamper-repaired", "message": "the recorder unit was modified and has been restored", "severity": "critical", "at": "2026-11-04T10:12:03.000Z" }, "url": "https://app.shellz.dev/servers/6f1c..." } ``` ### Verifying the signature When you add a webhook, shellz shows a signing secret once. Every delivery then carries two headers: - `X-Shellz-Timestamp`: the send time in milliseconds since the Unix epoch; - `X-Shellz-Signature`: the hex HMAC-SHA256 of `.`, keyed with the secret. ```js title="verify.js" import { createHmac, timingSafeEqual } from "node:crypto"; export function verify(secret, timestamp, rawBody, signature) { const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest("hex"); const fresh = Math.abs(Date.now() - Number(timestamp)) < 5 * 60_000; return fresh && timingSafeEqual(Buffer.from(expected), Buffer.from(signature)); } ``` Compute the HMAC over the body exactly as received, before parsing it. Rejecting old timestamps protects against replayed deliveries. ## Alert reference Severity is **critical**, **warning** or **info**. While a server or fleet member has an open critical alert, the dashboard shows it as **tampered**. ### Server health | Alert | Severity | Meaning | |---|---|---| | `heartbeat-missing` | critical | No heartbeat for 90 seconds. The server is down, cut off, or shellz was stopped. | | `heartbeat-recovered` | info | The server is sending heartbeats again. | | `recorder-started` | info | The recorder started. | | `policy-updated` | warning | The server applied a new signed policy. The message gives its revision. | | `update-rolled-back` | warning | A new version failed to start and the previous one was restored. | | `storage-quota-warning` | warning | The org is close to its storage quota. | | `storage-quota-exceeded` | critical | The org is over its storage quota. Servers keep recordings in their spool until there's room. | ### Tampering | Alert | Severity | Meaning | |---|---|---| | `tamper` | critical | The recorder was killed or crashed and was restarted. | | `tamper-repaired` | critical | A shellz file, setting, login shell or socket was changed and has been put back. The message says what. | | `uninstall-refused` | critical | Someone ran `shellz uninstall` without an approved removal. The message names the account. | | `enforcement-failed` | critical | shellz couldn't put its configuration back. | | `policy-rejected` | critical | The server received a policy that wasn't validly signed or was older than its current one. | | `org-key-rejected` | critical | The server received an org key that doesn't chain to the key it trusts. | | `immutable-unsupported` | warning | The immutable flag can't be set here (unprivileged LXC). Repairs still run. | ### Recording | Alert | Severity | Meaning | |---|---|---| | `unrecorded-session` | critical | A login avoided the recorder and was allowed to continue (fail-open). | | `unrecorded-session-terminated` | critical | A login avoided the recorder and was ended. | | `session-refused` | warning | A session was refused because recording wasn't possible. | | `recording-failed` | critical | Recording of a running session failed. | | `segment-signature-invalid` | critical | An uploaded segment failed its signature check and was rejected. | | `segment-conflict` | critical | An uploaded segment conflicts with one already stored. | ### Containers | Alert | Severity | Meaning | |---|---|---| | `docker-tcp-listener` | critical | Docker accepts API calls over TCP, which bypass recording. | | `unsupported-container-runtime` | warning | A container runtime or Docker setup that shellz can't record is in use. | | `unrecorded-container-exec` | critical | An exec into a container bypassed the shell shim. | | `unrecorded-noninteractive-exec` | warning | A command without a terminal ran while recording was unavailable. It was logged once recording was back. | | `member-lost` | warning | A fleet member stopped sending heartbeats without a clean stop. | | `member-disabled` | warning | A member is running with a revoked fleet token and refuses sessions. | | `fleet-full` | warning | A new container was refused because the fleet reached its member limit. | | `fleet-token-rejected` | warning | A container tried to join with a rotated, revoked or invalid token. | Repeated alerts of the same kind are grouped, with a count. # Team and access URL: https://shellz.dev/docs/guides/team > Invite teammates, choose roles, and see who viewed which recording. Everyone in your org can see servers and sessions. Roles decide who can change things, and every sensitive action needs a recent sign-in. ## Roles | Action | Owner | Admin | Member | |---|---|---|---| | See servers, fleets, sessions and alerts | ✓ | ✓ | ✓ | | Watch and replay recordings | ✓ | ✓ | ✓ | | Add servers and fleets, manage fleet tokens and policies | ✓ | ✓ | | | Remove servers, change server policies | ✓ | ✓ | | | Invite and remove teammates, change roles | ✓ | ✓ | | | Configure alert destinations | ✓ | ✓ | | | Read the audit log | ✓ | ✓ | | | Approve teammates and rotate the org key (Perfect Secrecy) | ✓ | ✓ | | | Turn on Perfect Secrecy | ✓ | | | Admins can't change or remove owners. An org always has at least one owner. ## Inviting teammates In **~/team**, under **Invite teammates**, enter an email address, choose **admin** or **member**, and click **Send invite**. The teammate gets an email from shellz, signs up with Google, GitHub or a passkey, and joins your org. Invites expire after 7 days; pending invites can be revoked from the same card. With [Perfect Secrecy](https://shellz.dev/docs/security/perfect-secrecy.md) turned on, a new teammate also needs to set up a passkey for encryption in **~/security**, and an owner or admin approves them under **Pending access approvals** in **~/team** after comparing key fingerprints. Until then, they can see sessions but not open recordings. ## Removing teammates Click **Remove** next to the teammate in **~/team**. They lose access immediately. With Perfect Secrecy, your browser also rotates the org key so future recordings use a key they never had. See [Removing a teammate](https://shellz.dev/docs/security/perfect-secrecy.md#removing-a-teammate). ## Recent sign-in Viewing recordings, and actions such as removing servers, changing policies, managing teammates and fleet tokens, and approving access, require a sign-in within the last 10 minutes. If yours is older, shellz asks you to confirm with your passkey or sign-in method before continuing. Sign-in sessions end after 7 days without use, or 30 days at most. You can see and end them in **~/security** under **Where you're signed in**. ## Audit log Owners and admins can read the audit log in **~/audit**. It records, with the actor, time and IP address: - every recording viewed; - sign-ins, failed sign-ins and passkey changes; - servers enrolled and removed, policies changed, rejected enrollments; - fleets created, tokens rotated and revoked; - invites, role changes and removals; - alert destination changes; - Perfect Secrecy setup, approvals, recovery and key rotations. # Updates and removal URL: https://shellz.dev/docs/guides/updates-and-removal > How shellz updates itself safely, and how to remove it from a server. ## Automatic updates On servers, shellz checks for a new release about every 6 hours (with some random delay, so a fleet doesn't update all at once), and five minutes after the recorder starts. Each update: 1. **Is verified.** The release manifest must be signed by the shellz release key pinned in the agent, and the download must match the signed SHA-256 hash and size. 2. **Only moves forward.** shellz refuses any version older than, or equal to, the newest one it has run, and any version that already failed on this server. 3. **Is tested before it's used.** The new binary runs a self-test before it replaces the old one, which is kept as a backup. 4. **Doesn't interrupt sessions.** The recorder switches to the new version once no recorded session is running, so an update never ends a session. 5. **Rolls back on failure.** If the new version doesn't start and report healthy within 2 minutes, or fails to start 3 times, the previous version is restored and an `update-rolled-back` alert is raised. To check or update by hand: ```bash sudo /usr/local/bin/shellz update --check # show whether an update is available sudo /usr/local/bin/shellz update # install it now ``` ### Containers Containers run the version that's in their image. To update, rebuild the image, which picks up the latest release with `get.shellz.dev/container`, or update the version and checksum in your `ADD` line. ## Removing shellz shellz can't be removed from a server without your org's approval. This stops someone with root access from quietly removing it. 1. **Approve the removal.** An owner or admin removes the server from your org, after confirming it's them. The server receives a signed policy that allows removal, on its next heartbeat. In Perfect Secrecy mode, the policy is signed in their browser. 2. **Uninstall on the server:** ```bash sudo /usr/local/bin/shellz uninstall ``` 3. shellz finishes any running sessions and uploads what's left in its spool (waiting at most 10 minutes), then: - restores every account's original login shell; - removes the SSH drop-in, the recorder unit, the SELinux module, the session shell, the binary and `/etc/shellz` and `/var/lib/shellz`; - on Docker hosts, gives the Docker socket back to Docker (see [Docker host](https://shellz.dev/docs/install/docker-host.md#removing-shellz-from-a-docker-host)); - reports `uninstalled` to your org. Recordings already uploaded stay in your org, under the retention lock. > [!NOTE] > Removing a server from the dashboard isn't available yet. Until it is, owners and admins can approve removal through the shellz API, or contact [hello@shellz.dev](mailto:hello@shellz.dev). If you run `shellz uninstall` without an approved removal, it's refused, nothing is changed, and an `uninstall-refused` alert names the account that tried. ### Fleets To stop recording a container workload, deploy it without the shellz entrypoint, then revoke or delete the fleet on its page. # CLI reference URL: https://shellz.dev/docs/reference/cli > Every shellz command on servers and in container images. The `shellz` binary is installed at `/usr/local/bin/shellz`. Server commands need root. On Rocky, Alma, RHEL and Fedora, call it by its full path with `sudo`, because sudo's `secure_path` doesn't include `/usr/local/bin`. ## On servers ### shellz status ```bash sudo /usr/local/bin/shellz status [--json] ``` Shows the enrollment, org key, policy, recorder state, active sessions, upload backlog, container recording state and recent alerts. `--json` prints the same as JSON, for monitoring. ### shellz update ```bash sudo /usr/local/bin/shellz update [--check] ``` Installs the latest signed release now, instead of waiting for the automatic check. `--check` only reports whether an update is available. See [Updates and removal](https://shellz.dev/docs/guides/updates-and-removal.md). ### shellz uninstall ```bash sudo /usr/local/bin/shellz uninstall ``` Removes shellz and restores the server, but only after an owner or admin has approved removal of this server. Otherwise it's refused and reported. See [Removing shellz](https://shellz.dev/docs/guides/updates-and-removal.md#removing-shellz). ### shellz decrypt ```bash shellz decrypt (--org-key FILE | --data-key KEY) [--host-key KEY] [--out DIR] [--session ID] ``` Verifies and decrypts recordings offline. It reads every `.seg` segment file under `` (any layout), checks the hash chain of each session, decrypts it, and writes `.cast` (asciicast v2) and `.txt` to `--out` (default: the current directory). Nothing is sent over the network. | Flag | Meaning | |---|---| | `--org-key FILE` | The org's X25519 private key: base64url, PKCS#8 PEM, or JSON (`{"keyId", "x25519Private"}`, or a list of them) | | `--data-key KEY` | A single session's data key (base64url) | | `--host-key KEY` | The server's Ed25519 public key (base64url). With it, segment signatures are checked too. | | `--out DIR` | Where to write the output | | `--session ID` | Only this session | `shellz decrypt` runs on Linux and doesn't need the server to be enrolled. ### shellz version ```bash shellz version [--short] ``` Prints the version. `--short` prints only the version number. ### Internal commands `shellz enroll ` is run by the installer. `shellz recorder` is run by systemd. `shellz shell` starts a recorded shell. You don't need to run these yourself. ## In container images ### shellz run ```dockerfile ENTRYPOINT ["/usr/local/bin/shellz", "run", "--"] CMD ["", ""] ``` Runs your command with session recording, normally as PID 1. It enrolls the container in its fleet using the fleet token, records shells opened by exec, forwards signals to your command, and exits with its exit code. See [Container image](https://shellz.dev/docs/install/container-image.md) and [Container settings](https://shellz.dev/docs/reference/container-settings.md). ### shellz image-install ```dockerfile RUN ["/usr/local/bin/shellz", "image-install"] ``` Puts the recording shim in front of the image's shells (`/bin/sh`, `bash`, `ash`, `dash`, `zsh`, `fish`), keeping the real shells next to them. It's safe to run more than once. `--undo` restores the original shells. The container installer `get.shellz.dev/container` runs it for you. ## Installers | Script | Use | |---|---| | `curl -fsSL https://get.shellz.dev \| sudo sh -s -- ` | Install on a server | | `curl -fsSL https://get.shellz.dev/container \| sh` | Add shellz to a container image, in a `RUN` step | Both verify the release signature with the shellz release key built into the script, and the SHA-256 checksum, before installing anything. # Container settings URL: https://shellz.dev/docs/reference/container-settings > Environment variables and files read by shellz run inside containers. `shellz run` needs only the fleet token. Everything else is optional. ## Fleet token | Source | Notes | |---|---| | `/run/secrets/shellz-fleet-token` | Recommended. A file containing the `szf_` token. | | `SHELLZ_FLEET_TOKEN` | Used if set, before the file. Visible to anyone who can exec into the container; shellz removes it from your app's and sessions' environments only. | ## Metadata These label members in the dashboard. shellz detects most of them on its own. | Variable | Default | Meaning | |---|---|---| | `SHELLZ_FLEET_PLATFORM` | Detected | `kubernetes`, `ecs`, `cloudrun`, `docker`, `devcontainer` or `other` | | `SHELLZ_POD_NAME` | `POD_NAME`, then the hostname | Kubernetes pod name | | `SHELLZ_POD_NAMESPACE` | `POD_NAMESPACE`, then the service account's namespace | Kubernetes namespace | | `SHELLZ_NODE_NAME` | `NODE_NAME` | Kubernetes node | | `SHELLZ_CONTAINER_NAME` | The hostname (ECS: from task metadata) | Container name on Docker, ECS and other platforms | | `SHELLZ_IMAGE` | Empty (ECS: from task metadata) | Image reference to show | Platform detection, in order: `KUBERNETES_SERVICE_HOST` means Kubernetes, `ECS_CONTAINER_METADATA_URI_V4` means ECS, `K_SERVICE` means Cloud Run, `REMOTE_CONTAINERS` or `CODESPACES` means a dev container, and `/.dockerenv` means Docker. ## Behaviour | Variable | Default | Meaning | |---|---|---| | `SHELLZ_DRAIN_SECONDS` | `20` | How long to wait for your app to exit after a stop signal before ending remaining processes and uploading the rest of the recordings. Keep the platform's stop timeout about 10 seconds longer. | ## Set by shellz Inside a recorded session, shellz sets `SHELLZ_SESSION` to the session ID. Don't set it yourself. ## Files | Path | Contents | |---|---| | `/usr/local/bin/shellz` | The shellz binary | | `/run/shellz/broker.sock` | The recorder socket (`/tmp/shellz/` if `/run` isn't writable) | | `/run/shellz/status.json` | Current status: enrollment, fleet, sessions, upload state | The member's signing key lives only in memory; nothing secret is written to the container's filesystem. ## Network Outbound HTTPS to `api.shellz.dev` and `ingest.shellz.dev`. Nothing listens for inbound connections. # Use the docs with AI URL: https://shellz.dev/docs/reference/ai > Read these docs from coding agents and AI assistants through the MCP server, llms.txt or plain Markdown. Every page of these docs is available as plain Markdown, so AI assistants and coding agents can read the same content you do: through an MCP server, through `llms.txt`, or page by page. ## MCP server The shellz docs MCP server is at: ```text https://shellz.dev/mcp ``` It uses the Streamable HTTP transport and needs no authentication. It's read-only and only serves these docs. | Tool | What it does | |---|---| | `search_docs` | Full-text search. Takes a `query` and an optional `limit`, and returns matching pages and sections with their paths. | | `list_docs` | The table of contents: every page with its title, description and Markdown URL. | | `read_doc` | The full Markdown of one page. Takes a `path` such as `/docs/install/kubernetes`, or a full URL. | Every page is also exposed as an MCP resource (`https://shellz.dev/docs/.md`, type `text/markdown`). ### Add it to your client ```json tab="Cursor" // ~/.cursor/mcp.json, or .cursor/mcp.json in your project { "mcpServers": { "shellz-docs": { "url": "https://shellz.dev/mcp" } } } ``` ```bash tab="Claude Code" claude mcp add --transport http shellz-docs https://shellz.dev/mcp ``` ```json tab="VS Code" // .vscode/mcp.json { "servers": { "shellz-docs": { "type": "http", "url": "https://shellz.dev/mcp" } } } ``` Other clients that support remote MCP servers over HTTP work the same way: add `https://shellz.dev/mcp` as the server URL. Then ask, for example: "Using the shellz docs, add shellz to our Kubernetes deployment" or "How does shellz's Perfect Secrecy use passkeys?" ### Discovery Clients and registries that support [MCP Server Cards](https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/seps/2127-mcp-server-cards.md) can find the server without configuration: - [`https://shellz.dev/.well-known/ai-catalog.json`](https://shellz.dev/.well-known/ai-catalog.json): the AI Catalog for shellz.dev, which links to the server card. - [`https://shellz.dev/mcp/server-card`](https://shellz.dev/mcp/server-card): the server card, with the server's name (`dev.shellz/docs`), endpoint and supported protocol versions. The card is also served at `/.well-known/mcp.json` and `/.well-known/mcp/server-card.json` for clients that use those earlier draft locations. ## llms.txt For tools that read [llms.txt](https://llmstxt.org): - [`https://shellz.dev/llms.txt`](https://shellz.dev/llms.txt): a short summary and a linked index of every page. - [`https://shellz.dev/llms-full.txt`](https://shellz.dev/llms-full.txt): the full text of every page in one file, to paste into a chat or load as context. ## Markdown for any page - Add `.md` to any docs URL: [`https://shellz.dev/docs/getting-started.md`](https://shellz.dev/docs/getting-started.md). - Or request a docs URL with the header `Accept: text/markdown`: ```bash curl -H 'Accept: text/markdown' https://shellz.dev/docs/install/linux ``` - On each page, **copy markdown** copies the page and **.md** opens it.