Trace a command on a production Linux server to the person who ran it

Bash keeps each user's commands in their own ~/.bash_history. It writes them there when the shell exits, so a session that's still open hasn't saved anything yet.

The user owns that file. history -c clears the list, history -d removes single entries, and unsetting HISTFILE means nothing is written at exit. If HISTCONTROL includes ignorespace, any command typed with a leading space is never saved.

Two shells open as the same user can also overwrite each other's history, unless the histappend option is on.

History is off in non-interactive shells. A command sent as ssh web-03 'rm -rf /srv/cache/*' runs and leaves no trace in anyone's history file.

sshd writes the start, authentication and end of each connection to the system log. Where it lands depends on the distribution: /var/log/auth.log on some, the systemd journal on others. last reads the wtmp file and lists who connected, from where, and for how long.

That's enough to learn alice was on db-02 from 01:58 to 02:14. It says nothing about what she ran. When four engineers and a CI job all use the deploy account, it says even less.

Every way into the box is covered: interactive SSH, single remote commands, SFTP uploads, su - and console logins. Each session is stored twice, as a full replay and as plain text, and the text is what you search later.

• ›Search for the command or the path it touched. Narrow by user, source IP, server, or container name and image.
• ›Open the session and replay it at whatever speed suits you. The transcript sits next to the player, so you can copy an error message exactly.
• ›Follow the trail. Container shells carry a pointer to the SSH session that spawned them, so you can walk from docker exec back to whoever logged in.
• ›Export the session as a .cast replay or a .txt transcript and attach it to the postmortem.

This timeline is invented for illustration. At 02:07 the API on web-03 starts returning errors because its cache directory is empty. The deploy user's bash history ends the previous afternoon.

A search for /srv/cache returns one session. It's a remote command from a CI runner's IP at 02:06, logged in as deploy. The replay shows a cleanup script printing the directory it was about to clear, then clearing it. The fix goes into the script, and the exported transcript goes into the incident doc.

Each session carries an integrity badge with one of three states: verified, gap or tampered. Recordings upload in pieces that the server signs and chains together, so a dropped or modified piece gets flagged on the session.

Nobody can alter or remove a recording after upload. Root changes to the recorder's settings get rolled back and flagged. If a server goes silent, an alert fires within a minute.

Root on a box can always fiddle with local software, and shellz makes sure the fiddling is caught and flagged. Viewing is audited too. Each playback is logged, so you can show who watched which recording during the investigation.

shellz records login sessions. A cron job, a systemd service or an agent running in the background never logs in, so nothing is recorded. auditd, the kernel's log of every program executed, covers that gap alongside shellz.

Input at hidden prompts, a sudo password for example, never reaches the recording. Output does, so handle recordings as sensitive data. They're encrypted before they leave the machine, and under Perfect Secrecy the decryption keys exist only on your teammates' passkeys.

Recordings exist only for sessions that happened after shellz was installed. Getting coverage means pasting one command on each machine. The free tier gives you 10 active devices and the most recent 50 recordings. Commercial charges $10 monthly per person on your team and $1 per GB, and keeps recordings as long as you want (plans).