Lesson 1 of 1 in The First Hour
Is It Still Happening
Three questions, in order, that decide everything you do next — and none of them is "how did they get in".
3 min read
Not yet reviewed
You have your snapshot. Now the temptation is to start hunting for the entry point, because that is the interesting question. It is also the slowest one, and it is not urgent. Three other questions are.
One — is somebody on the box right now
$ w
07:14:22 up 61 days, 3:02, 2 users, load average: 0.31, 0.28, 0.22
USER TTY FROM LOGIN@ IDLE WHAT
deploy pts/0 203.0.113.9 07:11 0.00s w
deploy pts/1 45.83.64.12 02:47 14:12 -bash
$ ss -tunp state established
tcp 0 0 10.0.0.4:39114 45.83.64.12:443 users:(("curl",pid=2481,fd=3))
Two sessions as deploy, from two different addresses. One of them is you. The other has been idle for fourteen minutes with a bash shell open, and something on this box is holding a connection out to the same address.
Note
An outbound connection is the tell people miss. Inbound access needs an open port and gets noticed; outbound looks like ordinary traffic and is how most access is actually held open — a shell that connects *to* the attacker rather than waiting to be connected to.
Two — what can they reach from here
Your server is a position, not a destination. Before you do anything else, write down what this machine holds credentials for: the database, the object store, the mail relay, the deploy key that can push to your repository, the API token in an environment file.
That list is the actual blast radius, and it is almost always larger than people expect. It is also the list you will be rotating in an hour, whether or not you ever find the entry point.
Take care
Rotate the credentials this machine holds before you finish the investigation, not after. The forensics can take a day; a leaked deploy key is being used within minutes. These are separate jobs and only one of them is urgent.
Three — what comes down
Now the judgement. "Preserve the evidence" is a strong default and it is not an absolute, and the difference is not technical.
Comes down immediately
Content that harms a person — doxxing, threats, images of abuse
A page collecting your users' passwords or card details
Anything actively attacking somebody else from your addressCan stay up while you work
A defacement that is only embarrassing
A machine that is compromised but serving normally
Spam that has already been sentThe left column is about somebody being hurt while you deliberate. The right column is about you being embarrassed, and embarrassment is not an emergency.
The second one bothers me. If I know it is compromised, isn't leaving it serving traffic negligent?
It would be if you left it alone. You are not leaving it alone — you are watching it while you find out how far in they are, which you cannot do once it is off. Taking it down also tells them you know, and somebody who knows they have been found starts deleting.
That is exactly what I do. The moment the site goes dark I assume somebody is reading the logs, and I clear my history and drop the cron. If you had waited an hour you would have had all of it.
Your blog is defaced with a rude message. Nothing else on the box is user-facing. What is the right first hour?
Only once those three are answered does the interesting question become worth your time. It is the subject of the rest of this course, and it starts in the access log.