Server down: what to check first, and in which order

This guide is about the server itself: a VPS, a dedicated or a cloud server that you run. If a single website fails and everything else on the machine works, or your site lives on shared hosting, start with the guide Website down: what to do instead. Here we look at the machine: does it answer at all, can you get in, and what stopped.

Step 1: what exactly is down?

"The server is down" can mean three very different things, and each has its own fix. Find out from outside first, from a network that is not yours. For websites on the server, the website down checker tells you within seconds whether the name resolves, whether the port answers and which status code comes back. For the machine, use ping and a port test from your own computer:

ping -c 3 203.0.113.10
nc -zv -w 5 203.0.113.10 22
nc -zv -w 5 203.0.113.10 443

Read the results together:

• No ping, no SSH, no web port: the machine is off, frozen, or the network in front of it is down. Go to step 2.
• Ping answers, SSH does not: the machine runs, but SSH is blocked, stopped or the server is too overloaded to answer. Go to step 3.
• SSH works, the website does not: the server is fine, a service is not. Go to step 4.

Some servers drop ping on purpose. If ping never worked, rely on the port tests.

Step 2: nothing answers

Open your provider's control panel or cloud console before anything else. Three questions settle most cases in a minute:

1. Is the server running? A stopped instance, a suspended account or a failed payment looks exactly like an outage from outside.
2. Is there a known incident? Check the provider's status page for your region or data center.
3. What does the console show? The web or serial console shows the screen of the machine even without network. A kernel panic, "Out of memory: Killed process" in a loop, or a boot stuck at a file system check each tell you where the problem is.

If the machine is frozen, a reboot from the console usually brings it back. Before you reboot, take a screenshot of the console: once it restarts, the evidence is gone. If it does not boot, most providers offer a rescue system from which you can mount the disk, read the logs and repair /etc/fstab or a broken network configuration.

Step 3: ping answers, SSH does not

The machine is alive, so the cause sits on it. Use the console to log in, then check the usual suspects:

sudo ufw status verbose
sudo iptables -S | head -n 30
sudo fail2ban-client status sshd
sudo systemctl status ssh sshd 2>/dev/null | grep -E 'Loaded|Active'

A firewall rule added in a hurry, your own IP address banned after a few wrong passwords, or an SSH configuration change that failed its test are the most common locks. Unban your address with sudo fail2ban-client set sshd unbanip YOUR_IP, and always run sudo sshd -t before restarting SSH. If SSH runs but is just very slow, the server is overloaded: go to step 4.

Step 4: you are in, now read four numbers

uptime
free -m
df -h
df -i
sudo dmesg -T | grep -iE 'oom|killed process|i/o error|read-only' | tail -n 20

• Load (uptime): far above the number of CPU cores means the machine is overloaded. top or htop shows which process is responsible.
• Memory (free -m): no free memory and no swap, plus "Killed process" in dmesg, means the kernel stopped a process to survive. Usually it was the database or PHP.
• Disk (df -h): a full disk is the most frequent cause of server problems. Databases stop writing, logs stop, sessions fail. df -i checks the inodes: a disk can be "full" with gigabytes free if millions of small files use up all inodes.
• I/O errors or a read-only file system in dmesg: the disk itself has a problem. Stop writing, make a backup if you still can and contact your provider.

To free space safely, start with the system journal and old archives:

sudo journalctl --vacuum-size=200M
sudo du -xh / --max-depth=2 2>/dev/null | sort -h | tail -n 15

Never delete database files or website files to make room.

Step 5: which service stopped, and why

systemctl --failed
sudo journalctl -u nginx -n 50 --no-pager
sudo nginx -t

systemctl --failed lists every service that stopped with an error. The journal of that service names the reason: a port already in use, a syntax error in a configuration file, a missing certificate, no space left. Test the configuration before every restart (nginx -t, apachectl configtest, php-fpm -t), then start it with sudo systemctl restart SERVICE. A restart without the test can take down a service that was still half working.

Step 6: what changed?

Outages have causes, and the cause is usually the last change. Look for it:

grep -E ' (install|upgrade) ' /var/log/dpkg.log | tail -n 20
last -n 10
sudo journalctl --since "2 hours ago" -p err --no-pager | tail -n 40

Automatic updates, a new kernel waiting for a reboot, a deploy, a cron job that filled the disk with backups, an expired certificate or a configuration edit by someone else: if one of these lines up with the time the outage started, you have your lead.

When the same server problems come back

A server that went down once because of a full disk or a stopped database will do it again. Checking from outside tells you when it is down; checking from inside tells you before. The server agent reports load, memory, disks and services from inside the machine and alerts you when a disk passes your threshold or a service you ticked stops, while there is still time to act. Server monitoring with the agent is part of the paid plans.

When you do not want to do this at 3 a.m.

If your server is down and you would rather have it fixed than debug it yourself, a quick fix handles the common causes at a fixed price: it restarts a stopped service after its configuration test passes, frees a full disk and renews an expired certificate, with a backup copy first. When that is not enough, a deep fix takes over around the clock, billed in 15-minute blocks, with the price shown before you order. Both work through the Responder, SSH as root or SSH access to your webspace. Details on the page server down fix and emergency server support.