Running your own web server (Part 2)

Blog: 31st Jan 2024

...there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don't know we don't know.

– Donald Rumsfeld, former U.S Secretary of Defence

Your man Don was right about all of that, particularly the unknown unknowns. His observation also applies to web server security. The main problem is that while some large percentage of your attackers are probably just script kiddies, some percentage of them may know a lot more than you do about securing (or failing to secure) your web server. And you don't know what you don't know about web server security. So what to do about it?

Googling for web security best practices is a depressing business. Someone asked this perfectly reasonable question on StackExchange: “Best practices for securing a public facing web server?” And one clown answered with “seriously, seriously too broad and open ended”. It’s a perfectly reasonable question, ffs! The internet isn’t the private playground of rude network security experts who are presumably running complex server farms that require the brain the size of planet to administer (in their own minds, at least).

And much as I love Hacker News, it too has its share of high priest bellends who provide non-answers dripping with acid scorn to people who dare ask a simple question.

Most other sources of answers to this question turn out to be ad-blog pages which, after a few paragraphs of copy/paste text, push you towards their commercial offerings. Maybe some are getting written by ChatGPT these days. To be fair, ChatGPT does a decent job of summarizing the main stuff you should be looking out for when I asked it that very question today.

In part 1, I discussed the basics to keep your new server safe. So here’s my small list of extra stuff you can do to reduce the attack surface area a bit more. Some of these are matters of opinion; you can find decent sources arguing strongly for or against them.

Change SSH to a non-standard port

Some would argue that switching SSH to a non-standard port is another piece of low-hanging fruit be plucked, but I’m not totally convinced. It only takes a second to port-scan a server and attempt an SSH connection on all the open non-standard ports. And you’d have to change every script or command you use to incorporate your non-standard port in daily use. On the plus side though, it would definitely keep most of the junk out of your SSH log. But journald takes care of recycling large system log files on Debian 12, and anyway, I find it occasionally interesting to read the SSH logs and observe the lowlife—in much the same way that 18th century Londoners would visit Bedlam asylum to be entertained by the inmates.

Install sshguard or fail2ban

Both protect your sshd from repeat attackers by making firewall entries to block them for a period (there are other variations on what they do, but that's the basics). I prefer sshguard because it’s a compiled C application rather than a Python script, but fail2ban is very popular. Some say they’re completely unnecessary if you only allow SSH key authentication, others say they they’re still useful to keep out repeat offenders from clogging your logs or overwhelming your server. And fail2ban can block attempts at getting into other services also.

Local or remote firewall

My virtual server is on Hetzner, who offer a tick-the-box firewall application which presumably works just inside their routers to protect their servers. Should I also make local firewall rules on my server as backup? I chose to only use the Hetzner firewall on the basis that:

Use a VPN for connectivity to the server

The idea here is to cut out web-facing SSH for logging in to your server and do it through a private tunnel of some sort. I’ve nothing against doing anything like this other than having to manage the extra complexity. Tailscale in particular seems to be loved by its users. In my case, the worst that can happen is that my site gets hacked and I have to install yesterday’s backup. No big deal, so simple SSH key authentication is good enough for me.

Use Cloudflare

Cloudflare offer a load of very useful features such as DDoS protection, a web application firewall, bot management, image compression, tunnelling, and all sorts of good stuff. But most of the good stuff costs money, and you may not yet need all that sophisticated tooling (I don’t—at least not for this site).

Only install what you need

It’s tempting to over-think what you need to install, but you should only install the bare minimum to meet your needs. You probably don't need services like: FTP, any kind of GUI system (Gnome, KDE etc.), print servers, LDAP, Samba, mail servers—you get the idea. If you don’t need it, don’t install it, or disable it if already installed. If you do need a service, then do your homework before installing it and read up on any known vulnerabilities and how to mitigate them.

If you need to move files to the server manually, do it over SSH using something like FileZilla using SFTP which can read your local SSH key and create a secure connection. Just don't use plain FTP—here's why:

There's really no good reason to install or use FTP these days.

Don’t blindly use copy/paste advice

This is a personal rule of mine transferred over from coding. If you don’t understand every command, option, and character in some randomly googled advice you found somewhere, then either:

This applies particularly to firewall rules and web server configuration, especially complex stuff like mod_rewrite if you're using Apache. Only Regular Expressions can match mod_rewrite for mind-boggling incomprehensibility. If the googled advice involves a configuration file line which looks like a coked-up squirrel danced across your keyboard, then dig in and work out exactly what it does before using it.

Conclusion

I started with a quotation, so let's end with one:

A man’s got to know his limitations

– St Clint of Hollywood

If your server is doing something important—like handling money or personal data—and you have any doubts at all about your competency to run it, then maybe you're better off buying a hosting service under the umbrella of a company who knows what they're doing in terms of security. Otherwise, if you're just hosting a hobby site like this, have fun and enjoy the freedom of running your own server. The internet is for everyone, so don't be put off by the miserable high priests of doom.

I’d love to hear from you if you think I’ve missed out anything obvious or if I’m totally wrong somewhere. Email details are on the main blog index page.