Paint-by-Number Web Servers on Linode

Posted on January 13, 2017

In general, I’m a big fan of managed hosting. For example, all of my apps are backed by FatFractal, my preferred backend-as-a-service provider, and some of my websites are hosted on WPEngine. Outsourcing these sorts of functions allows me to spend my time where it can do the most good for my business – building software, handling marketing, and developing strategy – rather than using it to recreate commoditized services. But these managed services aren’t always the right tools for the job.

Recently my business partner Joe Cieplinski and I decided to split our Release Notes website into several different sites. The main releasenotes.tv domain would continue to host our podcast’s WordPress site, while the websites for each edition of our conference would be split off on to its own subdomain. (For example, 2017.releasenotes.tv.) Unfortunately, the financial math becomes unattractive when hosting multiple websites on WPEngine. Hosting four sites on their service would cost $1,200 per year, far more than it is worth to me.

WPEngine’s service isn’t worth $1,200 per year to me, not because their service isn’t excellent – it is – but because the cost of their service in dollars scales up much faster than the cost of running additional websites on a server that I manage myself. Sure setting up a web server (and keeping it updated, monitoring it for downtime, etc.) is a bit of a pain, but once you have it set up, you can host lots and lots of websites on it for very little additional cost in dollars, time, and effort. In fact, I would suggest that if your business is going to have more than a couple websites, you owe it to yourself and your pocketbook to at least consider setting up your own server and self-hosting your site.

The problem with this, especially for developers who are more comfortable slinging bits than managing hardware, is where to begin. Luckily there are lots of companies that will sell you a virtual private server (VPS) for just a few dollars a month. My favorite is Linode (affiliate link), where at current rates you can get a VPS with 2 GB of memory for just $10 per month. (I’d also recommend that you spring for their automatic backup service, which will bring your cost up to $12.50 per month.) This is much cheaper (in dollars) than a managed host like WPEngine. In fact, Linode’s cost is comparable to what a shared web hosting provider like Dreamhost would charge you for a single website – and you can run as many sites as you want on your Linode VPS! So in dollars, Linode is a bargain.

There is an extra cost of running your websites on Linode, however, that comes from the time and expertise required to setup and maintain your server. You see, with your VPS, you are essentially just getting a basic installation of Linux. It’s a blank canvas for you to work with. Luckily, with modern Linux distributions that are designed for the server, it doesn’t require a lot of expertise to get things set up. In fact, if you’re a programmer who’s comfortable reading documentation, you’re probably more than capable of setting up a robust and secure web server on your VPS.

In setting up my own VPS, I found Linode’s documentation to be excellent, with lots of examples and tutorials for different flavors of Linux. There were, however, a few gotchas that I ran into, and it wasn’t always completely obvious in which order I should read and use different support docs that they provide. So I want to fully document below the steps I took to set up my Linode VPS with an Apache web server, as well as MySQL (actually, MariaDB), and PHP – your typical LAMP stack.

In the steps below, I’m not going to fully document each step. Instead, I’m going to link to the different pieces of documentation I used as reference during my setup and include additional notes to clarify what’s important and what you can safely ignore. I also hope to fill in a few gaps that I found in Linode’s documentation, so that you can benefit from the time I spent researching solutions to problems. So without further ado, here we go.

1. Provisioning Your Linode

  1. Provision and configure your Linode using the instructions in Getting Started with Linode.
    • Unless you expect tremendous amounts of traffic, choose Linode’s lowest tier 2GB Linode option. You can always upgrade it later if you need to.
    • In the “Deploying an Image” section, choose Centos 7 as your Linux distribution. From now on, when different instructions are presented for different distributions, always use the instructions for Centos 7.
    • In the “Find the IP Address of Your Linode” section, write down both your Linode’s IPv4 and IPv6 IP addresses. You’ll need those later.
  2. Turn on the Linode backup service using the instructions in Use the Linode Backup Service to Protect and Secure Your Data.
  3. Enable two-factor authentication on your Linode account using the instructions in Linode Manager Security Controls.
    • You only need to follow the instructions under “Two-Factor Authentication” in this document.
  4. Add Domain Name Service (DNS) records for for your Linode in your DNS host.
    • Add an A record for your Linode’s IPv4 IP address.
    • Add an AAAA record for your Linode’s IPv6 address.
  5. Configure Reverse DNS using the instructions in Reverse DNS.
    • Note: You won’t be able to configure Reverse DNS records until the DNS records you created above propagate to Linode, and that may take a few hours.

2. Security

  1. Secure your server using the instructions in Securing Your Server.
    • In the “Automatic Security Updates” section, configure yum-cron using these instructions for Fedora 21 or earlier versions
    • Skip the “Use Fail2Ban for SSH Login Protection” section. We will configure fail2ban below.
    • Skip the “Configure a Firewall” section. We will configure the firewall below.
  2. Configure fail2ban using the instructions in Using Fail2ban to Secure Your Server.
    • If you’re unsure what the fail2ban.local and jail.local configuration files should look like, you can use my configuration files as a starting point.
    • After adding your fail2ban.local and jail.local files, have fail2ban reload the files with this command: 
      sudo fail2ban-client reload
  3. Configure a firewall using the instructions in Configure Network Traffic with iptables.
    • Start with the section titled, “Deploy Your iptables Rulesets”.
    • If you’re unsure what ruleset files should look like, you can use my ruleset files as a starting point.

3. Configure Email

In general, it’s a bad idea for amateur system administrators to manage an email server. It’s really easy to misconfigure your mail transfer agent and cause a security vulnerability that threatens to turn your server into a gateway for spam. This is bad not just for those receiving spam through your server. It can also ruin the reputation of your email address and make it less likely that your email will be delivered. Because of these dangers, we will leave email delivery up to the pros at SendGrid, and refuse to accept email on this server.

  1. Create a free account with SendGrid.
    • As of this writing, you will probably only need SendGrid’s free one-month trial account. After one month, your SendGrid account will only be able to send 100 emails per month, but that is plenty for our purposes since we’re only sending occasional email alerts from the server.
  2. Install sendmail on your server by typing the following at the command line: 
    sudo yum install sendmail sendmail-cf m4 cyrus-sasl-plain
sudo systemctl start sendmail.service
sudo systemctl enable sendmail.service
  3. Configure sendmail to relay email through SendGrid using these instructions provided by SendGrid.
  4. Restart sendmail using this command: 
    sudo systemctl restart sendmail.service
  5. Prepare a test for sendmail by creating in your home directory a text file containing the following text: 
    To: your_email_here@example.com
Subject: Test Email
From: from_email_address@example.org

This is a test. Only a test.
    • Replace the email address in the To: field with your email address.
    • Replace the email address in the From: field with a different email address.
  6. Send an email with sendmail to test if it is working properly with SendGrid: 
    sendmail -vt < ~/email.txt

4. Install LAMP Stack

Why Apache and not Nginx? Mostly because I know Apache, but also because I'm conservative. Apache is battle tested, and documentation can be found for almost any use case. Finally, at the time of this writing, it is much easier to install a Let's Encrypt SSL certificate on Apache than on Nginx.

  1. Install a LAMP stack using the instructions in LAMP on Centos 7.
    • In the section "Configure Name-based Vitual Hosts", each virtual host definition block (<VirtualHost></VirtualHost>) must be in its own configuration file for Certbot to work correctly in a later step.
    • In the "MySQL / Maria DB" section, install MariaDB, not MySQL.
    • Skip the section titled "Create a MySQL/MariaDB Database".
  2. In vhosts.conf file created above, add this entry inside the <VirtualHost></VirtualHost> tags, replacing "/path/to/document/root/" with the actual document root: 
    <Directory "/path/to/document/root/">
    Options FollowSymLinks
    AllowOverride All

    Order allow,deny
    Allow from all
</Directory>
  3. Restart Apache with this command: 
    sudo systemctl restart httpd.service
  4. Create directories to store MariaDB backups by typing this at the command line: 
    sudo mkdir /var/backup
sudo chmod 700 /var/backup
  5. Schedule automatic MariaDB backups by adding these two lines to /etc/crontab: 
    30 1 * * * root /usr/bin/mysqldump -u root --all-databases | /bin/gzip > /var/backup/mariadb-`/bin/date +%Y-%m-%d_%H-%M-%S`.sql.gz 2>> /var/backup/mariadb.log
(Douar Ouled el Hadj, Tiaret, Algeria) * * * root /usr/bin/find /var/backup -name "mariadb*.sql.gz" -mtime +7 -delete >> /var/backup/mariadb.log 2>&1

5. Install WordPress

  1. Install and configure WordPress using the instructions found in How to Install and Configure WordPress.
  2. Install the PHP GD package, which is used by WordPress for image editing, by typing the following at the command line: 
    sudo yum install php-gd
sudo systemctl reload httpd.service
  3. You should now have a working WordPress site. Visit your site in a web browser to make sure that it is functioning.

6. Tune Apache for Performance

Although at this point Apache and WordPress should be working and ready for use, I found in practice that Apache was not performant under even light loads on a 2 GB Linode. In fact, I found that under even light loads, Apache would consume so much memory that MariaDB would be killed off due to memory pressure. This is obviously a problem since WordPress cannot function without a database backing it.

After research, I found that Apache's excess memory usage was due to two things. First, Apache as configured by default uses the "pre-fork" Multi-Processing Module (MPM) which fires up a new process for each web connection. This requires a lot of memory. Compounding this problem is that, in its default configuration, Apache loads many unneeded modules, including mod_php which in particular uses a lot of memory. Each of these unneeded modules is loaded by each Apache process which effectively multiplies the memory used by each module when the pre-fork MPM is used.

The solution to these problems is to unload unneeded modules, and to switch from the pre-fork MPM to the "event" MPM. The event MPM uses multiple threads (rather than multiple processes) for web connections. This drastically reduces the memory footprint of Apache.

In addition we will forego mod_php, and instead use PHP-FPM for PHP processing. Using PHP-FPM allows us to get PHP out of each Apache process and offload PHP processing to dedicated processes that are shared by all Apache processes and threads, thus making more efficient use of memory.

The end result of these performance tweaks is an Apache server that uses a fraction of the memory used by the default configuration, and remains performant under higher loads.

6a. Make Apache Use the Event MPM

  1. Open /etc/httpd/conf.d/php.conf in a text editor and find these two lines (they are probably at the end): 
    php_value session.save_handler "files"
php_value session.save_path "/var/lib/php/session"

    Replace them with the following lines:

    <IfModule mpm_prefork_module>
php_value session.save_handler "files"
php_value session.save_path "/var/lib/php/session"
</IfModule>
  2. Open /etc/httpd/conf.modules.d/00-mpm.conf in an editor and comment out the following line by putting a '#' character at the beginning of the line: 
    LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
  3. Find the following line in /etc/httpd/conf.modules.d/00-mpm.conf and uncomment it by removing the '#' character at the beginning of the line: 
    #LoadModule mpm_event_module modules/mod_mpm_event.so
  4. Add the following lines to /etc/httpd/conf/httpd.conf: 
    <IfModule event.c>
    StartServers             3
    MinSpareThreads         75
    MaxSpareThreads        250
    ThreadsPerChild         25
    MaxClients             400
    MaxRequestsPerChild   2000
</IfModule>

6b. Make Apache Use PHP-FPM

  1. Install PHP-FPM by typing the following at the command line: 
    sudo yum install php-fpm
sudo systemctl enable php-fpm.service
sudo systemctl start php-fpm.service
  2. Open /etc/php-fpm.d/www.conf and comment out each of the following lines (they may not consecutive lines) by putting a semicolon (';') character at the beginning of each line: 
    pm = dynamic
pm.max_children = 50
pm.start_servers = 5
pm.min_spare_servers = 5
pm.max_spare_servers = 35
pm.max_requests = 500
  3. Add the following lines to /etc/php-fpm.d/www.conf: 
    pm = ondemand
pm.max_children = 20
pm.process_idle_timeout = 10s
pm.max_requests = 200
  4. For any virtual host on which you want to use PHP, open the virtual host's .conf file (usually found in /etc/httpd/conf.d) and add these lines inside the <VirtualHost></VirtualHost> tags, replacing "/path/to/document/root/" with the actual path to that virtual host's document root: 
    <IfModule mpm_event_module>
    ProxyPassMatch ^/(.*.php(/.*)?)$ fcgi://127.0.0.1:9000/path/to/document/root/$1
</IfModule>

    As an example, your .conf file might look something like this:

    <VirtualHost *:80>
    ServerAdmin me@email.com
    ServerName foobar.com
    ServerAlias www.foobar.com
    DocumentRoot /path/to/document/root/
    ErrorLog /path/to/document/root/logs/error.log
    CustomLog /path/to/document/root/logs/access.log combined

    <IfModule mpm_event_module>
        ProxyPassMatch ^/(.*.php(/.*)?)$ fcgi://127.0.0.1:9000/path/to/document/root/$1
    </IfModule>

    <Directory "/path/to/document/root/">
        Options FollowSymLinks
        AllowOverride All

        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

6c. Disable Unneeded Apache Modules

  1. In the files within /etc/httpd/conf.modules.d/, comment out LoadModule lines (or rename files to end in an extension other than '.conf') so that only these shared Apache modules are loaded:
    • access_compat_module
    • alias_module
    • authz_core_module
    • autoindex_module
    • cache_module
    • deflate_module
    • dir_module
    • expires_module
    • headers_module
    • include_module
    • log_config_module
    • mime_module
    • mpm_event_module
    • negotiation_module
    • proxy_fcgi_module
    • proxy_module
    • rewrite_module
    • setenvif_module
    • slotmem_shm_module
    • socache_shmcb_module
    • ssl_module (This may not be installed yet. We will install it in a later step.)
    • systemd_module
    • unixd_module
    • vhost_alias_module

    Note: If you have deviated from the instructions above (for example, perhaps you decided to run a CMS other than WordPress) you may find that you need to leave some modules enabled in addition to the ones listed above. If you receive an error when restarting Apache (aka "httpd.service") try reenabling some or all of the modules disabled in this step.

  2. Restart PHP-FPM and Apache so that they use their new settings by typing the following at the command line: 
    sudo systemctl restart php-fpm.service
sudo systemctl restart httpd.service
  3. Test the new Apache and PHP-FPM configurations by visiting your WordPress site in a web browser.

7. Install an SSL Certificate

  1. Install Apache's mod_ssl using the following command: 
    sudo yum install mod_ssl
  2. Install and run certbot using instructions for "Apache" software on a "Centos/RHEL 7" system found on the Electronic Frontier Foundation's Certbot page.

In Conclusion

That's a lot of steps, I know. Following the instructions above, it takes me about half a day to set up a server from beginning to end. But I've done it a few times already. For someone who has never done it before, I'd guess that it will take a full day. I know that's a lot of time, but you'll only have to do it once. And once your server is set up, you'll have a web server that can likely host as many sites as you need.

For the record, I think the instructions above are complete and accurate. I have worked through them myself a few times now. But if you find any errors, or if you have any suggestions for improvements, please let me know. I plan to update this page in response to feedback.