HOWTO: Using nginx to accelerate Apache on Cpanel server

Nginx - the small, lightning fast and very efficient web server is usually used to serve static content or as a reverse proxy/load balancer for the Apache or other relatively slow backends. So it would be natural to use nginx as a frontend for Cpanel's Apache. It would save a substantial amount of memory and CPU time usually sucked by the numerous Apache children spoonfeeding content to the clients.

I always had this in mind, but until recently had no time to look closely at implementing it. Then I saw a forum post with a sample script for generating the nginx configuration file based on Cpanel account info, and then an onslaught of visitors on a shared Cpanel server I admin slowed it to a crawl, and I was forced to delve into the innards of Cpanel. As a result of this investigation I wrote the "nginx on Cpanel" HOWTO presented below.

---

Installing Apache module

First of all, when nginx is used as a reverse proxy to Apache, the visitors' IPs received by Apache are wrong - all requests to Apache come from nginx, so the main server IP will be logged.
To make Apache log the real IPs of the visitors instead of the main server IP, a special Apache module (mod_rpaf) is needed.
Download, untar, cd to the newly created directory and run this command as root:
/usr/local/apache/bin/apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
That will install the module into the Apache module directory.

Then go to WHM, Main >> Service Configuration >> Apache Configuration > Include Editor > Pre Main Include and add this section there, replacing LIST_OF_YOUR_IPS with the list of IP addresses managed by Cpanel:

LoadModule rpaf_module modules/mod_rpaf-2.0.so 

RPAFenable On
# Enable reverse proxy add forward
RPAFproxy_ips 127.0.0.1 LIST_OF_YOUR_IPS
# which ips are forwarding requests to us
RPAFsethostname On
# let rpaf update vhost settings
# allows to have the same hostnames as in the "real"
# configuration for the forwarding Apache
RPAFheader X-Real-IP
# Allows you to change which header mod_rpaf looks
# for when trying to find the ip the that is forwarding
# our requests

Apache configuration changes

Then we need to move Apache to another port, let's take 81 for example. You can simply edit it in the "Tweak Settings" page in WHM, replacing 0.0.0.0:80 with 0.0.0.0:81 or, doing it command line way, edit /var/cpanel/cpanel.config and change port 80 in apache_port assignment to 81:

apache_port=0.0.0.0:81

Run /usr/local/cpanel/whostmgr/bin/whostmgr2 --updatetweaksettings as advised at the top of that file.
Check /usr/local/apache/conf/httpd.conf for any occurences of port 80, and run /scripts/rebuildhttpdconf to make sure httpd.conf is up to date.

It also makes sense to reduce the number of Apache children, as nginx will take care of spoonfeeding the data to the clients connecting via the slow network links, freeing Apache children to do their backend work. Edit /usr/local/apache/conf/httpd.conf and replace prefork.c section with this (note that I used very modest values here, and your mileage may vary):

<IfModule prefork.c>
    StartServers 5
    MinSpareServers 2
    MaxSpareServers 5
    MaxClients 50
    MaxRequestsPerChild 0
</IfModule>

Run /usr/local/cpanel/bin/apache_conf_distiller --update --main to pick up the changes, and then /scripts/rebuildhttpdconf to make sure your changes are in.
Note that you will need to watch Apache extended server status at the peak load times to have an idea how many Apache children your server needs by default.

You'll also need to update the Apache port in /etc/chkserv.d/httpd and restart chksrvd with /etc/init.d/chksrvd restart

Generating nginx config files

The final step - we have to build the nginx config file based on the domains hosted on your server.
It is done by the simple script which will generate two configuration files for nginx - main one here: /usr/local/nginx/conf/nginx.conf and the include file with all virtual hosts: /usr/local/nginx/conf/vhost.conf

#!/bin/sh

cat > "/usr/local/nginx/conf/nginx.conf" <<EOF
user  nobody;
# no need for more workers in the proxy mode
worker_processes  1;

error_log  logs/error.log info;

worker_rlimit_nofile  8192;

events {
 worker_connections  512; # increase for more busy servers
 use rtsig; # you should use epoll here for Linux kernels 2.6.x
}

http {
 server_names_hash_max_size 2048;

 include    mime.types;
 default_type  application/octet-stream;

 sendfile on;
 tcp_nopush on;
 tcp_nodelay on;

 keepalive_timeout  10;

 gzip on;
 gzip_min_length  1100;
 gzip_buffers  4 32k;
 gzip_types    text/plain text/html application/x-javascript text/xml text/css;
 ignore_invalid_headers on;

 client_header_timeout  3m;
 client_body_timeout 3m;
 send_timeout     3m;
 connection_pool_size  256;
 client_header_buffer_size 4k;
 large_client_header_buffers 4 32k;
 request_pool_size  4k;
 output_buffers   4 32k;
 postpone_output  1460;

 include "/usr/local/nginx/conf/vhost.conf";
}

EOF

/bin/cp /dev/null /usr/local/nginx/conf/vhost.conf

cd /var/cpanel/users
for USER in *; do
 for DOMAIN in `cat $USER | grep ^DNS | cut -d= -f2`; do
  IP=`cat $USER|grep ^IP|cut -d= -f2`;
  ROOT=`grep ^$USER: /etc/passwd|cut -d: -f6`;
  echo "Converting $DOMAIN for $USER";

  cat >> "/usr/local/nginx/conf/vhost.conf" <<EOF
   server {
  access_log off;

  error_log  logs/vhost-error_log warn;
  listen    80;
  server_name  $DOMAIN www.$DOMAIN;

  # uncomment location below to make nginx serve static files instead of Apache
  # !WARNING!
  # it will make the bandwidth accounting incorrect as these files won't be logged!
  #location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css)$ {
  # root   $ROOT/public_html;
  #}

  location / {
   client_max_body_size    10m;
   client_body_buffer_size 128k;

   proxy_send_timeout   90;
   proxy_read_timeout   90;

   proxy_buffer_size    4k;
   # you can increase proxy_buffers here to suppress "an upstream response
   #  is buffered to a temporary file" warning
   proxy_buffers     16 32k;
   proxy_busy_buffers_size 64k;
   proxy_temp_file_write_size 64k;

   proxy_connect_timeout 30s;

   proxy_redirect  http://www.$DOMAIN:81   http://www.$DOMAIN;
   proxy_redirect  http://$DOMAIN:81   http://$DOMAIN;

   proxy_pass   http://$IP:81/;

   proxy_set_header   Host   \$host;
   proxy_set_header   X-Real-IP  \$remote_addr;
   proxy_set_header   X-Forwarded-For \$proxy_add_x_forwarded_for;
  }
 }
EOF
 done
done

Run /usr/local/nginx/sbin/nginx -t to check the configuration, and then /usr/local/nginx/sbin/nginx to start nginx. You are set!

If you don't care about the bandwidth consumed by the virtual hosts and are willing to lose the correct bandwidth calculation over the increased server performance, you can uncomment the <location> lines below the WARNING comment and watch the server picking up the speed. Beware of the two gotchas here: the sub domains most likely will not work as they have document root pointed to a different place, and as nginx doesn't support .htaccess files for the performance reasons, they won't be obeyed for the file types listed.

Obviously, the config file must be regenerated every time a new domain is added. The deleted and suspended domains should work just fine though.

Conclusion

You may ask, is it really worth the trouble? Here's a graph of the average load on the server where nginx was installed as a reverse proxy for Apache as described in this post, can you guess where the switch to nginx happened?