May 30, 2008

HOWTO: Using nginx to accelerate Apache on Cpanel server

Nginx - the small, lightning fast and very efficient web server is usually used to serve static content or as a reverse proxy/load balancer for the Apache or other relatively slow backends. So it would be natural to use nginx as a frontend for Cpanel's Apache. It would save a substantial amount of memory and CPU time usually sucked by the numerous Apache children spoonfeeding content to the clients.

I always had this in mind, but until recently had no time to look closely at implementing it. Then I saw a forum post with a sample script for generating the nginx configuration file based on Cpanel account info, and then an onslaught of visitors on a shared Cpanel server I admin slowed it to a crawl, and I was forced to delve into the innards of Cpanel. As a result of this investigation I wrote the "nginx on Cpanel" HOWTO presented below.



Installing Apache module


First of all, when nginx is used as a reverse proxy to Apache, the visitors' IPs received by Apache are wrong - all requests to Apache come from nginx, so the main server IP will be logged.
To make Apache log the real IPs of the visitors instead of the main server IP, a special Apache module (mod_rpaf) is needed.
Download, untar, cd to the newly created directory and run this command as root:
/usr/local/apache/bin/apxs -i -c -n mod_rpaf-2.0.so mod_rpaf-2.0.c
That will install the module into the Apache module directory.

Then go to WHM, Main >> Service Configuration >> Apache Configuration > Include Editor > Pre Main Include and add this section there, replacing LIST_OF_YOUR_IPS with the list of IP addresses managed by Cpanel:

LoadModule rpaf_module modules/mod_rpaf-2.0.so 

RPAFenable On
# Enable reverse proxy add forward
RPAFproxy_ips 127.0.0.1 LIST_OF_YOUR_IPS
# which ips are forwarding requests to us
RPAFsethostname On
# let rpaf update vhost settings
# allows to have the same hostnames as in the "real"
# configuration for the forwarding Apache
RPAFheader X-Real-IP
# Allows you to change which header mod_rpaf looks
# for when trying to find the ip the that is forwarding
# our requests


Apache configuration changes


Then we need to move Apache to another port, let's take 81 for example. You can simply edit it in the "Tweak Settings" page in WHM, replacing 0.0.0.0:80 with 0.0.0.0:81 or, doing it command line way, edit /var/cpanel/cpanel.config and change port 80 in apache_port assignment to 81:
apache_port=0.0.0.0:81
Run /usr/local/cpanel/whostmgr/bin/whostmgr2 --updatetweaksettings as advised at the top of that file.
Check /usr/local/apache/conf/httpd.conf for any occurences of port 80, and run /scripts/rebuildhttpdconf to make sure httpd.conf is up to date.

It also makes sense to reduce the number of Apache children, as nginx will take care of spoonfeeding the data to the clients connecting via the slow network links, freeing Apache children to do their backend work. Edit /usr/local/apache/conf/httpd.conf and replace prefork.c section with this (note that I used very modest values here, and your mileage may vary):
<IfModule prefork.c>
StartServers 5
MinSpareServers 2
MaxSpareServers 5
MaxClients 50
MaxRequestsPerChild 0
</IfModule>

Run /usr/local/cpanel/bin/apache_conf_distiller --update --main to pick up the changes, and then /scripts/rebuildhttpdconf to make sure your changes are in.
Note that you will need to watch Apache extended server status at the peak load times to have an idea how many Apache children your server needs by default.

You'll also need to update the Apache port in /etc/chkserv.d/httpd and restart chksrvd with /etc/init.d/chksrvd restart

Generating nginx config files


The final step - we have to build the nginx config file based on the domains hosted on your server.
It is done by the simple script which will generate two configuration files for nginx - main one here: /usr/local/nginx/conf/nginx.conf and the include file with all virtual hosts: /usr/local/nginx/conf/vhost.conf

#!/bin/sh

cat > "/usr/local/nginx/conf/nginx.conf" <<EOF
user nobody;
# no need for more workers in the proxy mode
worker_processes 1;

error_log logs/error.log info;

worker_rlimit_nofile 8192;

events {
worker_connections 512; # increase for more busy servers
use rtsig; # you should use epoll here for Linux kernels 2.6.x
}

http {
server_names_hash_max_size 2048;

include mime.types;
default_type application/octet-stream;

sendfile on;
tcp_nopush on;
tcp_nodelay on;

keepalive_timeout 10;

gzip on;
gzip_min_length 1100;
gzip_buffers 4 32k;
gzip_types text/plain text/html application/x-javascript text/xml text/css;
ignore_invalid_headers on;

client_header_timeout 3m;
client_body_timeout 3m;
send_timeout 3m;
connection_pool_size 256;
client_header_buffer_size 4k;
large_client_header_buffers 4 32k;
request_pool_size 4k;
output_buffers 4 32k;
postpone_output 1460;

include "/usr/local/nginx/conf/vhost.conf";
}

EOF

/bin/cp /dev/null /usr/local/nginx/conf/vhost.conf

cd /var/cpanel/users
for USER in *; do
for DOMAIN in `cat $USER | grep ^DNS | cut -d= -f2`; do
IP=`cat $USER|grep ^IP|cut -d= -f2`;
ROOT=`grep ^$USER: /etc/passwd|cut -d: -f6`;
echo "Converting $DOMAIN for $USER";

cat >> "/usr/local/nginx/conf/vhost.conf" <<EOF
server {
access_log off;

error_log logs/vhost-error_log warn;
listen 80;
server_name $DOMAIN www.$DOMAIN;

# uncomment location below to make nginx serve static files instead of Apache
# !WARNING!
# it will make the bandwidth accounting incorrect as these files won't be logged!
#location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp4|htm|html|js|css)$ {
# root $ROOT/public_html;
#}

location / {
client_max_body_size 10m;
client_body_buffer_size 128k;

proxy_send_timeout 90;
proxy_read_timeout 90;

proxy_buffer_size 4k;
# you can increase proxy_buffers here to suppress "an upstream response
# is buffered to a temporary file" warning
proxy_buffers 16 32k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

proxy_connect_timeout 30s;

proxy_redirect http://www.$DOMAIN:81 http://www.$DOMAIN;
proxy_redirect http://$DOMAIN:81 http://$DOMAIN;

proxy_pass http://$IP:81/;

proxy_set_header Host \$host;
proxy_set_header X-Real-IP \$remote_addr;
proxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;
}
}
EOF
done
done


Run /usr/local/nginx/sbin/nginx -t to check the configuration, and then /usr/local/nginx/sbin/nginx to start nginx. You are set!

If you don't care about the bandwidth consumed by the virtual hosts and are willing to lose the correct bandwidth calculation over the increased server performance, you can uncomment the <location> lines below the WARNING comment and watch the server picking up the speed. Beware of the two gotchas here: the sub domains most likely will not work as they have document root pointed to a different place, and as nginx doesn't support .htaccess files for the performance reasons, they won't be obeyed for the file types listed.

Obviously, the config file must be regenerated every time a new domain is added. The deleted and suspended domains should work just fine though.

Conclusion


You may ask, is it really worth the trouble? Here's a graph of the average load on the server where nginx was installed as a reverse proxy for Apache as described in this post, can you guess where the switch to nginx happened?



52 comments:

andphe said...

Hi, thanks for this post is a good resource, just consider incluide two extra extensions of static content to serve: .ico and .swf

Sam said...

Can you add some more features as provided by - http://www.webhostingtalk.com/showthread.php?t=708332 page 2 of the thread says, this how-to will fail at some condition.

figvam said...

OK, I read that thread but it seems the guy is just trying to downplay this HOWTO. Rewriting of the Location headers with embedded Apache port is handled by the proxy_redirect directives, so there is nothing to worry about here.

He also asserts "the guide has no configuration generation scripts" but that's what this HOWTO is all about - the said script is the main part of it!

In fact I already have a working nginx setup that has the same features as his paid installation service provides - slightly patched nginx that processes the static files but passes the dynamic content to Apache. I just don't have time to wrap it up.

Anonymous said...

why we need to change the port to 81?

figvam said...

> why we need to change the port to 81?

I picked up 81 as an example. You can set it to any unused port: 1234, 8080, etc.

As to why we need to change the port - that's because port 80 will be occupied by nginx serving as a proxy, so Apache needs some other port to listen and serve incoming requests from nginx.

Anonymous said...

thanks figvam,

can wed disable the Apache?

Anonymous said...

wed = we

Anonymous said...

If you want to disable apache, then this is not the guide for you.

figvam said...

> can we disable the Apache?

No, or at least not using the technique described in this HOWTO. Apache is needed to handle the dynamic files (PHP, shtml, cgi). Theoretically both shtml and PHP could be processed by nginx, but not CGI. Also, nginx doesn't support .htaccess files Cpanel uses.

Anonymous said...

Hello,is this tested with latest version of whm/cpanel?I cant find anywhere prefork.c
in conf files neither settings declared in it.When i add that to apache conf then sites didnt loaded at all.Also i noticed there is no need for update command,since apache port change works same moment after i put 81 into tweak settings.So i downloaded latest stable version of ngix and install but when i put that text into config i getting error when i type test command about unable to proccess cat directive unkown command.If i delete that line,that i get same error again but then with line where is rsig command.So tehnicly i wont need to change anything beside apache port and nginx config file since i dont need logging of ip adresses and changing prefork settings wont work anyway.So question is why prefork section doesnt work and why i getting error with nginx configuration.

Anonymous said...

Actualy i talk nonsense,i need to create file for generating config and not puting that text into config.

Anonymous said...

Ok now i managed to install it but there are still some questions remain.On my version of centos and whm chksrvd is called chkservd.
What to do with fork section on apache,and what is most important,how to make htaccess to work?When i put htaccess from apache it does work but then image from that domain not loading properly-some loading and some not.Any idea why is that happening considering this is not nginx install it is just proxy mode.

Anonymous said...

I added
location = /st/admin {
auth_basic "RESTRICTED ACCESS";
auth_basic_user_file /home/domain/public_html/st/admin/.htpasswd;
}
to section under domain which folder i want to protect,but when i enter username and password i getting 404 error.Any idea why ?

figvam said...

> In my version of centos and whm chksrvd is called chkservd.

Yes, apparently the service is called chkservd now. I'll update the guide.

> What to do with fork section on apache

Looks like you are not using prefork MPM. Then you have to update whatever MPM configuration your Apache is set up with. You can also leave the configuration unchanged, but in that case Apache will still consume the same amount of memory as before, removing the main advantage of the nginx proxying.

> how to make htaccess to work?When i put htaccess from apache it does work but then image from that domain not loading properly-some loading and some not.Any idea why is that happening considering this is not nginx install it is just proxy mode.

No idea here. Do you see this problem with the images only, or with the pages, too? Did you enable the handling of static file types by nginx (lines labeled with WARNING in the script)?

> I added
location = /st/admin {
auth_basic "RESTRICTED ACCESS";
auth_basic_user_file /home/domain/public_html/st/admin/.htpasswd;
}
to section under domain which folder i want to protect,but when i enter username and password i getting 404 error.Any idea why ?


Nginx doesn't support .htaccess and .htpasswd files (and all associated Apache config directives for the start). All authentication stuff will still be handled by Apache - Nginx will pass the authentication tokens from the browser to Apache. So you have to control the authentication stuff through Cpanel/WHM or by editing .htaccess files.

Anonymous said...

> Looks like you are not using prefork MPM. Then you have to update whatever MPM configuration your Apache is set up with. You can also leave the configuration unchanged, but in that case Apache will still consume the same amount of memory as before, removing the main advantage of the nginx proxying.

Actualy it seems you puted wrong name into if-it should mpm_prefork.c and not prefork.c.I dont know how did you got only prefork.c since i have mpm_prefork since ages.So i puted mpm_prefork and it works fine now.

> No idea here. Do you see this problem with the images only, or with the pages, too? Did you enable the handling of static file types by nginx (lines labeled with WARNING in the script)?

No i didnt enable handling of static files.

>Nginx doesn't support .htaccess and .htpasswd files (and all associated Apache config directives for the start). All authentication stuff will still be handled by Apache - Nginx will pass the authentication tokens from the browser to Apache. So you have to control the authentication stuff through Cpanel/WHM or by editing .htaccess files.

Actualy i does suport htpasswd files but not htaccess.I will try to repeat install of nginx this time with proper fork installation so let see what happening.
Also,i tried different apache configuration-instead suexec and dso and prefork i puted mpM_worker and mod_fcgi for apache and fast cgi for php.But the problem with fast cgi is it doesnt support well zend optimizer so script with zend doesnt work properly.But for someone who doesnt use zend application coulg come handy since it eliminate toomany connection problem(with it there is no slowdown and timeout)

Also one more thing:it seems there is no need to update vhost configuration when new host is added on apache.I removed one domain from vhost and it still worked fine.It seems nginx even if there is no details about domain in vhost it will redirect it properly.I think key is in ip adress,so instead puting details for each domain it should be enough to put ip info like this one:
server
{
listen 127.0.0.1:80;
server_name DOMAIN www.DOMAIN;
limit_conn zonetwo 200;

location /
{
proxy_pass http://127.0.0.1:81;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
# Redirects for this domain and its aliases added automatically
proxy_redirect http://DOMAIN:81/ http://$host:$server_port/;
proxy_redirect http://www.DOMAIN:81/ http://$host:$server_port/;

}

}

NIMHOST said...

problem with this one

# !WARNING!
# it will make the bandwidth accounting incorrect as these files won't be logged!
#location ~* \.(gif|jpg|jpeg|png|wmv|avi|mpg|mpeg|mp3|mp4|js|css|w3x|zip)$ {
#root $ROOT/public_html;
#}

when any hosting account is using addon domain :)

figvam said...

NIMHOST
> problem with this one when any hosting account is using addon domain :)


Yes, I warned about it in the last part.

NIMHOST said...

i'm interested to have chat with you about this, have you any IM to contact you and adding you ?

Anonymous said...

Houston we have problem again.
When i installed on server two,i had to revert to apache beacuse traffic was marked as 100% proxy traffic for some reason on all sites.So i install it to server one and there was'nt that problem,until i checked two sites where that problem showed again while other sites are ok even on same ip and domain account.Mod rpa is installed and properly configured.I noticed there is upstream response error for that two sites,and i fixed that by increasing proxy values to 64 128k
But even after that and reseting nginx i still getting all traffic marked as proxy traffic.Before i mentioned there was a problem with half pictures not showing.I resolved that problem by increasing number of workers to 2048(since i saw error about it in log)So i wonder could be toomuch workers problem with those two sites where all traffic is showed as proxy in trade script(that is script which tracks traffic)

Łukasz said...

But if i add new domain (virtualhost) by WHM then i should run your script manualy to generate vhost.conf for nginx? If, yes is there possibility to do that automaticlly when we add new domain (account) to WHM?

cucu said...

Please help,

I used the conf above and when running the test it failed:

unknown directive "cat" in /usr/local/nginx/conf/nginx.conf:4

how do i fix it ?

cucu said...

one question, before installing nignx i have a domain assigned with an ip and i can access it via the following path:

http://domain/test
http://ip/test

now after nginx install i get the 404 page not found error when try to go use the above path.

if i go to
http://domain/test
http://ip/test

then it give me the "welcome to nginx..." page

Isn't it suppose to work the same way? what am i doing wrong ?

Thank you

cucu said...

CORRECTION

if i go to

http://domain
http://ip

then it give me the "welcome to nginx..." page

Isn't it suppose to work the same way? what am i doing wrong ?

Thank you

figvam said...

cuci:
Yes, accessing by IP for the dedicated IP accounts doesn't work with the version of the config file generator I've published. However http://domain should always work.

You can make dedicated IP addresses work by altering the script a little:
- change "listen 80;" to "listen $IP:80;"
- add another line after two proxy_redirect lines:
"proxy_redirect http://$IP:81 http://$IP;"

Łukasz
> But if i add new domain (virtualhost) by WHM then i should run your script manualy to generate vhost.conf for nginx? If, yes is there possibility to do that automaticlly when we add new domain (account) to WHM?

Yes, you'll have to regenerate the nginx config on every domain addition. You can automate this by using Cpanel hooks:
http://www.cpanel.net/support/docs/hooks.htm#postwww

figvam said...

> You can make dedicated IP addresses work by altering the script a little

Oh, you'll need another addition to the script or Cpanel service check will fail repeatedly. Just before include "/usr/local/nginx/conf/vhost.conf"; line add this piece:

server {
access_log off;
error_log logs/error_log;
listen 80;
}

Anonymous said...

Hello,

This is a great tutorial, follow the steps and everything seem to be working. Just have a little problem with file downloading, mostly on slow connection.

On broad band connection i can download file (zip, rar) just fine. but on slow connection. the download get disrupt and the download could never get finish. It vary, for the same file some time it stop at 10% complete, sometime it at 30%. What setting should i change to get it working correctly with slow connection

figvam said...

Anonymous @23 January, 2009 06:30

If it happens with the large files only, then check the permissions on the proxy temp directory (/usr/local/nginx/proxy_temp by default).

Anonymous said...

what should be the correct permission for usr/local/nginx/proxy_temp

figvam said...

/usr/local/nginx/proxy_temp should be owned by "nobody" user with 0700 permissions. If you want to be on the safe side, you can do "chmod 777 /usr/local/nginx/proxy_temp" to allow all users to write to that directory.

Sushant said...

This is a great tutorial. I am having a little problem on the last step. I have create nginx.sh and copy-pasted the script to generate configuration files and ran it by giving 755 permission but I am getting these errors:

./nginx.sh: line 3: /usr/local/nginx/conf/nginx.conf: No such file or directory
/bin/cp: cannot create regular file `/usr/local/nginx/conf/vhost.conf': No such file or directory
Converting domain.com for user
./nginx.sh: line 59: /usr/local/nginx/conf/vhost.conf: No such file or directory
Converting domain.com for user
./nginx.sh: line 59: /usr/local/nginx/conf/vhost.conf: No such file or directory

Anonymous said...

A great tutorial, it's worked like a charm on my server.

One suggestion though that's worked out great for me - instead of generating the vhost config file for nginx based on cPanel data, why not just rip all the subdomain data from Apache's config itself, which would then have the proper document roots?

figvam said...

Sushant, it looks like nginx is installed in some other place than /usr/local/nginx on your server. You should correct the config file path in the script accordingly.

Anonymous @ 28 May, 2009
Yes, that would work, if you're able to write a good parser for httpd.conf. Personally I think it's a hassle, and a proper way would be to grab the domain data directly from /var/cpanel/userdata. It's stored in YAML so parsing it would be easier than httpd.conf, and not that ambiguous, too.

Anonymous said...

Will this step work with apache MPM worker?

Thanks alot.

figvam said...

Anonymous @ 21 July, 2009 16:28

You can use any Apache MPM with this guide, there is nothing specific to the prefork MPM except the StartServer/MaxClient tweaking part which obviously needs to be adjusted accordingly. Or you can leave it alone, it wouldn't affect the site operation, except for a small waste of memory on the extra Apache worker threads.

Anonymous said...

Thank you for this great howto.

So, just to confirm, apache in cpanel will handle the dynamic sites like php while nginx will handle static files right?

figvam said...

> So, just to confirm, apache in cpanel will handle the dynamic sites like php while nginx will handle static files right?

Nope :) Static files still go through Apache, but since nginx does much better job of spoonfeeding them to the clients, Apache processes are freed to handle next requests.

You can uncomment the lines denoted with WARNING in the script to make nginx handle the static content, but that will break the bandwidth accounting in Cpanel (the static files won't be counted).

Anonymous said...

Thanks Figvam for the reply.

coldplug said...

Hello, thx for how-to. One question: why not set apache listen to 127.0.0.1:80 and nginx to IP:80, that way they both coexist smootly, but apache does not listen on external interface which can be security advantage over configuration where apache listen on external interface's port 81 or whatever.

Am I right?

figvam said...

@coldplug: I haven't tried the loopback interface, but of course it's a viable option.

I don't see any security issues with Apache on the external interface though. After all that's how it is working in the default Cpanel.

Alexey said...

Thank you for your work!

I have a note - vhost generation script uses wrong $ROOT dir when root dir not within public_html.
DOMAIN can be in /home/ACCOUNT/ or /home/ACCOUNT/public_html/SUBDIR - but in all cases $ROOT will be /home/ACCOUNT/public_html.

That's broke many things.
Better to get real $ROOT from cPanel configs or maybe from httpd.conf ?

today said...

I have the same issue, how to get real root path for adddon domains?

figvam said...

@Alexey, @today:
Yes, I know about this shortcoming in the HOWTO. I described a possible way to deal with it in
this comment.

Alexey said...

Thank you.

However, I don't know how to parse /var/cpanel/userdata/ also.

today said...

Thanks,
I am bad in linux administration too.. Don't know how to do this :(

Also little notice
proxy_pass http://$IP:81/;
should be
proxy_pass http://$IP:81;
without slash at the end, in other way $_SERVER['REQUEST_URI'] variable is in low case.

Heru said...

http://www.cpanel.net/support/docs/hooks.htm#postwww

I'm confused how to hook my shell script to cPanel, there is no example on that link, can you help?

figvam said...

@Heru: apparently the hooks docs were moved here:
http://docs.cpanel.net/twiki/bin/view/AllDocumentation/AutomationIntegration/WebHome#Extending_cPanel_Functionality

Unfortunately I have no experience with Cpanel hooks so I can't help.

Anonymous said...

thanks for the tutorial, but why i still see .css / .js in access_log

please advise..

Anonymous said...

i mean apache access_log

Anonymous said...

Hi figvam,

Just want to ask. Will uploadprogress ( http://pecl.php.net/package/uploadprogress ) still works with nginx as proxy?

If not, can you include the configuration for this: http://wiki.nginx.org/NginxHttpUploadProgressModule


Much appreciated.

Ovidiu said...

how can I verify that my static components are actually delivered by nginx and not by apache? Just checking if I have set it up correctly..

Dave said...

Wow. I've now seen a BUNCH of newer posts on other blogs that are just copies of this one, with no updates AFAIK.

Is this still the latest and greatest -- it's been three years, does it still work this way?

Thanks...

EugeneWHZ said...

Hi,

I created a post that covers alternative methods of Nginx installation on cPanel server. Please feel free to check my post - Install Nginx on cPanel – Stop cPanel server load.

Thank you!