How to virtual host load balanced websites with ldirectord and Apache

I posted a while back on getting Heartbeat set up to add reliability to websites. After a few weeks of experience with the system, I thought I’d add a few additional tips on making the setup more reliable. There are already a few good guides on getting heartbeat set up. You could also read my original post on the subject if you don’t already have heartbeat load balancing your site. This post however, deals with the case when you are servicing more than one site per physical server.

We host three different websites on three different physical servers. Each physical server hosts two websites with Apache. Each website is hosted on two different physical servers. The sites are load balanced with ldirectord which resides on two different servers that manage the public IP addresses to our services with Heartbeat. If load increases on any of our services, we could always add additional physical servers relatively easily.

First, a quick refresher on how virtual hosting works with Apache. There are a couple ways to virtual host websites with Apache:

1. Use VirtualHost to host different sites on different IP Addresses.
   <VirtualHost 10.1.1.1:80>
 ...All the first hosts information...
</VirtualHost>
<VirtualHost 10.1.1.2:80>
 ...All the host other hosts information...
</VirtualHost>

2. Use NameVirtualHost : to host more than one site on the same IP Address.
   NameVirtaulHost 10.1.1.1:80
<VirtualHost 10.1.1.1:80>
# ServerName tells Apache to use this host when the host header matches
 ServerName onewebsite.example
 ...All the host information...
</VirtualHost>
<VirtualHost 10.1.1.1:80>
 ServerName theotherwebsite.example
 ...All the host information...
</VirtualHost>


Of course, you can use “*” instead of an IP address. You can also use combinations of the two forms above to build whatever combination of virtual hosts you like. For instance, you could have 3 websites on one IP address with NameVirtualHost, and then a forth website on it’s own 2nd IP Address.

Now, onto the important stuff. Originally, I was using NameVirtualHost to host two websites per machine. I quickly found that on occasion, the wrong website was served. A valid request (with the correct host header) to the Apache instance resulted in the incorrect host being served up. I took the following steps to remedy the situation.

1. Use Different IP Addresses

   This allows you to get rid of NameVirtualHost. The advantage is that a HTTP/1.0 request without a host header can be serviced correctly. Even though ldirectord has a mechanism to send a host header with its scheduled check test, I found that in the case one site isn’t working correctly, the check test may be serviced by another host.

   First, set up your computer to alias multiple IP addresses. Second, change each virtual host to have one unique IP address. (Like the first virtual host example above).

2. Don’t listen on localhost

   I found that this only causes confusion. If each service has its own distinct IP address, then which service gets localhost? I had a case where the default localhost page was served up instead of the actual virtual host site for the IP address. Listening only on the IP Addresses for the host sites fixed this for me.
   #Listen *:80 # comment out all of these
Listen 10.1.1.1:80 # and add individual IP addresses
<VirtualHost 10.1.1.1:80>
 ...All the first host information...
</VirtualHost>
Listen 10.1.1.2:80
<VirtualHost 10.1.1.2:80>
 ...All the other host information...
</VirtualHost>

   When you’ve got this set up correctly, you can verify that there is no listen on localhost.
   >netstat -ntl | grep ':80'
tcp 0 0 10.1.1.1:80 0.0.0.0:* LISTEN
tcp 0 0 10.1.1.2:80 0.0.0.0:* LISTEN


3. Create different up tests for ldirectord

   I had originally used the same up test on each site to let ldirectord know whether or not the site was working. I changed the receive line for the virtual configuration to be unique values across different websites. This allows ldirectord another disambiguation that can only help.
   # snipped host1.cf
 request = "/lbuptest"
 receive = "HOST1_OK"
# snipped host2.cf
 request = "/lbuptest"
 receive = "HOST2_OK"

   Of course, don’t forget to modify the real hosts on each physical machine to return the correct response.

After making these adjustments, I’ve had no additional problems virtual hosting sites with ldirectord and Apache. As a bonus, I created a script to make sure the load balancing is working correctly. In each VirtualHost section for each site, add an HTTP header to tell you which server is servicing the request:
Header set Real-Server phy1host1 # change the value to whatever you want.


Here is my test_rr.sh:
#!/bin/bash
do_get() {
 echo "HEAD / HTTP/1.1
Host: $1
Connection: close
" | nc $1 80 | grep Real-Server
}

x=0
while (($x<4)); do
 do_get $1
 x=$x+1
done


Just pass in the IP address of the site you want to test:
./test_rr.sh <public ip address>
Real-Server: w1
Real-Server: w3
Real-Server: w1
Real-Server: w3


Feel free to comment if I forgot anything.