In my production setup, I don't have a load balancer. This may be changed at some point, assuming that we can find one we like for the price we want to purchase it for, but in the mean time, I've been inspired to set up a virtual IP address across two machines that each can load balance between my real servers. Inspiration for this project came from this article. I'll be using Linux-HA to manage the virtual ip address with heartbeat.
I don't have any spare servers. I'm using linux-vserver to host more than one type of system on the same physical hardware. (I've got a post in the works about Xen vs Linux-Vserver too). So I've chosen two separate physical machines, both of which host virtual servers, to also host the heartbeat and director programs. I'll assume some knowledge of Linux-VServer and pretend I've already got a virtual server up and running with the hostname ha1. All of my virtual servers are running Gentoo Linux. Here is a good guide for getting Gentoo up and running inside of a linux-vserver.
Anyhow, assuming you have a vserver ready to go, here is the process.
- Make sure your vserver host is capable of IP Virtual Server stuff. The same configuration used in the basic HA loadbalancing setup for Gentoo will also work with the vserver-sources.
- Emerge the required packages in the guest vserver. I use gen-toolkit to set use flags.
>euse -E ldirectord # to pull in the ldirectord dependencies
>emerge -v heartbeat
Note that there are alternatives to ldirectord but I decided to try this 1st because it appears to be the Gentoo way. (There weren't use flags for anything else.)
Configure Heartbeat. I found sample files in /usr/share/doc/heartbeat/ I also found this page helpful.
Here are my configuration files:
auto_failback on # for me, I wanted two addresses,
# one on each machine. You could set this to off if you want
# the same machine to always have the ip address unless there
# is a failure
I'd prefer the ip addresses to be managed by node2 (ha2 for me) and I have two addresses to be managed by heartbeat.
1 sha1 whatever-key-you-want-in-text-format
Next, make sure the permissions on authkeys are correct:
chmod 600 authkeys
- Configure vserver. Ok, this part is probably somewhat controversial. According the the VServer documentation, giving the VServer extra capabilities is probably a security concern. I'll not address that here. To get heartbeat to start, the VServer needs additional system capabilities so it can work with eth0 and set itself to run at realtime priority.
On the vserver host (not in ha1):
>echo "NET_ADMIN" >> /etc/vservers/ha1/bcapabilities
>echo "NET_RAW" >> /etc/vservers/ha1/bcapabilities
>echo "SYS_NICE" >> /etc/vservers/ha1/bcapabilities
>echo "~HIDE_NETIF" >> /etc/vservers/ha1/flags
- Create the 2nd Virtual Server. You could start from scratch and install another virtual server, and perform the same configurations as above. I'm choosing to create a template from the 1st server and use that to populate ha2.
# I shut down the virtual server before doing this
tar -cvpf - * | ssh root@myothervserver "cat > /root/ha2.tar"
Next, I create ha2 on the other virtual server host:
>vserver ha2 build --context 1212 --hostname ha2.newduck.com --interface eth0:10.1.2.12/16 --initstyle gentoo -m template -- -d gentoo -t /root/ha2.tar
The only thing to do at this point is give the ha2 virtual server the same bcapabilities that ha1 needed. I also removed /var/lib/heartbeat/hb_* from ha2 (uuid needs to be regenerated)
- Progress Check I can restart either virtual server (ha1 or ha2) and make sure that the ip addresses float back and forth between the two and are always up. At this point I want to enabled my virtual servers to always come up on a reboot, and for heartbeat to start automatically:
# On ha1 and ha2's hosts
echo "default" >> /etc/vservers/ha/app/init/mark
# On ha1 and ha2
rc-update add heartbeat default
If this all isn't working at this poing, you may need to enable ip forwarding
# on the HOST
cat 1 > /proc/sys/net/ipv4/ip_forward
You can enable that in the guest too like this:
mkdir /etc/vservers/ha/sysctl/0 -p
cat "net.ipv4.ip_forward" > /etc/vserver/ha/sysctl/0/setting
cat "1" > /etc/vserver/ha/sysctl/0/value
For me, ip forwarding was already on though. There are other sysctl settings that may have impact, like send_redirects and some arp stuff, but for me, it is working. I may need to revisit the arp situation in case of an ha node failure. I'm not that far into my testing though.
- Set up ldirectord to balance and test real webservers.
The Gentoo HA failover HowTo is a good reference that covers much of the information needed for this step.
For me, with two sites I want balanced, I've changed haresources to the following:
ha1.newduck.com 10.1.2.100 ldirectord::site1.cf
ha2.newduck.com 10.1.2.101 ldirectord::site2.cf
Now, create site1.cf in /etc/ha.d/
checktimeout = 10
checkinterval = 30
virtual = 10.1.2.100:80
real= 10.1.0.26:80 gate 10
real= 10.1.1.15:80 gate 10
scheduler = wrr
protocol = tcp
request = "/lbuptest"
receive = "OK"
# repeated again for 10.1.2.101 w/ appropriate addresses
You can restart heartbeat now. Heartbeat manages ldirectord and ipvsadm internally. I noticed there are init.d scripts for those services but you should NOT turn them on.
- Check You can type ipvsadm to see output of the virtual server:
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.1.2.100:http lc
-> 10.1.0.26:http Route 10 0 2
-> 10.1.1.15:http Route 10 0 1
- Configure your real servers with ip aliases
Since I used ldirectord, which is the DirectRouting method. You need to tell the networks on the real server to respond to the virtual IP address. This took me a while to figure out because the guides I've linked to above don't mention this directly.
On the real servers, I did this:
config_dummy0=('10.1.2.100 netmask 255.255.255.255 broadcast 10.1.2.100')
I actually added both forwarded ip addresses.
And make the dummy interface start up by default
ln -s net.lo net.dummy0
rc-update add net.dummy0 default
Update: In addition, you need to tell the real servers not to respond to arp requests for ip addresses that are not on the eth that is actually on the network:
I added this to my /etc/sysctl.conf
Then refresh the sysctl settings:
I found this resource also helpful in making sure I had all the settings included that I need.
- That's it After setting up my firewall to direct my public ip addresses to the new virtual ip, I have two load balanced, redundant web servers with HA servers running on Linux Virtual Servers