Using Linux-HA for High Availability with Gentoo and Linux-VServer

In my production setup, I don’t have a load balancer. This may be changed at some point, assuming that we can find one we like for the price we want to purchase it for, but in the mean time, I’ve been inspired to set up a virtual IP address across two machines that each can load balance between my real servers. Inspiration for this project came from this article. I’ll be using Linux-HA to manage the virtual ip address with heartbeat.

I don’t have any spare servers. I’m using linux-vserver to host more than one type of system on the same physical hardware. (I’ve got a post in the works about Xen vs Linux-Vserver too). So I’ve chosen two separate physical machines, both of which host virtual servers, to also host the heartbeat and director programs. I’ll assume some knowledge of Linux-VServer and pretend I’ve already got a virtual server up and running with the hostname ha1. All of my virtual servers are running Gentoo Linux. Here is a good guide for getting Gentoo up and running inside of a linux-vserver.

Anyhow, assuming you have a vserver ready to go, here is the process.

  1. Make sure your vserver host is capable of IP Virtual Server stuff. The same configuration used in the basic HA loadbalancing setup for Gentoo will also work with the vserver-sources.
  2. Emerge the required packages in the guest vserver. I use gen-toolkit to set use flags.
    >euse -E ldirectord # to pull in the ldirectord dependencies
    >emerge -v heartbeat

    Note that there are alternatives to ldirectord but I decided to try this 1st because it appears to be the Gentoo way. (There weren’t use flags for anything else.)
  3. Configure Heartbeat. I found sample files in /usr/share/doc/heartbeat/ I also found this page helpful.

    Here are my configuration files:
    # /etc/ha.d/ha.cf
    use_logd on
    logfacility daemon
    bcast eth0
    keepalive 2
    warntime 10
    deadtime 30
    auto_failback on # for me, I wanted two addresses,
    # one on each machine. You could set this to off if you want
    # the same machine to always have the ip address unless there
    # is a failure
    node ha1
    node ha2

    I’d prefer the ip addresses to be managed by node2 (ha2 for me) and I have two addresses to be managed by heartbeat.
    # /etc/ha.d/haresources
    ha1 10.1.2.100
    ha2 10.1.2.101

    # /etc/ha.d/authkeys
    auth 1
    1 sha1 whatever-key-you-want-in-text-format

    Next, make sure the permissions on authkeys are correct:
    chmod 600 authkeys

  4. Configure vserver. Ok, this part is probably somewhat controversial. According the the VServer documentation, giving the VServer extra capabilities is probably a security concern. I’ll not address that here. To get heartbeat to start, the VServer needs additional system capabilities so it can work with eth0 and set itself to run at realtime priority.

    On the vserver host (not in ha1):
    >echo "NET_ADMIN" >> /etc/vservers/ha1/bcapabilities
    >echo "NET_RAW" >> /etc/vservers/ha1/bcapabilities
    >echo "SYS_NICE" >> /etc/vservers/ha1/bcapabilities
    >echo "~HIDE_NETIF" >> /etc/vservers/ha1/flags

  5. Create the 2nd Virtual Server. You could start from scratch and install another virtual server, and perform the same configurations as above. I’m choosing to create a template from the 1st server and use that to populate ha2.
    cd /vservers/ha1
    # I shut down the virtual server before doing this
    tar -cvpf - * | ssh root@myothervserver "cat > /root/ha2.tar"

    Next, I create ha2 on the other virtual server host:
    >vserver ha2 build --context 1212 --hostname ha2.newduck.com --interface eth0:10.1.2.12/16 --initstyle gentoo -m template -- -d gentoo -t /root/ha2.tar
    The only thing to do at this point is give the ha2 virtual server the same bcapabilities that ha1 needed. I also removed /var/lib/heartbeat/hb_* from ha2 (uuid needs to be regenerated)
  6. Progress Check I can restart either virtual server (ha1 or ha2) and make sure that the ip addresses float back and forth between the two and are always up. At this point I want to enabled my virtual servers to always come up on a reboot, and for heartbeat to start automatically:
    # On ha1 and ha2's hosts
    echo "default" >> /etc/vservers/ha[12]/app/init/mark
    # On ha1 and ha2
    rc-update add heartbeat default

    If this all isn’t working at this poing, you may need to enable ip forwarding
    # on the HOST
    cat 1 > /proc/sys/net/ipv4/ip_forward

    You can enable that in the guest too like this:
    mkdir /etc/vservers/ha[12]/sysctl/0 -p
    cat "net.ipv4.ip_forward" > /etc/vserver/ha[12]/sysctl/0/setting
    cat "1" > /etc/vserver/ha[12]/sysctl/0/value

    For me, ip forwarding was already on though. There are other sysctl settings that may have impact, like send_redirects and some arp stuff, but for me, it is working. I may need to revisit the arp situation in case of an ha node failure. I’m not that far into my testing though.

  7. Set up ldirectord to balance and test real webservers.
    The Gentoo HA failover HowTo is a good reference that covers much of the information needed for this step.

    For me, with two sites I want balanced, I’ve changed haresources to the following:
    ha1.newduck.com 10.1.2.100 ldirectord::site1.cf
    ha2.newduck.com 10.1.2.101 ldirectord::site2.cf

    Now, create site1.cf in /etc/ha.d/
    checktimeout = 10
    checkinterval = 30
    virtual = 10.1.2.100:80
      real= 10.1.0.26:80 gate 10
      real= 10.1.1.15:80 gate 10
      scheduler = wrr
      protocol = tcp
      request = "/lbuptest"
      receive = "OK"
      virtualhost="site1"
    # repeated again for 10.1.2.101 w/ appropriate addresses

    You can restart heartbeat now. Heartbeat manages ldirectord and ipvsadm internally. I noticed there are init.d scripts for those services but you should NOT turn them on.

  8. Check You can type ipvsadm to see output of the virtual server:
    # ipvsadm
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
    -> RemoteAddress:Port Forward Weight ActiveConn InActConn
    TCP 10.1.2.100:http lc
    -> 10.1.0.26:http Route 10 0 2
    -> 10.1.1.15:http Route 10 0 1
  9. Configure your real servers with ip aliases
    Since I used ldirectord, which is the DirectRouting method. You need to tell the networks on the real server to respond to the virtual IP address. This took me a while to figure out because the guides I’ve linked to above don’t mention this directly.

    On the real servers, I did this:
    # /etc/conf.d/net
    config_dummy0=('10.1.2.100 netmask 255.255.255.255 broadcast 10.1.2.100')

    I actually added both forwarded ip addresses.
    And make the dummy interface start up by default
    cd /etc/init.d
    ln -s net.lo net.dummy0
    rc-update add net.dummy0 default
    /etc/init.d/dummy0 start

    Update: In addition, you need to tell the real servers not to respond to arp requests for ip addresses that are not on the eth that is actually on the network:

    I added this to my /etc/sysctl.conf
    net.ipv4.conf.all.arp_ignore=1
    net.ipv4.conf.eth0.arp_ignore=1
    net.ipv4.conf.all.arp_announce=2
    net.ipv4.conf.eth0.arp_announce=2

    Then refresh the sysctl settings:
    sysctl -p
    I found this resource also helpful in making sure I had all the settings included that I need.

  10. That’s it After setting up my firewall to direct my public ip addresses to the new virtual ip, I have two load balanced, redundant web servers with HA servers running on Linux Virtual Servers
This entry was posted in System Administration and tagged , , , , , , , . Bookmark the permalink.

2 Responses to Using Linux-HA for High Availability with Gentoo and Linux-VServer

  1. Pingback: One way to unemerge lots of unneeded packages on Gentoo Linux | All My Brain

  2. Pingback: Linux-Vserver vs Xen | All My Brain

Comments are closed.