Linux-Vserver vs Xen

A while back, I found myself running out of hardware and wanting to host more sites than I currently was. In addition, I wanted to create a little bit more redundancy for some of the services I host.

At the time, I was hosting a number of services with Xen. One physical server hosted 3 or 4 virtual servers. After a certain amount of reading over different solutions, I decided to convert all my production virtual servers to Linux-vserver. I’m not advocating either solution here. I’m simply going to point out my reasons for changing and hopefully help my readers understand the issue more.

Continue reading “Linux-Vserver vs Xen”

Keeping a process running

Have you ever had a process that dies on occasion? For me, I hate that situation and prefer to fix the software as opposed to have a monitor that restarts the process when it dies. I’ve run into a case lately however, that has defied me for a solution to my dying process. I think it may be a hardware related issue but haven’t tracked down the cause yet. Anyhow, I read an email on the Provo Linux User Group in which the poster referred to PS-Watcher. I thought I’d give it a try for kicks.

After installing the program and reading through the documentation, I found that PS-Watcher is really quite nice. In addition to monitoring the results of the ps command, you can add custom actions that occur at the beginning or ending of the monitor cycle ($PROLOG and $EPILOG). You can also customize actions to be taken based on the number of processes, memory size, and a few other useful metrics.

For most situations where you want to monitor a process and take action, I think PS-Watcher will probably do the job nicely. After all this however, I decided what I really wanted was a little script that did a custom restart of my particular web server when the test URL wasn’t functioning properly. I decided to simply run it on a scheduled interval with cron. I’ve placed the script below for all to glean information from or make fun of as appropriate. Feel free to provide some additional tips as I don’t claim to be a “Bash Jedi Master”. The following script sends a request to the web server and parses the response for a string that lets us know the server is working properly.

#!/bin/bash

user="<the user my process is running under>"
port="<the port>"
okresp="^OK$" # I configured a test URL that returns OK if the server is up and running right.

# make a simple HTTP request to send
req="GET /lbuptest HTPP/1.0

"
# send it using netcat
resp=$(echo "$req" | nc localhost $port)
# test for the ok string
ok=0
echo "$resp" | grep $okresp 2>&1 >> /dev/null && ok="1"

# you could really place whatever actions you want here.
if [[ $ok != "1" ]]; then
/etc/init.d/<my process init script> restart
fi

The process I’m having trouble with is a TurboGears web application. I don’t think this is a Python problem however. Like I mentioned before, it only happens on this one server so I think I’ve got a hardware problem. Either way, if you found this page searching for TurboGears information, you might as well be interested in my TurboGears Init Scripts.

How to virtual host load balanced websites with ldirectord and Apache

I posted a while back on getting Heartbeat set up to add reliability to websites. After a few weeks of experience with the system, I thought I’d add a few additional tips on making the setup more reliable. There are already a few good guides on getting heartbeat set up. You could also read my original post on the subject if you don’t already have heartbeat load balancing your site. This post however, deals with the case when you are servicing more than one site per physical server.

We host three different websites on three different physical servers. Each physical server hosts two websites with Apache. Each website is hosted on two different physical servers. The sites are load balanced with ldirectord which resides on two different servers that manage the public IP addresses to our services with Heartbeat. If load increases on any of our services, we could always add additional physical servers relatively easily.

Continue reading “How to virtual host load balanced websites with ldirectord and Apache”

Using piped svndumpfilter commands to separate an svn repository

According to the documentation for svndumpfilter, you can include one subcommand when filtering a dumped repository. Suppose you have a repository that has a path “/some/path” that you’d like to separate out into its own new repository. From the documentation, you simply pipe the original dumped repository through the svndumpfilter command.

Example:
cat repos-dumpfile | svndumpfilter include some/path > new-dumpfile

The caveat is that if there are paths copied from other paths in your repository that the include argument does not cover, you’ll get an error. I got around this command by piping the output from one svndumpfilter command to another, each with exclude commands instead of include commands. The result leaves only the paths I want, but included the alternate copy branch that I had used part way through the coding process.

Example:
cat repos-dumfile | svndumpfilter exclude unwanted/one | svndumpfilter exclude unwanted/two | svndumpfilter exclude unwanted/three --drop-empty-revs --renumber-revs > new-dumpfile

Notice on the last svndumpfilter command, I added a couple arguments to renumber the repository revisions and drop the empty revisions. While these are of course optional, in my opinion, they make the new repository cleaner.

The subversion book states you can edit the Node-Path values in the dumped file to have the new repository imported at different paths. I chose to simply issue an “svn mv” command once I imported the repository.

Using tcpdump on a linux-vserver guest

To debug a problem I’m working on, I need to be able to see network traffic on an interface inside a linux-vserver guest. To do this, you have to enable the CAP_NET_RAW capability for that guest.
> echo "NET_RAW" >> /etc/vservers/myserver/bcapabilities

Then just restart the vserver.
I noticed you don’t have to enable NET_ADMIN, or unhide the interfaces. I’m not sure if there is much of a security risk on having NET_RAW enabled or not. You can always disable and restart after you’re done with tcpdump.

Gentoo Init Scripts for Cherrypy

A few of the web applications I am hosting/developing are written with TurboGears which uses Cherrypy as its applications server. I have a couple things that need fixed for hosting CherryPy web applications on my Gentoo systems. First, there are no init scripts for CherryPy. Second, I’m running mulitple web applications per machine. I run them as different users with virtual python environments to keep web applications stepping on each others toes.

You can customize the logging for a TurboGears application to have not logging to standard out, but you still have to deal with the python CherryPy process not running as a daemon. I chose rather to run the application in a screen session.

First, a quick script to start my web application in its virtual python environment in my web applications directory.
#!/bin/bash
PATH=/home/<myuser>/bin:$PATH
cd `dirname $0`
./start-<mywebapp>.py prod.cfg

And next, a real simply startup script that starts and stops the application in a screen session.

#!/sbin/runscript
depend() {
need net
}
start() {
su -l $APPUSER -c "screen -S $SESS_NAME -d -m /$APP_DIR/$SCRIPT_NAME"
}
stop() {
su -l $APPUSER -c "screen -S $SESS_NAME -X kill"
}

The variables can go in /etc/conf.d/<app_name> where the <app_name> is the same name as the init.d script. They will automatically be sourced and passed to the init script. For more information on Gentoo init scripts, check here.