All My Brain Where stuff from my brain lands

October 21, 2009

The Python install system needs an overhaul

Filed under: Programming — Tags: , , , — Dennis @ 10:58 am

Perhaps this is more of a rant than a useful blog post. I do plan on posting something useful though, so bear with me. First, a little background.

I’ve been working on installing a website based on Django for the last while. As part of the process, I wanted to bring in all the dependencies of my project with a file so I can easily move the site around to different development machines or server environments as needed. I wanted to make sure I didn’t install different versions on a different machine, so I’ve provided a packages directory and install the file with the dependencies locked at specific versions and the packages directory noted. Easy install then just uses the package I place in the packages directory and every place I install this website, I get the same setup.

My snippets

setup (
 # ...
  # etc 
 dependency_links = [ 'packages' ],
 # ...

So far so good. I should point out that some have suggested pip as a replacement to easy_install. For my rant portion of this post, both pip and easy_install don’t offer any help. If I’m wrong about pip, somebody please point out the proper way to do this..

My problem begins when I find a package with no file. The package I’m attempting to use is the forum module from Sphene Community Tools. As I download and go through their documentation, I find that they pretty much expect you to just add the sphene package to your python path. That doesn’t fit with my installable website model though so I want to add a simple file and install it with the rest of my packages.


  1. Package Data

    Turbogears contributers worked on a find_package_data function that can recursively add your package data. The fact that this isn’t included yet in the distutils api is quite annoying. I copied the file from another project [1]. I also added ‘.svn’ to the ignore directory list since I was building from the svn view.

  2. Extra Data

    Just as annoying as the package data problem is the static data. I originally thought I could just include the static directory in data files like this:

     data_files=[ ('static', findall('static') ]

    That just puts every file recursively found in the static directory all in one static directory for the install. Instead, I modified the result returned by the find_package_data function to use the data_files format. Here is my completed file:

    import os
    from setuptools import setup, find_packages
    from finddata import find_package_data
    static = find_package_data('static','sphene') 
    # not in correct format
    static_dict={} # dir -> files
    for path in static['sphene']:
       dir, file = os.path.split(path)
       dir = os.path.join('static', dir )
       files = static_dict.setdefault( dir, [] )
     scripts=['dist/scripts/', 'dist/scripts/' ],

    If someone else is interested, you can put and this in the sphene/communitytools directory and build your own egg. Not really the point of this post, but perhaps someone will find it useful. Actually, the Sphene tools should be installable I think. The expectation to just put the directory in your PYTHONPATH isn’t very flexible.

  3. Install from source

    Both easy_install and pip didn’t actually copy any of the package data or data files into the installed site-packages. I had to use setuptools bdist_egg and then installing the egg caused the files to be copied correctly.

  4. No uninstall

    This one isn’t really an issue with this install in particular, but it is always annoying.

Maybe one of these days some Python developer will come up with a good solution to the install mess. I’m too busy working on the projects I’ve got to get done aside form the occasional sidetrack that installing Python modules causes.

1: I copied paste.util.finddata

April 25, 2008

Using Multiple Python Environments With Gentoo

Filed under: Programming — Tags: , , , , — Dennis @ 12:09 pm

It’s been some time since Python 2.5 became stable and released. Version 2.5 has plenty of new features that have helped me in deciding that it was time to go ahead and start using it for primary development of all my new projects. One of the reasons I was still using version 2.4 is that Gentoo hadn’t upgraded 2.5 to the stable package system.

I decided to go ahead and unmask version 2.5 anyway. Installing Python 2.5 isn’t very complicated and I’ll leave the details out. I’ll just mention that after unmasking Python with the ~x86 keyword and installing the package, you’ll need to run the python-updater. Python-updater had it’s own problems with not being able to find some packages it thought needed to be re-emerged but I found pretty much all of those were unneeded old dependencies and I simply un-emerged them.

When you update Python, you can still get to your old version of Python by tacking on the version number to the python command, e.g., /usr/bin/python2.4. Since the Python updater uses emerge to install your python dependencies in the site-packages of your Python installation and emerge unmerges the old versions, your old Python probably doesn’t have all the site-packages any longer. This is only an issue if you find you need the old Python.

For me, I have a couple applications that didn’t quite want to work with Python 2.5 for some reason. I decided to use VirtualEnv to work on those applications.

The Steps

  1. To install virtualenv, you need setuptools. That package was one of the packages transfered to the 2.5 site-packages install and was no longer available with python2.4.

    To get around the issue, use to install a 2.4 version of setuptools instead of using emerge.
    cd <working dir>
    /usr/bin/python2.4 setuptools

  2. I simply used the virtualenv command that came with python2.5 but changed the interpreter to by python2.4 instead of python.

    cp /usr/bin/virtualenv .
    # edit virtualenv to have the correct interpreter line
    vim < or whatever editor > virtualenv
    ------ snip local virtualenv ------
    ------ snip ----------------------
    ./virtualenv --no-site-packages <virtual env install dir>
    cd <virtual env install dir>/bin/
    ln -s python2.4 python
    cd <working dir>

  3. Use the old python in it’s virtual environment:

    source <virtual env install dir>/bin/activate
    Python 2.4.4 (#1, Mar 5 2008, 10:47:15)
    [GCC 4.1.2 (Gentoo 4.1.2)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.

Anyway, that’s it. You can have a virtual environment that uses any version of Python you like on your system. I imagine this procedure would be somewhat similar on pretty much any Linux Distribution.

January 16, 2008

Virtual Hosting TurboGears Applications on Mac OS X Leopard

For a couple years now, I’ve been learning and applying various tricks for developing and hosting multiple Python web sites on my development machines. During that time, I made a migration to Mac OS X. Most setup files for python applications and libraries work out of the box on the Linux distributions I’ve tried. For OS X, you can find a lot of prebuilt packages for the necessary dependencies but those packages want to be installed in the OS X system library location for Python.

If you want to host or develop more than one site with Python, you’ll need to use virtual environments. This applies equally to any OS, not just Mac OS X. There seem to be a few more tricks to getting things up and running on OS X however. Here is the process I went through to get TurboGears applications hosted in virtual environments on OS X.


October 8, 2007

Gentoo Init Scripts for Cherrypy

A few of the web applications I am hosting/developing are written with TurboGears which uses Cherrypy as its applications server. I have a couple things that need fixed for hosting CherryPy web applications on my Gentoo systems. First, there are no init scripts for CherryPy. Second, I’m running mulitple web applications per machine. I run them as different users with virtual python environments to keep web applications stepping on each others toes.

You can customize the logging for a TurboGears application to have not logging to standard out, but you still have to deal with the python CherryPy process not running as a daemon. I chose rather to run the application in a screen session.

First, a quick script to start my web application in its virtual python environment in my web applications directory.
cd `dirname $0`
./start-<mywebapp>.py prod.cfg

And next, a real simply startup script that starts and stops the application in a screen session.

depend() {
need net
start() {
su -l $APPUSER -c "screen -S $SESS_NAME -d -m /$APP_DIR/$SCRIPT_NAME"
stop() {
su -l $APPUSER -c "screen -S $SESS_NAME -X kill"

The variables can go in /etc/conf.d/<app_name> where the <app_name> is the same name as the init.d script. They will automatically be sourced and passed to the init script. For more information on Gentoo init scripts, check here.

Powered by WordPress

%d bloggers like this: