The Python install system needs an overhaul

Perhaps this is more of a rant than a useful blog post. I do plan on posting something useful though, so bear with me. First, a little background.

I’ve been working on installing a website based on Django for the last while. As part of the process, I wanted to bring in all the dependencies of my project with a setup.py file so I can easily move the site around to different development machines or server environments as needed. I wanted to make sure I didn’t install different versions on a different machine, so I’ve provided a packages directory and install the file with the dependencies locked at specific versions and the packages directory noted. Easy install then just uses the package I place in the packages directory and every place I install this website, I get the same setup.

My setup.py snippets

setup (
 # ...
 install_requires=[
  'Django=1.1',
  'MySQL-Python==1.2.3c1',
  # etc 
 ],
 dependency_links = [ 'packages' ],
 # ...
)

So far so good. I should point out that some have suggested pip as a replacement to easy_install. For my rant portion of this post, both pip and easy_install don’t offer any help. If I’m wrong about pip, somebody please point out the proper way to do this..

My problem begins when I find a package with no setup.py file. The package I’m attempting to use is the forum module from Sphene Community Tools. As I download and go through their documentation, I find that they pretty much expect you to just add the sphene package to your python path. That doesn’t fit with my installable website model though so I want to add a simple setup.py file and install it with the rest of my packages.

Issues:

1. Package Data

   Turbogears contributers worked on a find_package_data function that can recursively add your package data. The fact that this isn’t included yet in the distutils api is quite annoying. I copied the file from another project [1]. I also added ‘.svn’ to the ignore directory list since I was building from the svn view.

2. Extra Data

   Just as annoying as the package data problem is the static data. I originally thought I could just include the static directory in data files like this:

    data_files=[ ('static', findall('static') ]
   

   That just puts every file recursively found in the static directory all in one static directory for the install. Instead, I modified the result returned by the find_package_data function to use the data_files format. Here is my completed setup.py file:

   import os
   from setuptools import setup, find_packages
   from finddata import find_package_data
   
   packages=find_packages('sphenecoll')
   package_data=find_package_data('sphenecoll')
   static = find_package_data('static','sphene') 
   # not in correct format
   static_dict={} # dir -> files
   for path in static['sphene']:
      dir, file = os.path.split(path)
      dir = os.path.join('static', dir )
      files = static_dict.setdefault( dir, [] )
      files.append(os.path.join('static',path))
   
   setup(
    name='sphene',
    version='0.6dev',
    packages=packages,
    package_data=package_data,
    package_dir={'sphene':'sphenecoll/sphene'},
    scripts=['dist/scripts/make-messages.py', 'dist/scripts/compile-all-sph-messages.py' ],
    data_files=static_dict.items()
   )
   

   If someone else is interested, you can put finddata.py and this setup.py in the sphene/communitytools directory and build your own egg. Not really the point of this post, but perhaps someone will find it useful. Actually, the Sphene tools should be installable I think. The expectation to just put the directory in your PYTHONPATH isn’t very flexible.

3. Install from source

   Both easy_install and pip didn’t actually copy any of the package data or data files into the installed site-packages. I had to use setuptools bdist_egg and then installing the egg caused the files to be copied correctly.

4. No uninstall

   This one isn’t really an issue with this install in particular, but it is always annoying.

Maybe one of these days some Python developer will come up with a good solution to the install mess. I’m too busy working on the projects I’ve got to get done aside form the occasional sidetrack that installing Python modules causes.

1: I copied paste.util.finddata http://trac.pythonpaste.org/pythonpaste/browser/Paste/trunk/paste/util/finddata.py