The Python install system needs an overhaul

Perhaps this is more of a rant than a useful blog post. I do plan on posting something useful though, so bear with me. First, a little background.

I've been working on installing a website based on Django for the last while. As part of the process, I wanted to bring in all the dependencies of my project with a file so I can easily move the site around to different development machines or server environments as needed. I wanted to make sure I didn't install different versions on a different machine, so I've provided a packages directory and install the file with the dependencies locked at specific versions and the packages directory noted. Easy install then just uses the package I place in the packages directory and every place I install this website, I get the same setup.

My snippets

  2. setup (
  3. # ...
  4. install_requires=[
  5. 'Django=1.1',
  6. 'MySQL-Python==1.2.3c1',
  7. # etc
  8. ],
  9. dependency_links = [ 'packages' ],
  10. # ...
  11. )

So far so good. I should point out that some have suggested pip as a replacement to easy_install. For my rant portion of this post, both pip and easy_install don't offer any help. If I'm wrong about pip, somebody please point out the proper way to do this..

My problem begins when I find a package with no file. The package I'm attempting to use is the forum module from Sphene Community Tools. As I download and go through their documentation, I find that they pretty much expect you to just add the sphene package to your python path. That doesn't fit with my installable website model though so I want to add a simple file and install it with the rest of my packages.


  1. Package Data

    Turbogears contributers worked on a find_package_data function that can recursively add your package data. The fact that this isn't included yet in the distutils api is quite annoying. I copied the file from another project [1]. I also added '.svn' to the ignore directory list since I was building from the svn view.

  2. Extra Data

    Just as annoying as the package data problem is the static data. I originally thought I could just include the static directory in data files like this:

    2. data_files=[ ('static', findall('static') ]

    That just puts every file recursively found in the static directory all in one static directory for the install. Instead, I modified the result returned by the find_package_data function to use the data_files format. Here is my completed file:

    2. import os
    3. from setuptools import setup, find_packages
    4. from finddata import find_package_data
    6. packages=find_packages('sphenecoll')
    7. package_data=find_package_data('sphenecoll')
    8. static = find_package_data('static','sphene')
    9. # not in correct format
    10. static_dict={} # dir -> files
    11. for path in static['sphene']:
    12. dir, file = os.path.split(path)
    13. dir = os.path.join('static', dir )
    14. files = static_dict.setdefault( dir, [] )
    15. files.append(os.path.join('static',path))
    17. setup(
    18. name='sphene',
    19. version='0.6dev',
    20. packages=packages,
    21. package_data=package_data,
    22. package_dir={'sphene':'sphenecoll/sphene'},
    23. scripts=['dist/scripts/', 'dist/scripts/' ],
    24. data_files=static_dict.items()
    25. )

    If someone else is interested, you can put and this in the sphene/communitytools directory and build your own egg. Not really the point of this post, but perhaps someone will find it useful. Actually, the Sphene tools should be installable I think. The expectation to just put the directory in your PYTHONPATH isn't very flexible.

  3. Install from source

    Both easy_install and pip didn't actually copy any of the package data or data files into the installed site-packages. I had to use setuptools bdist_egg and then installing the egg caused the files to be copied correctly.

  4. No uninstall

    This one isn't really an issue with this install in particular, but it is always annoying.

Maybe one of these days some Python developer will come up with a good solution to the install mess. I'm too busy working on the projects I've got to get done aside form the occasional sidetrack that installing Python modules causes.

1: I copied paste.util.finddata

3 Responses to “The Python install system needs an overhaul”

  1. matt harrison

    Sounds like a bug with sphene that it doesn’t have a file.

    Agreed, that data files in are a pain. If you care about changing that (or your other issues, unistall is a common one) you should at least mention it on distutils-sig since they are in the process of revamping everything….

  2. herbert poul

    well, i haven’t really had the request yet to add a to SCT – maybe because i don’t even have django installed, but am always using the latest trunk symlinked 🙂

    anyways… do you think your is stable enough so i can add it to the next SCT release? 🙂


  3. Dennis

    It works quite well for me. I haven’t done extensive testing though. I’m not sure if all of the required files and packages are included and copied correctly. What I needed for my own install is included properly. It would be quite easy to verify that everything else is included appropriately too though but I’m not familiar enough with all the features to authoritatively say.