The Python install system needs an overhaul

Perhaps this is more of a rant than a useful blog post. I do plan on posting something useful though, so bear with me. First, a little background.

I’ve been working on installing a website based on Django for the last while. As part of the process, I wanted to bring in all the dependencies of my project with a setup.py file so I can easily move the site around to different development machines or server environments as needed. I wanted to make sure I didn’t install different versions on a different machine, so I’ve provided a packages directory and install the file with the dependencies locked at specific versions and the packages directory noted. Easy install then just uses the package I place in the packages directory and every place I install this website, I get the same setup.

My setup.py snippets

setup (
 # ...
 install_requires=[
  'Django=1.1',
  'MySQL-Python==1.2.3c1',
  # etc 
 ],
 dependency_links = [ 'packages' ],
 # ...
)

So far so good. I should point out that some have suggested pip as a replacement to easy_install. For my rant portion of this post, both pip and easy_install don’t offer any help. If I’m wrong about pip, somebody please point out the proper way to do this..

My problem begins when I find a package with no setup.py file. The package I’m attempting to use is the forum module from Sphene Community Tools. As I download and go through their documentation, I find that they pretty much expect you to just add the sphene package to your python path. That doesn’t fit with my installable website model though so I want to add a simple setup.py file and install it with the rest of my packages.

Issues:

  1. Package Data

    Turbogears contributers worked on a find_package_data function that can recursively add your package data. The fact that this isn’t included yet in the distutils api is quite annoying. I copied the file from another project [1]. I also added ‘.svn’ to the ignore directory list since I was building from the svn view.

  2. Extra Data

    Just as annoying as the package data problem is the static data. I originally thought I could just include the static directory in data files like this:

     data_files=[ ('static', findall('static') ]
    

    That just puts every file recursively found in the static directory all in one static directory for the install. Instead, I modified the result returned by the find_package_data function to use the data_files format. Here is my completed setup.py file:

    import os
    from setuptools import setup, find_packages
    from finddata import find_package_data
    
    packages=find_packages('sphenecoll')
    package_data=find_package_data('sphenecoll')
    static = find_package_data('static','sphene') 
    # not in correct format
    static_dict={} # dir -> files
    for path in static['sphene']:
       dir, file = os.path.split(path)
       dir = os.path.join('static', dir )
       files = static_dict.setdefault( dir, [] )
       files.append(os.path.join('static',path))
    
    setup(
     name='sphene',
     version='0.6dev',
     packages=packages,
     package_data=package_data,
     package_dir={'sphene':'sphenecoll/sphene'},
     scripts=['dist/scripts/make-messages.py', 'dist/scripts/compile-all-sph-messages.py' ],
     data_files=static_dict.items()
    )
    

    If someone else is interested, you can put finddata.py and this setup.py in the sphene/communitytools directory and build your own egg. Not really the point of this post, but perhaps someone will find it useful. Actually, the Sphene tools should be installable I think. The expectation to just put the directory in your PYTHONPATH isn’t very flexible.

  3. Install from source

    Both easy_install and pip didn’t actually copy any of the package data or data files into the installed site-packages. I had to use setuptools bdist_egg and then installing the egg caused the files to be copied correctly.

  4. No uninstall

    This one isn’t really an issue with this install in particular, but it is always annoying.

Maybe one of these days some Python developer will come up with a good solution to the install mess. I’m too busy working on the projects I’ve got to get done aside form the occasional sidetrack that installing Python modules causes.

1: I copied paste.util.finddata http://trac.pythonpaste.org/pythonpaste/browser/Paste/trunk/paste/util/finddata.py

Posted in Programming | Tagged , , , | 3 Comments

Do we really need super-fast broadband?

Wouldn’t it be nice to download your favorite music track in just 11 seconds? Or a 60 minute TV show in just 1 minute 2 seconds? How about a High definition movie in just over 14 minutes? Is this ever possible? With 50Mb broadband, it is. Virgin Media currently has a 50Mb service that lets you download with speeds which we thought were possible only in our imagination. What it is about 50Mb Broadband anyway? Do we actually need it?

It is clear that the introduction of 50Mb broadband is the result of the ever increasing demand for online entertainment. Apart from the unbelievable download speeds, such broadband services take online gaming to a whole new level. When you can download a game is just under 2 minutes, what more can you ask for? For the hardcore gamer, 50Mb broadband will be a dream come true indeed! Why wouldn’t it be when he himself can host large games and play even the most complicated games without any visible glitches?

And as for online movies and streaming, an up to 50Mb service is more than just utilitarian, for you can stream HD content with almost zero buffering. Not just that, for large households where almost everyone wants a go at the Internet at the same time, an up to 50Mb broadband deal lets every user glide across the Internet through the same connection. When speeds of such order work wonders, one can imagine how a 160Mb broadband connection will perform. This is what 2.5 million Japanese customers have been enjoying for quite some time. Though Virgin media’s 50Mb deal appears to be a breakthrough, at least for the UK customers, it is gearing up for speeds between 100Mb and 150Mb by 2010.

Now do we actually need such super-fast broadband services? Why not, when we can get things done in the wink of an eye, when our homes can be equipped with business broadband speeds, when even the most elaborate virtual world can fly onto our home screen within seconds.

Posted in Web | Tagged , , | Comments Off on Do we really need super-fast broadband?

How to remove or edit a commit in your git repository

So you just committed 15 things to your git repository and now you want to push your changes. Oops, commit #2 added your password file. Or perhaps you misspelled words in the commit message. Now, being a git expert, you think to yourself, I’ll just create a temporary branch, cherry-pick the commits that are correct, re-commit correctly the incorrect commit, and then merge/push my temporary branch instead of this flawed branch I have. That all works fine, but luckily there is a much easier way to do it.

> git rebase -i 

When you use ‘-i’ with rebase, you get an interactive edit screen that gives you an option for each commit _after_ the sha1 you enter. For each commit you can either delete the entry entirely or replace the word “pick” with “edit”. Deleting an entry causes that commit to lost. Editing a commit gives you the option to unstage some of the files, edit the commit message, or really, to do whatever you want to change it how you intended. If you edit, or the rebase has a conflict, after setting things how you like, just use rebase –continue to go on your way.

Posted in Programming | Tagged , , | Comments Off on How to remove or edit a commit in your git repository

Example Image Upload with YUI Rich Text Editor 2.7.0

It’s somewhat slow coming, but I’ve checked compatibility with the image uploader and YUI version 2.7.0. If you haven’t read the original YUI Image Uploader page, start there. After that, you can use this page for an example getting the script to work with the latest YUI.

Continue reading

Posted in Programming | Tagged , , , , , | 43 Comments

Launching wxPython apps with an iPython shell

Suppose you want to run your fancy wxPython application but have a shell in the background to peek and poke at certains settings, help debug, and possibly even use an API that your program provides to automate tasks. iPython has built in wx support (as well as support for other GUIs and frontends). So anyway, I’ve been playing around with this lately and have noticed an inconsistency that needs addressed.

Launching other GUIs with iPython does not block. You instantly get a shell and your application loads. The “-wthread” option for wx applications executes any passed in script before dropping to the shell though.

Problem:

> ipython -wthread myapp.py
> # application runs here
> # no shell yet
> # you exit the application
"Welcome to iPython"
>>> 

If you happened to do things like this, you’d get the expected response:

> ipython -wthread
"Welcome to iPython"
>>> import myapp
# application loads
# you instantly have a prompt
>>>

Solution:

Tell iPython to launch your app after it starts:

> ipython -wthread -c "import myapp" -i
# app loads
"Welcome to iPython"
# you've got a shell
>>>

Anyway, for other threading backends in iPython, the shell doesn’t block while the application loads. Perhaps there is a way to get the wx backend to behave the same. In the mean time, I’m using this workaround without any serious issue. Perhaps I’ll find a better solution or submit a patch to iPython one of these days.

Posted in Programming | Tagged , , , , | Comments Off on Launching wxPython apps with an iPython shell

Fixing the code highlighter for WordPress

Many of my posts have some sort of code, shell script, or other syntax highlighted portion. I’ve loved the code highligher plugin for wordpress. The version I’ve been using (1.8) doesn’t seem to have had any updates for quite some time. The only reason that bothers me in the slightest is that there is a syntax highlighting bug in bash commenting. Since the highlighting for the plugin is done with Geshi, You can just download the latest Geshi and overwrite the plugin geshi folder.

> cd /your/wp/plugins/code-highligher
> wget http://superb-west.dl.sourceforge.net/sourceforge/geshi/GeSHi-1.0.8.4.tar.gz
> # untar geshi files to the current directory
> # note that the archive file has a top level geshi directory that the plugin file doesn't have.
> # now, if I did this right, my bash comments can contain 'strings' and reserved words df, ls, etc
Posted in Web | Tagged , , , , | Comments Off on Fixing the code highlighter for WordPress