Fetchez le Python

Pycon France will be starting Saturday in Paris. Make sure you’re there if you are in Paris. It’s Free.

Here are a list of talks I don’t want to miss (I want to see them all more of course):

• GUnicorn, the wsgi server
• MongoDB and MongoKit
• Python in Debian

The full program: http://www.pycon.fr/#talksschedule

Read more | Comment

Man this has been an amazing summer. We had 5 students working on Distutils 2 and the work done was really great. Most important, I think we managed to find new contributors for the future.

Thanks to Josip, Alexis, Konrad, Eric and Zubin ! Also, thanks to their respective mentors: Michael,  Fred, Titus and Lennart.

Let’s have a very high-level summary of the tasks that were done. As a reminder, Distutils2 aims to provide two things: a toolbox for third party projects that want to provide packaging features, and a full-featured package installer/manager for Python.

What was done

Distutils2 has now an index package, containing tools to work with PyPI (or any PyPI-like server). This package contains classes to work with the Simple Index protocol as well as the XML-RPC APIs PyPI has. We think that the latter should be replaced by static REST calls, but that’s another topic. If you want to build a tool that uses PyPI, that’s the package you want to use. See this doc (temporary location) for more info.

Another thing we added is an installer script. This script is a very light script that uses the index package I’ve mentioned before, but also a dependency graph builder we now have. The graph builder allows you to analyze relations between installed projects, but also any project you want to install. See this doc. This installation script will be completed by an uninstallation feature very soon hopefully. You might wonder why we did this since there are existing scripts like Pip or easy_install. The reasons are quite simple:

1)  we wanted to exercise all the modules Distutils2 provides to work with PyPI, metadata and installed projects in a high level script. The long-term goal is to have projects like Pip or Distribute use Distutils2-the-toolbox since they’ll use the same standards in the future.

2) Distutils2-the-package-manager wants to provide a basic installer/uninstaller.

Oh by the way, the depgraph tool is PEP 376 compatible, as we have now a new version of pkgutil that supports PEP 376. See this doc. The next task will be to put this new version in Python 3.2, and that should happen fairly soon.

Speaking of which, we are not sure yet if we are going to include an plugin system like the entry points in Distutils2. Discussions on this has been controversial on Python-dev. In the meantime you can enjoy the extensions project, which is a quick hack to have entry points without depending on Setuptools or Distribute –and it’s now Distutils2 and PEP 376 compatible–.

Distutils2 is almost Python 3 compatible. The student in charge is working hard to finish this task. See his repo. As usual, the last problems to be solved are about unicode, strings, bytes, you named it. But I expect to include this work in one of the alpha version of Distutils2 1.0.

Last but not least, we worked on the commands front of Distutils2:

• test was added — a command to run the project tests, inspired from Setuptools
• upload_docs was added — a command to upload docs at packages.python.org. Originally created by Jannis Leidel and present in Distribute
• check was added and improved — a command that I created to check a project before releasing/uploading it. It checks its metadata, its description reST compliancy etc.
• A command post/pre hook system — similar to what RPM has so you can point code to be run before and after a Distutils command is run. Very powerful and useful — but to be used with caution. You wouldn’t want to add to the install process a code dependency you are not sure to have. But in the meantime, being able to run a code every time a project is installed on your system is sweet

I am forgetting a million things, but I guess its not important, since all our students are blogging about it

Thanks to Google, the GSOC is a great program. Distutils2 just got a really serious boost, we are very close to a good, useful release.

As a conclusion, to make things clear for the few people that still see all these efforts as reinventing the wheel since packaging has been kind-of-solved already elsewhere: Every extensible system, whether its an OS like Debian or Fedora, or a language like Python, will always provide its own custom packaging system, that is built on the top of the community experience and needs. The only thing that really matters is to have this work based on standards so inter-operability is manageable. That’s the goal of some of the PEP we have written, like PEP 386 and PEP 345 and that will make OS packagers life easier in the future.  An universal packaging system is an utopia.

Read more | Comment

I have been working for a bit more than a month now on the next generation of the Firefox Sync server in Python and while the project is still in its early stages and subject to a lot of changes, I think it’s a good idea to share now about what we are building here at Mozilla. Maybe that’ll attract contributors !

About Sync

Firefox Sync (formerly Weave) let you synchronize your Firefox bookmarks, history, passwords, opened tabs etc. so you can have them on any computer, or even use them from your iPhone by using Firefox Home.

Clients that are syncing work with our servers at Mozilla by using the Sync and the User APIs defined in these documents:

• https://wiki.mozilla.org/Labs/Weave/User/1.0/API
• https://wiki.mozilla.org/Labs/Weave/Sync/1.0/API

The User APIs manage the users accounts and tell the client which server holds the data of a given user.  In other words, each user is tightly coupled to a single server when reading or writing data. This natural sharding is great for scaling Sync, and is possible because users don’t share data (yet… )

Another important point is that the data are encrypted on client side before they are sent over. That’s because one of the key concept of Sync is that your data should not be known by our servers, to protect your privacy.  Well, we could probably still know how many bookmarks you have by counting the number of entries in the DB, or how often you use your browser. But as soon as you use a service like that you have to give away these kind of information, most of the time just because they are useful to make the service faster or understand any potential problem. Read our privacy policy here.

And the good news is that you can set up your own Sync server and even implement it yourself if you want.

So, a Sync server a pretty passive storage server, that is quite easy to scale while keeping data consistency across clients.

About the code

The current implementation uses Apache, PHP, LDAP, MySQL and Memcached. For various reasons I won’t detail in this post –that might be another post– , it has been decided to switch the Sync server to Python

Python libraries

The Sync server is  composed of web services and a few screens used for the password reset process, so using a web framework would have been overkill. Although, writing a wsgi-enabled server made a lot of sense since it allows people to run our implementation on their laptop, or on any wsgi-compatible web server they wish to use.

So, I’ve picked :

• Routes, to dispatch requests to a few classes (controllers)
• WebOb to process incoming requests and build responses
• Paste. PasteScript, PasteDeploy, to group the configuration in an ini file and make it easy to run the application with a built-in server.

There are alternative routing systems, but Routes really fits my brain and make the dispatching quite simple. I really like the fact that you can optionally use regular expressions to validate URLs.  

WebOb is quite a standard library and make our life simple to read requests and write responses. The code in our controllers stays KISS with WebOb when you have to read incoming data: they’re all available in simple mappings. The response is also built by WebOb and you can forget about all the wsgi protocol details. We mainly return JSON dumps that WebOb wraps into responses.

Last, Paste is very handy to run the server locally, to initialize data, and handle multiple configurations. I should also say that my colleague Ian Bicking is behind the Paste and WebOb libs, and involved in the Sync project.  So those were quite natural choices.

The authentication process is a custom function that reads a basic authentication header and checks it using an authentication plugin (more on plugins later in this post.)

For the storage, I’ve picked SQLAlchemy and python-ldap.  I don’t really use the ORM part of SQLAlchemy and write pretty raw SQL queries to avoid any extra overhead. The benefit of the ORM was null here anyways, since all storage I/O are contained in a storage class that outputs simple mappings. I have created the mappers though, as they are useful to initialize a DB on a first run.

But when the server runs, SQLAlchemy is mainly used for:

• its connection pooling abilities.
• the nice parameters binding
• the ability to switch to any DB system via configuration (as long as the SQL is compatible of course)

As for python-ldap (I didn’t implement the LDAP part yet), it’s the standard connector I have always used with various flavors of LDAP servers (OpenLDAP, ActiveDirectories, etc.). I don’t think there is any competitor for this anyways.

Caching

The caching is currently done using Memcached. For instance, when clients are often asking for specific collection items, they end up in memcached to lower the number of queries made to MySQL. For the Python implementation though, I’ve decided to use Redis instead.

In terms of speed, Redis and Memcached are quite similar. Redis though has interesting extras:

• The data is saved to the disk, so you don’t lose your cache. The speed stays almost the same as memcached since the disk syncs are done asynchronously from time to time. Since a Sync user is tightly coupled to a storage server, that’s an interesting feature to have. And, hey, you can move data from a Redis DB to another, so migrating the cache to another server is even possible.
• Redis provides built-in APIs to work with sets and lists, which authorizes more complex caching without extra code. This will allow us to do more caching in the future.

Storage

The storage itself will stay on MySQL but we will probably explore alternative storages systems in the future. One requirement of Sync is to be able to write data as fast as possible so all clients can have access to them as soon as possible.  Right now, Sync provides immediate consistency, since all writes are done synchronously on a single server.

Plugins

The PHP application was built with extensibility in mind: the way Mozilla stores the data and authenticates users (a mix of LDAP and MySQL) might not work if the code is used by someone else. That’s why the code was built using abstractions for the storage and the authentication part, and the Python version took back this good idea.

Basically, you can write a new authentication or storage class, and configure Sync to use it. See the documentation I am building on this: http://sync.ziade.org/doc/storage.html (temporary location)

Web server

The web server that runs the Python application will stay Apache (with mod_wsgi) since it has proven to work very well with the current implementation. I might bench other servers in the future though, like Gunicorn + nGninx or uWSGI + nGninx. We now have a nice Grinder script that realistically mimics Sync users, so..

Doc and Code

I’ve started a documentation, the temporary location is at http://sync.ziade.org/doc and you can grab the code we are building at http://hg.mozilla.org/users/telliott_mozilla.com/sync-server. You can already use the server with your Firefox / Firefox Home, but this is still at development stage, so use at your own risks.

I would love to get some feedback on that work !

Read more | Comment

I have announced it already, and the dates have changed a bit. But let’s say it again since the date is approaching: we will have a sprint at my house the second week-end of September (17/18), focusing on packaging and testing (and other topics if people want).

This sprint will be a special event like last year: we will enjoy wines directly from the producers place (Gevrey-Chambertin anyone ?) and do some restaurants in the area.

Now its a good time to tell me if you are coming, by adding your name here: http://www.coactivate.org/projects/afpy-computer-camp-2010/project-home (make sure you are registered)

Read more | Comment

This blog entry was inspired by the discussion I just had with Michael on IRC, as he just added plugins in unittest2.

Setuptools’ entry points feature is hated and loved by developers. If you are not familiar with them, you can read this post from Armin.

Hated because when you install a project that contains entry points (let’s call them plugins), they can be used in another application without letting you know. So basically if a plugin sucks, it can break another application just by being installed in your Python. And it’s not easy to have an overview of what plugins are installed and potentially active.The worst is that projects that provide entry points, usually provide many other things. But if you want to deactivate the plugin, you have to remove the whole project… Note that plugins are not loaded at Python startup. What happens is that any application can iterate over the metadata of installed projects, looking for plugins, and eventually loading them if wanted.

Loved, because from a developer point of view you can have a new feature added in a program with no extra configuration at all. Take Nose. Thanks to entry points, it’s dead easy to create a plugin for this test runner, and tell people to pip-install this new project. Zero config. Nice. Another great thing is that it’s global to Python. Any application can consume any entry point. Entry points are implicit plugins I guess.

Distutils has a plugin system as well: you can add new commands by adding in distutils.cfg the path to the Python package containing the command. That’s an explicit plugin system since the end-user has to configure it manually so Distutils uses it. Mercurial uses the same technique: activating a plugin is done in .hgrc. I would call these explicit plugins.

I think we can get the benefits of entry points without their caveats really simply. And provide a generic plugin system for all. Let’s summarize what we want:

• being able to list all installed plugins for every Python application
• being able to remove a plugin or deactivate it. Without being forced to uninstall the project that provided it
• have a plugin automatically installed and activated when the project that provides it is installed

Here’s how we can do. That’s a brain dump, please give me some feedback !

Global plugin registry

Let’s have a .python-plugins.cfg file in the user’s home (and one global to Python. The user cfg is merged with the global one at startup, and overrides the values — thanks Mongoose_Q for mentioning this on Twitter). It’s a simple ini-like file like .hgrc, where each section represents a python application and a group name. A group is just a family of plugins. For instance ‘commands’ can be a group for the ‘distutils’ application. In this section, each line is a plugin, represented by a pointer to the module or class, followed by a label as the value.

Here’s an example for a distutils ‘i18n’ command. It’s a MyClass class, located in the foo package, in the bar module:

  [distutils:commands]
  foo.bar:MyClass = i18n

The link to the code comes first because some plugins could have no name:

  [app:group]
  package.module:Class =

Accessing the registry

distutils can provide an API to read the file, iterate and load the plugins:

    >>> from distutils2 import plugins
    >>> plugins.get('distutils', 'command')
    <iterator>
    >>> plugins.get('distutils', 'command').next()
    <Plugin "i18n" at foo.bar:MyClass>
    >>> plugin = plugins.get('distutils', 'command').next()
    >>> plugin.load()    # gets the code and loads it
    <MyClass Instance>

Installing the plugins

Last, distutils could provide a mechanism to automatically register a plugin.

Projects could describe their plugins in their setup.cfg:

  [plugins]
  distutils.commands.i18n =  foo.bar:MyClass

Then distutils would automatically inject them at installation time in .python-plugins.cfg only if the end user agrees:

  $ python setup.py install
  distutils has detected a "i18n" plugin for distutils:commands. Do you want to activate it (Y/n) ?

Read more | Comment

Working on the various parts of the Mozilla project requires some patch fu.

Basically, everything happens in bugzilla.mozilla.org, where you upload a patch and ask for a review. Once you have started to work on several patches, maintaining several mercurial clones can be tedious. That’s where queues are helping a lot.

Mozilla has a nice document about queues. I’ve also found this Sympy tutorial quite useful.

Here are my random notes about queues so far:

• a queue is a directory of patch files (in .hg/patches). You can qpush or qpull them in your local repo. qpush will apply the current patch and add an entry in the commit log. qpop undoes it.  pushing and poping will move you up or down in the stack.
• I enabled ‘color’ in mercurial and use “hg qseries” to know where I am in the stack
• I reorder patches by editing .hg/patches/series. Pretty rough but good enough. how come there are no q* command for that ?
• to delete a patch, I make sure there are no pending changes, then I do “hg qpop -a; hg qdelete the_patch”
• to import a patch from bugzilla, I use “hg qimport -n xxxx.patch https://bugzilla.mozilla.org/attachment.cgi?id=xxxx” where xxxx is the bug number

How do you work with queues ?

Read more | Comment

June is a very intense month for me, and the rest of the year should be way more intense.

For once, this entry is not about packaging, and a bit personal. But I want to share this with the Python community because that’s my second family.

First of all, Suki was born two days ago on June 17th ! That’s my lovely little girl, my second child. I am so happy about this (you bet). Her aunt, who is creating short movies for a living, has made a small video to welcome Suki to the world. Check it out, she’s done an amazing job.

I was looking for a job and I have found one. I have accepted a position at Mozilla Labs and I’ll start next Monday. I am very excited about this for many reasons. The projects I’ll be working on are great, and the Mozilla people & values match perfectly with how I feel about open source and the web. I’ll start talking about this here in a few weeks I guess, once things will be really started. Thanks to all the people that helped me in my job seeking during these last months !

Last, my latest book “Expert Python Programming” was translated in Japanese. I know some authors are translated in other languages quite often, but for me it was a big deal, because I have worked around with the translation team and the Japanese book is actually better than the original book, thanks to them, their feedback, passion about Python, ideas and work. Now I need to go to Japan to meet them

Read more | Comment

Note: if you are not familiar with PEP 345, you might want to read it to understand this entry. It adds for instance “Requires-Dist” that is similar to setuptools’ install_requires and provides a standard for dependencies description.

The GSOC has started and we are already working on a lot of tasks about packaging. The main difficulty is to make sure each student works without overlapping with others, and never get blocked. That’s why we will have weekly meetings with (almost) everyone. In parallel, the nice posse from the Montreal user group is organizing Distutils sprints quite often now. That means that we now have an important manpower for Distutils and things are starting to speed up.

There’s one controversial topic though, that we need to straighten up : do we want to add an installer in Distutils2 ? And since Distutils2 goal is to be back in the stdlib for Python 3.2, that means: do we want to add an installer in the stdlib ?

My answer so far is Yes. And that’s what I’ll be working on unless someone is able to change my mind

What is Distutils2 ?

Let me explain first what is the Distutils2 project, and what we want it to provide. Like its predecessor, Distutils2 wants to provide two things:

1. a toolbox for third packaging tools, whether they are simple installers or full featured package managers (PyPM, Pip, Enthought Installer etc..). This toolbox will include (if not already) reference implementation of PEP 345, PEP 376, PEP 386. In other words, if you want to create the next killer packaging system, you can use modules like distutils2.version (PEP 386) or distutils2.metadata (PEP 345) to build it, without depending on the “everything is a command” philosophy of Distutils.
2. a standalone tool that can be used to install or remove distributions. That’s what Distutils is and that’s what we want to provide in the future in Distutils2. The ability to install projects (and therefore its dependencies since this is a new metadata field we added in PEP 345).

The controversy is about 2. It’s controversial to provide a script that installs dependencies via PyPI into distutils2 because some projects like Pip already provides this feature.

Our current packaging ecosystem explained

A few years ago, before Setuptools added the ability to install dependencies via easy_install, installing a distribution of a given project was as simple as running a python setup.py install. This was installing the distribution in the target system, in proper locations defined by the install command. That’s it.

Setuptools grew organically on the top of Distutils to provide new metadata like the “install_requires” field, that lists dependencies. Setuptools provided two things:

• A new install command that triggers the installation of dependencies, by reading the setuptools-specific “install_requires” metadata, and fetching dependencies at PyPI and installing them recursively.
• An easy_install script that can be used to install a distribution located at PyPI. That’s just a bootstrap on the top of the new install command. In other words, it grabs the archive at PyPI, unpack it, and run “python setup.py install” on it.

In other words, your Python project setup.py is the installer itself because when you use setuptools, it calls its specific install command and triggers the installation chain.

That’s when the mess started: people that didn’t have setuptools installed couldn’t install projects that was using it of course. So the solution that was provided was to propose an ez_setup.py script that you have to include in your project and to run when setup.py is used, to be able to run your installation. In other words, your setup.py is bootstrapping the utilization/installation of setuptools.  And that turned out to be really messy since Setuptools has its own way for installing things. I hope I don’t sound harsh here, Setuptools is the best thing that happened to packaging in years. And a lot of our current work is to bring back its features into the “main stream”.

The result is that you, as a end user, do not control what installer is going to be used, and you end up with a site-packages that has projects installed differently, and that uses different installers.

I am strongly against this behavior because of the mess it creates. In my opinion a python source distribution should not embed an installer and force its usage like this. We need to separate concerns: a python source project should be a dumb container with the code, and with some metadata.

Then Pip showed up.

Pip is an installer script that grabs the project you want to install and run “python setup.py install” on it. That’s all it does when the project is a plain Distutils one. When it encounter Setuptools projects, it blocks the installation of the project’s dependencies I have described earlier, and installs it like a simple Distutils project. Then, it analyzes its dependencies and installs each one of them separately.

That’s really the way to go because it breaks what setuptools is enforcing: projects are not installing other projects in the process anymore. And in the long term, it will allow us to get rid of setup.py (but that’s another blog post). And I hope Pip will soon be able to install Distutils2 projects because it is providing unifi ed metadata (distutils+setuptools -> PEP 345).

Distutils2 vs Pip

So as I said before: it’s controversial to provide a script that installs dependencies via PyPI into distutils2 because Pip already provides this feature.

But one Distutils2 goal (like Distutils) is to provide a command to install a Distribution of your system so it works. And the concept of “Distribution” has evolved, thanks to PEP 345. this means that it needs to install dependencies now, exactly the way Pip does.

We could just tell people to install Pip on the top of the stdlib. But the goal is to provide in the stdlib a working packaging environment, that provides a minimum set of features. The goal is to have something that works when you install Python 3.2, like what was provided when distutils was brought in (eg batteries included).

Mac OS X includes easy_install, I don’t see any good reason not to include a package installer in the Python stdlib itself. At least, we will be able to have a control on what script gets installed by default with Python.

That’s why I have proposed to include Pip in Distutils2 but Ian and Carl seems a bit reluctant for various reasons. One of them is that having Pip included in the stdlib will slow down their work. I don’t think this is true as long as it’s included carefully. If Distutils2 allows its installer to be replaced through configuration by another one, then Pip can have new releases independently from the version included in the stdlib and people can upgrade their system without having to wait for the next Python release.

In any case, we are working on the various bits that are composing an installer in Distutils2 during GSOC since one of the goal of the project as I said earlier, is to provide a toolbox. So if the merge does not occur, it’s likely that we will start a installer/uninstaller script in Distutils2, and it will look a lot like Pip I guess.

EDIT: to make things clearer, when I am saying that both projects should merge, I am only referring to the raw “install with dependencies” features in Pip, and not all the other features.

Read more | Comment

Distutils makes some call to the PyPI server to register and upload projects. Distutils2 will also make some calls to packages.python.org to automate the upload of documentation. This feature was added by Janis a while ago in Distribute and is being backported in Distutils2 during the GSOC.

To test all these features, what I usually did in Distutils was to monkey-patch the code that calls the server and record in memory the exchanges to check them. The problem with this approach is that you have to be careful in the way you patch the APIs the client code use. A typical bug that can happen is to get a slightly different behavior making your code buggy or broken when it interacts with the real server. Of course you can run your test once with the real server then use some mock or stub techniques. But this work can be rather tedious and complex in my opinion.

What I tend to do these days is drop completely this client-side test fixture approach, and just run a local server that implements partially or fully the real server API.

This blog post is just to demonstrate how easy it can be to run your own test server.

For HTTP protocols, the standard library provides everything needed to write such a server in a few lines. The wsgiref module for instance, is great to get a web server up and running during your tests.

Here’s a full example (working for Python >= 2.6 and Python 3):

import threading
try:
    from urllib.request import urlopen
except ImportError:
    from urllib2 import urlopen

import time
from wsgiref.simple_server import make_server, demo_app

class AppRunner(threading.Thread):
    """Thread that wraps a wsgi app"""
    def __init__(self, wsgiapp=demo_app):
        threading.Thread.__init__(self)
        self.httpd = make_server('', 0, wsgiapp)
        self.address = self.httpd.server_address

    def run(self):
        self.httpd.serve_forever()

    def stop(self):
        self.httpd.shutdown()
        self.join()
        time.sleep(0.2)

_SERVER = None

def run_server():
    """Runs the server."""
    global _SERVER
    if _SERVER is not None:
        # we suppose it's running
        return _SERVER.address
    _SERVER = AppRunner()
    _SERVER.start()
    return _SERVER.address

def stop_server():
    """Stops the server."""
    global _SERVER
    if _SERVER is None:
        return
    _SERVER.stop()
    _SERVER = None

if __name__ == '__main__':
    # 1. set up
    print('Launching the test server')
    url, port = run_server()
    print('Test server running at %s:%d' % (url, port))

    # 2. test
    try:
        print('Testing..')
        res = urlopen('http://%s:%d' % (url, port))
        assert b'Hello world!' in res.read()
    finally:
        # 3. tear down
        print('Stopping the test server')
        stop_server()

run_server() and stop_server() are driving a thread that runs a wsgi application using the wsgiref helpers.

In the main section you can see an example of a test fixture, and of running a client test on the demo_app wsgi application wsgiref provides. It should return a “Hello world!” page. These functions are making sure there’s only one server running even if run_server() is called several times, so it’s dead easy to use it from a unittest class.

From there, writing a test server is just a matter of implementing a wsgi application that will replace the demo_app one. I find this technique superior to all the stub/mockup work required when you don’t run your own server.

For other protocols than HTTP, this work can be longer and more complex of course, unless the sdtlib or a third party project already has a server implementation you can use (like smtpd).

Read more | Comment

The GSOC list of accepted students for the PSF was released.

The Distutils2 project has 5 students that will work on various packaging tasks this summer.

Congratulations to :

• Konrad Delong
• Alexis Metaireau
• Eric Araujo
• Zubin Mithra
• Josip Djolonga

Also, a big thanks to the people that are going to mentor them:

• Fred Drake
• Titus Brown
• Lennart Regebro
• Michael Mulich

Last, a second important project will be done this summer, to create a PyPI Testing infrastucture. Congrats to Mouad Benchchaoui and Thanks to Jesse Noller !

I’ll be mentoring Alexis, and I’ll help other mentors and make sure we all synchronize our efforts. I’m pretty excited about this upcoming GSOC because it’s a true opportunity to speed up the work we need to do, in order to built the new packaging tools we all want to have.

Read more | Comment