Note: if you are not familiar with PEP 345, you might want to read
it to understand this entry. It adds for instance "Requires-Dist" that
is similar to setuptools' install_requires and provides a standard for
The GSOC has started and we are already working on a lot of tasks about
packaging. The main difficulty is to make sure each student works
without overlapping with others, and never get blocked. That's why we
will have weekly meetings with (almost) everyone. In parallel, the nice
posse from the Montreal user group is organizing Distutils sprints quite
often now. That means that we now have an important manpower for
Distutils and things are starting to speed up.
There's one controversial topic though, that we need to straighten up :
do we want to add an installer in Distutils2 ? And since Distutils2 goal
is to be back in the stdlib for Python 3.2, that means: do we want to
add an installer in the stdlib ?
My answer so far is Yes. And that's what I'll be working on unless
someone is able to change my mind :)
What is Distutils2 ?
Let me explain first what is the Distutils2 project, and what we want
it to provide. Like its predecessor, Distutils2 wants to provide two
1. a toolbox for third packaging tools, whether they are simple installers or full featured package managers (PyPM, Pip, Enthought Installer etc..). This toolbox will include (if not already) reference implementation of PEP 345, PEP 376, PEP 386. In other words, if you want to create the next killer packaging system, you can use modules like distutils2.version (PEP 386) or distutils2.metadata (PEP 345) to build it, without depending on the "everything is a command" philosophy of Distutils. 2. a standalone tool that can be used to install or remove distributions. That's what Distutils is and that's what we want to provide in the future in Distutils2. The ability to install projects (and therefore its dependencies since this is a new metadata field we added in PEP 345).
The controversy is about 2. It's controversial to provide a script that
installs dependencies via PyPI into distutils2 because some projects
like Pip already provides this feature.
Our current packaging ecosystem explained
A few years ago, before Setuptools added the ability to install
dependencies via easy_install, installing a distribution of a
given project was as simple as running a python setup.py install.
This was installing the distribution in the target system, in proper
locations defined by the install command. That's it.
Setuptools grew organically on the top of Distutils to provide new
metadata like the "install_requires" field, that lists dependencies.
Setuptools provided two things:
- A new install command that triggers the installation of dependencies, by reading the setuptools-specific "install_requires" metadata, and fetching dependencies at PyPI and installing them recursively. - An easy_install script that can be used to install a distribution located at PyPI. That's just a bootstrap on the top of the new install command. In other words, it grabs the archive at PyPI, unpack it, and run "python setup.py install" on it.
In other words, your Python project setup.py is the installer
itself because when you use setuptools, it calls its specific install
command and triggers the installation chain.
That's when the mess started: people that didn't have setuptools
installed couldn't install projects that was using it of course. So the
solution that was provided was to propose an ez_setup.py script
that you have to include in your project and to run when setup.py is
used, to be able to run your installation. In other words, your setup.py
is bootstrapping the utilization/installation of setuptools. And that
turned out to be really messy since Setuptools has its own way for
installing things. I hope I don't sound harsh here, Setuptools is the
best thing that happened to packaging in years. And a lot of our current
work is to bring back its features into the "main stream".
The result is that you, as a end user, do not control what installer is
going to be used, and you end up with a site-packages that has projects
installed differently, and that uses different installers.
I am strongly against this behavior because of the mess it creates.
In my opinion a python source distribution should not embed an
installer and force its usage like this. We need to separate concerns:
a python source project should be a dumb container with the code, and
with some metadata.
Then Pip showed up.
Pip is an installer script that grabs the project you want to install
and run "python setup.py install" on it. That's all it does when the
project is a plain Distutils one. When it encounter Setuptools projects,
it blocks the installation of the project's dependencies I have
described earlier, and installs it like a simple Distutils project.
Then, it analyzes its dependencies and installs each one of them
That's really the way to go because it breaks what setuptools is
enforcing: projects are not installing other projects in the process
anymore. And in the long term, it will allow us to get rid of setup.py
(but that's another blog post). And I hope Pip will soon be able to
install Distutils2 projects because it is providing unifi ed metadata
(distutils+setuptools -> PEP 345).
Distutils2 vs Pip
So as I said before: it's controversial to provide a script that
installs dependencies via PyPI into distutils2 because Pip already
provides this feature.
But one Distutils2 goal (like Distutils) is to provide a command to
install a Distribution of your system so it works. And the concept of
"Distribution" has evolved, thanks to PEP 345. this means that it needs
to install dependencies now, exactly the way Pip does.
We could just tell people to install Pip on the top of the stdlib. But
the goal is to provide in the stdlib a working packaging environment,
that provides a minimum set of features. The goal is to have something
that works when you install Python 3.2, like what was provided when
distutils was brought in (eg batteries included).
Mac OS X includes easy_install, I don't see any good reason not to
include a package installer in the Python stdlib itself. At least, we
will be able to have a control on what script gets installed by default
That's why I have proposed to include Pip in Distutils2 but Ian and
Carl seems a bit reluctant for various reasons. One of them is that
having Pip included in the stdlib will slow down their work. I don't
think this is true as long as it's included carefully. If Distutils2
allows its installer to be replaced through configuration by another
one, then Pip can have new releases independently from the version
included in the stdlib and people can upgrade their system without
having to wait for the next Python release.
In any case, we are working on the various bits that are composing an
installer in Distutils2 during GSOC since one of the goal of the project
as I said earlier, is to provide a toolbox. So if the merge does not
occur, it's likely that we will start a installer/uninstaller script in
Distutils2, and it will look a lot like Pip I guess.
EDIT: to make things clearer, when I am saying that both projects should merge, I am only referring to the raw "install with dependencies" features in Pip, and not all the other features.