Last year I've proposed a GSOC project that no student picked up. I
think the fact that it was titled "A Distutils Regression System" was
not really appealing. I should have called it a "PyPI testing
infrastructure" because that's what it is really. And I know today
exactly how the system should work, so I can make a clearer proposal.
The proposal page is here:
So basically, the idea is to run a service that will receive update
notifications everytime someone uploads a distribution at PyPI, to
perform a few tests on it and send a report to the author by email (or
display it on a website).
The tests we want to do are quite simple:
- does "python setup.py install" works or throw out some errors ? - what files are modified on the system upon installation ? - does the tests pass ?
For the latter, it supposes that we are able to detect and run the
tests. They are no conventions yet (but the TIP list is working on this)
but we can try out a few things on the source tree.
EDIT: for all the tests we want to run on a distribution, we should
look at the CheeseCake project (Thanks Richard for mentioning it in
Getting update notifications is fairly simple now that Martin added a
PubSubHubBub service at PyPI.
The core part of the project is to make sure the commands are run on a
clean system. The easiest way to do it is to use Virtual Machines
through a service like EC2: we can run the test, then rollback the VM
state so it's back in a clean state. There are a few libs in Python to
drive systems like EC2. We also need to secure the VM so the code that
is run doesn't make network calls. If the project tests are relying on a
network resource, those are bad tests anyways.
Last, but not least, if we want to get back interesting reports, we
need to probe what's going on. It's simple at Python level, but we never
know what can happen in a setup.py module. If a C module or an
external program is called somehow in there, it could perform operations
on the system we wouldn't be aware of. I think SystemTap is a great
tool to probe for all operations at the OS level. I am not sure how to
do it under Win32 though..
But having a working story under Linux would be already a great thing,
so I am not worrying about that at this point.
So, I've entered the topic in the GSOC list, and I am willing to mentor this if a student wants to do this. I know Titus is also very interested in this topic, so we might work together on this, as co-mentors, which would be great. That's a topic we've discussed several times in the past and I think Titus wants a similar system. Mmm.. wait, we all do, don't we ? ;)