Packaging !

tarek@ziade.org / @tarek_ziade

Agenda

  • Packaging
  • Python 3

Packaging

Packaging drive people insane

hate

5 Common Packaging Problems

  1. How do I define dependencies ?
  2. What's setup.py role ?
  3. What's installed ?
  4. What about versions schemes ?
  5. How to define data files ?

Dependencies

Distutils — module-level dependencies:

Done in setup.py:

1 from distutils.core import setup
2 setup(requires=['ldap'])

Then:

1 python setup.py sdist

⇒ Line "Requires: ldap" generated in PKG-INFO

⇒ PKG-INFO == Metadata v1.1

⇒ PKG-INFO installed in site-packages (Py>=2.5)

Dependencies

Setuptools — project-level dependencies:

1 from setuptools import setup
2 setup(install_requires=['lxml'])

⇒ "lxml" in requires.txt

⇒ requires.txt installed in site-packages

⇒ pkg_resources in setuptools to read it back.

Dependencies

Debian — system-level dependencies:

1 $ apt-get install libxslt-dev

⇒ "libxslt" installed on the OS.

Dependencies

  • Distutils — module-level dependencies

Gone !

  • Setuptools — project-level dependencies

New metadata field

  • Debian — system-level dependencies

Hints for system-level dependencies

PEP 345 - Metadata v1.2

  • project-level dependencies: Requires-Dist
  • Environment markers: OS-specific fields in pseudo-Python
  • hints for system-level dependencies: Requires-External
  • implemented in Packaging

PKG-INFO examples:

Requires-Dist: lxml (> 1.0)
Requires-Dist: zope.interface (> 1.0)
Requires-Dist: pywin32 (>1.0); 
               sys.platform == 'win32'
Requires-External: libxslt-dev

5 Common Packaging Problems

  1. How do I define dependencies ?
  2. what's setup.py role ?
  3. what's installed ?
  4. what about versions schemes ?
  5. how to define data files ?

setup.py ?

setup.py roles

  • register and upload a project at PyPI
  • build the project
  • develop with it
  • get information about the project (metadata, files..)
  • install the project
  • ...

"Da" setup.py

Setup.py

setup.cfg

  • static representation for:
    • metadata
    • project content
  • non-intrusive for older tools
  • clean separation from build process

Setup.cfg example

 1 [metadata]
 2 name = gitbuster
 3 version = 0.9
 4 author = Julien Miotte
 5 author_email = julien@gmail.com
 6 summary = Python Qt4 frontend 
 7 description_file = README.rst
 8 home_page = https://gitbuster
 9 requires_dist = gfbi_core (>=0.1)
10 requires_python = >=2.5

Setup.cfg example

 1 [files]
 2 packages =
 3     gitbuster
 4 scripts =
 5     gitbuster/gitbuster
 6 extra_files =
 7     setup.cfg
 8     setup.py
 9     README.rst
10     AUTHORS.txt
11     demo.sh
12     LICENSE.txt
13     MANIFEST.in

Wizard Demo !

5 Common Packaging Problems

  1. How do I define dependencies ?
  2. what's setup.py role ?
  3. what's installed ?
  4. what about versions schemes ?
  5. how to define data files ?

What's installed ?

What's installed ?

How to:

  • uninstall ?!?
  • list what's installed

Different Installation formats:

  • Distutils — nothing recorded (but --record exists!)
  • easy_install — installation not recorded
  • Pip — records in install-record.txt

PEP 376 - Database of installed projects

  • Standard Database of installed projects
  • Packaging implements it

Layout:

  • site-packages
    • Project.dist-info
      • RECORD
      • METADATA
      • INSTALLER
      • REQUESTED

PEP 376 features

  • Iterate on installed projects:
  • List installed files
  • Know which project uses which file
  • Uninstall !

5 Common Packaging Problems

  1. How do I define dependencies ?
  2. what's setup.py role ?
  3. what's installed ?
  4. what about versions schemes ?
  5. how to define data files ?

Version schemes

Real released versions at PyPI

  • working proof of concept
  • dev
  • TRUNKdev-r5
  • unreleased.unofficialdev
  • 1.0dev-BZR-r45-panta-elasticworld.org-20091021145839-1oceeh3stpvyl04t
  • 1.01b1+encoding_patch+removed_django_depends
  • 0.19 "Nose Furnace"
  • (tip)
  • Pre-Alpha

PEP 386 -- Standard version scheme

  • MAJOR.MINOR[.MICRO]...
  • Included in Metadata 1.2
  • PyPI implements it

Examples:

1.1 < 1.2a1 < 1.2 < 1.3.dev2 < 1.3 < 1.3.2

5 Common Packaging Problems

  1. How do I define dependencies ?
  2. what's setup.py role ?
  3. what's installed ?
  4. what about versions schemes ?
  5. how to define data files ?

Defining Data files

Distutils & Data Files

  • Python modules
  • Scripts
  • "Data" files -- explicit paths

Example of the __file__ hack:

1 import os.path
2 here = os.path.dirname(__file__)
3 
4 myfile = open(os.path.join(here, 'pic.jpg'))

Defining Data files, The New Way

One level of indirection:

  1. define resources in setup.cfg
  2. installation time: sysconfig.cfg lookup
  3. setup.cfg definitions copied in RESOURCES (metadata)
  4. runtime: use get_file

Step #1 -- You define in setup.cfg:

1 [files]
2 packages = foo
3 resources =
4   pic.jpg = {appdata}
5   app/app.icon = {icon}
6   somestuff.txt = {datadir}/{dist.name}

Defining Data files, The New Way

Step #2 -- The installer look at sysconfig.cfg (extract):

1 [posix]
2 datadir = /usr/share
3 appdata = {datadir}/{dist.name}

Example: where to install pic.jpg ?

  • project name : MyProject
  • platform: posix
  • setup.cfg definition: pic.jpg = {appdata}

⇒ pic.jpg installed in /usr/share/MyProject/

Defining Data files, The New Way

Step #3 -- You get your file back:

1 from packaging.database import get_file
2 
3 icon = get_file('MyProject', 'app/app.icon')
4 pic = get_file('MyProject', 'pic.jpg')

get_file process:

  1. locate MyProject, open its RESOURCES file
  2. find pic.jpg = {appdata}
  3. read sysconfig.cfg
  4. Find the file

Packaging Roadmap

  • backport in 2.x : "distutils2" -- ETA end of summer
  • version 1.0 in Python 3.3
  • support setuptools/distribute and packaging in your projects

5 tips for today

Tip #1 - use a pep 386-compatible scheme for your versions

Tip #2 - try to make setup.py as dumb as possible

Tip #3 - don't bet what installer people will use

Tip #4 - do not release unstable releases at PyPI

Tip #5 - be careful about your data files

Python 3

Python 3

  • py3k branch started in Q1 2006
  • 3.0 released in November 2008
  • 3.1 released in June 2009
  • 3.2 released in February 2011

Slow but constant uptake

py3pkgs

Available under Python 3

  • Distribute
  • Numpy/Scipy
  • SQLAlchemy
  • CherryPy
  • Bottle

See: https://py3ksupport.appspot.com/

Missing libs & frameworks

  • PyCrypto
  • WebOb (very soon)
  • NLTK
  • Twisted
  • Django
  • Pyramid (~8 months)

PyPy

  • Faster than CPython in some cases !
  • Syntax: Python 2.7
  • Django, ctypes, etc..
  • Features: sandboxing, stackless, etc
  • No support for gevent, some C extensions

Python 2.x v.s. Python 3 (vs PyPy)

どうも !

Questions ?

Twitter: @tarek_ziade Mail: tarek@ziade.org G+: +Tarek Ziade