rlimit support

Circus comes now with rlimit support. For example, if you want a specific process to have a limited number of open files to 100, you can use in your Circus configuration file the rlimit_nofile option:

[circus]
check_delay = 5
endpoint = tcp://127.0.0.1:5555

[watcher:myprogram]
cmd = myprogram
rlimit_nofile = 500

This feature is built-in in the Python standard library, in the resource module so that was easy to add. The next step here would be to see how Circus could interact with tools like cgroups.

stderr and stdout streaming

That feature is a must have, and we worked quite a bit on it to make sure it's fast even with hundreds of processes being watched : the ability to stream the standard output and standard error streams of all processes back into Circus.

For example, if you want all processes to write their stdout continously into a file, you can write:

[watcher:myprogram]
cmd = myprogram
numprocesses = 100

# will push in test.log the stream every 300 ms
stdout_stream.class = FileStream
stdout_stream.filename = myprogram.log
stdout_stream.refresh_time = 0.3

The core of this feature is a call to the select() function from the standard library on the PIPEs opened on each process.

The gist of this code is:

import fcntl

for pipe in pipes:
    fcntl.fcntl(pipe, fcntl.F_SETFL, os.O_NONBLOCK)

while True:
    rlist, __, __ = select(pipes, [], [])
    for pipe in rlist:
        try:
            data = pipe.read(self.buffer)
            redirect_to_circus(data)
        except IOError, ex:
            if ex[0] != errno.EAGAIN:
                raise

redirect_to_circus basically redirects the stream to whatever class you've configured in stdout_stream.class. You can provide your own class if you want to implement a specific stream handler.

We've implemented this stream redirector with one thread per watcher that operates the select() calls, but also have a Gevent implementation that uses greenlets instead of threads and Gevent's own select() implementation.

The threaded version is the default one, but you can pick the gevent backend with the stream_backend option.

Flapping

Yet another important feature to have: the ability to detect that a process that's launched constantly dies. That happens when the process command line is wrong or when a resource the program uses is not reachable for example.

Tools like daemontools will simply try again and again to run the service, eating in the process the whole CPU of your server.

With this feature built-in and enabled by default, Circus detects the flapping of processes, and try again later then eventually quit trying.

Of course, the flapping dance is published in the PUB/SUB channel so it's easy to build a script that will send you an alert when it happens.