tl;dr

I am building for Sagrada a web application that can be used to create and deploy web services through the web with a few Python functions.

See the prototype demo quick screencast -- If you can't view it in your browser, download it and use VLC -- The music is MC Jack IN the Box CC-AN 3.0

EDIT Here's a better demo, with Ace integration, where you can edit/create Python files in the browser -- http://ziade.org/redbarrel4.mpeg

Now read below for the long story :)

The Sagrada Project

We had a all-hands last week in San Jose, which gathered +600 Mozillians (yes, we're growing fast).

My team has started to brainstorm about Sagrada, the big project we're starting, and I've started to think about a few topics.

One long term goal of Sagrada is to let developers deploy on our servers their own web services. It's quite hard to put a strict definition for the term "web service", but the term is usually used these days to describe a server-side application that can be queried via the HTTP protocol, receiving and sending back Json objects.

So, the intent is to provide a Service Container for developers that need a server side for their applications.

The server side of Firefox Sync is a good example: It's a set of web services Firefox calls to power Sync. See the API definitions here.

For Sagrada the specific question I am interested to solve is :

What would be the easiest way to write and deploy web services into our infrastructure ?

To start off, here are a few assumptions I am making about what developers would probably expect from this kind of service -- but those are my own assumptions, and will change with the feedback I'll get in the process :

1. Building and deploying a web service should be easy to do whether you are a Python programmer or a JavaScript programmer.

While Python is our main target, being able to write JavaScript on server side should also be a goal -- And if the tool is extensible enough, why not other languages in the future ?

Moreover, most steps when building web services are not specific to a language. For instance, describing what HTTP method should be used, what's the web service URL, what kind of request body is expected, etc, is usually done in a specification document.

For example, when I've built the Easy Setup server for Sync, I've written this specification document : http://docs.services.mozilla.com/keyexchange/apis.html

It contains a description of each web service used by Firefox when you add a new device to your Sync account. The only thing that is not expressed in this document is the just the piece of code that does the job.

So, what if this document could be used directly by the server to run the application ? The only thing that would miss is a few functions.

The bottom line is that web services can be described in a Domain Specific Language (DSL) that can be used to generate the documentation automatically (and it stays accurate and up to date) but also to set up in the server things we usually do on the code side: the request dispatching.

In that case, the portion of code to write is reduced to building the response itself.

Even if we just support Python in a v1, if building a web service boils down to writing a specification document, and a few Python functions to build responses, I think that lowers the barrier enough for most developers.

And if we hide the DSL behind a nice user interface developers can use to build their apps, that's even better. That's what's happening in the screencast I've recorded. The forms generate portions of DSL.

2. The tool should be framework-agnostic if possible.

While tools like Node.js or Pyramid or <put your favorite framework here> provide great features to write web apps, web services people will build will be running in specific environments, where we will want to set up and control our own stack.

In other words, we'll want to isolate as much as possible the code written by developers.

Ideally, we'd just pass a request to a function and ask for a response. Python for instance has a CGI-like standard called WSGI, where the request is described in a mapping and the response is a sequence of string + a mapping of headers.

So telling developers: "Hey, your function will receive a WSGI request, send me back a WSGI response", sounds like a good basis.

I am pretty sure we can do the same in Javascript. And if we don't have such standard in Javascript, let's create it.

3. The tool should provide a Web UI, and CLI and a standalone server.

Web services should not be locked in our servers, developer should be able to edit them through the web, or via the console even if it's remote. They should be able to upload or download their apps in our environment, like what they would do with Google App Engine.

But they should also be able to create, run their apps on their own environment e.g. have a independent web server that can run their services. Not a toy server used for development only, but something they can really deploy in production.

For the remote console, Benoit was telling me this morning: why not iPython ? I dig this: editing web services through iPython would rock. If you don't know about this tool check it out it's amazing. It provides among other things, a way to build an interactive shell for a distant app.

A web service DSL

Back to our DSL. When you write a web service, it's always the same story no matter what framework you're using, you're basically doing these steps (I am over-simplying for now)
1. Define a route for your service on the server 2. Build the response 3. Send back the response

These steps can be described in a simple DSL.

Here's a basic example:

path hello_world (

 description "Simplest application: Hello World!",

 method GET,

 url /hello,

 use python:somemodule.hello

 );

With a function hello located in somemodule that can look like this :
def hello(request):

    return 'Hello World'

The application in that case is composed of
- a DSL file - a Python file with a single function

It's easy from there with the proper DSL parser to:
- deploy those two files in our infrastructure - run the app - provide auto-generated documentation for the service

Architecture

The prototype I've written for the demo does the following:
- The tool is a web application that provides forms to create Service Containers - Each Service Container has a unique root on the server - In each container you can add web services. - For each container, the DSL is built on the fly, then the corresponding AST is kept in memory. The web UI allow users to modify it on the fly - When a request comes in, it's rooted to the right Service Container depending on the beginning of the path, then the AST is used to find out what function(s) should be used. The function is then executed in a sandbox.

The prototype also provides a command-line tool to start a server by loading an arbitrary DSL file.

The DSL Parser

I will not go in to great details here, you can look at my previous posts mentioning RedBarrel.

The current DSL is implemented with PLY and can be found here.

Sandboxing the code

One thing we want to do when a developer uploads some code that is potentially going to be executed, is to sandbox it.

This is not really for security reasons, because we'll still need to protect our users by setting up VMs. This is mostly to
pre-configure what the user is allowed to do in the code and provide some useful feedback when he's doing things
we did not allow, like writing to the filesystem or using sockets -- I am not saying we will allow or disallow these
particular ones, I don't know yet.

In the current prototype, I've used pysandbox from Victor Stinner . It's used to load and execute the uploaded code.
see how I use it.

What's Next

Right now the web interface to build an application is simplistic compared to what the DSL can do. For instance you could chain several functions to perform pre- and post- processing for a request. So some functions can be reused in several services, like authentication.

Also, we want to provide our own Mozilla libs, like a way to authenticate against our own user database, or use a key-value storage. Basically, all the libraries we're currently building for Sagrada.

It's unclear at this point what imports we will allow in the scripts, and how we will publish our own libraries people will be able to use. I intend to clarify these in the upcoming days, and enhance the prototype to allow it to do more things.

Also, I'd like to write the Easy Setup server using this tool. I'll also try to organize a coding/brainstorming sprint since I have 5/6 people that worked with me on these topics to hack on this.

If you are interested or have some feedback, please comment !