Posts Tagged ‘cherrypy’

CherryPy, RESTFul Routing

October 12th, 2008 by ScottK | 2 Comments | Filed in CherryPy

Using CherryPy as a web server allows us to quickly create services and gain a lot more free time, or rest. I don’t miss those times where we worked none stop on other frameworks to provide services to other apps.  OK moving from on project immediately to another project takes up that free time and the rest in between is not what I mean by RESTFul routing.

The buzzword RESTFul routing is really just a concept that states actions that you perform for updating your data can be attributed to the state the browser asks you to do it. Along with parameters that it sends. I won’t go into any full explanation of what Representational State Transfer is, but you can find a good explanation here RESTFul.

I do find that that having RESTFul controllers are a big benefit to integrating with Ruby on Rails apps, yet I’m not so hooked on the idea that RESTFul is the end all be all of web programming. There are time in even rails apps that you need to break that convention and create special routing configurations. CherryPy, through the routes, fully supports those as well.

I’ve created the downloadable CherryPy Example 3 so you can follow along.

Let’s start by looking at the natural RESTFul controller: controllers/users_controller.py. You’ll find that it holds the full RESTFul actions you would expect to find. Create, Index, Show, New, Update, Edit, Delete. All of these are completely usable due to the @cherrypy.expose decorator. Look at the controllers/hello_worlds_controller.py as well. You’ll find that none of the RESTFul actions are there at all.

In order to use both of these controllers you need to step outside of the “default” CherryPy tutorial and dig a little deeper. Hence is the reason why you’ll find in the libs folder the route_mapper.py. It’s the core of setting up the RESTFul routing. So let’s walk through the example step by step because it all starts with the deploy.py and follows step by step.

In my last post about CherryPy, The Deploy Script I only setup the first stage of running a CherryPy app. Only hinting on how to set up RESTFul routing. I made mention that two methods were left out. Now those are important.

The first of the two is the “load_default_routes” method. This method does not accept any arguments and setups up all your RESTFul routes by default. The second of the the two is the “load_custom_routes” which is what you’ll use to set up all your custom routing, or named routing.

You’ll notice in the example that I import these controllers during these methods. This is intentional because we absolutely have to have the con figs loaded, example 2. Along with each of the methods we are building a list of dict pertaining to specific information. Name, path, controller for both default and named routes, as well as specific action for the named routes.

In the deploy script the “server_start” method calls these two methods. Which then passes these as arguments to the lib.route_mapper library. This is where the routes magic happens. In lib you’ll find the route_mapper.py file give that a look.

By default CherryPy uses it’s own dispatcher to set up routing. This is not necessarily RESTFul. In that using the default you would have to call /users/index to access the UsersController index action. What we wanted was the RESTFul GET method /users. What about different format types as well? The routes_mapper library throws away the CherryPy dispatcher and uses the routes dispatcher instead.

That what this is doing:

d = cherrypy.dispatch.RoutesDispatcher()

Now that we have the routes library dispatcher for CherryPy we can set up routing as we need using the connect method.

Certainly in setting this up I discovered some really strange things which I’ve fixed in your example. But what it boils down to if you want to set up your own RESTFul routing or named routing is this: Order of Operation.

1. (controller)/:id.:format
2.(controller):id

If I called event/1.html it would find number one first. If I called event/1 it would find number two.

1. (controller):id
2. (controller):id.:format

In either case of calling event/1 or event/1.html it would always find number one.

This is true of all methods as well. The order of Create, Index, New, Show, Edit, Update seems to be the best mix for setting up the Routes mapper. In different orders the correct actions may not be called correctly.

You’ve now seem me use :id, and :format. What do these mean? They are merely the arguments I am passing to the controller action. You can certainly use what ever you need to.

Given that if in my load_default_routes I have:

routes = [
{"name"  : "post_comments",
"path"  : "post/:post_id/comments/:id",
"controller" : CommentsController()
}
]

I would need to have a comments_controller class with a show action. This action would have the arguments of:

def show(self, post_id, id):
...

So anything you prepend with “:” thus becomes an argument to your action method.

Really by taking the routes dispatcher from the default CherryPy dispatcher you can implement RESTFul routing. It really isn’t difficult to do as long as you do it in order of actions and as long as you understand what Representational State Transfer is all about. Even named and custom routes can be used if you wish.

Tags: , , ,

CherryPy, The Deploy Script

September 28th, 2008 by ScottK | 3 Comments | Filed in CherryPy

To start a CherryPy application begins with a startup script for assigning all your controllers and generally telling it what to do. How your application runs is greatly dependent upon how this script operates. By default the CherryPy tutorial doesn’t allow for RESTFul routing or allowing the controllers access to the configuration files as they are imported.

I’m required to use different environments for building an app. Working locally to develop it I need development settings. Once done I send it off to QA which needs those settings. Finally in production it needs settings for that as well additional requirements as not to show tracebacks. This is easily done in the deploy script.

One of the much touted announcements that came from the release of CherryPy 3.1 stable was something called the “cherryd” script in which you could daemonize your application instead of running it under mod_python or as a background job. I looked at the script and was horrified to realize that it was so completely different from the deploy scripts I had written.

Then I looked at it even closer, following each line and suddenly realized that the cherryd was not a be all deploy script, it was merely a working script which was basic in it’s use. Having realized that only a few parts of it were needed to fully daemonize the deploy scripts I’ll show below while maintaining the RESTFul resources.

The CherryPy Example 2 code files for you to follow along. I’ve commented heavily in them for what each part does.

Before we begin with the actually deploy script let’s look at the configuration files themselves. I’ve intentionally left them very small but distinct enough so you can see how they work when running your app under the different environments.

The first configuration is the config/site_config.py. This holds all the settings common to all your running environments. Whether running in development or production the site_config will be the same for both.

site_config.py

[global]
server.thread_pool = 10
server.socket_host = "0.0.0.0"

The server.thread_pool is how many threads should your cherrypy app handle during operation. Having more threads allows your app to handle more requests and perform faster under heavy loads. I’m running applications that have 400 threads. On some machines I’ve had to lower that as they could handle so many of them. Play with it for the best performance of your application.

The server.socket_host set to 0.0.0.0 simply allows your app to run on any IP without you needing to assign it. Certainly this can be changed to 127.0.0.1 or even any other IP address. Leaving it at 0.0.0.0 means I don’t have to worry about it.

The deploy script first loads the site_config. After which it loads the environment config based upon what you specified in the startup command.

development.py

[global]
server.environment = "development"
server.socket_port = 7180

qa.py

[global]
server.environment = "production"
server.socket_port = 7181

production.py

[global]
server.environment = "production"
server.socket_port = 7182

As you can see each file holds the same settings. Yet these setting are different. Looking at the server.socket_port port assignment you can see that I can be developing the app while running the production app on the same machine since the running ports are different.

So now that we have the configuration files out of the way let’s look at the deploy script. The example deploy.py is complete, I’ll only discuss the relevant parts to load the configuration files and starting the app here.

deploy.py (partial)

if __name__ == '__main__':

    p = OptionParser()
    p.add_option('-d', action="store_true",
                 dest='daemonize', help="run the server as a daemon")
    p.add_option('-e', '--environment',
                 dest='environment', default=None,
                 help="apply the given config environment (development, staging, qa, production)")
    p.add_option('-p', '--pidfile',
                 dest='pidfile', default=None,
                 help="store the process id in the given file")

    options, args = p.parse_args()

    if not options.environment in ["development", "staging", "qa","production"]:
        print "A running environment is required"
        exit(1)

    """Load up the configuration files first before loading any
    controllers."""
    load_config(options.environment)

    """Since we've now loaded the configuration files we can load
    the controllers so they have access to the cherrypy.config.get(..)
    No need to use any PyYAML. :) """
    server_start()

    """When we are running in production mode we don not want tracebacks
    Additionally we want a redirect to the corresponding error page."""
    if options.environment == "production":
        cherrypy.config.update({'error_page.default': \
                                "%s/public/index.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.401': \
                                "%s/public/errors/401.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.404': \
                                "%s/public/errors/404.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.500': \
                                "%s/public/errors/500.html" % (os.path.abspath("."))}
                              )

    cherrypy.config.update({'log.error_file': \
                            "%s/logs/%s_error.log" % (os.path.abspath("."), options.environment)}
                          )
    cherrypy.config.update({'log.access_file': \
                            "%s/logs%s_access.log" % (os.path.abspath("."), options.environment)
                           }
                          )

    engine = cherrypy.engine

    """The plugins.Daemonizer handles all the daemon process setup for us.
    Additional processes and configs setup here when running under daemon"""
    if options.daemonize:
        cherrypy.config.update({'log.screen': False})
        plugins.Daemonizer(engine).subscribe()

    """Writes a pid file to the directory specified in the:
    -p /var/run/my_app.pid CLI option"""
    if options.pidfile:
        plugins.PIDFile(engine, options.pidfile).subscribe()

    """Setup the signal handler to stop the application while running"""
    if hasattr(engine, "signal_handler"):
        engine.signal_handler.subscribe()
    if hasattr(engine, "console_control_handler"):
        engine.console_control_handler.subscribe()

    """Finally start the app with engine.start(). Any kill signals
    are handled in the except. So if you need any other cleanup
    processes that need handled on shutdown put them there."""
    try:
        engine.start()
    except:
        sys.exit(1)
    else:
        engine.block()

So first off notice that we are going to run this from the command line “python ./deploy.py” This will never be imported for use elsewhere. To help out it also uses the OptionParser module to make it easier for command line assignment:

python ./deploy.py -e development  (Run the development configuration)
python ./deploy.py -e qa -d -p /var/log/qa.pid (Run the qa configuration as a daemon with pid file)

So with the above commands we’ve instructed the application to load the specific config file we need. That the job of function: load_config(options.environment).

I’m skipping the server_start function for now to revisit it when we discuss controllers.

Notice:

"""When we are running in production mode we do not want tracebacks
    Additionally we want a redirect to the corresponding error page."""
    if options.environment == "production":
        cherrypy.config.update({'error_page.default': \
                                "%s/public/index.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.401': \
                                "%s/public/errors/401.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.404': \
                                "%s/public/errors/404.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.500': \
                                "%s/public/errors/500.html" % (os.path.abspath("."))}
                              )

    cherrypy.config.update({'log.error_file': \
                            "%s/logs/%s_error.log" % (os.path.abspath("."), options.environment)}
                          )
    cherrypy.config.update({'log.access_file': \
                            "%s/logs%s_access.log" % (os.path.abspath("."), options.environment)
                           }
                          )

This section is special. If we are running the production environment then we want to setup the error handling pages for our application. Under all environments we also need to setup the corresponding log files for access or error. Why didn’t this get included in the configuration files?

Notice the os.path.abspath. In the configuration files we can not import other modules. Only use python functions that don’t need importing. Since other scripts can start the deploy script it’s difficult to tell what the file path would be in the config files unless you specifically say “/PATH_TO_DIRECTORY/logs/production_access.log”. That however, is not very portable is it.

This rest of the file has the comments on what each part means.

Give it a try. Run two commands:
python ./deploy.py -e development
python ./deploy.py -e production

view them at http://localhost:7180/ and http://localhost:7182/ . Notice that both give you the index.html page and that both are running. Now go to http://localhost:7180/nothing and http://localhost:7182/nothing.

Now do you see how the different environment config files come into play. The error for the development app shows you the error (404) tracebacks, whereas the production shows you the 404 html page instead.

Probably the most important file a CherryPy app requires is the deploy script. As developers we need a way to build in one environment and run production in another. This can be incorporated in the deploy script fairly easily and once done we can forget about it. Being able to daemonize the CherryPy app leads to huge performance increases. The cherryd script is merely a basic working script which we can incorporate into our own deploy scripts.

In the next post I’ll show you how to incorporate RESTFul controllers and custom named routes into the deploy script for a well rounded application.

Tags: , ,

CherryPy, The Setup

September 27th, 2008 by ScottK | No Comments | Filed in CherryPy

When the requirement was given to create a web application that was merely more than an API service CherryPy was chosen. CherryPy is an HTTP Framework without all the bells and whistles of database libraries, and templating libraries. As a web service provider it’s turned out to be very fast because of the lack of the other systems.

However what is the use of a web service that doesn’t have a connected database and some minor templating views? Since Ruby on Rails is the predominate framework in our shop and only a few of use work with Python I wanted to use CherryPy in the style of Ruby On Rails so that others transitioning from RoR would be familiar.

While I’m certainly not an evangelist for RESTFul resources I do find that it has it’s place and certainly helps me contemplate what is going to happen when a request is given for certain call methods, (POST, GET, PUT) etc. The CherryPy request dispatcher uses the routes library, which fully supports RESTFul controllers.

Another consideration is how to set up your environments; development, qa, production, etc. This was kind of tricky as a few others I talked to used the PyYAML library to inject global variables throughout their libraries. I found that this doesn’t have to be the case. Why bloat your application with another library with knowing when to import your controllers will allow you to have all that you need within standard CherryPy config.

When setting up your CherryPy application in a manner of the MVC framework you need to have the folder structure. I’ve included a downloadable zip that includes the folders neccessary for this post and subsequent posts. You’ll clearly see that it arranges every system rationally for easy look up and organisation. The key however is in the deploy.py file as it is the start and stop script for your application.

Feel free to download the CherryPy Example 1 file and get a feel for where I am going with the subsequent posts. Subsequent posts are going to cover everything from A-Z on setting up a RESTFul CherryPy application that you can daemonize for a really fast API server that is light on the templating side.

Tags: , , , , ,