When I started with CherryPy, luckily 3.1, I not only needed to learn how to work with it but also test my app as well. Others where using Nose and Twill with some success, the problem the other testing frameworks had seemed to surround the controller tests.

I discovered the testing framework within CherryPy and easily pulled it out to test my apps. There has never been a problem testing controllers, models, modules, etc. That is the what this CherryPy Example 4 is all about, testing your CherryPy app. You can find the follow along example here.

All the test case files will have to incorporate the cherrpy.test.helper class. This is in order for each of your test classes that must inherit the helper.CPWebCase class.

Controller Testing:

In the example program refer to the users_controller.py

@cherrypy.expose
    def index(self, format='html', **kwargs):
        if format == 'js':
            cherrypy.response.headers['Content-Type'] = "application/javascript"
            return "alert('format in JavaScript for index');"
        elif format == "xml":
            cherrypy.response.headers['Content-Type'] = "application/xml"
            return "Index"
        else:
            return "This is the Index in HTML"

This is the index action of the controller. We need to test it to make sure that it not only responds with XML/HTML/JavaScript accordingly.

def testIndexUSersWithXML(self):
    self.getPage("/users.xml", method='GET')
    self.assertStatus('200 OK')
    self.assertBody('Index')

def testIndexUSersWithHTML(self):
    self.getPage("/users", method='GET')
    self.assertStatus('200 OK')
    self.assertBody('This is the Index in HTML')

self.getPage is the call to the actively running application. assertStatus assures us that everything went ok. Of course you can also assert the other status as needed. assertBody test must match the entire response body. Generally I use assertInBody to make sure that what I’m looking for was returned rather than the whole response.

Prior to running the controller tests the controller does need to be registered with the dispatch. I’m still using the route_mapper as in CherryPy Example 3. So at the top of the controller test files I’ll include:

def setup_server():
    routes = [{"name" : "user", "path" : "user", "controller" : UsersController()}]

    conf = {'/': {
                    'request.dispatch' : rm.setup_routes(routes)
                 }
            }

    cherrypy.tree.mount(root=None, config=conf)

Then at the bottom of the file:

if __name__ == "__main__":

    setup_server()
    helper.testmain() //This calls for the program to run.

Outside of controllers, testing objects is pretty straight forward: assertEqual; assertNotEqual, etc. Again making sure that each test class inherits from helper.CPWebCase. test_user_model has the examples for testing a model.

As your application and test cases grows it becomes rather painful to keep running each file independently. I did this for a while with two dozen test files. Longing for a test runner I once again referred to the CherryPy module to see if they had something that would do this.

Sure enough the cherrypy.test has the method to run all your tests at once. However it required me to copy it and modify it. in the example tests folder you’ll find the test.py. This is the modified version of the cherrypy.test file.

The notable modification has to deal with the run function. Originally it contained a list of all the test cases to run at once. While this made it good as a test runner I also wanted to run singles as I was working on the model itself. So with the modification of test=None in the argument you can run the entire suite or just one test.

python test.py //Entire suite
pytnon test.py test_user_model //just the test_user_model.py file

All these tests are designed to be run in the tests folder using python test.py.

Testing CherryPy apps is really easy if you incorporate the CherryPy test suite. The мебелиCherryPy Example 4 has the necessary files in the tests folder to run them.

Using CherryPy as a web server allows us to quickly create services and gain a lot more free time, or rest. I don’t miss those times where we worked none stop on other frameworks to provide services to other apps.  OK moving from on project immediately to another project takes up that free time and the rest in between is not what I mean by RESTFul routing.

The buzzword RESTFul routing is really just a concept that states actions that you perform for updating your data can be attributed to the state the browser asks you to do it. Along with parameters that it sends. I won’t go into any full explanation of what Representational State Transfer is, but you can find a good explanation here RESTFul.

I do find that that having RESTFul controllers are a big benefit to integrating with Ruby on Rails apps, yet I’m not so hooked on the idea that RESTFul is the end all be all of web programming. There are time in even rails apps that you need to break that convention and create special routing configurations. CherryPy, through the routes, fully supports those as well.

I’ve created the downloadable CherryPy Example 3 so you can follow along.

Let’s start by looking at the natural RESTFul controller: controllers/users_controller.py. You’ll find that it holds the full RESTFul actions you would expect to find. Create, Index, Show, New, Update, Edit, Delete. All of these are completely usable due to the @cherrypy.expose decorator. Look at the controllers/hello_worlds_controller.py as well. You’ll find that none of the RESTFul actions are there at all.

In order to use both of these controllers you need to step outside of the “default” CherryPy tutorial and dig a little deeper. Hence is the reason why you’ll find in the libs folder the route_mapper.py. It’s the core of setting up the RESTFul routing. So let’s walk through the example step by step because it all starts with the deploy.py and follows step by step.

In my last post about CherryPy, The Deploy Script I only setup the first stage of running a CherryPy app. Only hinting on how to set up RESTFul routing. I made mention that two methods were left out. Now those are important.

The first of the two is the “load_default_routes” method. This method does not accept any arguments and setups up all your RESTFul routes by default. The second of the the two is the “load_custom_routes” which is what you’ll use to set up all your custom routing, or named routing.

You’ll notice in the example that I import these controllers during these methods. This is intentional because we absolutely have to have the con figs loaded, example 2. Along with each of the methods we are building a list of dict pertaining to specific information. Name, path, controller for both default and named routes, as well as specific action for the named routes.

In the deploy script the “server_start” method calls these two methods. Which then passes these as arguments to the lib.route_mapper library. This is where the routes magic happens. In lib you’ll find the route_mapper.py file give that a look.

By default CherryPy uses it’s own dispatcher to set up routing. This is not necessarily RESTFul. In that using the default you would have to call /users/index to access the UsersController index action. What we wanted was the RESTFul GET method /users. What about different format types as well? The routes_mapper library throws away the CherryPy dispatcher and uses the routes dispatcher instead.

That what this is doing:

d = cherrypy.dispatch.RoutesDispatcher()

Now that we have the routes library dispatcher for CherryPy we can set up routing as we need using the connect method.

Certainly in setting this up I discovered some really strange things which I’ve fixed in your example. But what it boils down to if you want to set up your own RESTFul routing or named routing is this: Order of Operation.

1. (controller)/:id.:format
2.(controller):id

If I called event/1.html it would find number one first. If I called event/1 it would find number two.

1. (controller):id
2. (controller):id.:format

In either case of calling event/1 or event/1.html it would always find number one.

This is true of all methods as well. The order of Create, Index, New, Show, Edit, Update seems to be the best mix for setting up the Routes mapper. In different orders the correct actions may not be called correctly.

You’ve now seem me use :id, and :format. What do these mean? They are merely the arguments I am passing to the controller action. You can certainly use what ever you need to.

Given that if in my load_default_routes I have:

routes = [
{"name"  : "post_comments",
"path"  : "post/:post_id/comments/:id",
"controller" : CommentsController()
}
]

I would need to have a comments_controller class with a show action. This action would have the arguments of:

def show(self, post_id, id):
...

So anything you prepend with “:” thus becomes an argument to your action method.

Really by taking the routes dispatcher from the default CherryPy dispatcher you can implement RESTFul routing. It really isn’t difficult to do as long as you do it in order of actions and as long as you understand what Representational State Transfer is all about. Even named and custom routes can be used if you wish.

Tags: cherrypy, dispatcher, restful, routes

To start a CherryPy application begins with a startup script for assigning all your controllers and generally telling it what to do. How your application runs is greatly dependent upon how this script operates. By default the CherryPy tutorial doesn’t allow for RESTFul routing or allowing the controllers access to the configuration files as they are imported.

I’m required to use different environments for building an app. Working locally to develop it I need development settings. Once done I send it off to QA which needs those settings. Finally in production it needs settings for that as well additional requirements as not to show tracebacks. This is easily done in the deploy script.

One of the much touted announcements that came from the release of CherryPy 3.1 stable was something called the “cherryd” script in which you could daemonize your application instead of running it under mod_python or as a background job. I looked at the script and was horrified to realize that it was so completely different from the deploy scripts I had written.

Then I looked at it even closer, following each line and suddenly realized that the cherryd was not a be all deploy script, it was merely a working script which was basic in it’s use. Having realized that only a few parts of it were needed to fully daemonize the deploy scripts I’ll show below while maintaining the RESTFul resources.

The CherryPy Example 2 code files for you to follow along. I’ve commented heavily in them for what each part does.

Before we begin with the actually deploy script let’s look at the configuration files themselves. I’ve intentionally left them very small but distinct enough so you can see how they work when running your app under the different environments.

The first configuration is the config/site_config.py. This holds all the settings common to all your running environments. Whether running in development or production the site_config will be the same for both.

site_config.py

[global]
server.thread_pool = 10
server.socket_host = "0.0.0.0"

The server.thread_pool is how many threads should your cherrypy app handle during operation. Having more threads allows your app to handle more requests and perform faster under heavy loads. I’m running applications that have 400 threads. On some machines I’ve had to lower that as they could handle so many of them. Play with it for the best performance of your application.

The server.socket_host set to 0.0.0.0 simply allows your app to run on any IP without you needing to assign it. Certainly this can be changed to 127.0.0.1 or even any other IP address. Leaving it at 0.0.0.0 means I don’t have to worry about it.

The deploy script first loads the site_config. After which it loads the environment config based upon what you specified in the startup command.

development.py

[global]
server.environment = "development"
server.socket_port = 7180

qa.py

[global]
server.environment = "production"
server.socket_port = 7181

production.py

[global]
server.environment = "production"
server.socket_port = 7182

As you can see each file holds the same settings. Yet these setting are different. Looking at the server.socket_port port assignment you can see that I can be developing the app while running the production app on the same machine since the running ports are different.

So now that we have the configuration files out of the way let’s look at the deploy script. The example deploy.py is complete, I’ll only discuss the relevant parts to load the configuration files and starting the app here.

deploy.py (partial)

if __name__ == '__main__':

    p = OptionParser()
    p.add_option('-d', action="store_true",
                 dest='daemonize', help="run the server as a daemon")
    p.add_option('-e', '--environment',
                 dest='environment', default=None,
                 help="apply the given config environment (development, staging, qa, production)")
    p.add_option('-p', '--pidfile',
                 dest='pidfile', default=None,
                 help="store the process id in the given file")

    options, args = p.parse_args()

    if not options.environment in ["development", "staging", "qa","production"]:
        print "A running environment is required"
        exit(1)

    """Load up the configuration files first before loading any
    controllers."""
    load_config(options.environment)

    """Since we've now loaded the configuration files we can load
    the controllers so they have access to the cherrypy.config.get(..)
    No need to use any PyYAML.  """
    server_start()

    """When we are running in production mode we don not want tracebacks
    Additionally we want a redirect to the corresponding error page."""
    if options.environment == "production":
        cherrypy.config.update({'error_page.default': \
                                "%s/public/index.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.401': \
                                "%s/public/errors/401.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.404': \
                                "%s/public/errors/404.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.500': \
                                "%s/public/errors/500.html" % (os.path.abspath("."))}
                              )

    cherrypy.config.update({'log.error_file': \
                            "%s/logs/%s_error.log" % (os.path.abspath("."), options.environment)}
                          )
    cherrypy.config.update({'log.access_file': \
                            "%s/logs%s_access.log" % (os.path.abspath("."), options.environment)
                           }
                          )

    engine = cherrypy.engine

    """The plugins.Daemonizer handles all the daemon process setup for us.
    Additional processes and configs setup here when running under daemon"""
    if options.daemonize:
        cherrypy.config.update({'log.screen': False})
        plugins.Daemonizer(engine).subscribe()

    """Writes a pid file to the directory specified in the:
    -p /var/run/my_app.pid CLI option"""
    if options.pidfile:
        plugins.PIDFile(engine, options.pidfile).subscribe()

    """Setup the signal handler to stop the application while running"""
    if hasattr(engine, "signal_handler"):
        engine.signal_handler.subscribe()
    if hasattr(engine, "console_control_handler"):
        engine.console_control_handler.subscribe()

    """Finally start the app with engine.start(). Any kill signals
    are handled in the except. So if you need any other cleanup
    processes that need handled on shutdown put them there."""
    try:
        engine.start()
    except:
        sys.exit(1)
    else:
        engine.block()

So first off notice that we are going to run this from the command line “python ./deploy.py” This will never be imported for use elsewhere. To help out it also uses the OptionParser module to make it easier for command line assignment:

python ./deploy.py -e development  (Run the development configuration)
python ./deploy.py -e qa -d -p /var/log/qa.pid (Run the qa configuration as a daemon with pid file)

So with the above commands we’ve instructed the application to load the specific config file we need. That the job of function: load_config(options.environment).

I’m skipping the server_start function for now to revisit it when we discuss controllers.

Notice:

"""When we are running in production mode we do not want tracebacks
    Additionally we want a redirect to the corresponding error page."""
    if options.environment == "production":
        cherrypy.config.update({'error_page.default': \
                                "%s/public/index.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.401': \
                                "%s/public/errors/401.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.404': \
                                "%s/public/errors/404.html" % (os.path.abspath("."))}
                              )
        cherrypy.config.update({'error_page.500': \
                                "%s/public/errors/500.html" % (os.path.abspath("."))}
                              )

    cherrypy.config.update({'log.error_file': \
                            "%s/logs/%s_error.log" % (os.path.abspath("."), options.environment)}
                          )
    cherrypy.config.update({'log.access_file': \
                            "%s/logs%s_access.log" % (os.path.abspath("."), options.environment)
                           }
                          )

This section is special. If we are running the production environment then we want to setup the error handling pages for our application. Under all environments we also need to setup the corresponding log files for access or error. Why didn’t this get included in the configuration files?

Notice the os.path.abspath. In the configuration files we can not import other modules. Only use python functions that don’t need importing. Since other scripts can start the deploy script it’s difficult to tell what the file path would be in the config files unless you specifically say “/PATH_TO_DIRECTORY/logs/production_access.log”. That however, is not very portable is it.

This rest of the file has the comments on what each part means.

Give it a try. Run two commands:
python ./deploy.py -e development
python ./deploy.py -e production

view them at http://localhost:7180/ and http://localhost:7182/ . Notice that both give you the index.html page and that both are running. Now go to http://localhost:7180/nothing and http://localhost:7182/nothing.

Now do you see how the different environment config files come into play. The error for the development app shows you the error (404) tracebacks, whereas the production shows you the 404 html page instead.

Probably the most important file a CherryPy app requires is the deploy script. As developers we need a way to build in one environment and run production in another. This can be incorporated in the deploy script fairly easily and once done we can forget about it. Being able to daemonize the CherryPy app leads to huge performance increases. The cherryd script is merely a basic working script which we can incorporate into our own deploy scripts.

In the next post I’ll show you how to incorporate RESTFul controllers and custom named routes into the deploy script for a well rounded application.

Tags: cherrypy, cherrypyd, script