Twitters downtime is infamous and I am fine with that. However when my browser starts crashing because of mistakes I take a huge offense to that. So now Twitter is throwing an SSL certificate error because they haven’t configured it correctly.

Do you see the localhost.localdomain on twitter.com?!?

What’s the point of this site, really.

This is due to the CherryPy RoutesDispatcher using the routes library. I’ve successfully setup dispatcher with RESTful routes and have tests to show that they work as planned.

I’ll do a follow up post with working examples this weekend.

Additionally, I’m working on my computer (localhost) so I need the SQLObject connections to point to my machine. When I send my work to QA the db connections are completely different. Production environment, again different. Traces, ip address, etc, all different from the various environments. CherryPy 3 does have a “global” config for all apps as well as a “site” config pertaining to your app, but no where does it suggest setting up these environments. Outside of the “environment = whatever” in the config.

Do I really have to write my code on my machine, send it to QA, log into shell and make the modifications, get a yay and send to production, log into shell there and make modifications? Do this across twenty developers as well?

No…

I hope you see the catastrophe that could ensue because an out of the box framework doesn’t allow environments to be set up within an app. Likewise I do judge a framework by the lack of code duplication (DRY). Searching the documentation for CherryPy as well as many other sites I found no information concerning my requirements. So it was time to start “playing”

CherryPy does allow you to remain DRY and set up environments!
Takes a little work around though, but with a deploy script it’s easy.

First let’s create the scripts:

# Site Config (site_config.py)

[global]
server.protocol_version = "HTTP/1.1"
tools.proxy.on = True
tools.proxy.local = "X-Forwarded-Host"
tools.proxy.remote =  "X-Forwarded-For"
log.error_file = "logs/error.log"
log.access_file = "logs/access.log"

#Development (development.py)

[global]
server.socket_host = "127.0.0.1"
server.socket_port = 8082
log.screen = True
tools.log_tracebacks.on = True

#QA (qa.py)

[global]
server.socket_host = "127.0.0.1"
server.socket_port = 8083
log.screen = True
tools.log_tracebacks.on = False

#routes.py (routes.py)

[/]

#whatever

#deploy script (deploy.py)

import sys
import cherrypy

import models
from models.base import Base
#add your additional models as well from models folder

if __name__ == '__main__':
    cherrypy.config.update(sys.argv[1] + ".py")
    cherrypy.config.update(sys.argv[2] + ".py")
    cherrypy.tree.mount(Base(), script_name="/", config="routes.py")
    cherrypy.engine.start()

So in the command line when you call:
>>python deploy.py site_config.py development
what happens is:

1. Loads the models as classes.
2. cherrypy loads the site wide config variables (site_config.py). These pertain to all the environments.
3. chrrypy loads, adds or overwrites, all the environment settings. Specific to development (locally), qa (hope it works), production (umm better work)
4. Mounts each model to the correct path as found in the routes.py file which contains settings for each route.

Of course it’s not completely DRY. As each new model is made you do have to update the deploy script as well as the routes.py, but it’s a lot better than doing all your environment files.

I do feel CherryPy would be much more powerful if out of the box we could set up situations like this as well as a plain routes file (Dispatcher) so hacks like this aren’t needed. However this hack at least kept me considering using CherryPy.

While displeased with this, I’m used to it. It also means that serious research is needed in order to work with the software. Being fully involved with Pylons, albeit new. I didn’t expect it to be RESTful; nor did I expect it to allow different extension type within an action. I am actually impressed with the Ruby on Rails RESTful implementation and was very surprised to find that Pylons itself has RESTful implementations itself.

However, my project requires me to call one action in the extension with “.js” for JSON and another extension of “.rss” for that version. The Pylons RESTful documentation lacks for this. I couldn’t find any other reference for this on the web either. While I don’t know for sure when Pylons added RESTful resources the documentation lacks any examples that I found for using different formats, so here it goes.

The Pylons wiki for RESTful controllers is found here: How map.resource enables controllers as services. It explains everything you want to know about how to use REST within Pylons. It does not explain my predicament on how to make the index action display JSON format or RSS format.

After a little testing and discovery I found out that you can, within an action, produce different views specific to the format of the action.  Given my requirements here is how I was able to use one RESTful action for many formats, i.e (.js, .gif, .rss, .rtf, etc).

1. Create your RESTful controller using the command: paster restcontroller <singular name> <plural name>
This is no different than the instructions in the map.resource link above.

2. In the action create an if…elsif…else to actually control the format:
def index(self, format=”html”):
if format == “js”:
response.headers[’Content-Type’] = ‘application/javascript’
return render(’/pubscript/indexjs.mako’)
elif format == “xml”:

response.headers[’Content-Type’] = ‘application/xml’
return render(’/pubscript/indexxml.mako’)
else:
abort(404, ‘Page Not Found’)

So as the index definition provides that any index call to the controller does perform the action, only the .js returns an js (JSON) file or an .xml renders an xml file format. Any other format or none provided sends a 404 File Not Found error. Now I had complete control of what the format is and control of any unexpected calls.

Pylons has evoled to the presedence set by Ruby on Rails in that RESTful controllers can be used. Even though the default format for Pylons is HTML you can detect such as present different formats based upon such.  Being new to the Pylon framework itself I wasn’t expecting this functionality. Discovering that it has it, and is acting on the different formats, I do believe that it has emerged as a mature framework that embraces best practices of web application programming.

Still highly complicated in setting up and a huge learning curve with little correct documentation though, don’t get me wrong there.

The ability in Python to explicitly and easily tell the interpreter to perform other actions on a called block of executable code in one line without changing any underlying logic is powerful. However the actually documentation or tutorials are vague on the actual inner workings of what goes on. How do you use the decorated function? You passed in arguments, are they available? I passed in a class, do I have access to it?

They answer is yes! Let’s set up a test case to walk through all of this and explain along the way.

1. I need a program that reads from a database x amount of records, and then converts them to the appropriate class object. Each object is then feed to a Queue for processing. Well that’s easy enough isn’t it.

2. Now as the Queue is iterated the object is removed from the Queue and an object method is called. Based upon the result of the method: place in the completed Queue; place in the failed Queue; place back into the processing Queue.

3. Run this until the processing Queue is empty.

A fairly simple program indeed. The complexity we need without Python decorators and maintaining OOP are two classes: A jobs class to load the special Queues, hold methods for switching Queues; the item class to hold the database information. Maintaining OOP means each class should not know about what happens to the other and each should perform only on itself.

Problem, the jobs class would need to know the result status of the item instance to correctly place it in the correct Queue. Or we have to pass in the different Queue to the item object during construction to be placed. Thereby locking in the item to the program and not very re-usable. Python decorators short cut this, we get to drop the jobs class and our code is more re-usable.

from Queue import Queue
import random

print "loading"
job_que = Queue()
failed_que = Queue()
completed_que = Queue()
print "queue complete"

def global_job(fn):
    def wrappedFN(arg1, arg2):
        eval(arg2 + "_que").put(arg1, block = True)
        result = fn(arg1, arg2)
        print "put into job complete"
        return result

    return wrappedFN

class SomeItem(object):

    _name = ""

    def __init__(self):
        self._name = "Me"

    def run_job(self):
        print ""
        num = random.sample([0, 1, 2],  1)
        if num == [0]:
            self._generic_job("failed")

        if num == [1]:
            self._generic_job("job")

        if num == [2]:
            self._generic_job("completed")

    @global_job
    def _generic_job(self, type):
        print "job " + type

    def load_queue():
        print "loading queue"
        for i in range(10):
            job_que.put(SomeItem(), block = True)

print "Starting Main"

load_queue()
print "finished loading jobs"

while not job_que.empty():
    print "processing size: " + str(job_que.qsize())
    for i in range(job_que.qsize()):
        job = job_que.get(i)      #job (class object) pulled the instance out of the Queue,
        job.run_job()
        job_que.task_done()

        print "processing size: " + str(job_que.qsize())
        print "completed size: " + str(completed_que.qsize())
        print "failed size: " + str(failed_que.qsize())
        job_que.join()

So there is the code to make this work and here’s how it does.

1. Load the processing Queue with the instances of “SomeItem”
2. Iterate each item in the Queue and remove the object from said Queue
3. Call object method “run_job”
4. The result within “run_job” can be one of three result: “failed”, “completed”, “job”.
5. run_job then calls self class method _generic_queue
6. Python decorator generic_job intercepts this call
7. _generic_queue gets executed
8. Python decorator global_job get executed and places in the correct Queue
9. Rinse and repeat job Queue until empty

All very cut and dry huh? Yep this is where my research into Python decorators ended. I mean I sent arguments to the class method that got intercepted and I need to perform an action in that method yet how does the decorator REALLY work?

Let’s start from the self._generic_job call. To keep the program DRY a singular function is called with arguments to process the results. In order to maintain the DRY’ness an argument of the result needs to sent. Being a class method we also need to def _generic_job with the first argument as “self”. The self reference is very import to the decorator. The second argument as the type of Queue we will be placing in.

We’ve called the self._generic_job, yet the global global_job has been wrapped around the _generic_job. So following the order of operation when we call self._generic_job(”completed”) what the interpreter did was global_job(<pointer>_generic_job). Awesomeness!

Taking a look at the actual decorator we can find out how to actually use it to process not only the class/function but also the arguments passed! Yes I made this difficult by passing a class instance over sending in a functiton! So let’s move on to the decorator itself.

def global_job(fn):
    def wrappedFN(arg1, arg2):
        eval(arg2 + "_que").put(arg1, block = True)
        result = fn(arg1, arg2)
        print "put into job complete"
        return result

    return wrappedFN

So the global global_job has intercepted the class method and wrapped itself around such. In the definition of global_job the argument of fn actually refers to the callback of the called object. global_job in and of itself returns a pointer to another nested function within itself which process the arguments. This nested function then takes the callback and can act on such along with the arguments to perform such.

I know, I know that was a very high level description. Just for brevity I swear. I got confused just writing it so here’s the actual processing.

self._generic_job was called with one argument. The actual definition takes two arguments: self; type. Both of these were passed to the global_job method.

So the global_job actually received the <object>.generic_job pointer reference. That would be the “fn”. The last line in global_job returns the reference to wrappedFN. Which basically means execute wrappedFN and here’s the arguments as well. So wrappedfn executes before the return.

You’ll notice that the wrappedFN is defined with two arguments: arg1, arg2. This is only for this example and need not be so for yours(*args, **kwargs) works also. But what are these arguments really? Really they are how the base function gets executed.

Keeping in mind that “fn” declared in generic_job is accessible to wrappedFN and that “fn” is the callback, then arg1 and arg2 are the method arguments. So in this case:

fn = SomeItem (instance)._generic_job
arg1 = self._generic_job(self<pointer>……
arg2 = self._generic_job(…, type)

So low and behold that in the def _generic_job method takes self as the first argument (naturally) and it is really the arg1 of the wrappedFN. Easy enough to use it as the reference pointer to the instance and of which can use as a full blown instance. Then arg2 must be the passed string argument of type. In this case we pass through the eval statement to place into the correct Queue as we need.

So we have the three key pieces: callback function; class instance; and the additional arguments it easy to place into the correct Queue. Yet how does the _generic_gob actaully get executed? That happens in line “result = fn(arg1, arg2)”. Keeping in mind that translates to SomeItem._generic_job(self, type). SomeItem isn’t a singelton object, it’s the actual job we pulled from the Queue and the self argument is actually the pointer referencing it.

Using Python decorators is a powerful option for extending programs but there aren’t any clear tutorials or articles explaining the way arguments and callback functions get executed. This article does just that as the decorator function does have access to the callback function and arguments passed. With just one line of code the call back function can be executed as well.

Two weeks ago I was put on a Twitter project written in Python and had a requirement of specifically using decorators for logging. I didn’t know anything about this strange functionality and was given a very brief overview of what they were. After the project meeting I hit the interwebs and played with some toy code to figure out just how they work I finally got it figured out to the point that “it worked”.

I was still left feeling like I didn’t understand what Python decorators were and really what the need for them are. Even worse I questioned the need for them anyhow. A short case

def global_some_function(fn):
def wrappedfn(*args, **kwargs):
#do stuff here
return wrappedfn

class base(object):
@global_function
def base_method(self):
pass

class ChildClass(base):
def call_base(self):
base.base_method()

So this is were I really felt like I was violating a bunch of programming rules. I mean I call ChildClass.call_base(), which then calls it’s parent class base_method. Yet that is intercepted by the Python decorator and global global_some_function. So effectively using this decorator the child class is not totally object oriented, and I would feel a lot more happier just having the Base.base_method class do the work instead of the decorator.

Feeling un-satisfied, confused, and like I had dirty code I pressed on and completed the project. The next project consisted of taking a Queue of classes, performing an action on them in a iteration, and then sorting based upon the result of the actions. Almost immediately I realized that this is were the power and flexibility of Python decorators comes in.

I mean based up the result of the action the object may be put back in the Queue, or it may go to another Queue for further processing. The object shouldn’t care about what Queue it’s in, so I needed a way to wrap in this functionality shared by by all Queues. Hitting the interwebs to truly understand what and how Python decorators work yeilded many examples but not a clear cut understanding of how decorators work.

So here my friends is how I figured out what, and how Python decorators work.

First and foremost when I say Python decorators I am not referring to the Decorator Design Pattern. Although I have found to be effectively the same. The Python decorator being a symbolic tag signifying to the Python interpreter that the following method is to be wrapped within an function or class for further processing. So while the Python decorator creates that link to another function, the other function then becomes the Decorated Design Pattern following that pattern.

def foo(fn):
print “Hello”

@foo
def bar(some_arg):
print some_arg

bar(”world”)

What would you expect with the above example? I initially expected the result to print foo(bar(”world”)), however that was not the case as only “Hello” was the result. This tripped me up for sometime and it’s were the online examples and documentation lacked. The reason is due to the decorator function “foo”. There are no functionalities to actually execute the original function “bar”. This pretty much where all the tutorials left off; if you’ve read this far you probably want me to get to the point now huh?

As I stated earlier Python decorators are not design patterns. They are merely a signal to the Python interpreter to wrap the call in other signified functions. There are at least two ingrained decorators that Python ships with: @staticmethod; @classmethod, but as programmers we’ll be dealing with our own custom decorators.

Using the Python decorator is relatively simple. Just declare the decorator with the @ symbol preceding the method to be wrapped. So:

def foo():
print “Hello World”

@foo
def bar():
pass

bar()

To the Python interpreter bar get wrapped with the function foo whenever the method bar is executed. So calling bar and following the order of exectution look s like this foo(bar). There is certainly no reason not to use only one to make if even more flexible:

def foo(fn):
print “world”

def bar(fn):
print “Hello”

@foo
@bar
def something():
print “I’m alive”

So executing something() would follow the order of execution of foo(bar(something)) would print:

Hello
World

but strangely “I’m alive” did not print! Why is that? It’s because wrapping the something function only sends the pointer to the function to the decorators, they are in charge of actually executing the received function. The other huge problem is how to deal with function arguements passed, and even bigger than that how to pass classes to global decorators while retaining the object.

Even though the name implies, Python decorators are not the design pattern. However usage is the same. Signify to the interpreter tho wrap this executable code within this executable code by using the @ character preceding the wrapping code block and preceding the code block to be wrapped.

This post is merely an introduction to what the Python decorators are and how they can be used. I’m following up this post with how to pass arguments through these decorators to really gain the power afforded by the decorator and to make your libraries more extend able. This post is here.

So started the experiment.

Using the traditional blogging methods: make a post; have it ping news aggregates, I created a link for than. I let it run for an hour to see if anything click the UrlBrief link and thus being recorded. The only two hits were from my Wordpress blog, none others.

After an hour and to the minute I made a Tweet with a fresh link for Come Adventuring. Absolutely no inclusion in any post. Even though I’m making this post twenty minutes early I have recorded 5 click thru’s to the post. One being googlebot.

After spending years in SEO/SEM I have become familiar with the traditional methods of code cleanliness, link campaigns, directory inclusions, ads, etc. I have never seen a faster generation of traffic than making a tweet on Twitter. This is truly amazing as Google must be sitting on Twitter and scraping.

Now keep in mind that UrlBrief also records for you to see where clicks have come from which make it very cool. So just as a grins and giggles idea I went back an hour later. Just to see. I about dropped out of my chair to see that google bot had already visited the links via the urlbrief url, and twice.

Not there are two ways that google bot could have done that: one is through the natural indexing of Come Adventuring, or; through Twitter. I’m guessing Twitter because Come Adventuring is way too new. Although it has a PR of 1, seriously it just doesn’t happen like that. Even by coincidence.

So I have another test running just in case. The Come Adventuring link above is url brief encoded. I’m going to let it run for an hour on this blog which Google has ranked spanked. I’ll post the details.

 Then I’m going to make a tweet about it. Then post after an hour.

I’ve spent years doing SEO/SEM and have never seen a quicker response from a SE. I can certainly attribute it to Twitter because, frankly, I don’t market this blog.

I found Firewall Script that can be used specifically to safe gaurd my personal site with dis-regard to all the others. I have not downloaded it yet but from the features it looks to be exactly what I was looking for. Protection from sql injection, XSS attacks, even DDOS protection. Not that this blog would really ever need that I do run other sites that would really require it if the server can’t handle it.

I wanted to get this news out as soon as possible because several of us Wordpress and site owners are experiencing a rise in intrusion attempts. I’m gonna give it a try as soon as possible and I hope that this may help you as well.

 I did find today that in fact a database injection had been done and the iframe was there. My mistake was that I only hit the home page and the iframe was on the previous page. The iframe source refers to 61.155.8.157 and after some examination of the obfuscated JavaScript, redirections were discovered. I’ve contacted the server abuse department and will follow up if these people are not shut-down.

Welcome to the interwebs. This is the second time I have been hijacked in the same way. This is a new theme that I have gone through and can not find malicious php code, and since it’s in the admin section I can only assume that it’s a Wordpress security problem. This however is the second time I’ve found hijacked stuff in my database!

While I have gone through every line of code in both this theme and the access logs for this site I can not find how this is happening. Which make me very cranky as it’s happening.

My apologies to anyone who has gotten alerts. Systems are in place to track and prevent these from happening anymore. I hate Wordpress 2.5 (I just set up a site using it) but if it’s a 2.3 problem I’ll have to upgrade.