Tech Ravings

One of the problems of learning Python decorators was figuring out how to not only pass arguments and use them, but also how to pass class objects to decorators and make them functional. I my post about Python Decorators, Wrapped with Goodness I introduced Python decorators. I left off with that article on actually how to pass through arguments for use in the decorator. This was intentional because, here comes another long post.

The ability in Python to explicitly and easily tell the interpreter to perform other actions on a called block of executable code in one line without changing any underlying logic is powerful. However the actually documentation or tutorials are vague on the actual inner workings of what goes on. How do you use the decorated function? You passed in arguments, are they available? I passed in a class, do I have access to it?

They answer is yes! Let’s set up a test case to walk through all of this and explain along the way.

1. I need a program that reads from a database x amount of records, and then converts them to the appropriate class object. Each object is then feed to a Queue for processing. Well that’s easy enough isn’t it.

2. Now as the Queue is iterated the object is removed from the Queue and an object method is called. Based upon the result of the method: place in the completed Queue; place in the failed Queue; place back into the processing Queue.

3. Run this until the processing Queue is empty.

A fairly simple program indeed. The complexity we need without Python decorators and maintaining OOP are two classes: A jobs class to load the special Queues, hold methods for switching Queues; the item class to hold the database information. Maintaining OOP means each class should not know about what happens to the other and each should perform only on itself.

Problem, the jobs class would need to know the result status of the item instance to correctly place it in the correct Queue. Or we have to pass in the different Queue to the item object during construction to be placed. Thereby locking in the item to the program and not very re-usable. Python decorators short cut this, we get to drop the jobs class and our code is more re-usable.

from Queue import Queue
import random

print "loading"
job_que = Queue()
failed_que = Queue()
completed_que = Queue()
print "queue complete"

def global_job(fn):
    def wrappedFN(arg1, arg2):
        eval(arg2 + "_que").put(arg1, block = True)
        result = fn(arg1, arg2)
        print "put into job complete"
        return result

    return wrappedFN

class SomeItem(object):

    _name = ""

    def __init__(self):
        self._name = "Me"

    def run_job(self):
        print ""
        num = random.sample([0, 1, 2],  1)
        if num == [0]:
            self._generic_job("failed")

        if num == [1]:
            self._generic_job("job")

        if num == [2]:
            self._generic_job("completed")

    @global_job
    def _generic_job(self, type):
        print "job " + type

    def load_queue():
        print "loading queue"
        for i in range(10):
            job_que.put(SomeItem(), block = True)

print "Starting Main"

load_queue()
print "finished loading jobs"

while not job_que.empty():
    print "processing size: " + str(job_que.qsize())
    for i in range(job_que.qsize()):
        job = job_que.get(i)      #job (class object) pulled the instance out of the Queue,
        job.run_job()
        job_que.task_done()

        print "processing size: " + str(job_que.qsize())
        print "completed size: " + str(completed_que.qsize())
        print "failed size: " + str(failed_que.qsize())
        job_que.join()

So there is the code to make this work and here’s how it does.

1. Load the processing Queue with the instances of “SomeItem”
2. Iterate each item in the Queue and remove the object from said Queue
3. Call object method “run_job”
4. The result within “run_job” can be one of three result: “failed”, “completed”, “job”.
5. run_job then calls self class method _generic_queue
6. Python decorator generic_job intercepts this call
7. _generic_queue gets executed
8. Python decorator global_job get executed and places in the correct Queue
9. Rinse and repeat job Queue until empty

All very cut and dry huh? Yep this is where my research into Python decorators ended. I mean I sent arguments to the class method that got intercepted and I need to perform an action in that method yet how does the decorator REALLY work?

Let’s start from the self._generic_job call. To keep the program DRY a singular function is called with arguments to process the results. In order to maintain the DRY’ness an argument of the result needs to sent. Being a class method we also need to def _generic_job with the first argument as “self”. The self reference is very import to the decorator. The second argument as the type of Queue we will be placing in.

We’ve called the self._generic_job, yet the global global_job has been wrapped around the _generic_job. So following the order of operation when we call self._generic_job(”completed”) what the interpreter did was global_job(<pointer>_generic_job). Awesomeness!

Taking a look at the actual decorator we can find out how to actually use it to process not only the class/function but also the arguments passed! Yes I made this difficult by passing a class instance over sending in a functiton! So let’s move on to the decorator itself.

def global_job(fn):
    def wrappedFN(arg1, arg2):
        eval(arg2 + "_que").put(arg1, block = True)
        result = fn(arg1, arg2)
        print "put into job complete"
        return result

    return wrappedFN

So the global global_job has intercepted the class method and wrapped itself around such. In the definition of global_job the argument of fn actually refers to the callback of the called object. global_job in and of itself returns a pointer to another nested function within itself which process the arguments. This nested function then takes the callback and can act on such along with the arguments to perform such.

I know, I know that was a very high level description. Just for brevity I swear. I got confused just writing it so here’s the actual processing.

self._generic_job was called with one argument. The actual definition takes two arguments: self; type. Both of these were passed to the global_job method.

So the global_job actually received the <object>.generic_job pointer reference. That would be the “fn”. The last line in global_job returns the reference to wrappedFN. Which basically means execute wrappedFN and here’s the arguments as well. So wrappedfn executes before the return.

You’ll notice that the wrappedFN is defined with two arguments: arg1, arg2. This is only for this example and need not be so for yours(*args, **kwargs) works also. But what are these arguments really? Really they are how the base function gets executed.

Keeping in mind that “fn” declared in generic_job is accessible to wrappedFN and that “fn” is the callback, then arg1 and arg2 are the method arguments. So in this case:

fn = SomeItem (instance)._generic_job
arg1 = self._generic_job(self<pointer>……
arg2 = self._generic_job(…, type)

So low and behold that in the def _generic_job method takes self as the first argument (naturally) and it is really the arg1 of the wrappedFN. Easy enough to use it as the reference pointer to the instance and of which can use as a full blown instance. Then arg2 must be the passed string argument of type. In this case we pass through the eval statement to place into the correct Queue as we need.

So we have the three key pieces: callback function; class instance; and the additional arguments it easy to place into the correct Queue. Yet how does the _generic_gob actaully get executed? That happens in line “result = fn(arg1, arg2)”. Keeping in mind that translates to SomeItem._generic_job(self, type). SomeItem isn’t a singelton object, it’s the actual job we pulled from the Queue and the self argument is actually the pointer referencing it.

Using Python decorators is a powerful option for extending programs but there aren’t any clear tutorials or articles explaining the way arguments and callback functions get executed. This article does just that as the decorator function does have access to the callback function and arguments passed. With just one line of code the call back function can be executed as well.

This post is really the first part of two. Here I’ll talk about what is this Python decorator that everyone is raving about. How does it work. What is it really. The second part I’ll show how to pass arguments and classes between Python decorators so that not only are the decorators executed but the original code block is executed.

Two weeks ago I was put on a Twitter project written in Python and had a requirement of specifically using decorators for logging. I didn’t know anything about this strange functionality and was given a very brief overview of what they were. After the project meeting I hit the interwebs and played with some toy code to figure out just how they work I finally got it figured out to the point that “it worked”.

I was still left feeling like I didn’t understand what Python decorators were and really what the need for them are. Even worse I questioned the need for them anyhow. A short case

def global_some_function(fn):
def wrappedfn(*args, **kwargs):
#do stuff here
return wrappedfn

class base(object):
@global_function
def base_method(self):
pass

class ChildClass(base):
def call_base(self):
base.base_method()

So this is were I really felt like I was violating a bunch of programming rules. I mean I call ChildClass.call_base(), which then calls it’s parent class base_method. Yet that is intercepted by the Python decorator and global global_some_function. So effectively using this decorator the child class is not totally object oriented, and I would feel a lot more happier just having the Base.base_method class do the work instead of the decorator.

Feeling un-satisfied, confused, and like I had dirty code I pressed on and completed the project. The next project consisted of taking a Queue of classes, performing an action on them in a iteration, and then sorting based upon the result of the actions. Almost immediately I realized that this is were the power and flexibility of Python decorators comes in.

I mean based up the result of the action the object may be put back in the Queue, or it may go to another Queue for further processing. The object shouldn’t care about what Queue it’s in, so I needed a way to wrap in this functionality shared by by all Queues. Hitting the interwebs to truly understand what and how Python decorators work yeilded many examples but not a clear cut understanding of how decorators work.

So here my friends is how I figured out what, and how Python decorators work.

First and foremost when I say Python decorators I am not referring to the Decorator Design Pattern. Although I have found to be effectively the same. The Python decorator being a symbolic tag signifying to the Python interpreter that the following method is to be wrapped within an function or class for further processing. So while the Python decorator creates that link to another function, the other function then becomes the Decorated Design Pattern following that pattern.

def foo(fn):
print “Hello”

@foo
def bar(some_arg):
print some_arg

bar(”world”)

What would you expect with the above example? I initially expected the result to print foo(bar(”world”)), however that was not the case as only “Hello” was the result. This tripped me up for sometime and it’s were the online examples and documentation lacked. The reason is due to the decorator function “foo”. There are no functionalities to actually execute the original function “bar”. This pretty much where all the tutorials left off; if you’ve read this far you probably want me to get to the point now huh?

As I stated earlier Python decorators are not design patterns. They are merely a signal to the Python interpreter to wrap the call in other signified functions. There are at least two ingrained decorators that Python ships with: @staticmethod; @classmethod, but as programmers we’ll be dealing with our own custom decorators.

Using the Python decorator is relatively simple. Just declare the decorator with the @ symbol preceding the method to be wrapped. So:

def foo():
print “Hello World”

@foo
def bar():
pass

bar()

To the Python interpreter bar get wrapped with the function foo whenever the method bar is executed. So calling bar and following the order of exectution look s like this foo(bar). There is certainly no reason not to use only one to make if even more flexible:

def foo(fn):
print “world”

def bar(fn):
print “Hello”

@foo
@bar
def something():
print “I’m alive”

So executing something() would follow the order of execution of foo(bar(something)) would print:

Hello
World

but strangely “I’m alive” did not print! Why is that? It’s because wrapping the something function only sends the pointer to the function to the decorators, they are in charge of actually executing the received function. The other huge problem is how to deal with function arguements passed, and even bigger than that how to pass classes to global decorators while retaining the object.

Even though the name implies, Python decorators are not the design pattern. However usage is the same. Signify to the interpreter tho wrap this executable code within this executable code by using the @ character preceding the wrapping code block and preceding the code block to be wrapped.

This post is merely an introduction to what the Python decorators are and how they can be used. I’m following up this post with how to pass arguments through these decorators to really gain the power afforded by the decorator and to make your libraries more extend able. This post is here.