One of the problems of learning Python decorators was figuring out how to not only pass arguments and use them, but also how to pass class objects to decorators and make them functional. I my post about Python Decorators, Wrapped with Goodness I introduced Python decorators. I left off with that article on actually how to pass through arguments for use in the decorator. This was intentional because, here comes another long post. 
The ability in Python to explicitly and easily tell the interpreter to perform other actions on a called block of executable code in one line without changing any underlying logic is powerful. However the actually documentation or tutorials are vague on the actual inner workings of what goes on. How do you use the decorated function? You passed in arguments, are they available? I passed in a class, do I have access to it?
They answer is yes! Let’s set up a test case to walk through all of this and explain along the way.
1. I need a program that reads from a database x amount of records, and then converts them to the appropriate class object. Each object is then feed to a Queue for processing. Well that’s easy enough isn’t it.
2. Now as the Queue is iterated the object is removed from the Queue and an object method is called. Based upon the result of the method: place in the completed Queue; place in the failed Queue; place back into the processing Queue.
3. Run this until the processing Queue is empty.
A fairly simple program indeed. The complexity we need without Python decorators and maintaining OOP are two classes: A jobs class to load the special Queues, hold methods for switching Queues; the item class to hold the database information. Maintaining OOP means each class should not know about what happens to the other and each should perform only on itself.
Problem, the jobs class would need to know the result status of the item instance to correctly place it in the correct Queue. Or we have to pass in the different Queue to the item object during construction to be placed. Thereby locking in the item to the program and not very re-usable. Python decorators short cut this, we get to drop the jobs class and our code is more re-usable.
from Queue import Queue
import random
print "loading"
job_que = Queue()
failed_que = Queue()
completed_que = Queue()
print "queue complete"
def global_job(fn):
def wrappedFN(arg1, arg2):
eval(arg2 + "_que").put(arg1, block = True)
result = fn(arg1, arg2)
print "put into job complete"
return result
return wrappedFN
class SomeItem(object):
_name = ""
def __init__(self):
self._name = "Me"
def run_job(self):
print ""
num = random.sample([0, 1, 2], 1)
if num == [0]:
self._generic_job("failed")
if num == [1]:
self._generic_job("job")
if num == [2]:
self._generic_job("completed")
@global_job
def _generic_job(self, type):
print "job " + type
def load_queue():
print "loading queue"
for i in range(10):
job_que.put(SomeItem(), block = True)
print "Starting Main"
load_queue()
print "finished loading jobs"
while not job_que.empty():
print "processing size: " + str(job_que.qsize())
for i in range(job_que.qsize()):
job = job_que.get(i) #job (class object) pulled the instance out of the Queue,
job.run_job()
job_que.task_done()
print "processing size: " + str(job_que.qsize())
print "completed size: " + str(completed_que.qsize())
print "failed size: " + str(failed_que.qsize())
job_que.join()
So there is the code to make this work and here’s how it does.
1. Load the processing Queue with the instances of “SomeItem”
2. Iterate each item in the Queue and remove the object from said Queue
3. Call object method “run_job”
4. The result within “run_job” can be one of three result: “failed”, “completed”, “job”.
5. run_job then calls self class method _generic_queue
6. Python decorator generic_job intercepts this call
7. _generic_queue gets executed
8. Python decorator global_job get executed and places in the correct Queue
9. Rinse and repeat job Queue until empty
All very cut and dry huh? Yep this is where my research into Python decorators ended. I mean I sent arguments to the class method that got intercepted and I need to perform an action in that method yet how does the decorator REALLY work?
Let’s start from the self._generic_job call. To keep the program DRY a singular function is called with arguments to process the results. In order to maintain the DRY’ness an argument of the result needs to sent. Being a class method we also need to def _generic_job with the first argument as “self”. The self reference is very import to the decorator. The second argument as the type of Queue we will be placing in.
We’ve called the self._generic_job, yet the global global_job has been wrapped around the _generic_job. So following the order of operation when we call self._generic_job(”completed”) what the interpreter did was global_job(<pointer>_generic_job). Awesomeness!
Taking a look at the actual decorator we can find out how to actually use it to process not only the class/function but also the arguments passed! Yes I made this difficult by passing a class instance over sending in a functiton! So let’s move on to the decorator itself.
def global_job(fn):
def wrappedFN(arg1, arg2):
eval(arg2 + "_que").put(arg1, block = True)
result = fn(arg1, arg2)
print "put into job complete"
return result
return wrappedFN
So the global global_job has intercepted the class method and wrapped itself around such. In the definition of global_job the argument of fn actually refers to the callback of the called object. global_job in and of itself returns a pointer to another nested function within itself which process the arguments. This nested function then takes the callback and can act on such along with the arguments to perform such.
I know, I know that was a very high level description. Just for brevity I swear. I got confused just writing it so here’s the actual processing.
self._generic_job was called with one argument. The actual definition takes two arguments: self; type. Both of these were passed to the global_job method.
So the global_job actually received the <object>.generic_job pointer reference. That would be the “fn”. The last line in global_job returns the reference to wrappedFN. Which basically means execute wrappedFN and here’s the arguments as well. So wrappedfn executes before the return.
You’ll notice that the wrappedFN is defined with two arguments: arg1, arg2. This is only for this example and need not be so for yours(*args, **kwargs) works also. But what are these arguments really? Really they are how the base function gets executed.
Keeping in mind that “fn” declared in generic_job is accessible to wrappedFN and that “fn” is the callback, then arg1 and arg2 are the method arguments. So in this case:
fn = SomeItem (instance)._generic_job
arg1 = self._generic_job(self<pointer>……
arg2 = self._generic_job(…, type)
So low and behold that in the def _generic_job method takes self as the first argument (naturally) and it is really the arg1 of the wrappedFN. Easy enough to use it as the reference pointer to the instance and of which can use as a full blown instance. Then arg2 must be the passed string argument of type. In this case we pass through the eval statement to place into the correct Queue as we need.
So we have the three key pieces: callback function; class instance; and the additional arguments it easy to place into the correct Queue. Yet how does the _generic_gob actaully get executed? That happens in line “result = fn(arg1, arg2)”. Keeping in mind that translates to SomeItem._generic_job(self, type). SomeItem isn’t a singelton object, it’s the actual job we pulled from the Queue and the self argument is actually the pointer referencing it.
Using Python decorators is a powerful option for extending programs but there aren’t any clear tutorials or articles explaining the way arguments and callback functions get executed. This article does just that as the decorator function does have access to the callback function and arguments passed. With just one line of code the call back function can be executed as well.
