Tech Ravings

In my belief that I could actually take some free time to explore new learning I set about last Monday night to set up a website that ran on a single port. I have never done this so I needed Apache to rewrite or proxy to that port so the application could work. It really was just a test application so I could learn how to do such things with apache.

Granted I have never set up a proxy server and found the warning posted all over every resource I read on the subject. Invariably yesterday I made a change that turned my personal web server into an open proxy; although I only realized this today. Never said I knew it all, other that JavaScript ;).

This morning I found that the one and only large website I run on my personal server was running a little slow I set about looking into it. That’s when I found out that I was processing about 9 requests per second on my box through the proxy. That is unheard of for this little 512M rdram box I’ve had from ‘98.

Of course at first I freaked not wanting all this traffic that was coming through, only thinking about how it would affect my network not to mention any hack attempts against my server. I tried desperately to revert all my changes to what they were before. To no avail. Apache kept serving an open proxy even with the old settings.

Then I noticed a trend in the proxies. A lot of the proxies seemed to be gaming affiliate systems. At that point I was all over turning this noobish episode into a honeypot. Seeings how one of my main considerations in setting up an affiliate system was to prevent fraud this was invaluable information on how gamers use proxies to gain money. I had a perfect opportunity to learn their signatures, if not their IP’s.

From my little bitty server I saw Alexa, Yahoo! get gamed. I saw click counts get inflated to several other sites. Even a few test the animosity of my new honeypot. Using the open proxy I accidentally created I became the hacker against it and sought to see if I could protect an application against this sort of gaming.

One of the not so striking lessons was that the user-agent was a standard browser user-agent *cough*FireFox*cough*. Although two requests per second from the same IPA with the same GET info over the total of five minutes, tends to lead you to believe that this wasn’t a person but a script bot instead.

If I’ve learned anything from Zookoda it’s that spammer busting is difficult, but can be done. I’ve had several “honeypot” blogs set up for a while to bust sploggers and comment spammers and that has been effective. Proxy spammers are just the next step in where I need to go.

I know I’m not a sys admin, but I want to learn. I also know an opportunity to crack a spammers ass when I see it as well!

I’m still plucking away at this project as time allows using CherryPy. This weekend I just started to use it and finally got a working environmental configuration system and then next step was to figure out if RESTful controllers could be used. As it turns out you can not only use RESTful controllers but have the additional benefit of format types as well (html, xml, json, etc).

This is due to the CherryPy RoutesDispatcher using the routes library. I’ve successfully setup dispatcher with RESTful routes and have tests to show that they work as planned.

I’ll do a follow up post with working examples this weekend.

Here’s my dilemma, I’m a developer working on code on my machine. Once I’m done I send the code to QA who then says yay or nay. Once I get a yay then the code goes into production on a live server. CherryPy has a site config for all apps as well as an app specific config, which pertains to the code I writing. Under the CherryPy documentation I would have to modify 3 different files in order to add 1 new model! That’s insane.

Additionally, I’m working on my computer (localhost) so I need the SQLObject connections to point to my machine. When I send my work to QA the db connections are completely different. Production environment, again different. Traces, ip address, etc, all different from the various environments. CherryPy 3 does have a “global” config for all apps as well as a “site” config pertaining to your app, but no where does it suggest setting up these environments. Outside of the “environment = whatever” in the config.

Do I really have to write my code on my machine, send it to QA, log into shell and make the modifications, get a yay and send to production, log into shell there and make modifications? Do this across twenty developers as well?

No…

I hope you see the catastrophe that could ensue because an out of the box framework doesn’t allow environments to be set up within an app. Likewise I do judge a framework by the lack of code duplication (DRY). Searching the documentation for CherryPy as well as many other sites I found no information concerning my requirements. So it was time to start “playing”

CherryPy does allow you to remain DRY and set up environments!
Takes a little work around though, but with a deploy script it’s easy.

First let’s create the scripts:

# Site Config (site_config.py)

[global]
server.protocol_version = "HTTP/1.1"
tools.proxy.on = True
tools.proxy.local = "X-Forwarded-Host"
tools.proxy.remote =  "X-Forwarded-For"
log.error_file = "logs/error.log"
log.access_file = "logs/access.log"

#Development (development.py)

[global]
server.socket_host = "127.0.0.1"
server.socket_port = 8082
log.screen = True
tools.log_tracebacks.on = True

#QA (qa.py)

[global]
server.socket_host = "127.0.0.1"
server.socket_port = 8083
log.screen = True
tools.log_tracebacks.on = False

#routes.py (routes.py)

[/]

#whatever

#deploy script (deploy.py)

import sys
import cherrypy

import models
from models.base import Base
#add your additional models as well from models folder

if __name__ == '__main__':
    cherrypy.config.update(sys.argv[1] + ".py")
    cherrypy.config.update(sys.argv[2] + ".py")
    cherrypy.tree.mount(Base(), script_name="/", config="routes.py")
    cherrypy.engine.start()

So in the command line when you call:
>>python deploy.py site_config.py development
what happens is:

1. Loads the models as classes.
2. cherrypy loads the site wide config variables (site_config.py). These pertain to all the environments.
3. chrrypy loads, adds or overwrites, all the environment settings. Specific to development (locally), qa (hope it works), production (umm better work)
4. Mounts each model to the correct path as found in the routes.py file which contains settings for each route.

Of course it’s not completely DRY. As each new model is made you do have to update the deploy script as well as the routes.py, but it’s a lot better than doing all your environment files.

I do feel CherryPy would be much more powerful if out of the box we could set up situations like this as well as a plain routes file (Dispatcher) so hacks like this aren’t needed. However this hack at least kept me considering using CherryPy.