Managing Database Migrations in Python, Name Me!

January 27th, 2009 by ScottK | No Comments | Filed in Python

I’ve devouted a great deal of time working with MVx frameworks in several languages: ColdFusion; PHP; Python; ASP .Net MVC; and of course Ruby on Rails. Clearly each has it’s own excellencies for what the app needs to accomplish, but when it comes to a well rounded application framework for building an application Ruby on Rails does the job. Albeit there are things I don’t really like about Ruby on Rails as well.

My “language” of choice is Python, yet I find the frameworks sub-ordinate in many areas for managing the apps. Especially in the area of database management. Clearly the ability to run db:migrate in Ruby on Rails to update your database to the current version via file naming is powerful, yet in case of trouble being able to rollback those changes is immensely so. The Python frameworks lack this ability and in managing several Python applications just to make the changes during a deploy is time consuming. God forbit that something goes wrong and I need to revert the changes (hasn’t happened yet, now I don’t have to worry).

So for those not familiar with the Ruby on Rails migration system let me explain. As you need changes to your database you place those changes in the “self.up” section of the file. Typically what you changed on the up in those files you also revert in the “self.down” section of the same file. Another crucial benefit is that you don’t need to know your database syntax, Ruby on Rails does it for you in their formatting, being that Ruby on Rails has database adapters.

db:migrate does this in one command, up/down, as well. If you have 20 migration files it’s one command versus 20 MySQL cli commands, and twenty password entries. It’s extremely powerful, I’ll admit, and I so need it for managing a few Python apps using CherryPy.

None of the Python apps I’ve used does anything close to this so I created one. Keeping in mind that this is totally independant from design from frameworks and it can be used in any situation or application. It fully supports updating you database schema to a new structure as well as reverting the schema via a record in a database table.

A sample migration looks like this:

[up]
add_column: users, registered_at, not_null : True, type : datetime
add_column: users, deleted_at, type : datetime
raw_sql: select * from users

add_index: users, my_stuff, unique : True, using : btree, columns : [account_key ASC, first_name DESC]
remove_index: users, my_stuff

change_column: users, account_key, type : varchar, length : 100, default : NULL

create_table: users_test, force : True, engine : InnoDB, char_set : latin1
    primary_key: id, account_key
    column: id, type : int, length : 11, not_null : True, auto : True
    column: account_key, type : varchar, length : 100, default : NULL
    column: first_name, type : varchar, length : 100, default : NULL
    column: updated_at, type : DATETIME, default : NULL

drop_table: users_test

[down]
remove_column: users, registered_at
remove_column: users, deleted_at

So if my database version was 002 then upgrading would run the up section. Creating, changing, dropping, etc. If I needed to go back down to version 001 then the down section would run, removing the registered_at and deleted_at columns.

I’m not releasing the code just yet. I need to DRY up the code and make sure a few other particulars are in order. I do have all those you see working correctly and I want to open the floor to a name. My particular choice is PyMigrate; but since I intend to let everyone use it I see reason to let everyone choose a name as wellr.

Until the initial release candidate it only supports MySQLdb as that is the db I am most familiar with. The database adapters are plugins so it will be really easy to include other database types before and after initial release. All I need  from you Python peeps is any ideas and to help me name this by Febuary 9th!

I’ve got some small refactoring to do plus lot’s of documentation to write to help get everyone started so check in here often! Leave your naming choices though in the comments as I’ll pick one by Feb. 9th.


Share Your Thoughts