A simple FormAlchemy tutorial

December 4th, 2009

Here’s a quick tutorial on the usage of FormAlchemy with Elixir and Pylons.

The model

To illustrate how things work, we will take a very simple example.

Let’s say you have a website where people have to sign in.

Here’s a very simple User class

# Entity binds a class to a table
from elixir import Entity
# Field represents bounds an attribute to a column in your table
from elixir import Field
# Unicode, Integer, DateTime etc. are column types. Pay attention to encoding when you’re using Unicode.
from elixir import Unicode
# This will allow us to represent relations between classes. Elixir will figure out how to represent them
# in the database, between the tables
from elixir import ManyToOne, OneToMany

import datetime # we will use this to put default values in DateTime columns

# Elixir will create a table for each class inherting form Entity
class User(Entity):
    # By default, your table names will look like "ModuleName_ClassName".
    # To change this, you can either pass in a shortnames=True argument to
    # the using_options function, which will tell elixir to create your tables
    # based on ClassName only, or a tablename=’My_Table_Name’ argument.
    # I’m used to have plural names for tables, and single names for classes,
    # so I’ll pass in a tablename arguemnt
   
    # Next thing is the inhertince argument. This will tell Elixir
    # that User can be subclassed.
    using_options (tablename=“Users”,inheritance=“multi”)
   
    # I like powers of 2
    username      = Field(Unicode(32))
    email         = Field(Unicode(64))
    password      = Field(Unicode(32))
    phone         = Field(Unicode(16))
    address       = Field(Unicode(64))
    # Many users may live in One City, so in this case it’s Many(users)ToOne(City)
    city          = ManyToOne(“City”)
    creation_date = Field(DateTime,default=datetime.datetime.now)

# Let’s make a distinction between Persons and Companies. Both are users, wich means they share a certain amount of
# properties, but they can also have different additional attribtues of their own.

class Person(User):
    using_options (tablename=“Persons”,inheritance=“multi”)
    first_name = Field(Unicode(32))
    last_name  = Field(Unicode(32))   

    def __str__(self):
        “”” This method is called when you try to print a user.”“”
        return “%s %s (%s)” % (self.first.encode(“utf-8″), self.last.encode(“utf-8″), self.username.encode(“utf-8″))
   
    def __repr__(self):
        “”” This method is called when you try to print a representation of a user.”“”
        return “” % str(self)   

class Company(User):
    using_options (tablename=“Users”,inheritance=“multi”)   
    logo    = Field(String(32)) # we store only the filename here.
    name    = Field(Unicode(64))
    website = Field(Unicode(64))

Things are looking pretty good so far ! Now let’s write a simple registration form.

The mako template

Let’s begin with the mako template :

<h2>Register to our great website !</h2>
<form action="${h.url_for(action=’processRegistrationForm’,controller=’registration’)}">
${c.form}
<input type="submit" value="go" />
</form>

Very simple code. c.form will contain the html code of the form, generated by formalchemy.

On to the controller action.

The controller

We’ll create a controller named registration for this purpose. Don’t forget to activate your virutal environement.

$(PYLONSENV) paster controller registration

Now let’s edit controller/registration.py

import logging

from pylons import request, response, session, tmpl_context as c
from pylons.controllers.util import abort, redirect_to
from myapp.lib.base import BaseController, render
# These imports up there are generated for you by paster
# Now you’ll need two other imports to generate the form : FieldSet and your model’s Person

# This is a customized FieldSet that will render itself following the template that you’ll find
# in templates/forms/fieldset.mako.
# I have edited that template to use semantic html elements like dl, dt and dd instead of bare div elements.
from myapp.forms import FieldSet

# This represents a form field, we will use this to create a password check form field.
from formalchemy import Field

from myapp.model import Person

log = logging.getLogger(__name__)

class RegistrationController(BaseController):

    def personRegistrationForm(self):
        “”
        Call this action in your mako templates to render the registration form
        “
“”
        fs = self.getPersonFieldSet()       
        c.form = fs.render()
        return render(“/pages/registration.mako”)

    def processRegistrationForm(self):
        “”
        Put this method in your registration’s form’s action attribute.
        <form action=”
/registration/processRegistrationForm“>
        or use
        <form action=”
${h.url_for(controller=‘registration’,action=‘processRegistrationForm’)}“>
        “
“”
        fs      = self.getPersonFieldSet()
        # This will populate the FieldSet with values found in request.params
        fs.data = request.params
        # This will populate the Person instance that FieldSet stores as an attribute named "model"
        fs.sync()
        # Here’s your object :) !
        user = fs.model
        # now do whatever you want with it. For this example, we’ll just return a simple message
        # saying that everything was ok.
        return “User (%s) successfully created” % str(user) # This will call user.__str__

    def getPersonFieldSet(self):
        “”
        Return the fieldset corresponding to a Person
        “
“”
        # Just pass in the class, not a specific instance
        fs = FieldSet(Person)
        # Now let’s add a password check field.
        # name is the attribute name that will be generated
        #
        # By default, formalchemy will append the class name as a prefix to every field name.
        # This is to prevent confusion when two classes may have the same name for one of their attribute.
        pwdcheck= Field(name=“pwdcheck”,type=Unicode)
        # We’ll add it to the FieldSet
        fs.append(pwdcheck)
        # Now we’ll customize the generated HTML with the conigure method.
        # the include argument tells FieldSet objects what fields you actually want to render.
        # A FieldSet consists of a group of Fields. Each field is accessed as an attribute of the FieldSet
        # it’s contained within. The attribute name of that field is the same as the Field name you used
        # in your class declaration.
        fs.configure(include=[fs.first,fs.last,fs.username,fs.city,fs.adresse,fs.email,
                              # Now here we’ll tell fs.password to render itself as a password field
                              # instead of a usual text field. This will return the modified field.
                              fs.password.password(),
                              # Now we’ll configure the pwdcheck field to have a meaningful and helpful label
                              fs.pwdcheck.password().label(“Confirm your password”)])
        return fs

That’s it :) you’re ready to go ! This is actually not too much code, let’s strip the comments to see how much code we needed.

Let’s recap’

the model code, model/__init__.py

from elixir import Entity, Field, Unicode,ManyToOne,OneToMany
import datetime

class User(Entity):
    using_options (tablename=“Users”,inheritance=“multi”)
    # I like powers of 2
    username      = Field(Unicode(32))
    email         = Field(Unicode(64))
    password      = Field(Unicode(32))
    phone         = Field(Unicode(16))
    address       = Field(Unicode(64))
    city          = ManyToOne(“City”)
    creation_date = Field(DateTime,default=datetime.datetime.now)

class Person(User):
    using_options (tablename=“Persons”,inheritance=“multi”)
    first_name = Field(Unicode(32))
    last_name  = Field(Unicode(32))   

    def __str__(self):
        “”” This method is called when you try to print a user.”“”
        return “%s %s (%s)” % (self.first.encode(“utf-8″), self.last.encode(“utf-8″), self.username.encode(“utf-8″))
   
    def __repr__(self):
        “”” This method is called when you try to print a representation of a user.”“”
        return “” % str(self)   

class Company(User):
    using_options (tablename=“Users”,inheritance=“multi”)   
    logo    = Field(String(32)) # we store only the filename here.
    name    = Field(Unicode(64))
    website = Field(Unicode(64))

the mako template

<h2>Register to our great website !</h2>
<form action="${h.url_for(action=’processRegistrationForm’,controller=’registration’)}">
${c.form}
<input type="submit" value="go" />
</form>

The Fieldset.mako template

The interresting part of the mako template for the form generation, templates/forms/fieldset.mako.
Here, I consider the form as a definition list, every label is a definition term, and the inputs are data definitions.

% for field in fieldset.render_fields.itervalues():
% if field.requires_label:
<dl>
<dt>
<label class="${field.is_required() and ‘field_req’ or ‘field_opt’}" for="${field.renderer.name}">${[field.label_text, fieldset.prettify(field.key)][int(field.label_text is None)]|h}</label>
</dt>
<dd>
${field.render()|n}
</dd>
% if ‘instructions’ in field.metadata:
<span class="instructions">${field.metadata[’instructions’]}</span>
% endif
% for error in field.errors:
<span class="field_error">${_(error)}</span>
% endfor
</dl>

The controller

The controller’s code, controller/registration.py. This is where formalchemy comes into play.

import logging
from pylons import request, response, session, tmpl_context as c
from pylons.controllers.util import abort, redirect_to
from formalchemy import Field
from myapp.model import Person
from myapp.forms import FieldSet
log = logging.getLogger(__name__)

class RegistrationController(BaseController):

    def personRegistrationForm(self):
        “”
        Call this action in your mako templates to render the registration form
        “
“”
        fs = self.getPersonFieldSet()       
        c.form = fs.render()
        return render(“/pages/registration.mako”)

    def processRegistrationForm(self):
        “”
        Put this method in your registration’s form’s action attribute.
        “
“”
        fs      = self.getPersonFieldSet()
        fs.data = request.params
        fs.sync()
        user = fs.model
        return “User (%s) successfully created” % str(user) # This will call user.__str__

    def getPersonFieldSet(self):
        “”
        Return the fieldset corresponding to a Person
        “
“”
        fs = FieldSet(Person)
        pwdcheck= Field(name=“pwdcheck”,type=Unicode)
        fs.append(pwdcheck)
        fs.configure(include=[fs.first,fs.last,fs.username,fs.city,fs.adresse,fs.email,
                              fs.password.password(),
                              fs.pwdcheck.password().label(“Confirm your password”)])
        return fs

This is an idiot mistake.

I wanted to create 3 Entities, all sharing a certain amount of behaviour and properties (that is methods and attributes). So I decided to a create a BaseClass where the common code is shared, then create 3 Classes that subclass the later.

So how’s the code gonna look like ?

At first guess, I thought “why would I subclass all my classes from Entity AND BaseClass ? wouldn’t it be better to just subclass BaseClass and then subclass BaseClass from Entity ? thus allowing all the subclasses of BaseClass be Entities ?”

class BaseClass(Entity):
    #Some useful methods and attributes here
    #…

class SubClassOne(BaseClass):
    # Your actual class
    # …

class SubClassTwo(BaseClass):
    # Your class here
    # …
#… more subclasses
 

Bad idea ! If you’re doing this, you are defining BaseClass as an entity, that will be created in the database. And so you’ll get an Exception like

Exception: Column ‘type’ already exist in ‘BaseClass’

when elixir will try to create the second subclass.

So, no, the only way to do it right is to subclass from Entity AND BaseClass.

class BaseClass:
    #Some useful methods and attributes here
    #…

class SubClassOne(BaseClass,Entity):
    # Your class here
    # …

class SubClassTwo(BaseClass,Entity):
    # Your class here
    # …
#… more subclasses
 

With the help of Ben bangert, I rewrote my code this way :

class UniqueMeta(EntityMeta):

    def __call__(cls,key):
        “”
        If it’s in the cache, return the cached version
        If not in the cache :
            If it’s in the database, retrieve it, cache it and return it
            If it’s not there, create it, cache it and return it
        “
“”
        thecache = cache.get_cache(cls.__name__,type=“memory”)
        def makeTag():
            theTag = cls.query.filter_by(**{cls.filterby:key}).first()
            if not theTag:
                #not in the database either
                theTag = type.__call__(cls,key)
                session.add(theTag)
            return theTag
        return thecache.get_value(key=key,createfunc=makeTag)
   
class Ville(Entity):
    using_options (tablename=“Villes”)
    __metaclass__ = UniqueMeta
    nom           = Field(Unicode(64))
    filterby      = “nom”
    def __init__(self,nom,*args,**kw):
        Entity.__init__(self,*args,**kw)
        self.nom = nom

    def __repr__(self):
        return “”self.nom
   
class TypeBien(Entity):
    using_options (tablename=“TypesBiens”)
    __metaclass__ = UniqueMeta
    type          = Field(Unicode(64))
    filterby      = “type”

    def __init__(self,type,*args,**kw):
        Entity.__init__(self,*args,**kw)
        self.type = type

    def __repr__(self):
        return “” % self.type

There’s an improvement here over the UniqueObject recipe : more classes can use the UniqueMeta metaclass if they define a filterby class attribute.

The code below is outdated, please see a recent version here

SQLAlchemy doesn’t provide a cahing mechanisme, because the designers thought that this is not an ORM’s business after all, and I think they are right.

Pylons offers you a way to cache arbitrary objects. This is handy to store server sessions objects, or, as described here, your SQLAlchemy mapped objects.

Now, I am a fan of the O3 principle, so I’m not arguing about speed performances/overheads, i’m arguing about doing things Once, and Only Once (O3), because I was taught to do so in my computing education.

It is frequent in applications to get the same rows from the database that have not been changed over and over again. Fortunatly, almost all databases are smart enough to figure that and implement a query cache to handle this situation. This means that if you repeat exactly the same query during a certain laps of time, it will return the results from the in-memory cache, not from the disk, which is times faster.

But some people do not like to talk to the database if the exact same queries have been processed in the near past. They like to implement an applicaiton-level cache, also referred to as a “second level cache”.

This is what I have tried to do with SQLAlchemy in a Pylons application. And here’s the result :

from elixir import Entity, EntityMeta,setup_all,create_all,metadata
from pylons import cache

class MetaCity(EntityMeta):
    cache = cache.get_cache(“cities”,type=“memory”)
    def __call__(cls,name):
        “”
        If it’s in the cache, return the cached version
        If not in the cache :
            If it’s in the database, retrieve it, cache it and return it
            If it’s not there, create it, cache it and return it
        “
“”
        theCity = MetaCity.cache.get_value(key=name,createfunc=lambda:None)
        if not theCity :
            #not in the cache
            theCity = cls.query.filter_by(name=name).first()
            if not theCity:
                #not in the database either
                print “not in the database”
                theCity = type.__call__(cls,name)
                session.add(theCity)
            #Adding it to the cache, after creating it in the database if it wasn’t there
            MetaCity.cache.set_value(key=name,value=theCity)           
        return theCity
   
class City(Entity):
    __metaclass__ = MetaCity
    using_options (tablename=“Cities”)
    name   = Field(Unicode(64))

    def __init__(self,name,*args,**kw):
        Entity.__init__(self,*args,**kw)
        self.name = name

    def __repr__(self):
        return “”self.name

metadata.bind = “mysql://username:password@localhost:3306/db”
metadata.bind.echo = True

And here’s the model I played with on my ipython interpreter (using paster shell).

NOTA : Ville is french for City and Villes (plural) for Cities. I also added some prints here and there to show where the objects come from.

(PYTHONENV)chaouche@la7lou:~/somedirectory/$ paster shell
VIRTUAL_ENV -> /home/chaouche/PYTHONENV/lib/python2.6/site-packages
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
Type “copyright”, “credits” or “license” for more information.

IPython 0.9.1 — An enhanced Interactive Python.
? -> Introduction and overview of IPython’s features.
%quickref -> Quick reference.
help -> Python’s own help system.
object? -> Details about ‘object’. ?object also works, ?? prints more.

All objects from interimmo.lib.base are available
Additional Objects:
mapper - Routes mapper object
wsgiapp - This project’s WSGI App instance
app - paste.fixture wrapped around wsgiapp

In [1]: from someproject.model.somemodel import *

In [2]: Ville(”Rouiba”)
not in the cache…
23:00:31,473 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] BEGIN
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Rouiba’
param.append(processors[key](compiled_params[key]))
23:00:31,480 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:00:31,480 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Rouiba’]
not in the database
adding it to the cache
Out[2]:

In [3]: Ville(”Rouiba”)
Out[3]:

In [4]: Ville(”Rouiba”)
Out[4]:

In [5]: Ville(”Kolea”)
not in the cache…
23:00:54,416 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:00:54,416 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Rouiba’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Kolea’
param.append(processors[key](compiled_params[key]))
23:00:54,656 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:00:54,656 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Kolea’]
not in the database
adding it to the cache
Out[5]:

In [6]: Ville(”Babezzouar”)
not in the cache…
23:01:15,955 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:01:15,955 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Kolea’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Babezzouar’
param.append(processors[key](compiled_params[key]))
23:01:15,962 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:01:15,962 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Babezzouar’]
not in the database
adding it to the cache
Out[6]:

In [7]: Ville(”Babezzouar”)
Out[7]:

In [8]: Ville(”Babezzouar”)
Out[8]:

In [9]: Ville(”Alger”)
not in the cache…
23:01:32,024 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:01:32,024 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Babezzouar’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Alger’
param.append(processors[key](compiled_params[key]))
23:01:32,029 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:01:32,030 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Alger’]
adding it to the cache
Out[9]:

In [10]: Ville(”Alger”)
Out[10]:

In [11]: Ville(”Kolea”)
Out[11]:

In [12]: Ville(”Rouiba”)
Out[12]:

Today’s video :

My notes :

SQLite introduction

db_row vs dtuple

August 29th, 2008

MySQLdb result sets do not offer a nice way to acess information : tuples. This means that whenever you change your SQL query, you have to modify the code that extract information from the data set because the position of elements may have changed.

result[0] was for example Name but now it is Password.

If you google a bit, you will pass by dtuple and db_row. Which one should you use ? I googled for a comparison of the two but found nothing in the first results. So here’s a little description of what I have found so far after testing the two :

dtuple is flexible, but column names are case sensitive. That means that row.Name or row[’Name’] is ok but not row.name nor row[’name’]. In the other hand, the results are mutable, this means that you can add some other field in the result set if another function or method needs it. On the contrary, db_row does not allow you to change the result set because they are immutable (they use python’s __slot__ thing that prevent you from adding new attributes to your object). The good thing is that this is faster because attribute lookup is boosted. Besides, db_row is memory friendly because it does not create an object per row. This means that if your SELECT returns 100.000 lines, it wont create 100.000 objects. db_row allows also case-insensitive key/attribute fetching, as a little “syntactic sugar”.

There’s a third alternative, sqlWrap, which offers very nice reporting solutions, like export to xhtml, pretty printing, and export to restructuredText. I did not try this one but looks fun.

Choose wisely.

Just a little memo…

greppy -l  “SESSION.get(idEntreprise)” | xargs sed -i -e “s/SESSION.get(idEntreprise)/SESSION.get(\”idEntreprise\”)/”

One handy alias :

alias greppy=‘find . -name "*.py" | grep -v svn | xargs grep –color’

Emacs is great for prototyping

August 14th, 2008

Split the window as you wish, this allows you to see different files at the same time to rapidly look for the information you need. I found this really useful yesterday when I was stuck on my application model. I needed to have a snapshot of the different classes and objects, when and where to store things in the session, and when to retrieve them etc. This window splitting system comes in handy when you need to do a global refactoring of your code and review the interactions and relations between your various objects.

emacs_split_window1.png

tidy –input-encoding utf8 –output-encoding ascii –doctype omit -i -o outputfile.html inputfile.html

–input-encoding : your encoding (actually supports utf-8, latin1 and latin15)
–ouptput-enconding : specifying ascii here forces tidy to convert non ascii chars to the corresponding html entities.

notice that

tidy -utf8 …

will output non-ascii characters as they are (no conversion) while

tidy -ascii …

will break you characters because he does not take the input file with the appropriate encoding.

That’s why you have to specify –input-encoding and –output-encoding to get the job done.

Why anybody thought about it before ? http://codepad.org/