Why should a blog have a title ?

Using beaker as a second level cache for sqlalchemy in pylons (better approach)

November 23rd, 2009

With the help of Ben bangert, I rewrote my code this way :

class UniqueMeta(EntityMeta):

def __call__(cls,key):
“”“
If it’s in the cache, return the cached version
If not in the cache :
If it’s in the database, retrieve it, cache it and return it
If it’s not there, create it, cache it and return it
““”
thecache = cache.get_cache(cls.__name__,type=“memory”)
def makeTag():
theTag = cls.query.filter_by(**{cls.filterby:key}).first()
if not theTag:
#not in the database either
theTag = type.__call__(cls,key)
session.add(theTag)
return theTag
return thecache.get_value(key=key,createfunc=makeTag)

class Ville(Entity):
using_options (tablename=“Villes”)
__metaclass__ = UniqueMeta
nom = Field(Unicode(64))
filterby = “nom”
def __init__(self,nom,*args,**kw):
Entity.__init__(self,*args,**kw)
self.nom = nom

def __repr__(self):
return “” % self.nom

class TypeBien(Entity):
using_options (tablename=“TypesBiens”)
__metaclass__ = UniqueMeta
type = Field(Unicode(64))
filterby = “type”

def __init__(self,type,*args,**kw):
Entity.__init__(self,*args,**kw)
self.type = type

def __repr__(self):
return “” % self.type

There’s an improvement here over the UniqueObject recipe : more classes can use the UniqueMeta metaclass if they define a filterby class attribute.

Bookmark to:

Posted by yassinechaouche Filed in pylons, DB, Code, Python

1 Comment »

Using beaker as a second level query cache for SQLAlchemy in Pylons

November 21st, 2009

The code below is outdated, please see a recent version here

SQLAlchemy doesn’t provide a cahing mechanisme, because the designers thought that this is not an ORM’s business after all, and I think they are right.

Pylons offers you a way to cache arbitrary objects. This is handy to store server sessions objects, or, as described here, your SQLAlchemy mapped objects.

Now, I am a fan of the O3 principle, so I’m not arguing about speed performances/overheads, i’m arguing about doing things Once, and Only Once (O3), because I was taught to do so in my computing education.

It is frequent in applications to get the same rows from the database that have not been changed over and over again. Fortunatly, almost all databases are smart enough to figure that and implement a query cache to handle this situation. This means that if you repeat exactly the same query during a certain laps of time, it will return the results from the in-memory cache, not from the disk, which is times faster.

But some people do not like to talk to the database if the exact same queries have been processed in the near past. They like to implement an applicaiton-level cache, also referred to as a “second level cache”.

This is what I have tried to do with SQLAlchemy in a Pylons application. And here’s the result :

from elixir import Entity, EntityMeta,setup_all,create_all,metadata
from pylons import cache

class MetaCity(EntityMeta):
cache = cache.get_cache(“cities”,type=“memory”)
def __call__(cls,name):
“”“
If it’s in the cache, return the cached version
If not in the cache :
If it’s in the database, retrieve it, cache it and return it
If it’s not there, create it, cache it and return it
““”
theCity = MetaCity.cache.get_value(key=name,createfunc=lambda:None)
if not theCity :
#not in the cache
theCity = cls.query.filter_by(name=name).first()
if not theCity:
#not in the database either
print “not in the database”
theCity = type.__call__(cls,name)
session.add(theCity)
#Adding it to the cache, after creating it in the database if it wasn’t there
MetaCity.cache.set_value(key=name,value=theCity)
return theCity

class City(Entity):
__metaclass__ = MetaCity
using_options (tablename=“Cities”)
name = Field(Unicode(64))

def __init__(self,name,*args,**kw):
Entity.__init__(self,*args,**kw)
self.name = name

def __repr__(self):
return “” % self.name

metadata.bind = “mysql://username:password@localhost:3306/db”
metadata.bind.echo = True

And here’s the model I played with on my ipython interpreter (using paster shell).

NOTA : Ville is french for City and Villes (plural) for Cities. I also added some prints here and there to show where the objects come from.

(PYTHONENV)chaouche@la7lou:~/somedirectory/$ paster shell
VIRTUAL_ENV -> /home/chaouche/PYTHONENV/lib/python2.6/site-packages
Python 2.6.2 (release26-maint, Apr 19 2009, 01:56:41)
Type “copyright”, “credits” or “license” for more information.

IPython 0.9.1 — An enhanced Interactive Python.
? -> Introduction and overview of IPython’s features.
%quickref -> Quick reference.
help -> Python’s own help system.
object? -> Details about ‘object’. ?object also works, ?? prints more.

All objects from interimmo.lib.base are available
Additional Objects:
mapper - Routes mapper object
wsgiapp - This project’s WSGI App instance
app - paste.fixture wrapped around wsgiapp

In [1]: from someproject.model.somemodel import *

In [2]: Ville(”Rouiba”)
not in the cache…
23:00:31,473 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] BEGIN
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Rouiba’
param.append(processors[key](compiled_params[key]))
23:00:31,480 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:00:31,480 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Rouiba’]
not in the database
adding it to the cache
Out[2]:

In [3]: Ville(”Rouiba”)
Out[3]:

In [4]: Ville(”Rouiba”)
Out[4]:

In [5]: Ville(”Kolea”)
not in the cache…
23:00:54,416 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:00:54,416 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Rouiba’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Kolea’
param.append(processors[key](compiled_params[key]))
23:00:54,656 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:00:54,656 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Kolea’]
not in the database
adding it to the cache
Out[5]:

In [6]: Ville(”Babezzouar”)
not in the cache…
23:01:15,955 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:01:15,955 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Kolea’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Babezzouar’
param.append(processors[key](compiled_params[key]))
23:01:15,962 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:01:15,962 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Babezzouar’]
not in the database
adding it to the cache
Out[6]:

In [7]: Ville(”Babezzouar”)
Out[7]:

In [8]: Ville(”Babezzouar”)
Out[8]:

In [9]: Ville(”Alger”)
not in the cache…
23:01:32,024 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] INSERT INTO `Villes` (nom) VALUES (%s)
23:01:32,024 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Babezzouar’]
/home/chaouche/PYTHONENV/lib/python2.6/site-packages/SQLAlchemy-0.5.5-py2.6.egg/sqlalchemy/engine/default.py:230: SAWarning: Unicode type received non-unicode bind param value ‘Alger’
param.append(processors[key](compiled_params[key]))
23:01:32,029 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] SELECT `Villes`.id AS `Villes_id`, `Villes`.nom AS `Villes_nom`
FROM `Villes`
WHERE `Villes`.nom = %s
LIMIT 0, 1
23:01:32,030 INFO [sqlalchemy.engine.base.Engine.0x…5d6c] [’Alger’]
adding it to the cache
Out[9]:

In [10]: Ville(”Alger”)
Out[10]:

In [11]: Ville(”Kolea”)
Out[11]:

In [12]: Ville(”Rouiba”)
Out[12]:

Bookmark to:

Posted by yassinechaouche Filed in pylons, Code, Python

1 Comment »

Why should nerds listen to classical music ?

May 16th, 2009

Thanks for the guy who searched for “Why should I like classical music” on Live search and reached here (found him on gostats last guests) , because on the search results I found a good article name “Why should nerds listen to classical music”. The author of this article gives 7 reasons for nerds to listen to classical music, one of the most interesting is “intellectual stimulation”. There’s even a link between classical music and computer science, this is illustrated in MIT’s open course ware on Godel, Escher, Bach. These video lectures are highly recommended for anyone interested in learning new intellectual concepts and structures.

By the way, there’s also a link between computer science and painting (this is the Escher part of Hofstadter’s book), this is probably why Paul Graham is also a painter :) and has written a good article about hacking and painting.

Bookmark to:

Posted by yassinechaouche Filed in classical music

No Comments »

CNet’s Startup Tips (2007)

May 13th, 2009

Not very up-to-date, but the advices are still valid. I found this document while reading simple entrepneur blog, a french blog that tells the story of the creation of a webservice called loomiz, but it is mainly about advices on the creation of a web startup.

O, did I say I was fan of mind maps ?