Python metaclasses in depth

Metaclasses have a reputation for being among the blackest of Python’s black magic. As it turns out, they’re actually pretty simple, but they require you to understand some concepts. There are two ways to learn metaclasses in Python:

  1. Skip the details and get straight to a useful implementation of a metaclass.
  2. Understand the core concepts behind metaclasses.

Number 1 is useful, but will only get you so far. If you really want to understand metaclasses, you need to take approach number 2.

With that in mind, I’m going to start with the basics of how classes are constructed in Python. Let’s consider the following class:

class SomeClass(object):
     x = 1
     y = 2

If you’re ready to learn about metaclasses, the above statement shouldn’t require much thought to understand. But let’s stop to think about it a bit anyway. In Python, everything is an object. Therefore, classes are also objects. But if SomeClass is an object, it must be an instance of some class. Let’s find out what that class is:

>>> type(SomeClass)
<type 'type'>

So apparently, it’s an instance of type. We just saw the most common usage of type: to query an object’s type. But have you ever read the help on that function?

>>> help(type)

Help on class type in module __builtin__:

class type(object)
 |  type(object) -> the object's type
 |  type(name, bases, dict) -> a new type
 ...

So it turns out that type is not only a builtin function, it’s also a class! How do we instantiate an instance of type? You’ve already seen the most obvious way: using a class statement. But did you know that you can create a class using type?

SomeClass = type('SomeClass', (object,), {'x' : 1, 'y' : 2})

For all intents and purposes, the above statement is equivalent to the prior definition of SomeClass. But this way is ugly and isn’t used very commonly. That said, we’ve demonstrated something: type isn’t just any class. It’s the class of classes. It’s a metaclass.

Let’s go a step further though. How does the compiler generate the dictionary that’s the third argument to type? As it turns out, classes have something in common with functions: they have local namespaces. You might have seen the locals function used like this:

def some_func():
    x = 1
    y = 2
    print locals()

If you execute this, the output is this:

>>> some_func()
{'y': 2, 'x': 1}

If classes have their own namespaces, then they must also be able to use the locals function as well:

>>> class SomeClass(object):
...     x = 1
...     y = 2
...     print locals()
...
{'y': 2, 'x': 1, '__module__': '__main__'}

The only difference here is that the namespace is generated at import time and passed into the type function.

So this is all interesting, but we haven’t really seen anything terribly useful yet. Let’s go further. We’ve seen that type is the class of classes. But if type is a class, then we must be able to subclass it. There are a number of reasons you might want to do this. For instance, you may sometimes want to attach a property to a class rather than to an instance. Let’s do just that:

class SomeMetaClass(type):
    @property
    def z(self):
        print 'In class property z'
        return self.x + self.y

>>> SomeClass = SomeMetaClass('SomeClass', (object,), {'x' : 1, 'y' : 2})
>>> SomeClass.z
In class property z
3

But we don’t want to use this same ugly notation for creating SomeClass. Python provides syntactic sugar for this. We can instead define SomeClass like this:

class SomeClass(object):
    __metaclass__ = SomeMetaClass
    x = 1
    y = 2

A more common use of metaclasses is to create a class constructor. Let’s attach z to the class directly rather than defining a property:

>>> class SomeMetaClass(type):
...         def __init__(self, name, bases, dict):
...             self.z = self.x + self.y
...
>>> class SomeClass(object):
...         __metaclass__ = SomeMetaClass
...         x = 1
...         y = 2
...
>>> SomeClass.z
3

As we can see, we’ve defined a constructor for SomeClass. Now let’s go a bit further. What if we want to change the base class of SomeClass? That can be done, but we have to use a __new__ method. I’m going to presume that you know a bit about __new__ methods. If you don’t, you might want to read up on them.

>>> class SomeMetaClass(type):
...     def __new__(cls, name, bases, cls_dict):
...         new_cls = type.__new__(cls, name, (dict,), cls_dict)
...         return new_cls
...
>>> class SomeClass(object):
...     __metaclass__ = SomeMetaClass
...     x = 1
...     y = 2
...
>>> x = SomeClass()
>>> x['foo'] = 'bar'
>>> x
{'foo': 'bar'}

That should hopefully give you an idea of what metaclasses are and how to use them. If you’re even more lost than you were before, don’t worry. This is just one of those things that requires an “aha!” moment. You might also want to check out Michael Foord’s great Meta-classes Made Easy for a different perspective.

7 tools for working with Python in Emacs (and Ubuntu)

I’ve been meaning to blog about this for some time, so I suppose now
is as good an opportunity as any. This is going to be a very “stream
of consciousness”-esque posting, so bear with me. Some of these are
things that have radically changed the way I use emacs. Others are
minor changes that I like. Feel free to pick and choose from them.

I’ll assume you’re running Ubuntu. Also, not all of these are Python
specific. But I feel that they will be useful to most Python
programmers who use emacs.

I should also note that (like most emacs users), most of these things
are tricks that I’ve picked up from various sources along the way. If
you wrote something that I put in here, thanks!

Ropemacs

Ropemacs is among the tools I love the most and hate the most. When
it works, it opens up a new world of automated refactorings for you.
But it seems to be a bit buggy at times. That said, setting it up is
really easy. Firstly, you need to have ropemacs installed. This is
pretty easy:

 
sudo apt-get install python-ropemacs 

After that, just a couple of lines of elisp in your .emacs file and
you’re good to go!

 
(require 'pymacs) 
(pymacs-load "ropemacs" "rope-") 
(setq ropemacs-enable-autoimport t) 

Anything

Anything is almost like Quicksilver for emacs. To begin, you need to
download anything.el and anything-config. I also use
anything-match-plugin. Then you just need the following lines of elisp:

 
(require 'anything-config) 
(require 'anything-match-plugin) 
(global-set-key "\C-ca" 'anything) 
(global-set-key "\C-ce" 'anything-for-files) 

Then, prepare to spend a lot less time searching for files!

Line number mode

When you’re pair programming, nothing is more helpful than being able
to direct people to a certain line of code. This lets you spend less
time saying “hey, see that over there? It’s about 3 lines up. No,
too far! go down another two lines.” Installing this is really easy.
You just need linum.el and two lines of elisp:

 
(require 'linum) 
(global-linum-mode 1) 

Flymake through pyflakes

In case you miss the automatic error highlighting of the Visual Studio
world, you should realize that emacs has a similar system built-in.
It’s called flymake. And you can make it work with Python as well. I
personally prefer to use pyflakes for this. All that’s needed is
pyflakes:

 
sudo apt-get install pyflakes 

…and some elisp:

 
(when (load "flymake" t) 
 (defun flymake-pyflakes-init () 
 (let* ((temp-file (flymake-init-create-temp-buffer-copy 
 'flymake-create-temp-inplace)) 
 (local-file (file-relative-name 
 temp-file 
 (file-name-directory buffer-file-name)))) 
 (list "pyflakes" (list local-file)))) 
 
 (add-to-list 'flymake-allowed-file-name-masks 
 '("\\.py\\'" flymake-pyflakes-init))) 

Uniquify

How many __init__.py buffers do you have open at this moment? If
you’re using emacs for Python programming, probably a lot. This is
where emacs’s uniquify functionality is useful. It gives you a more
useful name for your buffers other than just appending a number at the
end. I have mine use the reverse of the directory. For instance, if
I have foo/__init__.py and bar/__init__.py open, they will be named
__init__.py/foo and __init__.py/bar respectively.

You just need this in your .emacs:

 
(setq uniquify-buffer-name-style 'reverse) 
(setq uniquify-separator "/") 
(setq uniquify-after-kill-buffer-p t) ; rename after killing uniquified 
 
(setq uniquify-ignore-buffers-re "^\\*") ; don't muck with special 
buffers (or Gnus mail buffers) 

python-mode

To be totally honest with you, I haven’t used the built-in emacs mode
for python. I just installed python-mode because I was told it was
better. What I can tell you is that there is an occasional plugin
that requires python-mode. Installing it is easy. Just install
python-mode:

 
sudo apt-get install python-mode 

And add some elisp:

 
(autoload 'python-mode "python-mode" "Python Mode." t) 
(add-to-list 'auto-mode-alist '("\\.py\\'" . python-mode)) 
(add-to-list 'interpreter-mode-alist '("python" . python-mode)) 

Pylookup

Pylookup is useful for those moments when you find yourself asking
something like “Is join in os or os.path?” Unfortunately, the setup
can be complex, but well worth it. There are instructions here.