• Home
  • Publications
  • Subscribe
  • Power of metaclasses

    Introduction to metaclasses in Python. You'll learn what metaclasses are, how to construct them and at the end we will go through detailed example from Django ORM.

    This article is a transcript of my presentation for Mobile EMEA Summit - Czech Republic 2019.

    Metaclasses are deeper magic than 99% of users should ever worry about. If you wonder whether you need them, you don't. Tim Peters

    Everything is a class

    First thing you have to understand to unravel the mystery of meta classes is that in Python everything is an object. Try it yourself.

    isinstance(True, bool)
    # True
    # True is instance of class bool
    dir(True)
    # ['__abs__', '__add__', '__and__', '__bool__', '__ceil__', ...]
    # True has its own methods and attributes
    bool()
    # False
    # Calling constructor of class bool initilizes boolen with default value
    dir(None)
    # ['__bool__', '__class__', '__delattr__', '__dir__', '__doc__', ...]
    # Even None is an object

    Every object has a class from which it was created. Think of a class as a cookie cutter and instance as a cookie. You can use a cookie cutter to create many cookies, but each cookie has to be created using cookie cutter. Likewise, each class can create many instances but each instance (or object) has a class from which it was created.

    And because we already know that everything in Python is an object, we can deduce that everything in Python has a class.

    To find out what class was used to create an object, we can use function type() (it's not a regular function as we will find out).

    For some objects it's easy to guess their class (type):

    class Klass: pass
    type(A())
    # <class '__main__.Klass'>
    # Type of instance of user-defined class is the class itself
    type(1)
    # <class 'int'>
    # Type of build in variable type is some build-in class
    type([])
    # <class 'list'>
    # The same applies for build-in structers
    type(None)
    # <class 'NoneType'>
    # As we shown before, even None has it's own type

    For other objects it's not so obvious, what's their type:

    def func(): pass
    type(func)
    # <class 'function'>
    # Functions are instances of class function 
    type(open)
    # <class 'builtin_function_or_method'>
    # But build-in functions have their own special class
    type(exit)
    # <class '_sitebuiltins.Quitter'>
    # Some build in functions are instance of even more special classes
    
    import math
    type(math)
    # <class 'module'>
    # And lastly, modules are instances of module class.

    What are metaclasses?

    You are probably asking now "So what's a class of a class?". Smart question.

    Classes are objects too (everything is an object, remember?) and as such they have a class. Class of class is called metaclass.

    Any class whose instances are themselves classes, is a metaclass. Class is an instance of its metaclass.

    Let's try to look at some metaclasses.

    class Klass: pass
    type(Klass)
    # <class 'type'>

    If you are not familiar with metaclasses, this probably surprised you. But it's correct, type of classes is called type.

    It's probably good time to look at the function type() we kept calling all the time. As I suggested earlier, it's not a regular function.

    type(type)
    # <class 'type'>

    In fact, type is not a function. It's a class. And not only that it is its own type. It is a top level class used to create all other classes. As with most classes, we can call its constructor to create new instances. And that's what we've been doing the whole time. Calling type(12) simply creates a class from given arguments. When we pass number 12 for example, Python uses the value to figure out what class we want - in this case it's <class 'int'> that already existed. But we can also use type to create a brand new classes.

    Creating classes in the runtime

    Most often we will use type with one argument to find out a type of some object. We can also use it to create new classes in the runtime by passing 3 arguments.

    type('Klass', (), {})
    # <class '__main__.Klass'>
    # type creates new class
    Klass()
    # <__main__.Klass object at 0x10d1edac8>
    # we can use the new class to create new instances

    We will look at the arguments closer but for now, the first argument is the name of our new class, second is a tuple of the parent classes and third is a dictionary of all the attributes, methods etc.

    How does Python construct new classes?

    Have you ever think about what happens when you write class MyClass(ParentClass): ... in Python? We can simplify it to 5 steps:

    1. Python sees class definition
    2. Determinates metaclass - either from named argument metaclass or default type
    3. Reads list of base classes - either as positional arguments of default object
    4. Reads attributes and methods
    5. Internally calls metaclass(name, bases, attributes)
    class Troll(User, metaclass=type):
    	def taunt(self):
    		print('trololo' * 1000)
    
    # is equivalent to this
    
    Troll = type('Troll', (User,), {'taunt': <function 0x108b20510>})

    As you can see, creating new class is similar to creating new instance of metaclass. In reality, instance initialization is little bit more complicated then that. In reality, calling Class() is a shortcut for calling __new__ and __init__. The code above is therefore equivalent to this:

    Troll = type.__new__(type, 'Troll', {'taunt': <function 0x108b20510>})
    type.__init__(Troll, 'Troll', (Person,), {'taunt': <function 0x108b20510>})

    Custom metaclasses

    Now you should have pretty clear idea of what metaclasses are and why are they important. But what can we use them for? Custom metaclasses are useful when we want to modify the behavior of our own classes. Instead of trying to explain what that means, I'll give you an example of real code.

    Django framework contains ORM. It allows you define your model objects as classes and Django can generate database schema from your classes, makes sure all the relations are consistent and allows you to request related model objects. It looks something like this:

    from django.db import models
    
    class Person(models.Model):
    	name = models.CharField(max_length=50)
    	age = models.PositiveIntegerField()
    	pronoun = models.CharField(max_length=300)

    When you create new instance of Person it automatically has setters and getters for name, age and pronoun. It also automatically validates the inputs and adds methods for saving the instance to database. Let's see how we can create something similar using metaclasses.

    Django ORM has way too many features. We will limit ourselves to implementing a Model class and an IntegerAttribute that will validate that given value is an int bellow some maximum value. We will also allow the attribute to have some default value.

    Class IntegerAttribute:
    	# This class will represent the integer attribute
    	pass
    
    class ModelMeta(type):
    	# This is our metaclass for creating model classes
    	def __new__(meta, name, bases, dct):
    		# We can use either __new__ or __init__ method to modify the behaviour. Let's use __new__
    		return super().__new__(meta, name, bases, dct)

    We created a metaclass. If we want to use it right now, we can do it like this:

    class Person(metaclass=ModelMeta): pass

    Adding named attribute to each class declaration is too cumbersome, using inheritance is easier. We can use the fact that metaclasses are also inherited and simplify the code.

    Class IntegerAttribute: pass
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta):
    	pass

    If we inherit from Model, we will also automatically use its metaclass so we can now create Person class simply by sub-typing Model.

    class Person(Model): pass

    Now that all the preparation is ready, let's code! First thing we need to do is to remove all defined attributes from the model class because later we will replace it with getters and setters.

    class IntegerAttribute: pass
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		for key in list(dct):
    			# iterate all attributes and remove all IntegerAttributes
    			attr = dct[key]
    			if isinstance(attr, IntegerAttribute):
    				del dct[key]
    		# We create the class with modified attribute list
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta): pass

    We can test all IntegerAttributes are removed:

    class Person(Model):
    	age = IntegerAttribute()
    	name = 'Joe'
    
    print(Person.name)
    # Joe
    
    print(Person.age)
    # AttributeError: type object 'Person' has no attribute 'age'

    We created a metaclass that removes some attributes. In next step we will replace them with a method that will set an instance attribute.

    class IntegerAttribute: pass
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		for key in list(dct):
    			attr = dct[key]
    			if isinstance(attr, IntegerAttribute):
    				def setter(this, value):
    					setattr(this, key, value)
    				dct[f'set_{key}'] = setter
    				del dct[key]
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta): pass

    Now all instance of the class User have a set_age method.

    class User(Model):
    	age = IntegerAttribute()
    
    user = User()
    user.set_age(42)
    print(user.age)
    # 42

    Similarly, to the setter we create a getter. Also, we will move both the setter and getter to the IntegerAttribute class where they belong. They can't be part of the class itself; we have to create them in the runtime.

    class IntegerAttribute:
    	def get_setter(self, name):
    		def setter(this, value):
    			setattr(this, name, value)
    		return setter
    
    	def get_getter(self, name):
    		def getter(this):
    			return getattr(this, name)
    		return getter
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		for key in list(dct):
    			attr = dct[key]
    			if isinstance(attr, IntegerAttribute):
    				dct[f'set_{key}'] = attr.get_setter(key)
    				dct[f'get_{key}'] = attr.get_getter(key)
    				del dct[key]
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta): pass

    We cannot set and retrieve attribute value by special methods.

    class User(Model):
    	age = IntegerAttribute()
    
    user = User()
    user.set_age(42)
    print(user.get_age())
    # 42

    In next step, we'll add the input validation. We'll check that the input is an integer and also add the possibility set maximum value.

    class IntegerAttribute:
    	def __init__(self, maximum=None):
    		self.maximum = maximum
    
    	def get_setter(self, name):
    		def setter(this, value):
    			if not isinstance(value, int):
    				raise TypeError(f'{value} is not of type int')
    			if self.maximum is not None and value > self.maximum:
    				raise ValueError(f'{value} is larger than {self.maximum}')
    			setattr(this, name, value)
    		return setter
    
    	def get_getter(self, name):
    		def getter(this):
    			return getattr(this, name)
    		return getter
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		for key in list(dct):
    			attr = dct[key]
    			if isinstance(attr, IntegerAttribute):
    				dct[f'set_{key}'] = attr.get_setter(key)
    				dct[f'get_{key}'] = attr.get_getter(key)
    				del dct[key]
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta): pass

    And again, we can test the code works as expected.

    class User(Model):
    	age = IntegerAttribute(maximum=100)
    
    user = User()
    user.set_age('Joe')
    # TypeError: Joe is not of type int
    user.set_age(105)
    # ValueError: 105 is larger than 100

    Last thing I promised you we'll implement together is the possibility to have a default value. It's a simple change and you probably already know how to do that.

    class IntegerAttribute:
    	def __init__(self, default=None, maximum=None):
    		self.default = default
    		self.maximum = maximum
    
    	def get_setter(self, name):
    		def setter(this, value):
    			if not isinstance(value, int):
    				raise TypeError(f'{value} is not of type int')
    			if self.maximum is not None and value > self.maximum:
    				raise ValueError(f'{value} is larger than {self.maximum}')
    			setattr(this, name, value)
    		return setter
    
    	def get_getter(self, name):
    		def getter(this):
    			return getattr(this, name, self.default)
    		return getter
    
    
    class ModelMeta(type):
    	def __new__(meta, name, bases, dct):
    		for key in list(dct):
    			attr = dct[key]
    			if isinstance(attr, IntegerAttribute):
    				dct[f'set_{key}'] = attr.get_setter(key)
    				dct[f'get_{key}'] = attr.get_getter(key)
    				del dct[key]
    		return super().__new__(meta, name, bases, dct)
    
    class Model(metaclass=ModelMeta): pass

    I will not show you how to test this change as you certainly have your own ideas. If not, you can download this code with all the unit-tests from my github.

    Where to go from here

    Together we explored the possibilities of metaclasses in Python and we implemented simple model class. If you find this interesting, you can continue a add some more features. Here are some ideas what to do.

    To find out more about metaclasses I recommend reading Python documentation on Data model.