When to use metaclasses in Python: 5 interesting use cases
Metaclasses are mentioned among the most advances features of Python. Knowing how to write one is perceived like having a Python black belt. But are they useful at all outside job interviews or conference talks? Let's find out! This article will show you 5 practical applications of metaclasses.
What metaclasses are - quick recap
Assuming you know the difference between classes and objects, metaclasses should not be that difficult - they are classes for classes (hence "meta" in their name).
Simply saying - while classes are blueprints for objects, metaclasses are blueprints for classes. Class acts as a blueprint when we create an instance of it whereas metaclass acts as a blueprint only when a class is defined.
The simplest implementation of a metaclass that does nothing looks as follows:
# inherit from type
class MyMeta(type):
# __new__ is a classmethod, even without @classmethod decorator
def __new__(cls, name, bases, namespace):
# cls - MyMeta
# name - name of the class being defined (MyClass in this example)
# bases - base classes for constructed class, empty tuple in this case
# namespace - dict with methods and fields defined in class
# in this case - {'x': 3}
# super().__new__ just returns a new class
return super().__new__(cls, name, bases, namespace)
class MyClass(metaclass=MyMeta):
x = 3Avoiding decorators repetition or decorating all subclasses
Let's say you fall in love with relatively recent dataclasses stdlib module or you use much more advanced attrs. Or you just use a lot of repetitive decorators on your classes:
@attr.s(frozen=True, auto_attribs=True)
class Event:
created_at: datetime
@attr.s(frozen=True, auto_attribs=True)
class InvoiceIssued(Event):
invoice_uuid: UUID
customer_uuid: UUID
total_amount: Decimal
total_amount_currency: Currency
due_date: datetime
@attr.s(frozen=True, auto_attribs=True)
class InvoiceOverdue(Event):
invoice_uuid: UUID
customer_uuid: UUIDMaybe it's not too much repetition, but we can still get rid of it we we wrote a metaclass for Event:
class EventMeta(type):
def __new__(cls, name, bases, namespace):
new_cls = super().__new__(cls, name, bases, namespace)
return attr.s(frozen=True, auto_attribs=True)(new_cls) # (!)
class Event(metaclass=EventMeta):
created_at: datetime
class InvoiceIssued(Event):
invoice_uuid: UUID
customer_uuid: UUID
total_amount: Decimal
total_amount_currency: Currency
due_date: datetime
class InvoiceOverdue(Event):
invoice_uuid: UUID
customer_uuid: UUID
The most important line, `attr.s(frozen=True, auto_attribs=True)(new_cls)` deals with decoration of our subclasses. The key to understanding this example is the fact that syntax with "at sign" (@) is merely a syntax sugar over such construct:
class Event:
created_at: datetime
Event = attr.s(frozen=True, auto_attribs=True)(Event)Validation of subclasses
Speaking of classes, let's think about inheritance in a context of Template Method design pattern. Simply saying, we define an algorithm in the base class, but we leave one or more steps (or attributes) as abstract methods (or properties), to be overridden in a subclass. Consider this example:
class JsonExporter(abc.ABC):
def export(self):
# do some stuff
with open(self._filename) as f:
for row in self._build_row_from_raw_data():
pass # do some other stuff
@abc.abstractmethod
@property
def _filename(self):
pass
@abc.abstractmethod
def _build_row_from_raw_data(self, raw_data):
pass
class InvoicesExporter(JsonExporter):
_filename = 'invoices.json'
def _build_row_from_raw_data(self, raw_data):
return {'invoice_uuid': raw_data[0]}We have a simple base class for JSON exporters. It is enough to subclass it and provide an implementation for the _filename property and the _build_row_from_raw_data method. However, abc only provides validation for the absence of these parameters. If we, for example, would like to check more things, like the uniqueness of filenames or its correctness (e.g. always ends with .json) we can write a metaclass for that as well:
import inspect
"""
We inherit from abc metaclass (ABCMeta) to avoid metaclasses conflicts
"""
class JsonExporterMeta(abc.ABCMeta):
_filenames = set()
def __new__(cls, name, bases, namespace):
# first execute abc logic
new_cls = super().__new__(cls, name, bases, namespace)
"""
There is no need to run validations against abstract class
"""
if inspect.isabstract(new_cls): # 2
return new_cls
"""
Validate if _filename is a string
"""
if not isinstance(namespace['_filename'], str):
raise TypeError(f'_filename attribute of {name} class has to be string!')
"""
Validate if a _filename has a .json extension
"""
if not namespace['_filename'].endswith('.json'):
raise ValueError(f'_filename attribute of {name} class has to end with ".json"!')
"""
Validate uniqueness of _filename among other subclasses.
This uses a metaclass attribute _filenames - a set of strings
to remember all _filenames of subclasses
"""
if namespace['_filename'] in cls._filenames:
raise ValueError(f'_filename attribute of {name} class is not unique!')
cls._filenames.add(namespace['_filename'])
return new_cls
"""
Now, we will not be inheriting from abc.ABC but we will use our new metaclass instead
"""
class JsonExporter(metaclass=JsonExporterMeta):
pass # The rest of the class remains unchanged, so I skipped it
class BadExporter(JsonExporter):
_filename = 0x1233 # That's going to fail one of the checks
def _build_row_from_raw_data(self, raw_data):
return {'invoice_uuid': raw_data[0]}Speaking of ABC, this is another thing that's implemented using metaclasses. See this talk by Leonardo Giordani - Abstract Base Classes: a smart use of metaclasses.
Registering subclasses - extendable strategy pattern
Using metaclasses attributes we can also write a clever, Open-Closed implementation of a factory. The idea will be based on keeping a registry of concrete (non-abstract) subclasses and building them using a name:
class StrategyMeta(abc.ABCMeta):
"""
We keep a mapping of externally used names to classes.
"""
registry: Dict[str, 'Strategy'] = {}
def __new__(cls, name, bases, namespace):
new_cls = super().__new__(cls, name, bases, namespace)
"""
We register each concrete class
"""
if not inspect.isabstract(new_cls):
cls.registry[new_cls.name] = new_cls
return new_cls
class Strategy(metaclass=StrategyMeta):
@property
@abc.abstractmethod
def name(self):
pass
@abc.abstractmethod
def validate_credentials(self, login: str, password: str) -> bool:
pass
@classmethod
def for_name(cls, name: str) -> 'Strategy':
"""
We use registry to build a better class
"""
return StrategyMeta.registry[name]()
class AlwaysOk(Strategy):
name = 'always_ok'
def validate_credentials(self, login: str, password: str) -> bool:
# Imma YESman!
return True
# example
Strategy.for_name('always_ok').validate_credentials('john', 'x')Warning: for this to work, we need to import all of the subclasses. If they were not loaded to the memory of interpreter, they simply will not be registered.
A less abstract example could be a plugin system for a linter (think about Pylint or flake8). By subclassing abstract Plugin class, we would be not only providing our custom checks but also registering the plugin. By the way, if you are looking how to write such a plugin for Pylint - check out my article on Writing custom checkers for Pylint.
A declarative way of building GUI
Credit for this way amazing metaclasses application goes to Anders Hammarquist - author of EuroPython talk Metaclasses for fun and profit: Making a declarative GUI implementation.
Basically, the idea is to turn imperative code responsible for building a GUI out of components...
class MyWindow(Gtk.Window):
def __init__(self):
super().__init__(self, title="Hello, window!")
self.box = Gtk.VBox()
self.add(self.box)
self.label = GtkLabel(label="Hello, label!")
self.box.add(self.label)...into this:
class Top(Window):
title = "Hello, window!"
class Group(VBox):
class Title(Label):
label = '"Hello, label!"'Just wow. It's impressive, because not only simplifies the resultant code, but also fits our minds much better. Visual composition by nesting classes looks more natural to us, given that the end result is very similar - components nested within each other.
If you are interested about implementation details (and problems author had to overcome) see the talk. Code is available on bitbucket: https://bitbucket.org/iko/ep2016-declarative-gui/
Adding attributes - Django ORM's Model.DoesNotExist
If you have some experience with Django, you surely noticed that each model class gets dedicated DoesNotExist exception. The latter is an attribute on the class. But where does it come from? Well, Django uses metaclasses. For models, it does a lot of things from validation to dynamically adding a few attributes, i.e. DoesNotExist and MultipleObjectsReturned exceptions.
# django/db/models/base.py:128
if not abstract:
new_class.add_to_class(
'DoesNotExist',
subclass_exception(
'DoesNotExist',
tuple(
x.DoesNotExist for x in parents if hasattr(x, '_meta') and not x._meta.abstract
) or (ObjectDoesNotExist,),
module,
attached_to=new_class))
new_class.add_to_class(
'MultipleObjectsReturned',
subclass_exception(
'MultipleObjectsReturned',
tuple(
x.MultipleObjectsReturned for x in parents if hasattr(x, '_meta') and not x._meta.abstract
) or (MultipleObjectsReturned,),
module,
attached_to=new_class))Honourable mention: __init_subclass__
As you have noticed, metaclasses are quite verbose. For example, if we want to affect the entire hierarchy of classes, we need at least two classes (one for the metaclass and another one for the base class).
There is also a risk of falling into metaclasses conflict if we try to apply it in the middle of a hierarchy. For example, you won't be able to just use a custom metaclass for your Django model. You would have to use the trick I was leveraging - create a subclass of django.db.models.Model (django.db.models.base.ModelBase), write your own logic in __new__ and then create your own base class for all models, instead of using django.db.models.Model. Sounds like a lot of work.
Luckily for us, since Python3.6 there is another hook available: __init_subclass__. It is able to replace majority (if not all) metaclasses.
class Strategy(abc.ABC):
_registry: Dict[str, 'Strategy'] = {}
def __init_subclass__(cls, **kwargs):
"""
This is implicitly a classmethod. It will be called only for classes lower in hierarchy, not for Strategy.
"""
super().__init_subclass__(**kwargs)
Strategy._registry[cls.name] = cls
@property
@abc.abstractmethod
def name(self):
pass
@abc.abstractmethod
def validate_credentials(self, login: str, password: str) -> bool:
pass
@classmethod
def for_name(cls, name: str) -> 'Strategy':
return StrategyMeta.registry[name]()
class AlwaysOk(Strategy):
name = 'always_ok'
def validate_credentials(self, login: str, password: str) -> bool:
return True
# End result is the same
Strategy.for_name('always_ok').validate_credentials('john', 'x')For details, see PEP 487.
Summary
I know what you are thinking - all these metaclasses are useful only if you are writing a framework. That would be correct, but up to the point. If you discover a repetitive pattern in your project, why not think about metaclasses?
The "meta" part is the key to see their usefulness ;) When you see that you're spending a lot of time or making mistakes while doing "the work", maybe there's a room for some "metawork"!
Metaclasses can be used to the enforcement of conventions, guiding implementation or enable easy extendability.
Do you know any other cool application of metaclasses? Share them in comments!
Source of image: https://pixabay.com/pl/illustrations/excalibur-miecz-polanie-kaprys-3445952/
Comments powered by Disqus.