Custom exceptions in Python - how and what for?

Posted May 2, 2021

By Sebastian Buczyński 6 min read

Exceptions are a standard way of signalling errors in Python. If you have ever written some code in this language, I bet you saw at least a couple of them. :) Python has quite a few built-in exception classes for all occasions. For example, there's ZeroDivisionError raised when you try to divide by zero. Or ValueError raised on many occasions - also when you try to convert a string to an integer that doesn't look like one:

1 / 0  # doomed to fail
# Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
# ZeroDivisionError: division by zero

int("Python")  # can't do this
# Traceback (most recent call last):
#  File "<stdin>", line 1, in <module>
# ValueError: invalid literal for int() with base 10: 'Python'

Exceptions can be handled using try..except block. With aforementioned examples that's pretty simple to imagine:

from datetime import timedelta, date


while True:
    try:
        year_of_birth = int(input("Enter your year of birth and I'll tell you how old are you!"))
    except ValueError as exc:
        print("Invalid value! Try again.")

    break

...  # rest ommitted, non-essential for the blog post

Did you know you could handle SyntaxError?

What may be surprising, situations like the inability to import an object (ImportError), handling CTRL+C (or CMD+C on MacOS) - KeyboardInterrupt or even wrong syntax - SyntaxError are also signalled through exceptions. This means you can catch them

try:
    eval("""
    def syntactically_wrong_function(): this syntax does not exist
    """)
except SyntaxError as exc:
    print("Oh no!", exc)

I don't recommend doing so. However, handling ImportError is quite a common pattern in libraries to handle optional dependencies. For example, aiohttp has several optional dependencies, e.g. cchardet or brotli. See handling presence or absence of dependency here. The same approach could be used to handle differences between Python 2 and 3 - some modules were moved.

I don't encourage you to catch SyntaxError but be aware that Python is consistent when it comes to handling errors. It's very convenient! Here's a full exceptions hierarchy.

Should you add custom exceptions?

Creating a new exception is trivial - just subclass Exception class:

class MyCustomError(Exception):
    pass

Many libraries choose to allow you for precise error handling, e.g. requests or sqlalchemy. If you delve into the source code, you'll see that their custom exceptions follow Python's conventions. For instance, HTTPError from requests inherits from built-in IOError. Also, exceptions classes bundled with libraries follow naming schema by adding "Error" suffix. The bottom line is that without custom exceptions you would not be able to effectively do any error handling with 3rd party libraries. Built-in exceptions are already used in the standard library, so if you would try to handle IOError, you wouldn't easily know if you are handling HTTP request failure or a problem with reading a file from a hard drive.

So the answer to question whether you should add custom exceptions is yes.

Expressiveness thanks to custom Python exceptions

If by any chance you are writing code that fulfils business requirements, custom exceptions can make the code cleaner, self-explanatory and more intention-revealing. Consider the following project:

An auction allows bidders to compete for an item. Auction has some starting price to encourage bidders and secure against giving item away for very little money. Bidders place bids. If a bid is bigger than current price (inititally equal to starting price), then a bid's amount becomes new current price and a bidder becomes a winner. If a new bid is lower than current price, no change is made to the auction and bid is rejected. Auction lasts until moment of time specified when an auction was started.

Now read that again and think what's the happy path. After that, think about all cases when something may go wrong... and write it as exceptions classes! My propositions:

class BidOnEndedAuctionError(Exception):
    pass

class BidLowerThanCurrentPriceError(Exception):
    pass

class AlreadyAWinnerError(Exception):
    pass

While exceptions like HTTPError or ZeroDivisionError are very technical, BidOnEndedAuctionError is a completely different kind - so-called domain exception. This type of errors belongs to the domain language of the project. Hence, we can use them in conversations with users and stakeholders and be well-understood.

For convenience, I encourage you to introduce a common base exception class for this type of errors:

class DomainException(Exception):
    pass

class BidOnEndedAuction(DomainException):
    pass

...and optionally, strip "Error" suffix. It doesn't add much value, really.

What can you do with domain exceptions?

Obviously, you can catch and handle them whenever this is needed. However, remember about one rule about dealing with exceptions:

Catch exceptions only when you can do something about them

If you cannot handle the exception, let it go through. At least you'll get a much clearer error message. In fact, not only you - often domain exceptions can be safely shown to users. Recently, I've been working on a project with GraphQL with graphene-python and FastAPI. Handling errors there is a bit problematic - by default, the exception's message is returned on the API. It can leak database errors (or worse password or who knows what other sensible information). To mitigate this risk, I introduced DomainException base class. On the API, I only show messages of exceptions that inherit from that class. For others, I return a generic message and report to Sentry.

Catching exceptions just to silence them is super evil and gonna bite you sooner or later. Don't do that.

If you are writing Pythonic code using the Clean Architecture, then custom domain exceptions are an absolute must :)

How to avoid falling victim to defensive programming?

"But Sebastian, what about all those errors that CAN happen? How can you sleep at night without explicitly handling all possible errors?!"

I choose not to. And I recommend you do the same.

This is not C, where a simple mistake can lead to memory access violation and killing the entire program by an operating system. I've been there, done that. In C, a more defensive programming style is justified. But we're talking about Python.

First of all, recall the rule of catching exceptions - do that only if you can do something about it. If not - let it through. Even if you do nothing, in 99% of cases it will be handled higher, especially if you use some kind of a framework. There will be an HTTP 500 error, but no casualties. You should also have some error reporting like Sentry to notify you about it. You won't be able to foresee all possible errors anyway.

Obviously, there is a type of errors that even the framework cannot reasonably handle. But then there is a bigger issue and STILL, you cannot do anything in your code about it.

So don't worry and enjoy your custom exceptions in Python :)