Modular monolith in Python

Posted Dec 8, 2020 Updated Jan 17, 2024

By Sebastian Buczyński 10 min read

Microservices are not the only way

I remember when the microservices boom started. Truth to be told, it still echoes strongly to this day. You could see conferences agendas packed with talks about microservices, articles galore, finally books and frameworks. At some point, I was afraid to open my fridge. Everyone and their dog wanted to work with microservices. But who worked with microservices done wrong, no longer laughs at the circus.

Don't get me wrong - microservices are great, but they cost a lot and distributed architectures are inherently complex. The decision to distribute application should not be made lightly as it takes an enormous toll. To make an informed decision, we might use so-called architectural drivers. Simply saying, these are factors favouring microservice architecture:

Different security concerns for parts of the system,
Need to scale independently (but remember microservice are NOT a magic pixie dust! distributing application alone will make things slower, not faster)
Need to use another programming language or concurrency paradigm
Need separate deployments, e.g. due to various risk
etc - read more here

A short comment about startups. Even if we have a gut feeling that the project we work on eventually will need to be distributed across a cluster of services, do we really want to do it in the most critical moment of a startup (i.e. before it starts bringing revenue)? Setting up Kubernetes, delivery pipelines and other stuff takes time. Wouldn't it be more productive to spend that time on producing something that will potentially bring value for the end-user...?

If not microservices, what other option we have? Not-so-good, ol' scary monolith? Nah. There is a third way - modular monolith AKA modulith!

Modular monolith (or modulith)

The idea is simple. Keep codebase together but thoughtfully packed into separate components. To make boundaries sharp, we put some extra restrictions about using code from one component in another and very consciously model communication between components of Pythonic modular monolith.

Although microservices transformation did not work out for everyone, we learnt some valuable lessons along the way. Features of a good microservice were identified - sometimes the hard way. These are:

autonomy
loosely-coupled with other microservices
communication only via APIs (no direct access to database)

Components of a modular monolith also have these qualities.

How does a component look like inside?

Each component has a public API and private, internal details. The former is meant to be used from the outside while the latter must not be touched. This is a concept of encapsulation known to you (I hope!) from classes applied at a level of the entire component. You can read more about encapsulation in Python here.

Interestingly, each component can have different internal architecture. For instance, the core component(s) with business-critical stuff or the most complex can implement the Clean Architecture. It fosters testability and puts business rules before infrastructural, lower-level concerns. You can see an example here - auctions component implementing the Clean Architecture.

├── application
│   ├── queries
│   ├── repositories
│   └── use_cases
├── domain
│   ├── entities
│   ├── events
│   ├── exceptions
│   └── value_objects
└── tests

Alternatively, we can use a lightweight approach with database models as our domain objects. Yet another case is when our component is merely a wrapper around a 3rd party API. See payments example.

├── api
│   ├── __init__.py
│   ├── consumer.py
│   ├── exceptions.py
│   ├── requests.py
│   └── responses.py
├── config.py
├── dao.py
├── events.py
├── facade.py
├── models.py
└── tests

Allowing for different architectures in each component of a modular monolith is a pragmatic approach. Implementing the Clean Architecture is an investment which may not pay off in less important or simpler parts of the project. For the sake of sanity, I'd limit internal architectures to 2 - the Clean Architecture & some simpler approach.

How does a component look like from the outside?

As mentioned earlier, each component has its own API. Components that implement the Clean Architecture will have a series of Use Cases (possibly followed by Queries)

└── application
    ├── queries
    │   ├── get_single_auction.py
    │   └── get_all_auctions.py
    └── use_cases
        ├── beginning_auction.py
        ├── ending_auction.py
        ├── placing_bid.py
        └── withdrawing_bids.py

while components with less sophisticated internal architecture can use Facade design pattern:

class PaymentsFacade:

    def get_pending_payments(self, customer_id: int) -> List[PaymentDto]:
        ...

    def start_new_payment(self, payment_uuid: UUID, customer_id: int, amount: Money, description: str) -> None:
        ...

    def charge(self, payment_uuid: UUID, customer_id: int, token: str) -> None:
        ...

    def capture(self, payment_uuid: UUID, customer_id: int) -> None:
        ...

It is a bad idea to expose database models or entites/domain objects on the API. Use Data Transfer Objects that can be implemented using attr.s or poor-man's counterpart from the std lib - dataclasses!

@dataclass(frozen=True)
class PaymentDto:
    id: UUID
    amount: Money
    description: str
    status: str

You can use it for both input and output data. TypedDict also works, especially for output data from queries/query methods of Facades.

Testing modular monolith in Python

Each component can have it's own test suite that will exercise it thoroughly. One note, though - focus on testing via public API of a component. Some unit tests may be helpful for nailing down edge cases in certain internal classes/functions, but don't focus on testing each and every piece or you will set your codebase in concrete. The same applies to every codebase! An entire component is a handy UNIT for testing.

Of course, this is not sufficient - we know well that 1 integration test >>> 2 unit tests. So we also need a couple of higher-level tests what will see if our components correctly talk to each other.

How components in a modular monolith written in Python communicate?

The best communication (and dependency) between components is none. However, it cannot be avoided. Let's do it thoughtfully!

Direct call

The simplest variant - we just import Facade/Use Case from other module and call it. It introduces a tight coupling and heavy dependency on the called component. (Which is not a bad thing in itself, think of it as a design trade-off).

# web_app
from auctions import PlacingBid, PlacingBidInputDto

# we can get PlacingBid instance thanks to flask-injector/other Dependency Injection lib
def some_view(request: Request, placing_bid: PlacingBid) -> Response:
    dto = PlacingBidInputDto(...)
    placing_bid.execute(dto)
    ...

Definitely, it's a way to go for REST/GraphQL API component - it does not have the logic of their own, but they are interface for the outer world. See how web_app component calls code from other components.

Direct call via abstract class

It can be useful if a core component that implements the Clean Architecture needs to use another component's functionality. The Clean Architecture is very thoughtful about external dependencies and another component is such a thing.

# e.g. in auctions
class PaymentsPort(abc.ABC):
    @abc.abstractmethod
    def pay(bidder_id: int, amount: Money) -> None:
        pass

# in payments
class PaymentsAdapter(PaymentsPort):
    def pay(bidder_id: int, amount: Money) -> None:
        ... reuse Facade's methods or call 3rd party API right here

Note that the language of auctions (bidder_id) is imposed on payments which burdens payments component with knowledge about auctions.

Events

Last but not least - if something in a component of a modular monolith in Python should trigger some action elsewhere in another component, but implementing port/adapter seems too much, one can resort to events. Framework-agnostic solution can leverage Blinker library:

from blinker import signal

# this creates a named signal which can be referred in other
# area of code without importing anything
# (we just need its name)
bidder_has_been_overbid = signal('bidder_has_been_overbid')

I recommend to use immutable Data Transfer Object for passing event data around:

@dataclass(frozen=True)
class BidderHasBeenOverbid:
    auction_id: int
    bidder_id: int
    new_price: Money
    auction_title: str

Modelling communication best practices

Avoid circular dependencies. Think twice when the language of one component leaks to another. Prefer inferior components to accept the language of superior, more important ones.

For component's unit tests, prefer using stubs of other components. See more details in this article - warning against overusing mocks.

How do I enforce architectural rules?

As for today, I can see two viable ways to do it in Python. First, is to write each component as a separate installable package. I did a similar thing in a project that illustrates my book - Implementing the Clean Architecture. Then, you can install each component separately during a CI build and validate if you don't try to use anything that's not mentioned in the component's requirements.

The second approach is simpler - you can use a plugin for pylint I wrote - pylint-forbidden-imports. It lets you specify allowed imports for each component. Example config:

allowed-modules-dependencies=
    customer_relationship->auctions,  ; customer_relationship can import code from auctions
    web_app->*,  ; web_app can import everything
    *->foundation,  ; foundation can be imported from everything

How do I assemble Pythonic modular monolith?

Just like you assemble objects when dealing with composition, you can assemble components.

In several projects, I used dependency injection library called injector which proved to be extremely valuable as it is not tied to any particular framework. Although it has integrations for Flask and Django as far as I remember. It is also relatively easy to integrate with others. I did in Pyramid and FastAPI.

For this to work, you define a injector.Module in each of your components, e.g. for auctions:

class Auctions(injector.Module):
    @injector.provider
    def placing_bid_uc(
        self,
        repo: AuctionsRepository
    ) -> PlacingBid:
        return PlacingBid(boundary, repo)

Here, you have a provider (factory) for PlacingBid Use Case class which requires a repository. Injector uses type annotations magic to find a corresponding definition of a dependency. It can be found in AuctionsInfrastructure:

class AuctionsInfrastructure(injector.Module):
    @injector.provider
    def auctions_repo(self, conn: Connection) -> AuctionsRepository:
        return SqlAlchemyAuctionsRepo(conn)

All you have to do is to create an instance of injector.Injector, passing list of instances of injector.Modules:

container = injector.Injector(
    [
        Auctions(),
        AuctionsInfrastructure(),
    ]
)

# Then you can get an instance with all dependecies using
containger.get(PlacingBid)

# your code should not use Injector directly - only inject at a top level (e.g. to a view), preferrably using framework integration

I recommend doing it in a separate, framework-agnostic component called for example main. See full example here. Then, different delivery methods of the application (APIs, background task queue, CLI) use main to assemble app and attach proper interface atop of it.

Why should I code using modular monolith approach?

First of all, it is much simpler and cheaper than microservices but allows for nice code separation. It puts some order to chaos. And that's always welcomed.

If we don't make any intentional effort towards keeping project clean, we'll create another scary monolithic ball of mud. Divide and conquer!

Even if the project will have to be split eventually, with modular monolith along the way you can experiment with the design and pay much less price if you get the boundaries wrong.

Where can I go from there?

Being able to organize and write code is one thing, but how does one know where to put what...? That's another story which will be described in another article. For now, I can give you a sneak peek - strategic Domain-Driven Design. It's a pretty formal and well-recognized methodology of taming large projects.

Monolith is dead, long live modulith!