Leveraging Typing.Protocol: Faster Error Detection And Beyond Inheritance

Introduction

Two weeks ago I wrote an article about ABCs and interface enforcement. Shortly after I learned that you can do this as well with protocols.

Python 3.8 introduced quite a groundbreaking feature that further advanced the language’s capabilities in type checking: the typing.Protocol which allows Python developers to define and enforce interface contracts in their code.

Unlike traditional dynamic typing, which Python is well-known for, typing.Protocol brings a layer of static type checking that allows for more explicit, readable, and robust code by defining expected behaviors and structures.

This feature not only enhances code quality and developer productivity but also aligns Python more closely with the benefits seen in statically-typed languages, all while maintaining Python’s expressive and dynamic nature.

In this article, we delve into the potential of typing.Protocol through the lens of refactoring the Pybites Search feature, showcasing the immediate benefits of early error detection.

Before we dive in, in case you missed last article, let’s first clarify what “enforcing an interface” actually means …

What is interface enforcement?

Interface enforcement in object-oriented programming (OOP) is a design principle where you define a contract or a blueprint for what methods and properties should be present in a class.

This contract ensures that all classes implementing the interface adhere to a specific structure, promoting consistency and predictability across your codebase.

For example, if you define an interface for a Search functionality, every class implementing this interface must provide the specific methods defined, such as match_content.

This approach allows programmers to design components that can interact seamlessly, knowing the expected behaviors and data types, thereby reducing errors and improving code maintainability.

The ABC / Abstract Method Way

Remember how we implemented PyBites Search originally using Abstract Base Classes (ABCs) to enforce a consistent interface for search functionality:

from abc import ABC, abstractmethod

class PybitesSearch(ABC):
    @abstractmethod
    def match_content(self, search: str) -> list:  # >= 3.9 you can use list over typing.List
        ...

While ABCs served us well here, the interface enforcement only happens at runtime.

Static type checking vs runtime checks

Static type checking offers several advantages over runtime checks, primarily by shifting error detection earlier in the development process.

By identifying type mismatches and potential bugs during coding or at compile time, developers can address issues before they manifest in a running application, leading to more stable and reliable software.

Static type checking also enhances code readability and documentation, as type annotations provide clear expectations for variables, arguments, and return types.

This explicitness improves developer collaboration and facilitates easier maintenance and debugging of the codebase.

Moreover, static type checking can lead to performance optimizations since the interpreter or compiler can make certain assumptions about the types, potentially streamlining execution.

If you are new to Python type hints, check out our article.

Detecting Errors Earlier

Although ABCs and abstract methods enforce an interface, one limitation is that errors can slip through into production.

What if we can detect interface contract breaches earlier?

Enter typing.Protocol! This new feature allows for static type checking, ensuring that not only our class implements the enforced interface, it also does it with the right method signatures.

Here’s how we could refactor Pybites Search to use this (code):

from typing import Protocol

class PybitesSearchProtocol(Protocol):
    def match_content(self, search: str) -> list[str]:
        """Implement in subclass to search Pybites content"""
        ...

class CompleteSearch:
    def match_content(self, search: str) -> list[str]:
        # Implementation of search method
        return ["result1", "result2"]

class IncompleteSearch:
    # Notice that we don't implement match_content here
    pass

def perform_search(search_tool: PybitesSearchProtocol, query: str) -> None:
    results = search_tool.match_content(query)
    print(results)

# Static type checking will pass for CompleteSearch
perform_search(CompleteSearch(), "Python")

# Static type checking will fail for IncompleteSearch
perform_search(IncompleteSearch(), "Python")

In this refactored version, PybitesSearchProtocol defines an interface using typing.Protocol.

Unlike ABCs, classes like CompleteSearch and IncompleteSearch don’t need to inherit from PybitesSearchProtocol to be considered compliant.

Instead, compliance is determined by whether a class implements the required methods, in this case, match_content.

When the perform_search function is called with an instance of CompleteSearch, static type checkers like mypy will confirm that CompleteSearch satisfies the PybitesSearchProtocol because it implements match_content.

However, passing an instance of IncompleteSearch, which lacks the match_content method, will result in a type-checking error.

This approach, using typing.Protocol, offers a more flexible way of enforcing interfaces, particularly useful in scenarios where rigid class hierarchies are undesirable or unnecessary.

It aligns well with Python’s dynamic nature while still leveraging the benefits of static type checking to ensure code correctness.

Static Type Checking

Now let’s see how this protocol is enforced as I try to implement.

Step 1: Missing Method Implementation

Initially, when a class like IncompleteSearch does not implement the required match_content method.

class IncompleteSearch:
    pass

Failing to implement a required method could lead to AttributeError exceptions at runtime, disrupting user experience and potentially halting application functionality.

Running mypy, it catches this error:

$ mypy script.py
script.py:24: error: Argument 1 to "perform_search" has incompatible type "IncompleteSearch"; expected "PybitesSearchProtocol"  [arg-type]
Found 1 error in 1 file (checked 1 source file)

Step 2: Incorrect Method Signature

Next we implement the method but with an incorrect signature:

class IncompleteSearch:
    def match_content(self):
        pass

Implementing a method with the wrong signature may result in a TypeError that is only caught when the specific code path is executed, risking data inconsistencies or application crashes in production environments.

Mypy catches this too:

$ mypy script.py
script.py:25: error: Argument 1 to "perform_search" has incompatible type "IncompleteSearch"; expected "PybitesSearchProtocol"  [arg-type]
script.py:25: note: Following member(s) of "IncompleteSearch" have conflicts:
script.py:25: note:     Expected:
script.py:25: note:         def match_content(self, search: str) -> list[str]
script.py:25: note:     Got:
script.py:25: note:         def match_content(self) -> Any
Found 1 error in 1 file (checked 1 source file)

Step 3: Incompatible Return Type

Finally, we correct the method signature but we return an incorrect type.

class IncompleteSearch:
    def match_content(self, search: str) -> list[str]:
        return (1, 2)

Returning incorrect types from methods can lead to subtle bugs, such as incorrect data processing or application logic failures, which may not be immediately apparent and could lead to significant issues over time.

Once again, mypy flags it:

$ mypy script.py
script.py:15: error: Incompatible return value type (got "tuple[int, int]", expected "list[str]")  [return-value]
Found 1 error in 1 file (checked 1 source file)

Incorporating typing.Protocol into our Python codebase not only facilitates earlier error detection and circumvents the inheritance requirement but also subtly enforces the Liskov Substitution Principle (LSP).

By ensuring that objects are replaceable with instances of their subtypes without altering the correctness of the program, typing.Protocol aids in creating more reliable and maintainable code.

This alignment with SOLID principles highlights the broader impact of adopting modern type checking in Python, enhancing both code quality and developer productivity.

Enhancing Protocols with Runtime Checkability

To bridge the gap between static type checking and runtime flexibility, Python’s typing module includes the @typing.runtime_checkable decorator.

This feature allows protocols to be used in runtime type checks, similar to abstract base classes.

Applying @typing.runtime_checkable to a protocol makes it possible to use isinstance() and issubclass() checks, offering a layer of dynamic validation that complements static type checking.

However, it’s important to note the limitation: while runtime checkable protocols can verify the presence of required methods at runtime, they do not validate method signatures or return types as static type checking does.

This offers a practical yet limited approach to enforcing interface contracts dynamically, providing developers with additional tools to ensure their code’s correctness and robustness.

One practical use case for @typing.runtime_checkable is in developing plugins or extensions. When you’re designing an API where third-party developers can provide plugin implementations, runtime checks can be used to verify that an object passed to your API at runtime correctly implements the expected interface.

This can be especially useful in dynamically loaded modules or plugins where static type checks might not catch mismatches. For example, before invoking plugin-specific methods, you might use isinstance(plugin, PluginProtocol) to ensure that the plugin adheres to your defined protocol, enhancing reliability and reducing the risk of runtime errors.

Structural Subtyping and Embracing Python’s Duck Typing

The concept of structural subtyping, formalized through typing.Protocol, is a testament to Python’s commitment to flexibility and the “duck typing” philosophy.

Structural subtyping allows a class to be considered a subtype of another if it meets certain criteria, specifically if it has all the required methods and properties, irrespective of the inheritance relationship.

This approach enables developers to design more generic, reusable components that adhere to specified interfaces without being bound by a strict class hierarchy.

It essentially allows objects to be used based on their capabilities rather than their specific types, echoing the Pythonic saying, “If it looks like a duck and quacks like a duck, it’s a duck.”

By leveraging typing.Protocol, Python developers can enjoy the benefits of static type checking while maintaining the language’s dynamic, expressive nature, ensuring code is both flexible and type-safe. 😍 📈

Conclusion

Adopting typing.Protocol in the PyBites Search feature showcased not just an alternative to ABCs and abstract methods, but also illuminated a path toward more proactive error management and a deeper alignment with Python’s dynamic and flexible design ethos.

By embracing typing.Protocol, we not only enhance our code’s reliability through early error detection but also open the door to more robust design patterns that leverage Python’s strengths in readability and expressiveness. 💡

The exploration of runtime checkability with @typing.runtime_checkable and the discussion around structural subtyping and duck typing further underscore the versatile and powerful nature of Python’s type system. 😍

These features collectively foster a development environment where code is not just correct, but also elegantly aligned with the principles of modern software design.

As Python continues to evolve, tools like typing.Protocol are invaluable for developers aiming to write high-quality, maintainable code that adheres to both the letter and spirit of Python’s design philosophy. 🐍 📈

Leveraging typing.Protocol: Faster Error Detection And Beyond Inheritance

By Bob Belderbos on 9 February 2024

Introduction

What is interface enforcement?

The ABC / Abstract Method Way

Static type checking vs runtime checks

Detecting Errors Earlier

Static Type Checking

Step 1: Missing Method Implementation

Step 2: Incorrect Method Signature

Step 3: Incompatible Return Type

Enhancing Protocols with Runtime Checkability

Structural Subtyping and Embracing Python’s Duck Typing

Further reading

Conclusion

Introduction

What is interface enforcement?

The ABC / Abstract Method Way

Static type checking vs runtime checks

Detecting Errors Earlier

Static Type Checking

Step 1: Missing Method Implementation

Step 2: Incorrect Method Signature

Step 3: Incompatible Return Type

Enhancing Protocols with Runtime Checkability

Structural Subtyping and Embracing Python’s Duck Typing

Further reading

Conclusion

Related articles