Assertions About Exceptions With pytest.raises()

I got some feedback related to Bite 243 recently. Since that’s a testing bite, it means working with pytest and specifically checking for exceptions with pytest.raises(). The comment got me to look at this handy feature of pytest with fresh eyes, and it seemed like a trip worth sharing!

pytest.raises() as a Context Manager

We can uses pytest.raises() to assert that a block of code raises a specific exception. Have a look at this sample from the pytest documentation:

def test_recursion_depth():
    with pytest.raises(RuntimeError) as excinfo:

        def f():
            f()

        f()
    assert "maximum recursion" in str(excinfo.value)

Is that test reasonably clear? I think so. But see how that assert is outside the with block? The first time I saw that sort of assertion, it felt odd to me. After all, my first exposure to the with statement was opening files:

with open('my_delicious_file.txt') as f:
    data = f.read()

When we get comfortable using open() in a with block like that, we pick up some lessons about context manager behavior. Context managers are good! They handle runtime context like opening and closing a file for us, sweeping details under the rug as any respectable abstraction should. As long as we only touch f inside that with block, our lives are long and happy. We probably don’t try to access f outside the block, and if we do things go awry since the file is closed. f is effectively dead to us once we leave that block.

I didn’t realize how much I had internalized that subtle lesson until the first time I saw examples of pytest.raises. It felt wrong to use excinfo after the with block, but when you think about it, that’s the only way it can work. We’re testing for an exception after all – once an exception happens we get booted out of that block. The pytest docs explain this well in a note here:

Note

When using pytest.raises as a context manager, it’s worthwhile to note that normal context manager rules apply and that the exception raised must be the final line in the scope of the context manager. Lines of code after that, within the scope of the context manager will not be executed. For example:

>>> value = 15
>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...     assert exc_info.type is ValueError  # this will not execute

Instead, the following approach must be taken (note the difference in scope):

>>> with raises(ValueError) as exc_info:
...     if value > 10:
...         raise ValueError("value must be <= 10")
...
>>> assert exc_info.type is ValueError

Under the Covers

What I didn't think about until recently is how the open()-style context manager and the pytest.raises() style are mirror-world opposites:

How does this work under the covers? As the Python documentation notes, entering a with block invokes a context manager's __enter__ method and leaving it invokes __exit__. Check out what happens when the context manager gets created, and what happens inside __enter__:

def __init__(
    self,
    expected_exception: Union["Type[_E]", Tuple["Type[_E]", ...]],
    message: str,
    match_expr: Optional[Union[str, "Pattern"]] = None,
) -> None:
    ... snip ...
    self.excinfo = None  # type: Optional[_pytest._code.ExceptionInfo[_E]]

def __enter__(self) -> _pytest._code.ExceptionInfo[_E]:
    self.excinfo = _pytest._code.ExceptionInfo.for_later()
    return self.excinfo

So that excinfo attribute starts empty - good, there's no exception yet! But in a nod to clarity, it gets a placeholder ExceptionInfo value thanks to a for_later() method! Explicit is better than implicit indeed!

So what happens later when we leave the with block?

    def __exit__(
        self,
        exc_type: Optional["Type[BaseException]"],
        exc_val: Optional[BaseException],
        exc_tb: Optional[TracebackType],
    ) -> bool:
        ... snip ...
        exc_info = cast(
            Tuple["Type[_E]", _E, TracebackType], (exc_type, exc_val, exc_tb)
        )
        self.excinfo.fill_unfilled(exc_info)
        ... snip ...

Pytest checks for the presence and type of an exception, and then it delivers on its for_later() promise by filling in self.excinfo.

A Summary in Three Parts

With all that background out of the way, we can see the three-act play of excinfo's life - from nothing, to empty, to filled:

def __init__(...):
    self.excinfo = None  # type: Optional[_pytest._code.ExceptionInfo[_E]]

def __enter__(...):
    self.excinfo = _pytest._code.ExceptionInfo.for_later()
    return self.excinfo

def __exit__(...):
    self.excinfo.fill_unfilled(exc_info)

Which shows up in our test code as:

with pytest.raises(RuntimeError) as excinfo:  # excinfo: None
    # excinfo: Empty
    def f():
        f()

    f()
# excinfo: Filled
assert "maximum recursion" in str(excinfo.value)

And that's a beautiful thing!

-- AJ

References

With Statement Context Managers (python docs)
pytest.raises (pytest docs)
Assertions about excepted exceptions (pytest docs)
PEP 343 - The "with" statement