Building a 500 line API regression test suite

This article appeared as a Pybites email first. If you like it consider joining our friends list for weekly Python, developer and (!) mindset content …

Last year I built a cool API to post code images using our pybites-carbon tool. It will store the tip code in a database and store the code image in an S3 bucket.

A few weeks ago I finally made it open source so I am happy to receive contributions etc. 🎉

I am not sure why I kept it private for so long but one reason was a much needed cleanup 😅🤯

OK some lines are black formatting, but ~500 of those lines were me adding a test suite to feel more comfortable about the code (“if it’s not tested it’s broken” < literally, keep reading … 😱) and help make it more robust for the future (there are definitely cool features we can add …)

In this article I wanted to share a few tips how to test API code (this is an API after all) and overall some things I learned (and refreshed) building this test suite 💪

Here we go:

1. Dependency injection

FastAPI is blessed with sublime documentation so the first thing that’s super helpful is its dependency injection system so you can use a different database when running the app vs testing it.

The docs show how to do the DB override using app.dependency_overrides() – you can see this here.

Thanks to this I use an in memory sqlite db for testing which is faster and has no side effects on the development database.

By the way, I put some shared fixtures in conftest.py which pytest automatically picks up.

2. Fixtures

The hard work writing a test suite is getting your code in a certain state, hence apart from the two fixtures in conftest.py the first 100 lines of test_api.py is fixture code.

I needed different users to test different things: unverified user, verified user, user hitting rate limit, etc.

The good news is that once you have your fixtures in place it becomes much easier to write your tests.

For perspective, I really like what Simon Willison said in his brilliant Djangocon talk “Increase your productivity on personal projects with comprehensive docs and automated tests” (a must watch!):

The hard bit of testing is getting a testing framework setup in the first place … Once that’s in place, adding new tests becomes really easy.

https://simonwillison.net/2022/Nov/26/productivity/

So it’s all work upfront but adding more tests became relatively easy at the tail end of this effort.

If you’re new to pytest you definitely need to learn about how to write fixtures, check out our article.

3. Mocking

I had to do some mocking here. The create_tip endpoint uses pybites-carbon (Selenium) and posts the image to an S3 bucket (boto3).

As I just wanted to test the end-to-end API + DB round-trip without external network calls, I ended up with a triple @patch decorator (I also had to mock out os module / file system stuff) which you can see here.

4. Coverage

I added a Makefile with coverage alias to get some metrics.

Although it does not say anything about the quality of the tests per se, it’s still and indication of how well you’re going with the test suite.

And always do PRs 🙂 – one of our coaches, Hugh, taught me about the nice --cov-fail-under switch:

# Makefile
.PHONY: cov
cov:
  pytest --cov=tips --cov-report=term-missing --cov-fail-under=80

5. Naming matters

Overall looking back at the code, just a reminder that it’s ok to use longer test function names: it expresses intent better and they are easier to filter if you want to run them in isolation (using pytest -k).

Also it was nice not to worry about formatting too much, for example with long expected nested data structures, I had black taking care of it (see here for example).

Again, testing is critical. Not surprisingly I caught a few bugs, here is one:

It’s a great feeling when you can merge a PR that gives your code 96% code coverage. And it will be much easier to keep evolving this project from here on.

I hope this inspires you to keep writing tests for all code that you ship 😍

– Bob

Testing is a critical developer skill. There are a couple of more of these holistic skills critical to succeed as a Python developer.

Check out our developer mindset training to learn more and keep your Python journey focused and fun.