Running tests in parallel with pytest

Running tests takes time. Even if they are insanely fast. The execution time adds up once you add more and more tests to your project. You might not notice that when you have 100 tests. You'll for sure notice it once you have 10000 tests to run. So what can we do to minimize the execution time? The first obvious answer is – make tests faster. That's a great thing to do. Anyhow, there's a limit on how fast your tests can run. Fortunately, we can parallelize test executions. In this article, we'll take a look on how to run tests in parallel with pytest.

Running tests in parallel using pytest and multiple CPU cores

The first option for test parallelization is utilization of multiple available CPU cores. Fortunately, there are pytest plugins that can help you with that. The most widely used one is pytest-xdist. You can install it like any other Python package:

$ pip install pytest-xdist

Once installed, you can use it to execute tests in parallel:

$ pytest -n auto

pytest-xdist first checks the number of CPU cores. Then it spawns the number of workers that matches number of CPU cores and randomly distributes collected tests across the workers. Bear in mind that this comes with an overhead of multiprocessing. Therefore, it doesn't make sense to use it if your tests execute fast (e.g., ~10 seconds). Nevertheless, using pytest-xdist can decrease execution time when there are lot of tests to run. In such case, multiprocessing overhead becomes neglectable.

Note: pytest-django supports test execution with pytest-xdist. It creates database for each worker.

Running tests in parallel using pytest and multiple CI/CD jobs

Spreading test execution across multiple CPU cores is great. Anyhow, it might not the best solution for every problem. In CI/CD pipelines, it can be more effective to split tests into groups and run each group inside its own CI/CD job. Jobs can be executed across multiple different machines.

There, you have two options: pytest-test-groups or pytest-shard. One or the other, the idea is the same. All tests are collected, but only a subset of tests (a test group) is executed – 1-100, 101-200, … Each group has the same size.

Note: If you're using CircleCI, you might want to look into pytest-circleci-parallelized.

Using test groups, you can utilize CI/CD job parallelization to execute all groups. Example for Gitlab CI/CD would look like this:

tests:
  image: python:3.10-slim-bullseye
  parallel: 3
  before_script:
    - pip install -r requirements.txt
  script:
    - pytest --test-group-count $CI_NODE_TOTAL --test-group=$CI_NODE_INDEX

CI_NODE_TOTAL and CI_NODE_INDEX are environment variables provided by Gitlab CI/CD. You have something similar for CircleCI – CIRCLE_NODE_INDEX and CIRCLE_NODE_TOTAL.

Which one to choose?

Two slightly different approaches that address the same problem from slightly different angle. So the question is which one to choose? If you want to speed up test execution on a single machine, pytest-xdist is the right choice. If you want to run your tests across parallel CI/CD jobs, use test grouping. Feel free to combine pytest-xdist and test grouping to squeeze the maximum performance out of your CI/CD pipeline.