What Is Test Prioritization in Software Testing?

As software projects grow, test suites grow with them. Running thousands of tests sequentially takes time — sometimes hours. Developers waiting for CI feedback slow down: they context-switch, work on unrelated tasks, or lose the flow of the change they were making.

Test prioritization addresses this by running the tests most likely to fail (or most likely to reveal relevant information) first. When a failure occurs early in the run, the CI build fails fast, the developer gets feedback within minutes rather than hours, and the rest of the suite can be skipped.

This matters especially for large-scale software systems where a full test run might take 30 minutes or more.

History-based prioritization — tests that have failed recently are likely to fail again. Sort by recent failure rate.

Coverage-based prioritization — tests that cover the code changed in the current commit are most likely to catch regressions related to that change. Sort by coverage overlap with the diff.

Fault-detection maximization — use code coverage data to order tests so that each successive test covers the maximum number of yet-uncovered statements.

Risk-based prioritization — weight tests by the risk of the code they cover (e.g., critical payment paths ranked above rarely-used admin features).

Order-dependent flaky test prioritization — a specialised strategy: identify pairs of tests most likely to have an order-dependency relationship, and prioritise running those pairs in the orderings most likely to surface the dependency. This is the approach explored in my published research.

Detecting order-dependent flaky tests requires running a test suite in many different orderings — a potentially enormous search space. For a suite of N tests, there are N! orderings to explore.

Prioritization dramatically reduces this space. By analysing which tests access shared state (databases, statics, environment variables), we can predict which test pairs are likely to have a polluter-victim relationship. These pairs are then prioritised for re-execution in orderings that would expose the relationship.

My research at IIT, University of Dhaka (published at the International Flaky Tests Workshop, IEEE/ACM ICSE 2025) proposes a prioritization approach that significantly reduces the number of test re-runs required to detect OD flaky tests, making the detection process practical for large CI/CD pipelines.

Most modern CI systems support some form of test prioritization.

Jest (JavaScript) — supports running failed tests first via --testPathPattern and supports custom sequencers that can implement history-based ordering.

pytest (Python) — the pytest-ordering and pytest-randomly plugins provide ordering control. The pytest-failed-first plugin implements history-based prioritization.

JUnit 5 (Java) — the @TestMethodOrder annotation supports custom ordering strategies including random and display-name-based.

CI/CD tools — GitHub Actions, CircleCI, and BuildKite all support test parallelisation with automatic splitting based on historical timing data, which is a form of implicit prioritization.