Characterization Tests: The Regression Tests Nobody Writes

8 min read

Came here from Instagram, LinkedIn, X or YouTube? Here is the full source list you came for. Everything below is built from these, nothing else.

The article

• The Regression Tests Nobody Writes: Capturing How Software Actually Behaves, Stack Overflow Blog, Oct 22 2026

Before you unplug anything behind your TV, you take a photo of the cables.

You don't know why the HDMI cable goes into port 3 instead of port 1. Maybe it matters. Maybe it doesn't. The photo doesn't care. It doesn't say the setup is right. It just records how things are, so if something stops working after you move the TV, you can compare and see exactly what changed.

A characterization test is that photo, taken of your code.

It doesn't check what the code should do. It records what the code does right now, quirks and bugs included, and complains the moment that changes. A recent Stack Overflow Blog piece makes the case that this is the regression test almost nobody writes, and that it guards the exact gap where the nastiest regressions live. Here is the idea, how it works, when it's worth it, and a small example you can copy.

Key takeaways

• Most tests record intent. Characterization tests record actual behavior, which is what your users really depend on.
• Hyrum's Law (named after Hyrum Wright at Google): once enough people use your API, every observable behavior gets depended on by somebody.
• The idea was named by Michael Feathers in Working Effectively with Legacy Code: before you touch code you don't fully understand, pin what it does now.
• You pin the behavior even when it looks like a bug. The point is that any change becomes a decision, not an accident.
• Recording real production traffic turns a whole service into a characterization suite, including the database calls and messages it sends.
• They pay off most in 4 situations: legacy refactors, framework upgrades, rewrites, and AI-assisted changes.

What is a characterization test?

A characterization test captures how a piece of code behaves today and fails when that behavior shifts.

That sounds like any test. The difference is where the expected value comes from. In a normal test, you decide the answer first and make the code live up to it. In a characterization test, you run the code, look at what comes out, and write that down as the answer.

The Stack Overflow piece frames it as a mental flip. A normal test says "here's what I want, code, please live up to it." A characterization test says "here's what's real, and if you change it, you'd better have a reason."

Michael Feathers gave them the name in Working Effectively with Legacy Code. His use case was simple: you're about to change code you don't fully understand, so first you build a safety net out of what it currently does. Correctness is not the goal. Noticing change is.

Why don't your existing tests catch these regressions?

Because your existing tests only check what somebody thought to check.

Most regression tests get written alongside a feature. This function returns that. This endpoint rejects bad input. Those are good tests. But a system that has run in production for a few years picks up a second layer of behavior that nobody planned and nobody wrote down. The article lists the kind of thing it means: a default value someone picked in 2019 for reasons nobody remembers, weird edge cases, the order results come back in.

That is where Hyrum's Law bites. Once enough people use your API, somebody depends on all of it, not just the documented contract:

• The order results come back in
• The exact wording of an error
• How long a call takes
• Whether an empty field shows up as null or is missing entirely

If you've maintained anything popular, you know the pattern. You fix something that was obviously wrong, ship it, and within the hour someone reports that you broke their integration. It was built on the wrong behavior. From their side, you did break it.

The article's summary is the one worth remembering: every long-lived codebase has two specs. One lives in its tests. The other lives in its behavior. Most regression testing only protects the first.

How does a characterization test work, step by step?

Here is the loop, in the order you actually do it.

1. Pick code you're scared to change. A gnarly function, a report generator, a service nobody wants to touch.
2. Throw inputs at it. Normal ones, empty ones, odd ones. The more strange cases, the better the net.
3. Record every output. Don't predict. Run the code and look.
4. Pin what you saw. Write each output into a test as the expected value.
5. Pin the bugs too. If an output looks wrong, pin it anyway and leave yourself a note. You're not saying it's fine. You're making sure that if it changes, someone chose to change it.
6. Strip out the noise. Timestamps and generated IDs change on every run. Ignore those specific fields and nothing broader, or the real signal drowns.
7. Make your change and read the diff. A red test here doesn't mean "bug". It means "behavior changed".
8. Decide. If you meant it (you fixed the quirk on purpose and warned the people relying on it), the new output becomes the baseline. If you didn't, you just caught a regression no intent-based test would ever have noticed.

Step 5 is the one that feels wrong. Writing a test that asserts behavior you're fairly sure is a bug goes against every developer instinct. That, plus the sheer slog of steps 2 to 4, is why the article says almost nobody writes these.

How do golden masters and snapshot tests make this cheaper?

The slog has had fixes for years. Golden master testing, which some people call approval testing, captures a large chunk of output once, saves it as the blessed version, and compares every future run against it. Frontend snapshot tests run on the same instinct.

Both cut the cost because you capture everything at once instead of asserting one field at a time. They work best on things you can call directly: a function, a report generator, a rendered component.

Can you characterize a whole service?

This is the part of the article I found most useful. A service is harder, because its behavior isn't only what it returns. It's which queries it runs, which other services it calls, and which messages it fires off.

The article's answer: the best record of how your service behaves is already in its traffic. Every real request and response is a characterization test that hasn't been written yet. Here it is as a diagram in words:

• Record: real request comes in → service responds → capture both, plus every call the service makes to its database, queues and other services along the way.
• Replay: send the same recorded requests to the changed service → answer its dependency calls from the same capture → compare.
• Diff: different response? It shows. Same response, but an extra database call? That shows too.

Recording at the network level means you don't touch application code, and it picks up database and queue traffic along with HTTP. Because dependencies are answered from the capture, every run behaves the same way.

That last diff case deserves attention. Some of the most expensive regressions keep the response identical while changing what happens underneath, like a new query that is harmless in testing and brutal at scale. An intent-based test that only checks the response will pass. A traffic-based suite won't.

When should you use characterization tests?

The article names four situations where they earn their keep.

Situation	Why characterization tests help
Refactoring legacy code	The original case Feathers wrote about. Pin the behavior, then move things around without the cold sweat.
Framework and library upgrades	A new major version of your web framework or ORM can change how something serializes or what order results come back in. Intent tests rarely look at that.
Rewrites	When you port a service to a new language or architecture, the old service's recorded traffic becomes the spec. Send the same requests to both. Every difference is either a deliberate improvement or a regression.
AI-assisted changes	When an assistant drafts a big refactor and updates the tests in the same pass, a baseline captured beforehand is one of the few checks that doesn't share the change's assumptions.

The last row is the one I'd underline right now. If the same tool writes the change and edits the tests, the tests can quietly agree with the change. A snapshot taken before anyone touched anything can't.

When should you skip them?

They are not a replacement for normal tests, and they are not free. Some honest limits:

• Brand new code. If you're writing a feature today, you know the intent. Write intent-based tests. There is no hidden layer of behavior yet to capture.
• Code you're about to change on purpose everywhere. If every output is meant to change, a characterization suite will just go fully red and tell you what you already know. It's more useful to pin the parts that should stay the same.
• When you can't control the noise. If you can't isolate the fields that change every run, the suite fails constantly and people stop reading the diffs. The fix is to ignore those specific fields only. Ignoring whole sections makes the suite quietly dishonest.
• When nobody will review the diff. The value is in the conversation: did we actually want this? If red results get blindly re-approved, you have a ritual, not a safety net.

That third point is where most teams go wrong. Ignore the timestamp field, not the whole response.

What does a characterization test look like in code?

Here is a small Python example. Say you inherit this function and need to refactor it:

# legacy.py
def display_name(user):
    first = user.get("first", "").strip()
    last = user.get("last", "").strip()
    if not first and not last:
        return "Guest"
    if not last:
        return first
    return f"{last.upper()}, {first}"

You don't write tests for what it should do. You call it with a spread of inputs, look at what comes back, and pin every answer, including the odd one:

# test_display_name.py
from legacy import display_name

def test_full_name():
    assert display_name({"first": "Asha", "last": "Rao"}) == "RAO, Asha"

def test_first_name_only():
    assert display_name({"first": "Asha"}) == "Asha"

def test_no_name():
    assert display_name({}) == "Guest"

def test_last_name_only_quirk():
    # Looks like a bug (trailing comma and space). Pinned on purpose.
    # Change it only if we decide to, and tell whoever reads this output.
    assert display_name({"last": "Rao"}) == "RAO, "

That last test is the whole idea. Somebody downstream might be splitting on that comma. Now, if your refactor "fixes" it, the test goes red and you have to make a call instead of shipping a surprise.

For bigger outputs, the golden master version captures everything at once and ignores only the fields that change every run:

# test_report_golden.py
import json
from pathlib import Path

from reports import build_report, sample_input

GOLDEN = Path("tests/golden/report.json")
NOISY_FIELDS = {"generated_at", "request_id"}

def scrub(value):
    if isinstance(value, dict):
        return {k: "<ignored>" if k in NOISY_FIELDS else scrub(v) for k, v in value.items()}
    if isinstance(value, list):
        return [scrub(v) for v in value]
    return value

def test_report_matches_golden():
    actual = scrub(build_report(sample_input()))
    if not GOLDEN.exists():
        GOLDEN.parent.mkdir(parents=True, exist_ok=True)
        GOLDEN.write_text(json.dumps(actual, indent=2))
    expected = json.loads(GOLDEN.read_text())
    assert actual == expected

The first run writes the blessed file. Every run after that compares against it. Lists are compared in order, so if your ORM upgrade quietly changes the order results come back in, this test catches it. Commit report.json and the diff shows up in your pull request, which is where the article says the healthy habit forms: behavior changes stop being accidents and start being decisions.

reports, build_report and sample_input stand in for your own code. Swap in whatever you're scared to touch.

FAQ

What is the difference between a characterization test and a regression test?

A typical regression test records intent: what someone decided the code should do when they wrote the feature. A characterization test records actual current behavior, including quirks and bugs. Both catch changes, but only the second catches changes to behavior nobody ever wrote down.

Should a characterization test pin a bug?

Yes. Pinning it doesn't mean the bug is acceptable. It means that if the behavior changes, someone decided to change it, ideally after warning anyone who relies on it. Leave a note in the test so the next person knows it was deliberate.

Are snapshot tests the same as characterization tests?

They come from the same instinct. Snapshot tests and golden master (approval) tests are cheap ways to characterize output, because they capture a large chunk at once and compare future runs against it. For a whole service, you also need to capture what it sends to its dependencies, which is where recording real traffic comes in.

How do I stop timestamps from breaking characterization tests?

Ignore the specific fields that change every run, like timestamps and generated IDs, and nothing broader. If you ignore whole sections to make the noise go away, you also hide the real changes the suite exists to catch.

Pick one function or service you're a little scared to change, capture what it really does before your next refactor, and read every line of the diff. Then browse the free tools on Art of Code for the small jobs around it, or look at developer jobs if you want to work on codebases where this matters.

Written by Adithya Guttha, Founder of Art of Code.