It Worked Yesterday

Why Prompt Regression Testing Is the Canary in the AI Coal Mine

“The prompt hasn’t changed.” “We didn’t update the code.” “But the output is completely different…”

Is this your new reality as well? You are not alone (if that matters to you). Welcome to Building with Large Language Models (LLMs), where the prompt that saved you an hour yesterday now yields gibberish, all thanks to progress through silent model updates.

Prompt-based applications are drifting. And without strategies to catch that drift, we are flying blind.

Potential Failures (That Will Happen to Someone)

The Legal Summary That Forgot the Fine Print

Your app summarizes NDAs into bullet points for product managers. A prompt that once reliably included "termination clauses" suddenly stopped including them after last week's model update. Nobody noticed until someone violated the NDA.

The Email Copilot That Turned Passive-Aggressive

A customer support tool goes from “We appreciate your patience” to “You should’ve read the manual.” Same prompt. Different tone. A former client.

The Code Generator That Started Gaslighting Coders

A prompt used to return accurate SQL queries. Suddenly, it starts JOINing on the wrong column. The QA team is confused. The developers are confused. The LLM… has no memory.

So Why Aren’t We Testing Prompts Like Code?

In traditional software development, we write unit tests, integration tests, and build CI/CD pipelines to catch errors like this for us. With LLMs, at least for now, most of us are skipping this step.

Why? Because prompt engineering felt like having a conversation with a junior developer (kind of like you told us to treat it). But now:

  • We have hundreds of prompts scattered across dozens of applications.
  • Prompt-based agents are making decisions in production.
  • APIs and models are updated behind the scenes on a weekly basis.

We wouldn’t deploy microservices without tests. So why are we flying blind with the AI that sends our emails and summarizes our contracts?

What a Manual Prompt Test Looks Like

Let’s say we want to test this:

Prompt: "Summarize the NDA below for a junior product manager."
Input: [Full NDA]
Expected Output: Includes 'termination clause', 'third party', avoids legalese

Step-by-step workflow:

  1. Run the prompt manually against GPT-4.
  2. Copy/paste the output into a document.
  3. Compare against last week’s output.
  4. Highlight missing/changed phrases.
  5. Guess whether the changes are harmful.
  6. Repeat for 73 more prompts.

You’ll burn out before you catch the silent hallucination that ships to users.

🤖 What an Automated Prompt Regression Test Looks Like

  1. ✅ Store each prompt in versioned YAML/JSON.
  2. ✅ Run it daily or per-model-release across GPT-4, Claude, Gemini.
  3. ✅ Compare outputs:
    • Exact match? ✅
    • Semantic similarity > 0.90? ✅
    • Key phrases still present? ✅
    • Tone classifier still friendly? ✅
  4. ❌ Detects drift. Flags for review.
  5. 🚨 Slack alert: “Prompt summarize_nda regressed on model gpt-4-june5. Missing: ‘termination clause’.”

What Happens Without It?

  • Drifted prompts silently erode user trust
  • Compliance tools become liabilities
  • “Hallucinations” get blamed on end users
  • Everyone loses confidence in the tools they’re supposed to trust

The Ask

If you are already building LLM-backed tools, ask these questions:

  • Do we know which prompts are business-critical?
  • Can we detect if their behavior changes tomorrow?
  • Do we have a playbook—or even a canary test—to catch LLM drift?

If not, the consequences will show up in your inbox—or worse, your legal department.