AI-generated code fails in predictable ways. Deterministic checks catch them before you ship

Most of the code in a vibe-coded app was written by a model. The review, if there was one, was done by the same model that wrote it.

That setup has a problem you can measure. Ask an LLM to review the same file twice and you get two different lists of findings. The model that produced the bug also tends to miss it on review, because the blind spot that created it is still there. And when something breaks in production, you have no record of what was checked before you shipped.

I run strategy at Quality Clouds. We've spent 9 years scanning code on some of the biggest enterprise platforms, across 950+ instances, and we've cataloged more than 13 million issues doing it. This year we took that experience and built Norma, a deterministic checker for AI-generated code that runs inside the agent loop. Here's what we learned about how this code actually fails, and how we think review has to change when an agent writes the code.

The failure patterns are boring, and that's the point

AI-generated code rarely fails in exotic ways. It fails in patterns, and the same ones keep coming back:

• Secrets where they don't belong. The classic in Supabase projects is a service role key that ends up reachable from the client bundle, which lets anyone bypass row level security from the browser.
• Missing security headers on generated front ends. Vite scaffolds ship without a Content-Security-Policy unless someone adds one, and agents almost never do.
• Validation that exists in the happy path and nowhere else. The model writes the form, writes the endpoint, and quietly trusts the input.
• Dead weight. Duplicated helpers, unused dependencies, and copy-pasted patterns that make the next agent session worse, because the model learns from the mess it's editing.

A simple rule finds every one of these, and a rule runs the same way on every pass, no matter how long the session has gone on.

Why deterministic beats "ask the model to review it"

The common answer to code quality in agent workflows is a rules file: drop your standards in CLAUDE.md or .cursorrules and tell the model to follow them. That's worth doing. The standards reach the model.

The verdict is still probabilistic. Run the same review twice and the findings shift. Which means you can't build a process on it: you can't tell a customer, an auditor, or your own future self that a given check passed, because next run the same code gets a different answer.

A deterministic engine gives you two things a model can't:

1. Repeatability. Same code in, same findings out, every single run.
2. A record. Each check is logged: what ran, on what code, with what result. When someone asks "was this reviewed?", you have an answer that isn't "the agent said it looked fine."

That second point matters more than indie builders expect. The moment your side project gets a paying customer with a security questionnaire, "deterministic checks with an audit record" is the difference between a 10-minute answer and a lost deal.

What "inside the loop" means in practice

Checks that run after the pull request is open are checks the agent never sees. By then the code is written, the context is gone, and the fix is a new conversation.

So we built Norma as an MCP server. Your agent connects to it like any other tool, and review happens while the code is being written:

• In Cursor, Claude, or Windsurf, the agent calls a Livecheck on the file it just wrote, gets structured findings back, and fixes them in the same session, while it still has the context.
• In Lovable, one standing instruction ("check every change with Norma before finishing") makes enforcement part of the project, and the agent follows it on each iteration.
• In VS Code, an extension runs Livecheck on the active file and puts findings in the Problems panel, with a "Fix with AI" action that hands the finding to your editor's AI chat. It never edits your code itself.

Norma detects the repo's stack on its own and picks the matching rulesets. Today that covers JavaScript, TypeScript, Python, and PHP, with rules for React, Node, Vite, FastAPI, SQLAlchemy, Supabase, and Magento, plus a ruleset for the prompts themselves. A full repo scan produces a Production-Ready Score, a 0 to 100 diagnostic across the areas that matter before real users touch the thing. It's a diagnostic, so it tells you where you stand; it doesn't block your deploy.

Two design decisions I'll state plainly, because indie makers ask about them first: the GitHub App requests read-only access to code and pull requests, and repository code is deleted after each scan.

Try the pattern, with or without us

Even if you never touch Norma, steal the process. Put your standards in a rules file so the model has them. Add a deterministic check the agent can call while it works, so the verdict stops depending on the model's mood. Keep a record of what was checked, so shipping fast doesn't mean shipping blind.

If you want that without building it, we built Norma to do exactly this: deterministic checks for AI-generated code, with a permanent free tier. To start straight away, sign up, connect the MCP server at api.qualityclouds.ai/mcp to Cursor, Claude, or Lovable, and open a demo repository to see what a deterministic pass finds in code an agent wrote.

The agents are getting better at writing code. Review is the part of the loop they shouldn't be grading alone.

Albert Franquesa, Chief of Strategy & Board Member, Quality Clouds