1+1 is always 2: Bringing determinism back to AI-Assisted engineering

AI agents can now build a working application from a handful of prompts, and vibe coding demos produce whole systems from scratch that the agents even manage to keep running for a while. What these demos rarely show is the quality of the generated code, because at the proof-of-concept stage nobody looks at it closely.

In a production codebase that many teams change every day, code quality determines how stable the system is, how much it costs to maintain and whether engineers still know what the code does. Among developers, 46% distrust the accuracy of AI tools, against 33% who trust it, and 66% name output that is almost right, but not quite, as their biggest frustration [1]. An analysis of 211 million changed lines of code found that the share of changes associated with refactoring fell from 25% in 2021 to under 10% in 2024, while copy-pasted lines grew from 8.3% to 12.3% [2].

My team builds AI workflows for a large monorepo that holds many projects, technologies and domains. Our conclusion is that an AI agent becomes dependable in a large codebase only when the structure around it is deterministic: The context it receives, the order in which the work runs and the way its consistency is measured.

My team's goal set a high bar for reliability. We wanted a developer who is new to an area of the codebase to deliver a large task while spending as little time as possible reading the plan, since nobody wants to read long plans and a newcomer cannot judge whether a plan is right. By a large task I mean work that, before LLMs, took a very experienced developer three or more weeks of intensive effort. Our target is to deliver it in less than a day.

We tried many techniques, including published best practices from Anthropic and OpenAI, and most worked only part of the time. Instructions in a general rules file were not reliably followed, and skills, the packaged instructions an agent can call, run only when the model decides to call them. In a monorepo the agent can also miss knowledge that the code does not reveal, and even after many runs it cannot be certain that what it found is correct. Piling more instructions into one session does not solve this, because a model's context is a finite resource and its recall of specific details drops as the context grows [3].

The answer we found is path-scoped rules. Claude Code, one of the AI coding tools we use, loads a rule file automatically when the agent reaches files that match a path pattern defined in that rule [4]. As the agent moves through the code, the relevant rules enter its context on their own, every time.

Each rule carries only knowledge the model cannot easily work out from the code, and legacy code is the clearest case. Real codebases keep old solutions nobody has refactored, and if the agent reads one of them while analyzing a task, it will likely copy their outdated patterns and utilities into the new feature. The rule for that area states which implementation is correct today and which one to avoid. We keep every rule as short and simple as possible, because long text does not produce better results, and wherever we can, each domain gets a dedicated rule file. A single large rule file is the one most likely to be ignored.

LangGraph, an open-source framework for multi-step AI workflows, became the second part of our setup because the workflow could not depend on a single AI coding tool such as Claude Code or Codex, and switching models had to be easy.

The first thing the framework gave us is determinism, which I did not expect to have in an AI project. A run executes its steps in exactly the order we defined, every time, and we can debug it like any other deterministic software. Each step's state is saved [5], so we can measure every step and rerun it on its own. Each step can also use its own model, which lets us run benchmarks and quality evaluations and use cheaper models for steps that do not need the strongest one.

Our workflow starts with the user's requirements and an analysis of the code for similar functionality, after which the user confirms which projects the change will affect. Each project gets its own analysis and its own design, and the designs are combined into one document that the user reviews until it is ready. Only then is the design split into small and medium tasks, which are implemented in parallel and validated.

Every step runs in its own agent session and receives only the input it needs. The context stays short, so the tool never has to compress a long session and lose details. The same control lets us inject system prompts that give our rules more weight. For a small bug with a narrow blast radius, none of this machinery is needed.

My team then had to prove that the workflow gives consistent results. The obvious test was to run the same task many times and compare the code, but each run takes about an hour, our Claude Code subscription does not have enough tokens for that many runs, and we did not want to wait a day or more for results. The test also had to show where a codebase this size still lacked rules.

We made one assumption that changed the approach. The structure of an implemented feature is decided in its design document, and once the design is correct, implementation is relatively simple for most modern AI coding tools. So the design became the thing we measure.

A design is mostly text, which is hard to compare, but ours also contains two structured blocks: A file tree of the files to be changed, created or removed, and a dependency diagram that shows how the new code will be used. If the process is consistent, these two blocks should come out identical or very close every time the design for the same feature is generated.

To test this, we take a pull request that engineers checked by hand and accepted as the expected implementation of a feature, our ground truth. A LangGraph script creates a git worktree (a separate working copy of the repository), gives the agent the ticket for that feature and runs the original workflow up to the design phase several times in parallel. A deterministic script then compares each run's file tree and dependency diagram with the ground truth and produces a report with metrics for every file and every dependency path.

The report shows how consistent the design phase is in each area of the code and points to the areas where rules are missing or poorly written. Every fix can then be checked against the same benchmark.

Our work with AI continues as models and tools gain new capabilities every day, but I keep reminding engineers on my team that the code we ship has to return the expected result every time. In arithmetic, 1+1 is always 2, and the software that businesses depend on deserves the same certainty.

Maxim Magaziner is a Staff Software Engineer and Tech Lead at Taboola in Los Angeles, with more than ten years of experience in software engineering. His work focuses on frontend architecture, developer productivity in large codebases and AI-assisted engineering. He holds a B.S. in Computer Software Engineering from SCE College of Engineering, where he received the Best Final Project Award.

This article was written in cooperation with Maxim Magaziner