How to structure an AI engineering team

Every team shipping an LLM feature eventually hits the same fight. A PM edits a system prompt on Thursday afternoon. The engineer who runs the evals is on holiday. Three product squads are blocked on Friday morning. The retro on Monday concludes "we need better process". Two weeks later the same thing happens with a different prompt.

The pattern is not a process problem. It is an ownership problem, and it hides inside disagreements that sound technical. Whoever edits the prompt is making a product decision. Whoever writes the eval is making that decision on a different day against a different rubric. Whoever picks the embedding model is deciding what retrieval quality even means. In most teams shipping LLM features today, none of these people have been named as the accountable owner of the thing they are deciding.

Issue 003 was the technical piece about what "good" means. This issue is the org-chart piece about who gets to answer. The answer changes with team size, so the rest of the issue lays out a RACI matrix (Responsible, Accountable, Consulted, Informed) for four load-bearing activities at the three team sizes (five, fifteen, fifty engineers) where the structure most often has to be redrawn out loud.

Why AI work resists a normal org chart

Three things about AI engineering work make ownership harder to pin down than in the pre-AI software you grew up shipping.

Start with the artefacts. Ownership normally follows the artefact: whoever exports a function owns it; whoever writes to a table owns the schema. That heuristic breaks the moment you look at a system prompt. The prompt is read by every product team that calls the LLM, edited by whoever felt strongest in the last design review, evaluated by whoever wrote the eval set, and silently broken by whoever last shaved a hundred tokens off it without rerunning the golden set. No team ships a prompt the way a team ships a schema, and the absence shows up as ownership fights nobody was staffed to have.

Then there is churn. A backend service contract changes once a quarter and everyone downstream gets a sprint of warning. A system prompt can change three times on a Monday afternoon, invisible to every reader until the next request lands. The volume of "who changed this and why" that follows a live prompt through a product-development week dwarfs the equivalent for any traditional artefact, and the tools engineers use to manage that volume (code review, semantic versioning, deprecation windows) do not really fit the shape of the work.

The third one is cost, and it is the one that hurts most. A backend engineer who adds a JOIN knows roughly what it costs to run. A product manager who adds three helpful sentences to a system prompt is making a decision worth a few thousand dollars a month at any real traffic level, and the feedback loop that would tell them so arrives on a bill six weeks later. Ownership of the prompt has been assigned, implicitly, to the person with the least visibility into what the prompt costs. Nobody would design this on purpose.

AI work expands until it fills the gaps in the org chart, and the gaps are exactly the places where ownership matters. The rest of this issue closes them with names.

The four activities you have to name

Before you name owners, you name the work. Four activities are load-bearing in any team shipping LLM features. The tell that any of them is missing an owner is the same: point at it, ask "who owns this?", the room goes quiet for a beat.

Prompts. System prompts, tool definitions, few-shot examples, and templates that condition every model call. The artefact has awkward parenthood: it is code (version-controlled, reviewable, deployable) and product specification (encodes intent, captures behavioural rules), and which parent claims it decides how the team ships. Engineering-owned prompts change slowly. Product-owned prompts change constantly and break the cache. Nobody-owned prompts drift.

Evals. The golden set, the three-layer evaluator from Issue 003, the assertions gating merges, the regression alerts. The eval set is the most consequential artefact in the whole stack because it encodes what "good" means, and whoever owns it is making a stream of product decisions dressed as testing decisions. Do not stop at "we have evals". Ask who edits the rubric.

Retrieval. The embedding model, the vector database from Issue 002, the chunking strategy from Issue 007, the hybrid retrieval pattern from Issue 006, the reranker from Issue 005. Retrieval feeds every product feature that uses RAG, which makes it a platform concern by construction. When each squad owns its own retrieval stack, the team ends up with three pipelines that all mostly work, share no evals, and give the LLM three different flavours of not-quite-right context.

Ops. Cost monitoring, scaling, vendor management, the streaming architecture from Issue 008, the prompt caching from Issue 009, the observability stack from Issue 004. Ops is the activity nobody volunteers for until the bill arrives or the pager fires, which is exactly why it gets assigned by default to whoever cannot decline.

The RACI matrix, sized by headcount

Every team-size breakpoint is the moment an existing structure has stopped working and nobody has said so yet. The three below are where AI work most consistently forces the conversation. The numbers are rough (five is really three to six, fifteen is really twelve to eighteen, fifty is really forty to sixty), but the transitions are real, and the sooner they are named, the cheaper they are to navigate.

Five engineers

At five engineers there is no platform team and the founder still knows every commit. Specialisation is a luxury the team cannot afford. The only ownership pattern that survives contact with reality: name a single accountable owner for each activity, accept the responsible work is shared across the whole team, and put the names somewhere everyone can see.

You will notice the founding AI engineer sits on the accountable line for three of the four rows. That is not an accident of drafting; it is what actually happens at this team size, and it is the team's largest bus-factor risk. The trick is not to spread accountability out (nobody else has the context to hold it), it is to admit the concentration is real and start recruiting for the seat that will inherit half of it.

The failure mode is easy to spot once you know what you are looking at. The PM waits half a day for the AI engineer to update a prompt. Eval-set additions get promised for "later this week" and land three sprints later. Retrieval tweaks are batched into a Friday afternoon because that is the only free block on the calendar. When two or three of those signs show up inside a fortnight, the team has outgrown the pattern.

Fifteen engineers

At fifteen engineers the founding AI engineer has stopped scaling and the team has quietly noticed. The pattern that works next is a small AI platform team of two to four people that owns prompts, evals, and retrieval as services the product squads consume. The team usually splits two ways: two engineers with strong AI-application chops plus one or two who lean toward ops or infrastructure. This fits alongside two to three product squads of four to five engineers each.

The tradeoff is worth naming before anyone signs up. Friction goes up: a product squad that used to Slack a prompt change to the AI engineer now files a ticket. Quality goes up too: the eval set gets rerun before every merge instead of after the next incident. The bottleneck moves from "one person's time" to "the platform team's roadmap", which is a better place for a bottleneck to live but is still a bottleneck.

The failure mode is the platform team becoming a service desk. If the platform team's quarterly roadmap is nine feature requests from product squads and zero long-running platform work, the failure mode has arrived. Two other tells: product squads write their own prompts in application code because the platform team is too slow to file with, and retrieval upgrades everyone agrees on keep sliding by a quarter because nobody has the calendar space. When those show up, the fifty-engineer pattern is closer than the team wants to admit.

Fifty engineers

By fifty engineers the team has usually shipped two or three AI-facing products, the platform team has become the bottleneck it used to solve, and the pattern that unblocks it is decentralisation with guardrails. A fully staffed AI platform team of five to eight people owns the cross-cutting infrastructure. Each product squad picks up one or two "AI champions" who own the prompts and evals for their squad's product against a platform-provided framework. The shape is common in platform engineering: the platform team owns the paved road, the product squads own their own cars.

Prompts and evals decentralise because each product's system prompt encodes product-specific rules that the platform team cannot know as well as the squad shipping the feature; the AI champion is the person who does know. Retrieval and ops stay centralised because the underlying infrastructure is genuinely shared, and forking it by squad is how a fifty-engineer org accidentally becomes a hundred-engineer org overnight.

The failure mode at fifty engineers is divergence you did not see happening. Squads pick different vendor SDKs. Eval rubrics diverge to the point where cross-product comparisons stop meaning anything. Two safeguards keep this in check. The platform team publishes a "golden path" stack (one prompt framework, one eval format, one retrieval client) and a "diverge cost" policy that names what a squad gives up by going off the golden path. The AI champions meet weekly, without engineering leadership in the room, so patterns spread horizontally rather than being enforced from above.

Fifty engineersAI platform teamowns retrieval, ops,frameworkProduct AI championsown product prompts andevalsProduct squadsbuild featuresFifteen engineersAI platform teamowns prompts, evals,retrievalProduct squadsconsume servicesInfra leadowns opsFive engineersFounding AI engowns prompts, evals,retrievalCTOowns ops

The diagram tracks the same activities through three team sizes. What moves is where accountability lives: one person at five, one team at fifteen, one team plus embedded champions at fifty. What stays the same is that every activity has exactly one accountable owner at any moment, and every transition between sizes is a moment when that ownership has to be redrawn out loud rather than left to shift by accident.

Common mistakes

A handful of failure modes recur enough across teams that they are worth naming directly.

Prompts anyone can edit. A system prompt everyone can edit becomes a system prompt no one understands, and the diff a week later reads as a puzzle. The fix is not heavy gatekeeping (that just moves the problem into a review queue), it is a named accountable owner plus a CI check that runs the golden-set evaluation on every prompt change.

The eval-set owner reporting into product. When the eval owner sits under product, the eval set drifts toward becoming a velocity blocker rather than a quality gate, because the product side has structural incentives to relax assertions that slow releases. The eval owner should sit in engineering, or in a cross-functional role with engineering's air cover, so the quality gate stays a quality gate on the days it is inconvenient.

Retrieval treated as a product-squad concern. From the moment the team has two products that use RAG, every squad that owns its own embedding model, chunking, and vector store duplicates work, fragments cost, and prevents the cross-product evaluation that would catch regressions early. Retrieval is a platform concern by construction, and treating it otherwise is expensive in ways only visible on the third product.

Nobody owns cost. A cost graph without an owner is a dashboard nobody looks at, and the bill arrives as a surprise. Name a person. Give them per-feature, per-tenant, per-template visibility from Issue 004's observability stack. Put a cost threshold in the production checklist the owner has to sign off on. Cost that nobody owns is cost that grows.

Summary

The shape of the answer is simpler than the shape of the question. The four activities (prompts, evals, retrieval, ops) need names on them. The names change with team size: one accountable owner at five engineers, a small platform team at fifteen, a platform team plus embedded AI champions at fifty. Every team-size transition is a moment when the names have to be redrawn, and the RACI conversation is the cheapest way to redraw them before the friction that follows becomes structural. Write the names down before the next retrospective. The retrospective will find them anyway.

Production checklist

• Name a single accountable owner for each of the four activities (prompts, evals, retrieval, ops). Write the names down where everyone can see them, and revisit the list every quarter.

• Confirm the current team size and pick the matching RACI pattern. Do not skip a stage because the team is "almost" the next size up; the transitions are where structure breaks.

• Wire the golden-set evaluation from Issue 003 into the CI for every prompt change, so the accountable owner is not the only line of defence.

• Move the eval-set owner into engineering or an explicit cross-functional role, not under product, so the quality gate stays accountable to quality on the days it is inconvenient.

• Centralise retrieval (embedding model, chunking, hybrid retrieval, reranking) as soon as the team has two products that use RAG. Do not let each squad own its own retrieval stack.

• Name an owner of cost. Give them visibility into per-feature, per-tenant, per-template cost from Issue 004's observability stack, and put a cost threshold in the production checklist the owner has to sign off on.

• At fifty engineers, publish a "golden path" stack (one prompt framework, one eval format, one retrieval client) and a "diverge cost" policy that names what a squad gives up by going off the golden path.

• Schedule an explicit RACI review at every team-size transition (passing five, fifteen, and fifty engineers). The transitions are where the structure breaks; the review is the cheapest way to catch the break early.

Further reading

• Will Larson, "Staff Engineer" - staffeng.com

• Hamel Husain, "Your AI product needs evals" - hamel.dev/blog/posts/evals

• Eugene Yan, "Patterns for building LLM-based systems and products" - eugeneyan.com/writing/llm-patterns