From the outside an AI setup looks like it consists of a model and a plugin. That is usually how it is explained to managers too. In every environment I have built, however, four separate layers emerged. The most important one came last every time: nothing works without the first three, but without the fourth nobody knows what the agent is touching.
In part one I described the system as three parts: gateway, rulebook, independent control. Those three parts sit on top of these four layers. Independent control is fed directly by the fourth layer.
In this part I will describe the options at each layer, what I chose and what I deliberately did not choose. At the end there is my setup recommendation for a team starting from scratch.
Layer one: the model
This is the most visible layer and, ironically, the least critical. The question “where is the model, which model, on which hardware?” has three answers:
- Frontier models in the cloud. Highest quality, but the code leaves.
- The vendor’s model in your data centre. A door that opened in 2026 but is still narrow. OpenAI announced a partnership with Dell in May to bring Codex in-house, details still unclear; Anthropic has no publicly available on-premises product. High quality, partial control, high cost.
- Open-weight models on your own hardware. Full control, but quality is a generation or two behind and the operational load is entirely yours.
If your code constraint is firm, you start with an open model; there is no other way. But not with a single model. Depending on the type of work you need several: a small, fast model for code completion, a large one for reasoning, a separate embedding model for context search.
This is hard to plan up front, because the need emerges from use. Usually you start with one model. A month later come the latency complaints on small jobs and the quality complaints on large ones, and diversification is born from those complaints.
What I deliberately did not do was fine-tune my own model. The idea “let it learn the bank’s code” comes up in every meeting and sounds very good. In practice, data preparation, version tracking and maintenance are a load a small team cannot carry. Besides, carrying the institution’s knowledge to the model on every request, that is, building the third layer, is both cheaper and always current, compared with teaching it to the model once.
Layer two: the developer interface
Where the developer talks to the model: an IDE plugin, a terminal agent or a web interface. The real job of this layer is not user experience but determining what goes to the model. There are three options:
- Off-the-shelf commercial plugins. The context-gathering logic belongs to the vendor; your control is limited.
- Open-source plugins. They connect to a local model and give more control, but quality varies.
- Your own plugin. Control is entirely yours, and so is the development cost.
In a .NET-heavy banking codebase made up of dozens of solutions, I had to write my own Visual Studio extension. The ready-made ones either did not connect to a local model or sent half the repo to the model while gathering context. In my own extension I decided which file enters the context, which directory is never read, and how every request is logged. This also made my job easier later, when building the gateway.
There is one more thing learned at this layer in 2026: the configuration files the agent reads are now an attack surface. CLAUDE.md, .claude/settings.json, hook definitions, .mcp.json. Most of the CVEs I listed in part one came through these files: hooks that run as soon as the project is opened, permission settings that take effect before the trust dialog appears. LayerX also showed in April 2026 that an instruction silently added to a repo’s CLAUDE.md passed unnoticed to everyone who cloned the project.
Nor is it only configuration files. Any text the agent reads, a README, an issue description, a dependency’s documentation, can carry instructions. That was the trigger for the sandbox escapes in part one as well.
So the question at this layer is no longer only “what enters the context?” but “which file does the agent take instructions from?” The answer is clear: these files are reviewed like code, belong to the admin, and the agent cannot change them.
What I did not choose was giving every developer the freedom to pick their own tool. It looks flexible but it is not governed: five plugins means five context policies and five log formats. Tool variety is possible, and I will describe how in part five, but on the condition that every tool goes through the gateway.
Layer three: context
The model does not know your institution. It does not know the business rules, the naming, the “why we do it this way here”. The context layer carries this knowledge to the model on every request. Its common name is RAG: documents are chunked, indexed with an embedding model, and when a request arrives the relevant chunks are found and added to the prompt.
Three sources worked for me: technical analysis documents, comments in the code and regulation summaries. It works up to a point. The model learns how the bank names one of its products, but it cannot learn which end-of-day job will break when a method is changed. Because RAG finds text; it does not understand structure. That is the limit that makes the fourth layer mandatory.
What I did not choose was indexing everything: the mail archive, meeting notes, old documents. As noise grows, the chunks the model finds “relevant” start to mislead. Few but current sources always beat many but stale ones.
Then there is maintenance. Documents change; if the index is not updated, the model answers confidently with old information. A RAG with no owner starts doing harm within six months. I will describe how that happened to me in part seven.
Industry data also says that investment in this layer is more effective than investment in the model. In Stack Overflow’s survey published in October 2026, developers pointed to project goals and requirements as the most reliable context source, at 85%. Faros’s telemetry report on 22,000 developers also argues that quality must be addressed at the stage where code is written, not afterwards. In other words, the thing that raises productivity the most without changing the model is what you give the model.
Layer four: system understanding
In most setups this layer does not exist, and that is the cause of most failures. In systems made up of dozens of solutions, such as banking or insurance software, a change can break an end-of-day process three solutions away. Letting an agent touch the code without understanding this structure is gambling.
The system-understanding layer is a graph extracted from the code itself: which class calls which service, which table is read in which batch job, which endpoints are affected when a method changes.
The graph is also given to the model as context, but its real value is elsewhere: impact analysis rests on the graph, not on the model’s guess. This is what I called “independent control” in part one.
What I did not choose was expecting the model to do this on its own. “Give it the whole repo, it will understand”, people say. But in a multi-solution system the code neither fits in the window nor, if it did, would the result be reliable. I will describe how the graph is built in part three.
Which can you start without
Building all four layers at once is six months of work, so order matters.
- You do not start without layers one and two. A local model and an interface you can control will run in the first week with an open-source plugin.
- Leave layer three to the second month. Watching what users ask in the first month and building the index around those questions gives far better results than starting with “index everything”.
- Layer four is essential in a multi-solution system, a luxury in a single repo. If you have a single monorepo, the model’s context window will do for a while. But if you have ten solutions and end-of-day processes, do not let the agent touch production code without the graph.
Something I noticed when all four were running: every layer produced traces, but the traces lived in separate places. I could not see from one place which user had sent what to which model. The story of the gateway in part six starts exactly there.
A starter setup
The recommendations below were written with a mid-sized .NET team of roughly 50 developers in mind. Hardware and models change every few months; look at the reasoning behind the numbers rather than the numbers.
Hardware. This area changes every quarter, so instead of listing card models let me describe how I think. The first decision is one GPU or two. For a team I recommend two data-centre-class cards: one for the large reasoning model, the other for code completion and embeddings. That way, while one is busy with a long analysis, line completion on the other does not slow down. If you will start with a single card, keep the reasoning model mid-sized; large models take up a significant share of memory with their weights and leave little room for long-context requests. How many people you can serve is determined less by model size than by how many long requests are open at the same time. Consumer cards are fine for a prototype, not enough for a team.
Model serving. Starting with Ollama is easy; ideal for trying things out with a few people. When the number of users grows and requests start to pile up, you need to move to a server designed for concurrency, such as vLLM. The thing to keep in mind when choosing a model is the type of work. Line completion wants a small, fast model, because the developer will not wait more than a few seconds. Analysis and document work want a large model and a long context, but latency matters less there; nobody minds waiting a minute for an analysis document. For embeddings a small, multilingual model is enough; if you have Turkish documents, multilingual is a must. Which model is best today changes every few months. So build the architecture so that changing the model is a configuration change.
Developer interface. For day one an open-source IDE plugin is enough. What matters is connecting it to your local server and narrowing what it reads. I recommend limiting the context at first to the open file and the directory the developer selects. It sounds restrictive, but it is the easiest way to know for certain which file goes to the model; widening later is always easier than narrowing later. As the team matures two paths open up: either you write your own plugin, or you use a ready-made tool with its managed settings. Whichever you choose, the tool must sit behind the gateway.
Context layer. You may not need to set up anything new for the vector database: if you already use PostgreSQL, pgvector will do; if not, Qdrant is a good start. The real decision is not the tool but what you put in the index. For the first index, technical analysis documents and code comments are enough. The questions users ask will tell you what is missing anyway. Also refresh the index weekly and give it an owner; I will describe in part seven how an ownerless index rots.
System understanding. When extracting the graph, use the compiler itself: Roslyn in .NET, similar analysers in Java. That way you work with what the code actually calls, not with text matching. At the start you do not need a separate database to store the graph either; in a small system an edge list in a relational table is enough. As the number of solutions and connections grows, Neo4j makes queries easier. Expose the graph as an MCP server so that whichever tool is used, the same question gets the same answer. I will describe what those questions are in part three.
The one thing that must be in place from the start: the log. A log of every request: user, model, files that entered the context, token count, duration. The gateway comes later, but the log must be there from day one. Otherwise you too will live through the “no before data” problem I will describe in part seven. A warning: this log itself contains code and customer data. Keep its access at least as tight as the data the agent reaches.
The next part is about tools: a tool comparison with a regulated institution’s questions, and the job no tool does, the code graph.
Sources: Check Point Research and Dark Reading (CVE-2025-59536, CVE-2026-21852, February 2026); stack.watch (CVE-2026-33068); DevOps.com (LayerX, CLAUDE.md, 9 April 2026); BleepingComputer (Pillar Security, 20 July 2026); OpenAI (Dell partnership, 18 May 2026); Stack Overflow Developer Survey 2026 (6 October 2026); Faros AI, AI Engineering Report Q2 2026 (22,000 developers, 4,000 teams).