Stop optimizing prompts in isolation—a peer-reviewed study of 9,649 experiments shows model choice outweighs prompt format by 21 percentage points (p=0.484 for format effects). Context engineering designs the full information environment: retrieval architecture, memory management, tool selection, and output schemas. Production agents spend roughly 5% of their context budget on the actual prompt; the other 95% determines success or failure. Invest in context infrastructure, not wordsmithing.
Last month we rebuilt the AI agent pipeline for a logistics company that had spent three months A/B testing prompt variations. They'd tried chain-of-thought, few-shot, zero-shot, XML tags, markdown formatting—every prompt engineering technique in the playbook. Their agent's task success rate sat stuck at 62%, regardless of the prompt. The fix wasn't a better prompt. It was redesigning what information the agent received before it ever saw the prompt: the retrieval pipeline, the tool descriptions, the memory layer. Two weeks of context engineering lifted accuracy to 89%. The prompt barely changed.
This pattern keeps repeating across our client engagements at Particula Tech. The era where tweaking a system prompt could meaningfully move the needle on production AI systems is ending. A new discipline—context engineering—is taking its place, and the research now confirms what practitioners have been discovering through expensive trial and error.
Why Prompt Engineering Hit a Ceiling
The 9,649-Experiment Study That Proved It
In February 2026, researcher Damon McMillan published "Structured Context Engineering for File-Native Agentic Systems"—a peer-reviewed study that tested prompt and context variables at a scale that makes most prompt engineering advice look anecdotal. The study ran 9,649 experiments across 11 models including Claude Opus 4.5, GPT-5.2, Gemini 2.5 Pro, DeepSeek V3.2, and Llama 4. It tested four data formats (YAML, Markdown, JSON, TOON), two retrieval architectures, and schema scales from 10 to 10,000 database tables. The headline finding: model choice created a 21-percentage-point accuracy gap between frontier and open-source models. That gap is larger than any format, prompt structure, or retrieval architecture effect combined. Even more telling: prompt format had no statistically significant aggregate effect (p=0.484). Whether you structured your context in YAML, JSON, or Markdown—it didn't matter. The study found one format-related quirk: TOON, a compact token-oriented notation, caused a "grep tax" where models spent extra tokens reasoning about the unfamiliar format. But the core finding was unambiguous—how you phrase and format your prompt matters far less than what information you provide and which model processes it. This doesn't mean prompts are irrelevant. But it means the cottage industry built around optimizing instruction phrasing has been polishing the hood ornament while ignoring the engine.
The Context Is the Product
Andrej Karpathy—former Director of AI at Tesla and founding member of OpenAI—distilled this shift into a definition that's become a rallying cry: "Context engineering is the delicate art and science of filling the context window with just the right information for the next step." The emphasis is on "just the right information." Not "as much information as possible." Not "the most cleverly worded instruction." The right information—retrieved, compressed, structured, and positioned—for the specific step the model needs to take next. In production AI systems, the prompt itself—the instruction text developers write—typically accounts for roughly 5% of the context window. The remaining 95% is system instructions, conversation history, retrieved documents, tool definitions, memory from previous interactions, and output schemas. Context engineering is the discipline of designing that 95%.
What Context Engineering Actually Means
Context engineering is not a rebranding of prompt engineering. It's a fundamentally different engineering discipline, closer to systems architecture than copywriting. Where prompt engineering asks "how should I phrase this instruction?", context engineering asks "what information environment does the model need to succeed at this task?"
At Particula Tech, we've structured context engineering around four infrastructure layers. Each one represents a distinct engineering problem that prompt optimization alone cannot solve.
Retrieval Architecture
The most impactful context decision is what external information to retrieve and include. McMillan's study found that file-based context retrieval improved frontier model accuracy by 2.7%—but decreased open-source model accuracy by 7.7%. The same retrieval strategy that helps one model hurts another. In practice, this means your retrieval pipeline needs to be model-aware. We've moved clients from static RAG implementations—where every query retrieves the same number of chunks from the same vector store—to dynamic retrieval that adjusts based on query complexity and the target model's strengths. A frontier model can handle more context without degradation; a smaller model needs tighter, more relevant retrieval. For teams already running retrieval pipelines, the first optimization is almost always compression. Our experience across dozens of deployments matches what we detailed in our guide on prompt compression and context window optimization—reducing context by 50-80% while maintaining accuracy isn't unusual when you remove the noise.
Memory Management
Production agents accumulate state across turns. Tool outputs, user corrections, intermediate reasoning steps, failed attempts—all of it enters the context window. Without active memory management, this state grows until it drowns the signal in noise. This is what researchers call "context rot." The LOCA-bench study—published in February 2026 by Weihao Zeng, Yuzhen Huang, and Junxian He—benchmarked language agents under extreme context growth and confirmed what practitioners have experienced: agent performance degrades as environment states grow complex. But the study also showed that advanced context management techniques substantially mitigate the degradation. The scaffold around the model matters as much as the model itself. We implement memory at two levels. Short-term memory uses rolling buffers and periodic summarization to keep the current context window focused on the active task. Long-term memory stores extracted facts, user preferences, and resolved decisions in vector databases and structured stores for retrieval only when relevant. The full architecture is covered in our guide on how AI agents maintain context across conversations—the principles there have become foundational to context engineering practice. The practical question for every turn of an agent loop: does this information help the model complete the next step? If not, compress it, store it externally, or discard it.
Tool Selection and Routing
A production agent might have access to 20, 50, or 200 tools. Listing all of them in the context window wastes tokens and confuses models. Context engineering treats tool definitions as dynamic context—expose only the tools relevant to the current step. This technique, sometimes called Tool RAG, uses the query or current agent state to retrieve a subset of tool definitions, much like document retrieval in a standard RAG pipeline. A financial analysis agent doesn't need database migration tools in its context. A code review agent doesn't need customer lookup APIs. We've found that agents with 5-8 tools in context outperform identical agents with 40+ tools, even when the larger set includes the correct tool. The noise from irrelevant tool definitions degrades the model's ability to select and parameterize the right one. Fewer, better-selected tools beat comprehensive tool catalogs every time.
Output Schemas and Constraints
The final layer defines what the model should produce. Output schemas—JSON structures, typed responses, constrained formats—are part of the context, and their design affects both accuracy and downstream system reliability. This is where context engineering overlaps with system prompt design but extends beyond it. A well-designed output schema doesn't just tell the model what format to use—it constrains the model's reasoning to produce outputs that integrate cleanly with the rest of your system. For high-throughput structured extraction, we deploy Particula-JSON specifically because it was trained to respect output schemas reliably without the format drift that general-purpose models exhibit after extended conversations.
LOCA-bench: The Evidence That Context Rot Kills Agents
LOCA-bench deserves a closer look because it quantifies a problem most teams experience but few measure. The benchmark evaluates language agents as combinations of models and "scaffolds"—the context management strategies that wrap around the model.
The study's core insight: the same model with different scaffolds produces dramatically different success rates. The model is often not the bottleneck. The context infrastructure surrounding it—what it sees, when, and how much—determines whether it succeeds or fails.
As Google DeepMind researcher Philipp Schmid put it: "Most agent failures are not model failures anymore, they are context failures."
This reframing is critical for how engineering teams allocate resources. If you're spending 80% of your AI engineering time on prompt optimization and 20% on context infrastructure, you've inverted the priority. The evidence consistently shows context architecture drives more outcome variance than prompt phrasing.
Decision Framework: When Prompting Is Enough
Context engineering isn't always necessary. Overengineering a simple classification task with memory layers and dynamic tool routing wastes resources. Here's the decision framework we use with clients:
Use prompt engineering when the task is well-defined, single-turn, and the model has everything it needs from the instruction alone. A well-crafted prompt with the right approach—whether few-shot examples, chain-of-thought, or structured output—remains the fastest path to production for these use cases.
Use context engineering when your system accumulates state, uses tools, retrieves external knowledge, or needs to maintain coherence across multiple steps. If your system has a retrieval pipeline, a memory layer, or a tool catalog, you're already doing context engineering—the question is whether you're doing it intentionally or accidentally.
| Characteristic | Prompt Engineering Sufficient | Context Engineering Required |
|---|---|---|
| Turns | Single-turn | Multi-turn |
| State | Stateless | Stateful across turns |
| Tool use | None or 1-2 fixed tools | Multiple dynamic tools |
| Knowledge | None or static few-shot examples | Dynamic retrieval from external sources |
| Memory | Not needed | Conversation history, user preferences |
| Error handling | Retry with modified prompt | Self-correction with updated context |
| Examples | Summarization, translation, classification | Support agents, research assistants, coding agents |
Getting Started: A Context Engineering Audit
If you're building production AI agents, here's how to transition from prompt-centric to context-centric development.
These ratios shift by use case, but the pattern holds: the prompt is a small fraction of a well-engineered context window.
Step 1: Instrument Your Context Window
Before optimizing, measure. Log the full contents of every context window your system constructs. Categorize each section: system instructions, conversation history, retrieved documents, tool definitions, few-shot examples, output schemas. You can't engineer what you don't observe.
Step 2: Measure Context-Outcome Correlation
Not all context helps. Run ablation experiments that systematically remove context sections and measure impact on task success. We consistently find that 30-50% of what teams include in their context windows has zero or negative impact. Removing irrelevant context doesn't just save tokens—it improves accuracy.
Step 3: Build Context Infrastructure
Replace ad-hoc context assembly with engineered pipelines:
- Retrieval: Use embedding-based search with reranking to select the most relevant documents per query—not the most recent or the most popular
- Memory: Implement tiered storage—working memory in the context window, session memory in a fast cache, long-term memory in a vector database for semantic retrieval
- Tool routing: Build a selection layer that exposes only relevant tools based on the current task state
- Compression: Summarize and compress context between turns to prevent rot
Step 4: Budget Your Context
Set explicit token budgets for each context layer. For a 128K context window, a practical starting allocation:
| Context Layer | Budget | Purpose |
|---|---|---|
| System instructions | 2-5% | Core behavior, constraints, persona |
| User prompt | 2-5% | Current task or query |
| Retrieved documents | 30-40% | Relevant knowledge for this step |
| Conversation history | 20-30% | Compressed recent context |
| Tool definitions | 10-15% | Dynamically selected tools |
| Output schema + examples | 5-10% | Format constraints and demonstrations |
The Shift Is Already Here
Context engineering isn't a future trend—it's what successful production AI teams are already doing, even if they don't use the term. Every team that's built a RAG pipeline, implemented conversation memory, or dynamically selected tools has practiced context engineering. McMillan's study and LOCA-bench just gave us the evidence to name it and prioritize it correctly.
The companies that will build the most reliable AI agents in 2026 won't be the ones with the cleverest prompts. They'll be the ones with the best context infrastructure—the retrieval pipelines, memory managers, tool routers, and compression layers that ensure the model sees exactly what it needs, and nothing else.
At Particula Tech, we've shifted our AI development practice accordingly. When a new client engagement starts with "we need better prompts," we start by auditing their context. The prompt usually isn't the problem. The information environment surrounding it almost always is.
Frequently Asked Questions
Quick answers to common questions about this topic
Context engineering is the discipline of designing the entire information environment an LLM operates in—not just the prompt. It encompasses system instructions, conversation history, retrieved documents, tool definitions, memory stores, few-shot examples, and output schemas. As Andrej Karpathy defined it: 'the delicate art and science of filling the context window with just the right information for the next step.' Unlike prompt engineering, which optimizes a single text input, context engineering treats the full context window as infrastructure that determines model behavior.