ML System Design Interview Guide

10 real-world ML systems from Waymo, Anthropic, Google DeepMind, Cursor, LangChain, and OpenClaw — with architecture diagrams, design decisions, and interview talking points.

The 7-Step System Design Framework

Problem Formulation — Clarify requirements, define metrics, frame as ML task
Data — Sources, collection, labeling, preprocessing, storage
Feature Engineering — Feature selection, embeddings, representation
Model Architecture — Model choice, training strategy, tradeoffs
Serving & Inference — Latency, throughput, scaling, caching
Evaluation & Monitoring — Offline metrics, online metrics, A/B testing, drift detection
Iteration & Improvement — Feedback loops, retraining, failure modes

Cursor

Real-Time Code Completion System — 1M+ QPS LLM serving with speculative edits

▼

graph TD subgraph Client["Client Layer - VS Code Fork"] U["User types code"] --> CC["Context Collector"] CC --> |"Gathers surrounding code,
imports, file tree"| ENC["Local Encryption"] end subgraph Infra["Infrastructure Layer"] CF["Cloudflare
Reverse Proxy / TLS / DDoS"] end ENC --> CF CF --> BE["Monolithic Backend
TypeScript + Rust"] subgraph Server["Server Layer - AWS CPU + Azure H100 GPUs"] BE --> DEC["Decrypt Request"] DEC --> ROUTER["Model Router"] ROUTER --> |"Autocomplete
low-latency"| FW["Fireworks
Fine-tuned Models"] ROUTER --> |"Chat / Summarize"| OAI["OpenAI
GPT Models"] ROUTER --> |"Reasoning"| ANT["Anthropic
Claude Models"] ROUTER --> |"Multimodal"| GCP["Google Vertex AI
Gemini"] FW --> SPEC["Speculative Edits Engine"] SPEC --> |"Fine-tuned Llama-3-70b
13x speedup"| RESP["Response Builder"] OAI --> RESP ANT --> RESP GCP --> RESP end RESP --> CF CF --> |"Sub-second latency"| RENDER["Render Suggestions
in Editor"] subgraph Indexing["Codebase Indexing Pipeline"] FILES["Local Codebase Files"] --> MERKLE["Merkle Tree Hash
Sync every 3 min"] MERKLE --> |"Detect changed files"| CHUNK["Code Chunker"] CHUNK --> EMB["Embedding Model"] EMB --> TP["Turbopuffer
Vector DB"] end TP --> |"Semantic code search
for context retrieval"| CC subgraph Shadow["Shadow Workspace"] EDIT_REQ["Edit Request"] --> SW["Hidden VSCode Window"] SW --> |"AI applies edits"| LINT["Lint Check"] LINT --> |"Pass/Fail + diagnostics"| REPORT["Report Back to User"] end subgraph Training["Custom Training Pipeline"] UD["Real User Data"] --> CB["Cursor Bench
Custom Benchmark"] CB --> RL["RL Training
PyTorch + Ray"] RL --> |"Thousands of GPUs"| CM["Composer Model"] CM --> |"Evaluates correctness +
codebase abstraction adherence"| FW end subgraph Monitoring["Observability"] DD["Datadog Monitoring"] PG["PostgreSQL
Primary Datastore"] end BE --> DD BE --> PG style Client fill:#1a2744,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Server fill:#1a2233,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style Indexing fill:#1a2a22,stroke:#3fb950,stroke-width:2px,color:#e6edf3 style Shadow fill:#2a2219,stroke:#d29922,stroke-width:2px,color:#e6edf3 style Training fill:#2a1a22,stroke:#f85149,stroke-width:2px,color:#e6edf3 style Monitoring fill:#1a2233,stroke:#8b949e,stroke-width:2px,color:#e6edf3 style Infra fill:#1a2233,stroke:#39d2c0,stroke-width:2px,color:#e6edf3

Problem Statement

Build a real-time code completion system that serves 1M+ queries per second at peak, providing context-aware suggestions with sub-second latency. The system must deeply understand an entire codebase (not just the open file), support multi-model routing for different task types (autocomplete, chat, reasoning, code edits), and maintain strict user privacy through client-side encryption.

Core ML tasks: (1) Next-token code prediction at extreme throughput, (2) Semantic code retrieval via embeddings for context augmentation, (3) Reinforcement learning from real user interactions to continuously improve suggestion quality.

Architecture Overview

Client (VS Code fork): TypeScript/Electron app collects code context and encrypts it locally before transmission.
Codebase Indexing: Files are chunked, embedded, and stored in Turbopuffer (vector DB). Merkle tree hash syncs every 3 minutes; only modified files are re-indexed. Raw code is never stored server-side.
Infrastructure: Cloudflare for TLS/DDoS. AWS for CPU workloads; Azure for tens of thousands of H100 GPUs (inference). Monolithic TypeScript + Rust backend backed by PostgreSQL.
Model Router: Fireworks for low-latency autocomplete, OpenAI GPT for chat, Anthropic Claude for reasoning, Google Vertex AI for Gemini.
Speculative Edits: Novel variant of speculative decoding for code. Fine-tuned Llama-3-70b via Fireworks achieves 13x speedup over vanilla, 9x over GPT-4.
Shadow Workspace: Hidden VSCode window where AI performs edits, runs lint checks, and reports diagnostics without affecting the user's active session.
Custom Training: Composer model trained via RL on real user data (PyTorch + Ray). Custom benchmark "Cursor Bench" evaluates correctness + codebase abstraction adherence.

Key Design Decisions

Decision	Why	Tradeoff
Monolithic backend	Maximizes developer velocity for small team	Sacrifices independent scaling; acceptable because dev speed is the bottleneck
Multi-provider model routing	Each provider excels at different tasks	Vendor dependency; mitigated by owning the critical path (autocomplete via Fireworks)
Speculative decoding with long windows	Existing code provides strong prior for speculation	Wasted compute when wrong; 13x speedup shows hit rate is high for code
Merkle tree + 3-min sync	Efficiently detects changed files without full-codebase diff	Up to 3 min staleness; acceptable since active file has real-time context
Client-side encryption + embeddings-only storage	Enterprise privacy requirements	Limits server-side debugging; necessary for enterprise adoption
RL on real user data with custom benchmark	Standard benchmarks don't capture real editing patterns	Expensive to collect; far more realistic than synthetic benchmarks

Interview Talking Points

Scale separation: Autocomplete (1M+ QPS, ~100ms) is fundamentally different from chat/reasoning (~1 QPS, seconds). Different model providers handle each.
Speculative decoding for code: Code edits are uniquely suited — existing code acts as a strong prior, enabling longer speculation windows and 13x speedup.
Merkle tree indexing: Same data structure as Bitcoin — detects changed files efficiently without diffing entire codebases.
Privacy-preserving retrieval: Only embeddings stored server-side. Discuss how this constraint shapes the architecture.
Shadow Workspace: Using existing tools (VSCode's lint engine) as AI infrastructure rather than rebuilding them.
Monolith over microservices: At their team size, developer velocity outweighs operational benefits of independent scaling.
Custom benchmarks: "Cursor Bench" uses real agent requests, evaluating correctness + codebase convention adherence.
Infrastructure pragmatism: Yugabyte → PostgreSQL migration — choose the simplest technology that solves your problem.

Evaluation Metrics

Acceptance rate: % of suggestions accepted by users (primary online metric)
Characters saved: How much typing the system saves per session
Latency P50/P95/P99: Sub-second target for autocomplete; tracked per model provider
Edit distance: How close the suggestion is to the final user code
Cursor Bench accuracy: Correctness + codebase abstraction adherence on real user tasks
Cache hit rate: Merkle tree sync efficiency — % of requests served with fresh embeddings

Sources:
Cursor: Shadow Workspace Fireworks: Speculative Decoding for Cursor ByteByteGo: Cursor Architecture Pragmatic Engineer: Building Cursor

Anthropic

Contextual Retrieval (RAG) — 67% fewer failed retrievals with hybrid search + reranking

▼

graph TD subgraph INGEST["Ingestion Pipeline"] A["Raw Document"] --> B["Chunking"] B --> C["Chunks c1, c2, ... cN"] C --> D["LLM Context Generation"] A -- "Full document as cached prefix" --> D D --> E["Contextual Chunks"] E --> F["Embedding Model"] E --> G["BM25 Tokenizer"] F --> H["Vector Index"] G --> I["BM25 Index"] end subgraph CACHE["Prompt Caching Optimization"] J["Full Document - cached"] --> K["+ Individual Chunk - variable"] K --> L["Generated Context Snippet"] L --> M["~90% cost reduction per chunk"] end subgraph QUERY["Query Pipeline"] N["User Query"] --> O["Query Embedding"] N --> P["Query Tokens"] O --> Q["Semantic Search"] P --> R["Keyword Search - BM25"] Q --> S["Merge Results"] R --> S S --> T["Reranker - e.g. Cohere"] T --> U["Top-K Contextual Chunks"] U --> V["LLM Answer Generation"] N --> V end H -.-> Q I -.-> R style INGEST fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style CACHE fill:#1c2333,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style QUERY fill:#1c2333,stroke:#3fb950,stroke-width:2px,color:#e6edf3

Problem Statement

Traditional RAG splits documents into chunks for retrieval, but individual chunks lose critical context. A chunk stating "Revenue increased 3% over the previous quarter" becomes useless without knowing which company or quarter. The right chunk exists but cannot be retrieved — "failed retrieval" is the dominant error in production RAG.

Architecture Overview

Contextual Embeddings: Before embedding each chunk, an LLM generates a context snippet situating it within the source document. The enriched chunk is then embedded.
Contextual BM25: Same enriched chunks indexed for keyword/lexical search — catching exact entity names and domain terms.
Hybrid Search + Reranking: Query runs against both indexes. Results merged, then a cross-encoder reranker re-scores candidates. Top-K passed to LLM for answer generation.

Strategy	Failed Retrieval Reduction	What It Adds
Contextual Embeddings only	35%	Richer semantic representations
+ BM25 (Hybrid)	49%	Keyword matching for exact terms
+ Reranking	67%	Cross-attention re-scoring

Key Design Decisions

Why hybrid search? Embeddings miss exact keywords (entity names, acronyms). BM25 misses paraphrases. Combining covers the full spectrum — 14pp improvement from adding BM25.
Why prepend context? Model-agnostic (works with any embedding model), no retraining, directly addresses the root cause — the chunk text itself lacks context.
Why reranking? Cross-encoder jointly attends to query+chunk pairs (much more accurate than bi-encoder). Only applied to top candidates after fast first-stage retrieval.
Cost-quality tradeoff: Prompt caching keeps the document in KV cache, varying only the per-chunk suffix. ~90% cost reduction, but requires sequential batch processing.

Interview Talking Points

Root cause: Failed retrievals — right chunk exists but isn't found — are more damaging than generation errors.
Enrichment is ingestion-time, not query-time: One-time cost per chunk, not per query.
Hybrid search: ~1/3 of remaining failures after embeddings were keyword-match problems.
Reranking as precision layer: Cross-encoder too expensive for full corpus, highly effective on top candidates.
Cost engineering: Prompt caching cuts enrichment cost by ~90% via KV cache reuse.
Layered improvement: Each technique compounds: 35% → 49% → 67%. Great ablation study example.
Model-agnostic: Works with any embedding model — no fine-tuning required.
Chunk size matters: Smaller chunks improve retrieval precision but lose broader context. Larger chunks retain context but reduce precision. This is a fundamental tradeoff to discuss.

Evaluation Metrics

Recall@K: % of relevant chunks retrieved in top-K results
Precision@K: % of top-K results that are actually relevant
MRR (Mean Reciprocal Rank): How high the first relevant result appears
Answer correctness: End-to-end accuracy of LLM-generated answers (human eval or LLM-as-judge)
Failed retrieval rate: % of queries where the correct chunk exists but was not retrieved (Anthropic's primary metric)
Ingestion cost per chunk: $ cost of LLM enrichment call (optimized by prompt caching)

Sources: Anthropic — Contextual Retrieval

Anthropic

Multi-Agent System — Orchestrating parallel agents with context engineering

▼

Architecture Overview

Composes simple patterns — orchestrator-workers, parallel exploration, tool use, and context engineering — into a system where multiple agents cooperate on complex research queries. An outer agent harness provides checkpoint/resume durability.

graph TD subgraph Harness["Agent Harness - Long-Running Durability"] direction TB CP["Checkpoint / State Serialization"] RS["Resume and Retry on Failure"] MW["Multi-Context-Window Persistence"] end UQ["User Query"] --> Router["Router Agent
Classify and Route"] Router --> Orchestrator["Orchestrator Agent
Decompose into subtasks"] subgraph ParallelExploration["Parallel Exploration - Fan-Out"] direction LR W1["Worker Agent A
Angle 1"] W2["Worker Agent B
Angle 2"] W3["Worker Agent C
Angle 3"] end Orchestrator --> W1 Orchestrator --> W2 Orchestrator --> W3 subgraph ToolUse["Tool Use Layer"] direction TB Think["Think Tool
Stop and Reason"] MCP["MCP Code Execution
Write code to invoke tools"] DirectAPI["Direct Tool Calls
Search, Retrieve, Execute"] end W1 --> Think W2 --> MCP W3 --> DirectAPI subgraph ContextEng["Context Engineering Layer"] direction TB Summarize["Summarization
Compress prior turns"] Retrieve["Retrieval
Pull relevant context"] Curate["Curate and Write Context
Finite window management"] end Think --> Summarize MCP --> Retrieve DirectAPI --> Curate Summarize --> Fusion["Fusion Agent
Aggregate and Synthesize"] Retrieve --> Fusion Curate --> Fusion subgraph QualityLoop["Evaluator-Optimizer Loop"] direction LR Eval["Evaluator Agent
Score quality"] Refine["Optimizer Agent
Refine output"] Eval -->|"Below threshold"| Refine Refine -->|"Re-evaluate"| Eval end Fusion --> Eval Eval -->|"Meets threshold"| FinalAnswer["Final Comprehensive Answer"] CP -.->|"Wraps entire pipeline"| Router MW -.->|"Checkpoints at each stage"| Fusion style Harness fill:#1a1a2e,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style ParallelExploration fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style ToolUse fill:#1c2333,stroke:#3fb950,stroke-width:2px,color:#e6edf3 style ContextEng fill:#1c2333,stroke:#d29922,stroke-width:2px,color:#e6edf3 style QualityLoop fill:#1c2333,stroke:#f85149,stroke-width:2px,color:#e6edf3

Composable Agent Patterns

Pattern	Structure	When to Use	Example
Prompt Chaining	LLM → gate → LLM (sequential)	Fixed sequential steps	Generate outline → validate → write draft
Routing	Classify → specialized handler	Distinct input categories	Query → billing / technical / account
Parallelization	Fan-out → aggregate	Independent subtasks	Research from 3 angles simultaneously
Orchestrator-Workers	Central LLM delegates to workers	Dynamic subtask decomposition	Code refactor: plan + parallel file edits
Evaluator-Optimizer	Generator + Evaluator loop	Clear quality criteria	Generate code → evaluate → refine

Key Design Decisions

Simple composable patterns vs frameworks: Primitives are easier to debug, test, and reason about. Start simple, add complexity only when measured improvement justifies it.
Parallel exploration + fusion: Reduces wall-clock latency linearly. Produces diverse perspectives without anchoring bias.
Code execution via MCP: Agent writes code to invoke tools — scales to 100+ endpoints. Direct calls work for small tool sets only.
Multi-context-window persistence: Complex tasks exceed single context windows. Checkpoint/resume preserves progress across crashes and context limits.
Curated context engineering: Context windows are finite — curated context keeps signal-to-noise high and leaves room for reasoning.

Interview Talking Points

Start simple: An agent is LLM + tools + loop + stopping condition. Everything else is implementation detail.
Parallel exploration: Multiple agents explore different angles simultaneously — diverse perspectives without anchoring bias.
Context engineering: Same model produces dramatically different results depending on what's in the context window.
Code execution scales: Direct function calling works for 5-10 tools, breaks at 100+. Code lets agents compose, loop, and branch.
"Think" tool: Explicit pause-and-reason mechanism prevents agents from acting before thinking. Simple but measurably effective.
Agent harness: Checkpointing + state serialization borrowed from distributed systems. Treats agents like long-running processes.
Evaluator-optimizer loops: LLM outputs are stochastic. Separate evaluator catches errors and iteratively refines.
Error propagation: When one worker agent produces incorrect output, how does the system detect and prevent it from cascading through fusion into the final answer? Evaluator loop is the safety net.

Evaluation Metrics

Task completion rate: % of queries that produce a fully satisfactory answer
Answer quality score: LLM-as-judge or human evaluation of comprehensiveness and accuracy
Latency (wall-clock): End-to-end time from query to final answer (parallel exploration reduces this)
Tool call success rate: % of tool invocations that return valid results
Context utilization: How much of the context window is used effectively vs. wasted on irrelevant content
Checkpoint recovery rate: % of failures successfully recovered by the agent harness

Sources:
Anthropic — Building Effective Agents Anthropic — Multi-Agent Research System Anthropic — Context Engineering

LangChain

Agent Design Patterns — Context engineering for long-running autonomous agents

▼

Architecture Overview

As agents tackle longer tasks, model performance degrades with expanded context windows. This drives a core principle: context must be treated as a finite resource with diminishing marginal returns. These design patterns — drawn from Claude Code ($1B run rate), Manus ($2B acquisition), and other production agents — show how to strategically populate the context window with only essential information for the agent's immediate next step.

Based on Agent Design Patterns by Lance Martin (Jan 2026)

Seven Agent Design Patterns

These patterns address the central challenge of context engineering — getting the right information into the context window at the right time, while keeping it lean enough for high-quality reasoning.

Give Agents a Computer — OS-layer access

1. Give Agents a Computer

Multi-Layer Action Space — hierarchical tools

2. Multi-Layer Action Space

Progressive Disclosure — on-demand tool retrieval

3. Progressive Disclosure

4a. Offload Context (Summarization)

Offload Context — tool result offloading

4b. Offload Context (Tool Results)

5. Cache Context

6a. Isolate Context (Sub-Agents)

Isolate Context — Ralph Wiggum loop pattern

6b. Isolate Context (Ralph Wiggum Loop)

7. Evolve Context

Pattern	Core Idea	How It Works	Example
Give Agents a Computer	OS-layer access (filesystem + shell)	Agents interact with the operating system via CLI, gaining persistent storage and executable capabilities beyond isolated tool sets	Claude Code, Manus — agents that write, execute, and iterate on code directly
Multi-Layer Action Space	Hierarchical tools instead of flat tool lists	6-20 atomic tool definitions at the top layer; complex workflows executed via shell/code at the computer layer (CodeAct pattern)	Agent calls a "run_code" tool that executes a script invoking dozens of APIs, avoiding intermediate tool result bloat
Progressive Disclosure	Reveal information on demand, not upfront	Tool indexing retrieves definitions on demand; skill folders store detailed docs agents access selectively; agents call --help flags when needed	Cursor Agent syncs MCP tool descriptions to folders, providing abbreviated lists with full docs retrievable on demand
Offload Context	Write old results to filesystem storage	Agent writes tool results and trajectories to files. Plans stored as files and periodically reloaded to reinforce objectives. Selective summarization only when offloading value diminishes	Agent writes intermediate research findings to scratch files, reads them back when synthesizing final answer
Cache Context	Prompt caching changes agent economics	Resume from cached prefixes instead of replaying linear chat history. Manus identified cache hit rate as the most critical production metric. Without caching, coding agents become economically prohibitive	Higher-capacity model with caching can cost less than lower-capacity model without caching
Isolate Context	Sub-agents with independent context windows	Each sub-agent has its own context, tools, and instructions. Enables parallelizable tasks and long-running sequential loops. Git-backed coordination communicates progress across instances	"Ralph Wiggum" pattern: sequential agents each tackle one discrete plan item, coordinating via git history
Evolve Context	Continual learning in token space	Analyze past sessions, extract learnings, update master documentation. Diary-based memory distills sessions into concise entries. Skill extraction saves reusable procedures as new skills	GEPA framework: collect trajectories, score outcomes, refine prompt variants over time

Context Engineering Deep Dive

The fundamental insight: context engineering is not prompt engineering. It is the discipline of building systems that populate the context window with exactly the right information at each step of agent execution.

Finite Resource Model: Context windows have diminishing marginal returns — adding more information beyond a threshold actively degrades performance. Every token in the window must earn its place.
CodeAct Pattern: Instead of processing intermediate tool results in context, agents execute code that captures only final outputs. This avoids context bloat from verbose API responses and lets agents compose, loop, and branch over tool calls in code.
Cache Hit Rate as North Star: Manus identified prompt cache hit rate as the single most important metric for production agent economics. High cache hit rates make high-capability models affordable; low hit rates make even cheap models expensive at scale.
Selective Summarization: Not all context should be summarized — compress only when the cost of keeping full context exceeds the information loss from summarization. Plans and critical decisions should be preserved verbatim.
File-Backed Memory: The filesystem serves as an external memory bank. Agents write plans, intermediate results, and learnings to files, then selectively read them back. This decouples memory capacity from context window size.

Emerging Frontiers

Learned Context Management: Instead of hand-crafted compression strategies, models may learn their own context management. Recursive Language Models (RLM) suggest LLMs could absorb scaffolding currently embedded in agent harnesses. Sleep-time compute enables agents to reflect offline, consolidating memories without explicit prompting.
Multi-Agent Coordination at Scale: Scaling to concurrent agent swarms introduces shared-context and conflict-resolution challenges. Gas Town demonstrates coordination using git-backed tracking, a specialized "Mayor" agent maintaining workspace context, and merge queues for parallel work.
Infrastructure for Long-Running Agents: Production requirements include observability into agent behavior, human-review hooks, graceful degradation frameworks, standardized debugging interfaces, and human-in-the-loop monitoring — most of which remain immature.

Key Design Decisions

Decision	Chosen Approach	Alternative	Rationale
Agent-computer interface	OS-level access (filesystem + shell)	Sandboxed tool-only access	OS access provides persistent storage, executable capabilities, and scales to arbitrary workflows. Tool-only access limits agents to predefined actions.
Action space design	Multi-layer: minimal tools + computer layer	Flat list of all available tools	Dozens of tool definitions pollute context. A hierarchical action space keeps the prompt lean while enabling complex workflows via code execution.
Information loading	Progressive disclosure (retrieve on demand)	Load all tool/context upfront	Upfront loading wastes context on information the agent may never need. On-demand retrieval keeps context focused on the current subtask.
Memory architecture	File-backed offloading + selective recall	Keep everything in context window	Context windows are finite with diminishing returns. Filesystem storage provides unlimited capacity; selective recall retrieves only what's relevant.
Cost optimization	Prompt caching with cache hit rate as KPI	Use cheaper models to reduce cost	A high-capability model with high cache hit rate can cost less than a low-capability model without caching, while producing better results.
Multi-agent coordination	Isolated contexts + git-backed coordination	Shared context window across agents	Isolated contexts prevent cross-contamination and enable parallelism. Git provides durable, auditable coordination without shared-state complexity.

Interview Talking Points

Context is a finite resource with diminishing returns: Adding more information to the context window eventually degrades performance. The core skill of agent design is curating what goes in — not maximizing what goes in. This principle drives every pattern in this section.
Give agents a computer, not just tools: The most successful agents (Claude Code, Manus) interact at the OS layer — filesystem for persistence, shell for execution. This is fundamentally different from giving an LLM a list of API tools. It enables agents to write their own tools, store intermediate results, and compose arbitrary workflows.
Multi-layer action spaces prevent context pollution: Instead of loading 50+ tool definitions into context, use 6-20 atomic tools at the top layer and let the agent execute complex workflows via code at the computer layer. The CodeAct pattern avoids processing verbose intermediate results by capturing only final outputs.
Cache hit rate is the most important production metric: Manus found that prompt cache hit rate determines agent economics more than model choice. High cache hit rates make powerful models affordable; without caching, even coding agents become economically prohibitive. Design your agent's prompt structure to maximize cache hits.
Progressive disclosure keeps context lean: Don't load all tool definitions and documentation upfront. Use tool indexing, skill folders, and help flags to let agents retrieve information on demand. This mirrors how human developers work — you don't read every man page before starting a task.
Evolving context is the path to continual improvement: Agents that reflect on past trajectories, extract reusable skills, and update master documentation improve over time without retraining. This operates in "token space" — updating what goes into context rather than updating model weights.
Multi-agent coordination is an unsolved problem at scale: Isolated context windows with git-backed coordination works for small agent teams, but scaling to concurrent swarms requires new infrastructure — observability, merge queues, human-review hooks, and graceful degradation. This is an active area of research worth flagging in an interview.

Sources:
Lance Martin — Agent Design Patterns (Jan 2026)

DeepMind

Gemini MoE Architecture — Sparse expert routing for efficient trillion-parameter models

▼

graph TD subgraph INPUT["Input Pipeline"] A["Input Tokens"] --> B["Embedding Layer"] B --> C["Transformer Layers x N"] end subgraph MOE_LAYER["Sparse MoE Layer - replaces dense FFN"] C --> R["Router Network - Learned Gating"] R --> |"Softmax over logits"| G["Top-K Selection - K=2 of 64"] G --> E1["Expert FFN 1"] G --> E2["Expert FFN 2"] G -.-> |"Not activated"| E3["Expert FFN 3...64"] E1 --> W["Weighted Combination"] E2 --> W W --> OUT["Layer Output"] R --> |"Auxiliary loss"| LB["Load Balancing Loss"] end subgraph TRAINING["Training Infrastructure"] TPU["TPU v5p / Ironwood Pods"] --> JP["Jupiter Network Fabric"] JP --> DP["Data Parallelism"] JP --> EP["Expert Parallelism - experts across devices"] DP --> JAX["XLA/JAX Compiler"] EP --> JAX JAX --> MP["Mixed Precision Training"] AC["AlphaChip AI-Designed Layout"] -.-> TPU end subgraph SERVING["Serving Pipeline"] PW["Pathways Distributed Runtime"] --> MH["Multi-Host Inferencing"] MH --> SD["Speculative Decoding"] SD --> CH{"MoE Challenge:
Draft tokens activate
more experts"} CH --> |"Verification overhead 2-3x"| S1["Adaptive Grouped Speculation"] CH --> S2["Context-Aware Scheduling"] CH --> S3["Confidence-Based Deferral"] MH --> VL["vLLM on TPU"] end OUT --> PW style INPUT fill:#1c2333,stroke:#58a6ff,color:#e6edf3 style MOE_LAYER fill:#1c2333,stroke:#3fb950,color:#e6edf3 style TRAINING fill:#1c2333,stroke:#bc8cff,color:#e6edf3 style SERVING fill:#1c2333,stroke:#d29922,color:#e6edf3 style E3 fill:#161b22,stroke:#6e7681,color:#6e7681

Problem Formulation

How do you scale a language model to trillions of parameters without proportionally scaling inference cost? The key insight: not all parameters are needed for every token. Replace dense FFN layers with multiple expert FFNs gated by a learned router.

Aspect	Dense Transformer	Sparse MoE
Params activated	100% per token	~2-10% (top-K experts)
FLOPS per token	Proportional to total params	Proportional to K * expert_size
Total capacity	Limited by compute	10-100x larger at same per-token cost
Memory	All params loaded	All params loaded (serving challenge)

The Speculative Decoding + MoE Problem

Standard speculative decoding uses a small draft model to generate candidates cheaply. But with MoE, draft tokens collectively activate many more experts — verification time increases 2-3x due to data movement. When throughput gains don't offset overhead, speculation causes slowdowns up to 1.5x.

Solutions: Adaptive grouped speculation, context-aware scheduling, divided rollout for load balancing, confidence-based deferral, blockwise parallel decoding.

Interview Talking Points

Capacity vs compute: 1T-parameter model activating only 100B per token. Different tokens need different expertise.
Router is learned: Neural network trained end-to-end. Load balancing loss prevents expert collapse.
Memory-bound inference: All expert weights in memory even though fraction activated. Bottleneck is data movement, not compute.
Speculation breaks with MoE: Draft tokens collectively activate more experts → 2-3x verification overhead.
Expert parallelism ≠ tensor parallelism: EP places whole experts on different devices with all-to-all token routing.
Hardware co-design: AlphaChip designs TPU → Jupiter interconnect → XLA/JAX → Pathways runtime. Each layer optimized for MoE.
Why not just bigger dense? Dense models double cost linearly. MoE scales capacity sublinearly at the cost of system complexity.
Expert count vs expert size: More smaller experts = finer specialization but more routing overhead. Fewer larger experts = less routing complexity but coarser specialization. Typical: 64 experts with K=2.

Evaluation Metrics

Perplexity: Language modeling quality on held-out data
Downstream task accuracy: Performance on benchmarks (MMLU, HumanEval, etc.)
Expert utilization uniformity: Variance in tokens routed per expert (lower = better load balancing)
FLOPS per token: Actual compute used vs. total model params (MoE efficiency metric)
Tokens per second (throughput): Inference speed under different batch sizes
Memory bandwidth utilization: How much of HBM bandwidth is used during inference (MoE is memory-bound)

Sources: DeepMind Models Gemini 2.5 Technical Report

DeepMind

Gemini Storybook — Multimodal pipeline: text + image + audio in seconds

▼

System Architecture

graph TD subgraph Input["User Input Layer"] UP["User Prompt"] PHOTO["Photo Upload - optional"] STYLE["Style Selection
pixel art, comics, etc"] end subgraph Encoding["Multimodal Encoding"] TE["Text Encoder"] PE["Photo Encoder"] UE["Unified Embedding Space"] end subgraph StoryPlanning["Story Planning Module"] NP["Narrative Planner
10-page arc"] CCS["Character Consistency
Registry"] SC["Style Conditioning
Vector"] end subgraph MoEBackbone["Sparse MoE Backbone"] ROUT["Expert Router"] EX1["Text Expert"] EX2["Image Expert"] EX3["Style Expert"] CMA["Cross-Modal Attention"] end subgraph PerPage["Per-Page Generation Loop - pages 1-10"] TG["Text Generation"] IG["Image Generation
discrete tokens"] AG["Audio Narration
USM, 45+ langs"] CONSIST["Cross-Page
Consistency Check"] end subgraph Safety["Content Safety Layer"] TF["Text Safety Filter"] IF2["Image Safety Filter"] AF["Audio Safety Filter"] end subgraph AssemblyOut["Assembly and Output"] LAYOUT["Page Layout Engine"] SYNC["Text-Image-Audio Sync"] BOOK["10-Page Storybook"] end UP --> TE PHOTO --> PE STYLE --> SC TE --> UE PE --> UE UE --> NP SC --> NP NP --> CCS CCS --> ROUT NP --> ROUT ROUT --> EX1 ROUT --> EX2 ROUT --> EX3 EX1 <--> CMA EX2 <--> CMA CMA --> TG CMA --> IG PE -.->|"photo conditioning"| IG SC -.->|"style conditioning"| IG TG --> AG TG --> CONSIST IG --> CONSIST CONSIST -.->|"feedback"| TG TG --> TF IG --> IF2 AG --> AF TF --> LAYOUT IF2 --> LAYOUT AF --> SYNC LAYOUT --> SYNC SYNC --> BOOK style Input fill:#1a3a2a,stroke:#3fb950,color:#e6edf3 style Encoding fill:#1a2a3a,stroke:#58a6ff,color:#e6edf3 style StoryPlanning fill:#2a1a3a,stroke:#bc8cff,color:#e6edf3 style MoEBackbone fill:#3a2a1a,stroke:#d29922,color:#e6edf3 style PerPage fill:#1a3a3a,stroke:#39d2c0,color:#e6edf3 style Safety fill:#3a1a1a,stroke:#f85149,color:#e6edf3 style AssemblyOut fill:#1a2a2a,stroke:#58a6ff,color:#e6edf3

Problem Formulation

Given a text prompt (and optionally photos + style), generate a 10-page illustrated storybook with coherent narrative, consistent artwork, and read-aloud audio — all within seconds. 45+ languages, diverse visual styles, child-safe content.

Note: The architecture details below are inferred from Gemini's published capabilities and general multimodal system design principles. Google has not published the specific internal architecture of Storybook.

Key Architecture Decisions

Decision	Gemini Approach	Alternative	Tradeoff
Multimodal strategy	Single unified model	Separate models (GPT + DALL-E + TTS)	Unified enables cross-modal attention; separate is easier to debug
Image generation	Discrete image tokens (autoregressive)	Latent diffusion	Tokens integrate natively; diffusion may have higher fidelity for single images
Consistency	Character registry + cross-page attention	Per-page independent + post-hoc correction	Registry is coherent; post-hoc is simpler but inconsistent
Style transfer	Conditioning vector (45+ styles in one model)	Fine-tuned model per style	One model, many styles; per-style fine-tuning yields higher per-style quality

Interview Talking Points

Unified model vs separate models: Cross-modal attention enables tight coherence between text and images. Separate models require brittle prompt engineering.
Cross-page consistency: The hardest problem — character registry + cross-page attention prevents visual drift across 10 pages.
Sparse MoE for latency: Only ~10-20% of parameters activate per token, making seconds-level generation feasible.
Safety is non-negotiable: Multi-layer filtering (text + image + audio) for children's content. Adversarial testing essential.
Pipeline parallelism: Story planning is sequential, but per-page generation can be parallelized once the arc is established.
Photo conditioning: User photos encoded into dense features that condition image decoder — identity preservation across art styles.
45+ styles via conditioning vectors: One model, many styles — contrast with maintaining 45 separate fine-tuned models.

Sources: Google: Gemini Universal AI Assistant DeepMind Models

DeepMind

Nano Banana — Gemini Image — Reasoning-first image generation with autoregressive + diffusion hybrid

▼

graph TD subgraph NanoBanana["Nano Banana Pipeline — Reasoning-First"] direction TB PROMPT["Text Prompt"] --> GEMINI["Gemini LLM Backbone"] subgraph Think["Think Phase — Scene Planning"] GEMINI --> REASON["Reasoning Core"] REASON --> PHYSICS["Physics and Gravity"] REASON --> SPATIAL["Spatial Layout"] REASON --> TEXTPLAN["Text Rendering Planning"] REASON --> WORLD["World Knowledge"] end PHYSICS --> PLAN["Unified Scene Plan"] SPATIAL --> PLAN TEXTPLAN --> PLAN WORLD --> PLAN subgraph AutoReg["Autoregressive Token Generation"] PLAN --> TOK1["Image Token 1"] TOK1 --> TOK2["Image Token 2"] TOK2 --> TOK3["Image Token 3"] TOK3 --> TOKN["Image Token N"] TOK1 -.-> |"Each token aware
of prior tokens"| TOKN end subgraph DiffHead["Diffusion Head — High-Fidelity Rendering"] TOKN --> DIFF["Diffusion Decoder"] DIFF --> DENOISE["Iterative Denoising"] DENOISE --> PIXELS["Pixel Synthesis"] end PIXELS --> OUTPUT["Output Image
Native 2K / 4K"] end subgraph Traditional["Traditional Diffusion - e.g. Stable Diffusion"] direction TB TPROMPT["Text Prompt"] --> CLIP["CLIP Text Encoder"] CLIP --> NOISE["Random Gaussian Noise"] NOISE --> UNET["U-Net Denoising
No scene planning"] UNET --> TDEC["VAE Decoder"] TDEC --> TIMG["Output Image"] end subgraph Comparison["Key Differences"] D1["Nano Banana: Reasons THEN renders"] D2["Diffusion: Denoises noise directly"] D3["Nano Banana: Accurate text in images"] D4["Diffusion: Struggles with text"] end style NanoBanana fill:#1a2233,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style Think fill:#1a2a22,stroke:#3fb950,stroke-width:2px,color:#e6edf3 style AutoReg fill:#1a2744,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style DiffHead fill:#2a1a22,stroke:#f85149,stroke-width:2px,color:#e6edf3 style Traditional fill:#2a2219,stroke:#d29922,stroke-width:2px,color:#e6edf3 style Comparison fill:#1a2233,stroke:#8b949e,stroke-width:2px,color:#e6edf3

Problem Statement

"Nano Banana" is the public codename for Google DeepMind's Gemini Image models — first appearing anonymously on Chatbot Arena in August 2025 before being officially revealed.

Build an image generation system that produces photorealistic, physically accurate images with flawless text rendering and conversational editing. The system must understand the semantics of what it generates — reasoning about physics, spatial relationships, and causal logic before pixel generation.

Architecture Overview

Gemini LLM Backbone: Built on the Gemini transformer — inherits world knowledge, reasoning, and multimodal understanding. Knows what "1960s diner at golden hour" looks like.
"Think" Phase: Mandatory scene planning — evaluates physics, spatial layout, text rendering, world knowledge before any pixels. Cannot be disabled in Nano Banana Pro.
Autoregressive Image Tokens: Sequential token generation with full awareness of prior tokens. Each sub-region is distinct and coherent with the rest.
Diffusion Head: Converts planned tokens into high-fidelity pixels — textures, lighting, sharp edges. Conditioned on the token plan (far more directed than unconditional denoising).
Native 2K/4K: No post-hoc upscaling. Every detail planned by the reasoning core.
Conversational Editing: Multimodal LLM natively supports multi-turn: "make the sky more dramatic," "add a reflection."

Product Evolution

Version	Model	Date	Key Advances
Nano Banana	Gemini 2.5 Flash Image	Aug 2025	First release; viral "3D figurine" images. Native 2K.
Nano Banana Pro	Gemini 3 Pro Image	Nov 2025	4K output, improved text rendering, mandatory thinking. Under 10s.
Nano Banana 2	Gemini 3.1 Flash Image	Feb 2026	Pro-quality at Flash speed.

Key Design Decisions

Decision	Why	Tradeoff
Autoregressive + diffusion hybrid	AR planning provides global coherence; diffusion head handles pixel fidelity	AR is sequential (adds latency). Quality gains justify it.
Mandatory reasoning	Prevents errors impossible to fix during denoising (e.g., misspelled text)	Adds latency even for simple prompts. Quality-over-speed choice.
LLM backbone for images	Inherits world knowledge and reasoning. Enables conversational editing.	Tightly coupled; larger/more expensive than standalone diffusion models.
Native high resolution	Upscaling hallucinations are a common complaint. Every detail is planned.	Dramatically more compute per image. Justified for professional use cases.

Interview Talking Points

AR + diffusion hybrid: Sketching composition before adding detail — mirrors how humans create images. "System 2" thinking for image generation.
Why diffusion fails at text: Pure diffusion denoises all pixels simultaneously with no character-level planning. Nano Banana plans text layout first.
Foundation model bet: Image generation built into the LLM backbone. Model knows what "subsurface light scattering in jade" looks like from multimodal training.
Mandatory thinking: Product decision disguised as architecture — removing the skip option ensures consistent quality. Like chain-of-thought for images.
Conversational editing as moat: Standalone diffusion models need ControlNet/InstructPix2Pix bolted on. Shared backbone enables it natively.
Arena-first launch: Anonymous blind evaluation on Chatbot Arena before reveal — eliminates brand bias and provides genuine quality signal.

Sources:
DeepMind: Gemini Image Google: Nano Banana Pro Google: Nano Banana 2

Waymo

Autonomous Driving System — Perception, prediction, and planning at 200M+ autonomous miles

▼

Architecture Overview

The Waymo Driver is a modular-hybrid autonomous driving system built on a Foundation Model with a "Think Fast, Think Slow" (System 1 / System 2) architecture. A sensor fusion encoder handles real-time perception while a driving VLM (fine-tuned from Gemini) reasons about rare and complex scenarios. Large teacher models are distilled into efficient students for onboard deployment under 10ms latency.

Based on Demonstrably Safe AI for Autonomous Driving and The Waymo World Model

graph TD subgraph Sensors["6th-Gen Sensor Suite"] direction LR CAM["13 Cameras
17MP, HDR"] LID["4 LiDARs
360 FOV, 300m+"] RAD["6 Radars
Imaging, all-weather"] EAR["Audio Receivers
Siren detection"] end subgraph Foundation["Waymo Foundation Model"] direction TB subgraph Sys1["System 1: Think Fast"] SFE["Sensor Fusion Encoder
Camera + LiDAR + Radar over time"] SFE --> OBJ["3D Objects + Semantics
+ Rich Embeddings"] end subgraph Sys2["System 2: Think Slow"] VLM["Driving VLM
Fine-tuned from Gemini"] VLM --> SEM["Complex Semantic Reasoning
Novel/rare situations"] end end subgraph Pipeline["Perception - Prediction - Planning"] direction LR PERC["Perception
3D Detection, Tracking,
Lane/Sign/Light"] PRED["Prediction
Multimodal Trajectory
Forecasting 6-64 modes"] PLAN["Planning
IL + RL Hybrid
Trajectory Generation"] end subgraph Safety["Safety Architecture"] direction LR VAL["Onboard Validation Layer
Independent trajectory verification"] FALL["Deterministic Fallbacks
Safety-critical path"] end subgraph Virtuous["Virtuous Cycle"] direction TB DRIVER["Waymo Driver"] SIM["Waymo Simulator
World Model / Genie 3"] CRITIC["Waymo Critic
Driving quality evaluation"] DRIVER --> SIM --> CRITIC --> DRIVER end CAM --> SFE LID --> SFE RAD --> SFE EAR --> SFE CAM --> VLM OBJ --> PERC SEM --> PERC PERC --> PRED --> PLAN PLAN --> VAL VAL --> FALL subgraph Distill["Teacher-Student Distillation"] direction LR TEACH["Large Teacher Models
Max quality, cloud-trained"] STUD["Small Student Models
Real-time onboard, under 10ms"] TEACH -->|"Distill"| STUD end Foundation -.-> TEACH style Sensors fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Foundation fill:#1a1a2e,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style Sys1 fill:#1c2333,stroke:#3fb950,stroke-width:2px,color:#e6edf3 style Sys2 fill:#1c2333,stroke:#d29922,stroke-width:2px,color:#e6edf3 style Pipeline fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Safety fill:#1c2333,stroke:#f85149,stroke-width:2px,color:#e6edf3 style Virtuous fill:#1c2333,stroke:#39d2c0,stroke-width:2px,color:#e6edf3 style Distill fill:#1c2333,stroke:#d29922,stroke-width:2px,color:#e6edf3

Sensor Suite & Perception

The 6th-generation Waymo Driver achieves a 42% reduction in total sensors vs. 5th-gen while improving performance, at under $20K per unit. Overlapping fields of view cover up to 500 meters in all conditions.

Sensor	Count	Key Specs	Role in Perception
Cameras	13 (down from 29)	17MP imager, HDR, superior low-light	Texture, color, traffic lights, signs, lane markings
LiDAR	4 (down from 5)	360° FOV, 300m+ range, cm-scale accuracy	3D geometry, precise depth, object detection regardless of appearance
Radar	6	Imaging radar, unprecedented resolution	Instantaneous velocity, all-weather robustness (rain/fog/snow)
Audio (EARs)	Array	External Audio Receivers	Emergency vehicle siren detection

SWFormer (Sparse Window Transformer): Converts 3D LiDAR points into sparse voxels, processes with self-attention within spatial windows plus cross-window correlation. 73.4 L2 mAPH on Waymo Open Dataset.
PVTransformer: Improved point-to-voxel aggregation achieving 76.1 L2 mAPH (+1.7 over SWFormer).
Sensor Fusion Encoder (System 1): Rapidly fuses camera, LiDAR, and radar inputs over time into objects, semantics, and rich embeddings. Runs in real-time onboard.
AutoML / NAS: Neural architecture search finds optimal quality-latency tradeoffs for onboard deployment.

Motion Forecasting

Given the current scene (road geometry, traffic lights, agent histories), predict the future trajectories of all agents as multimodal distributions. Models produce 6-64 trajectory hypotheses per agent with associated probabilities.

Model	Architecture	Key Innovation	Finding
MultiPath++	Multi-Context Gating (MCG)	Efficient fusion between agents and road elements; latent-space trajectory anchors	State-of-the-art on multiple benchmarks
Scene Transformer	Attention-based joint prediction	Predicts all agent trajectories jointly to capture interactions; supports conditioned prediction	Unified architecture for multi-agent forecasting
Wayformer	Multimodal attention family	Factorized attention + latent query attention (16x compression, no quality loss)	Early fusion, despite being simplest, achieves state-of-the-art

Planning & Decision Making

The planning system determines the ego vehicle's trajectory given perception outputs and predicted agent trajectories. Waymo combines imitation learning with reinforcement learning, validated by an independent onboard safety layer.

ChauffeurNet: Deep RNN trained via imitation learning on a bird's-eye-view representation. Trained on 30M+ expert driving examples. Key innovation: synthesizing perturbations (collisions, going off-road) to make IL robust.
IL + RL Hybrid: RL fine-tuning on top of imitation learning achieves a 38% reduction in safety events on the most difficult scenarios. Trained on 100K+ miles of real-world urban driving data.
Scaling Laws: Planning performance improves as a power-law function of compute. Unlike LLMs, optimal AV models are relatively smaller in size but require significantly more data — model size scales 1.5x faster than dataset size.
Onboard Validation: An independent validation layer verifies ML-generated trajectories before execution, with deterministic fallbacks for safety-critical situations.

EMMA: End-to-End Multimodal Model

Built on Gemini, EMMA takes raw camera images and text inputs and outputs trajectories, 3D detections, and road graph elements — all as natural language text tokens. A research model exploring the end-to-end frontier.

Chain-of-thought reasoning: 6.7% improvement in planning performance by letting the model reason before acting.
Joint training: Training on planning + perception + road graph understanding simultaneously improves all tasks vs. training individual models.
Current limitations: Camera-only (no LiDAR/radar), processes limited frames, computationally expensive — not yet suitable for real-time onboard inference.
Strategic role: Research vehicle for pushing performance boundaries; insights feed back into the modular production system.

World Model & Simulation

Built on Google DeepMind's Genie 3, the Waymo World Model generates photorealistic, controllable, multi-sensor driving scenes at scale — including scenarios impossible to capture in reality. The fleet drives ~20 million simulated miles per day.

Multi-sensor generation: Jointly produces temporally consistent camera imagery and LiDAR point clouds aligned with Waymo's real sensor stack — no domain gap.
Driving action control: Run "what if" counterfactual scenarios to test whether the Waymo Driver could have driven more confidently on alternative routes.
Scene layout control: Customize road geometry, traffic signal states, and other road user behavior through selective placement and layout mutations.
Language control: Adjust time-of-day, weather, or generate entirely synthetic scenes via natural language prompts (dawn/morning/noon/afternoon/evening/night; cloudy/foggy/rainy/snowy/sunny).
Long-tail generation: Synthesizes never-before-seen conditions — tornadoes, floods, elephants, T-rex costumes, car-sized tumbleweeds — by transferring world knowledge from 2D video pre-training into 3D LiDAR outputs.

Scale & Infrastructure

Metric	Value
Real-world autonomous miles	200M+ fully autonomous on public roads
Simulation miles	15B+ total; ~20M miles/day
Driving data corpus	500,000+ hours of driving data
Data rate per vehicle	Up to 1.8 TB/hour
Onboard inference latency	<10ms for most neural nets
Sensor suite cost (6th-gen)	<$20K (50%+ reduction from 5th-gen)
Safety record (56.7M miles)	84% fewer airbag-deployment crashes, 73% fewer injury crashes vs. humans
Paid rides per week	~400K (targeting 1M by end of 2026)

Key Design Decisions

Decision	Chosen Approach	Alternative	Rationale
Architecture	Modular-hybrid: Foundation Model trained end-to-end, deployed in modular structure with safety layers	Monolithic end-to-end (Tesla FSD)	"A monolithic architecture makes it easy to get started, but is wildly inadequate for safe, at-scale full autonomy" — Dmitri Dolgov. Modular allows independent testing, debugging, and safety certification per component.
Sensor suite	LiDAR + cameras + radar + audio (multi-sensor fusion)	Camera-only (Tesla)	LiDAR provides cm-scale 3D accuracy; radar sees through rain/fog/snow; redundant sensing critical for safety. Cost reduced to <$20K (6th-gen). Disengagement rate 0.0004/mile vs. higher for camera-only.
Planning training	Imitation learning + RL fine-tuning	Pure imitation learning	IL alone is brittle in edge cases. RL fine-tuning on safety-critical scenarios yields 38% reduction in safety events on the hardest scenarios.
Onboard deployment	Teacher-student distillation with NAS	Deploy full-size models with powerful hardware	Distillation achieves large-model quality at small-model latency (<10ms). NAS finds Pareto-optimal quality-latency architectures. Better scaling laws for distilled students.
Long-tail handling	Generative world model (Genie 3) + VLM reasoning (System 2)	Collect more real-world data	Rare events (<0.03% frequency) are impractical to capture at scale. World Model synthesizes arbitrary scenarios; VLM transfers internet-scale world knowledge to understand novel objects/situations.
Simulation fidelity	Generative world model producing real sensor outputs (camera + LiDAR)	Physics-based simulation (Carcraft)	Generative model eliminates sim-to-real domain gap — downstream autonomy systems consume simulation like real sensor logs. Language-controllable for rapid scenario iteration.

Interview Talking Points

Modular-hybrid beats pure end-to-end for safety-critical systems: Waymo trains a Foundation Model end-to-end for representation quality, but deploys it in a modular structure with an independent validation layer. This enables per-component testing, failure attribution, and deterministic fallbacks — essential for a system where errors cost lives, not just user experience.
Multi-sensor fusion provides irreplaceable redundancy: LiDAR gives cm-scale depth that monocular cameras cannot match; radar provides instantaneous velocity and sees through weather; cameras capture texture and semantics. The 6th-gen system proves cost is solvable (<$20K, 42% fewer sensors) — the safety margin is not. Disengagement rate of 0.0004/mile speaks for itself.
Teacher-student distillation is the deployment strategy: Train the largest, most accurate teacher models in the cloud, then distill to small students that run onboard in <10ms. This decouples model quality from inference latency — you can keep scaling teachers without redesigning the onboard stack. NAS further optimizes the quality-latency Pareto front.
Scaling laws for AV differ from LLMs: Waymo showed planning performance follows power-law scaling, but optimal AV models are relatively smaller and need much more data than LLMs. Model size scales 1.5x faster than dataset size. This insight shapes compute allocation: invest more in data diversity and coverage than raw model parameters.
World models solve the long-tail simulation problem: Rare events (<0.03% frequency) are the hardest to handle but most dangerous. The Waymo World Model (built on Genie 3) generates photorealistic multi-sensor scenarios — including physically impossible conditions (snow on Golden Gate Bridge, elephants on highways) — with zero sim-to-real domain gap because it outputs real sensor formats.
The virtuous cycle is the competitive moat: Driver, Simulator, and Critic share the same Foundation Model. Real-world driving feeds data mining, which feeds simulation, which feeds training, which improves the driver. 200M+ real miles and 15B+ sim miles create a flywheel competitors cannot easily replicate.
RL on top of IL is the planning breakthrough: Pure imitation learning is brittle at the edges — the model has never seen the expert recover from near-collisions it was never in. RL fine-tuning on safety-critical scenarios yields a 38% reduction in safety events. ChauffeurNet's innovation of synthesizing perturbations during IL training addresses the same insight from a data perspective.
EMMA shows where end-to-end is heading: Building on Gemini, EMMA achieves 6.7% planning improvement through chain-of-thought reasoning and proves joint training helps all tasks. But it's camera-only, processes few frames, and is too expensive for real-time — showing why the modular-hybrid approach remains necessary for production while E2E research advances.

Sources:
Waymo — Demonstrably Safe AI for Autonomous Driving (Dec 2025) Waymo — The Waymo World Model (Feb 2026) Waymo — Introducing EMMA (Oct 2024) Waymo — Scaling Laws in Autonomous Driving (Jun 2025) Waymo — Meet the 6th-Generation Waymo Driver (Aug 2024)

Anthropic

AI Agent Evaluations — Systematic eval pipelines for non-deterministic AI agents

▼

Architecture Overview

Agent evaluations provide a systematic framework for measuring AI agent performance across non-deterministic outputs. The core pipeline flows from task definition through an agent harness that produces transcripts, which are then scored by one or more graders to produce outcomes. Suites of 100s-1000s of tasks run in parallel, with pass@k and pass^k metrics capturing both capability ceilings and reliability floors. Anthropic advocates a Swiss Cheese model where automated evals, production monitoring, A/B testing, user feedback, manual review, and human studies form complementary layers — no single method catches everything.

Based on Demystifying evals for AI agents (Jan 2026)

graph TD subgraph Suite["Evaluation Suite"] direction TB TASK["Task Definition
Input + expected behavior"] TRIAL1["Trial 1"] TRIAL2["Trial 2"] TRIALN["Trial N"] end subgraph Harness["Agent Harness"] direction TB ENV["Environment Setup
Isolated sandbox per trial"] AGENT["Agent Under Test
Model + tools + prompts"] EXEC["Execution
Multi-step tool use"] end subgraph Transcript["Transcript"] direction TB STEPS["Step-by-step log
Actions, tool calls, outputs"] end subgraph Graders["Grading Layer"] direction LR CODE["Code-Based Grader
Deterministic checks,
unit tests, regex"] MODEL["Model-Based Grader
LLM judge with rubric,
semantic matching"] HUMAN["Human Grader
Gold-standard review,
calibration baseline"] end subgraph Outcome["Outcome and Metrics"] direction LR SCORE["Score per Trial
Pass/fail or 0-1"] PASSK["pass@k
At least 1 success in k trials"] PASSK2["pass^k
All k trials succeed"] end TASK --> TRIAL1 TASK --> TRIAL2 TASK --> TRIALN TRIAL1 --> ENV TRIAL2 --> ENV TRIALN --> ENV ENV --> AGENT --> EXEC EXEC --> STEPS STEPS --> CODE STEPS --> MODEL STEPS --> HUMAN CODE --> SCORE MODEL --> SCORE HUMAN --> SCORE SCORE --> PASSK SCORE --> PASSK2 style Suite fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Harness fill:#1c2333,stroke:#3fb950,stroke-width:2px,color:#e6edf3 style Transcript fill:#1c2333,stroke:#d29922,stroke-width:2px,color:#e6edf3 style Graders fill:#1c2333,stroke:#bc8cff,stroke-width:2px,color:#e6edf3 style Outcome fill:#1c2333,stroke:#39d2c0,stroke-width:2px,color:#e6edf3

Eval Terminology

Term	Definition
Task	A single problem instance with defined inputs and success criteria. The atomic unit of evaluation.
Trial	One execution of a task by the agent. Multiple trials per task capture non-deterministic variation.
Grader	A function that scores agent output against expected behavior. Can be code-based, model-based, or human.
Transcript	The full step-by-step record of an agent's actions, tool calls, and outputs during a trial.
Outcome	The grader's verdict on a trial — typically pass/fail or a score between 0 and 1.
Evaluation harness	The infrastructure that orchestrates task selection, trial execution, grading, and metric aggregation.
Agent harness	The wrapper that provides the agent with its environment (tools, sandbox, credentials) for each trial.
Evaluation suite	A curated collection of 100s-1000s of tasks designed to measure a specific capability or track regressions.

Types of Graders

Choosing the right grader type is a core design decision. Most production eval suites use a combination, with code-based graders for objective checks and model-based graders for subjective quality assessment.

Dimension	Code-Based	Model-Based	Human
Methods	Unit tests, regex matching, string comparison, assertion checks, execution-based verification	LLM-as-judge with rubric, pairwise comparison, semantic similarity scoring	Expert review with scoring rubric, A/B preference testing, calibration sessions
Strengths	Fast, deterministic, zero cost per run, perfectly reproducible, no false positives from grader errors	Handles open-ended outputs, captures nuance and partial credit, scales to 1000s of tasks	Gold-standard accuracy, catches subtle quality issues, provides calibration baseline for other graders
Weaknesses	Cannot assess subjective quality, brittle to valid alternative solutions, high upfront authoring cost	Non-deterministic (grader itself varies), can hallucinate scores, requires careful prompt engineering	Expensive ($5-50 per task), slow (hours to days), does not scale, inter-rater disagreement

Evaluating Agent Types

Different agent categories require distinct eval strategies because their outputs, environments, and failure modes vary significantly.

Agent Type	Focus	Key Benchmarks	Grading Strategy
Coding Agents	End-to-end code generation, bug fixing, test writing	SWE-bench (resolve real GitHub issues), HumanEval	Code-based: run test suites against generated code. Pass/fail based on test outcomes.
Conversational Agents	Multi-turn dialogue, tool use in conversation, user satisfaction	tau-Bench (simulated retail and airline customer service)	Model-based: LLM judges conversation quality. Hybrid with code checks for tool call correctness.
Research Agents	Information gathering, synthesis, multi-step web browsing	BrowseComp (find hard-to-locate facts on the web)	Code-based for factual accuracy (exact match). Model-based for synthesis quality and completeness.
Computer Use Agents	GUI interaction, form filling, multi-app workflows	WebArena (web tasks), OSWorld (desktop OS tasks)	Code-based: check final application state (DOM, file system). Screenshot comparison for visual tasks.

Non-Determinism: pass@k vs pass^k

AI agents are inherently non-deterministic — the same task can produce different results across runs. Two complementary metrics capture different aspects of this variability:

pass@k (capability ceiling): The probability that at least 1 out of k independent trials succeeds. Answers "can the agent do this at all?" A task with pass@5 = 80% means the agent can solve it most of the time if given multiple attempts. Useful for capability evals where you want to know the frontier of what the agent can achieve.
pass^k (reliability floor): The probability that all k independent trials succeed. Answers "can the agent do this reliably?" A task with pass^5 = 30% means the agent only consistently solves it 30% of the time across all attempts. Useful for regression evals and production readiness where every run must succeed.
The gap between pass@k and pass^k reveals flakiness: If pass@5 is 90% but pass^5 is 20%, the agent has the capability but lacks reliability. This gap is the signal to investigate — read transcripts of failing trials to find the failure modes causing inconsistency.
Practical guidance: Run at least 3-5 trials per task. For high-stakes production decisions, run more. Track both metrics over time — a rising pass@k with flat pass^k means the agent is getting more capable but not more reliable.

The 8-Step Eval Roadmap

Anthropic recommends this progression for building an eval suite, from first prototype to production-grade evaluation infrastructure:

Start early: Build evals from day one, not after the agent is "ready." Even 5-10 hand-written tasks provide signal that manual testing cannot.
Harvest manual tests: Every time you manually test the agent, convert that interaction into a reproducible eval task. Your manual testing backlog is your best task source.
Write unambiguous tasks: Each task must have a clear, objectively verifiable success criterion. Ambiguous tasks produce noisy signals that waste engineering time.
Build balanced problem sets: Include easy (sanity checks), medium (core capability), and hard (frontier) tasks. An all-hard suite cannot distinguish "completely broken" from "slightly degraded."
Stabilize the harness: Invest in reproducible environments — deterministic setup, isolated sandboxes, pinned dependencies. Flaky infrastructure produces flaky results indistinguishable from agent flakiness.
Design thoughtful graders: Match grader type to output type. Use code-based graders for verifiable outputs, model-based for subjective quality. Calibrate model graders against human judgments.
Read transcripts: Aggregate metrics tell you what failed; transcripts tell you why. Regularly read 10-20 failing transcripts to identify systematic failure patterns.
Monitor saturation: When pass rates approach 90-95%, the suite is saturating. Add harder tasks or move to the next capability frontier. A saturated eval gives false confidence.

Key Design Decisions

Decision	Chosen Approach	Alternative	Rationale
Grading approach	Code-based graders for verifiable outputs	Model-based (LLM-as-judge)	Code graders are deterministic and free to run. Use model graders only when output is subjective or open-ended — they add non-determinism to the eval itself.
Eval type	Capability evals for development, regression evals for CI	Single eval suite for both	Capability evals explore frontiers (pass@k matters); regression evals guard against breakage (pass^k matters). Mixing them conflates "can it?" with "does it still?"
Success metric	pass@k for capability, pass^k for reliability	Single pass rate (1 trial per task)	Single-trial pass rate conflates capability with luck. Running k=3-5 trials and tracking both metrics separates "the agent can do it" from "the agent reliably does it."
Task design	Narrow, specific tasks with unambiguous success criteria	Broad, realistic end-to-end scenarios	Narrow tasks produce cleaner signal for development. Broad tasks are better for final validation but harder to grade and debug. Start narrow, add broad tasks later.
Environment	Isolated sandbox per trial (fresh state)	Shared state across trials	Shared state creates ordering dependencies — trial 3 might pass only because trial 2 left artifacts. Isolation ensures each trial is an independent measurement.
Grader calibration	Human-calibrated model graders (validate against human scores)	Fixed rubric without human baseline	Model graders drift and hallucinate scores. Periodic human calibration (score 50-100 tasks manually) catches grader bugs before they corrupt metrics.

Interview Talking Points

Agent evals require fundamentally different infrastructure than model evals: Traditional model evals test single input-output pairs. Agent evals must orchestrate multi-step execution with tool use, manage sandboxed environments, capture full transcripts, and handle non-deterministic outputs across 3-5+ trials per task. The evaluation harness is itself a complex system.
The pass@k vs pass^k gap is the most actionable metric for agent reliability: If pass@5 = 90% but pass^5 = 20%, the agent has the capability but fails 4 out of 5 times. This 70-point gap directly quantifies flakiness and tells you exactly where to invest — read the failing transcripts to find the systematic failure mode.
Code-based graders should be the default, not model-based: Model-based graders add a second source of non-determinism on top of the agent's own variability. For coding agents, SWE-bench uses test suite execution — deterministic, zero-cost, and perfectly reproducible. Reserve LLM judges for genuinely subjective outputs like conversation quality.
The Swiss Cheese model of evaluation layers catches what no single method can: Automated evals, production monitoring, A/B testing, user feedback, manual review, and human studies each have blind spots. Like Swiss cheese slices, each layer has holes — but stacking 6 layers makes it unlikely a serious regression passes through all of them undetected.
Eval suite saturation is a hidden failure mode: When pass rates hit 90-95%, teams celebrate — but the eval has stopped providing signal. Saturated benchmarks give false confidence. The fix is continuously adding harder tasks and retiring solved ones, treating the eval suite as a living system that evolves with the agent.
Harvesting manual tests is the highest-ROI eval strategy: Every manual test session generates 5-10 potential eval tasks for free. Teams that systematically convert manual testing into automated evals build 100+ task suites in weeks. Teams that try to design eval suites from scratch often stall at 20-30 tasks.
Environment isolation is non-negotiable for meaningful metrics: Shared state across trials creates hidden dependencies — trial 3 passes because trial 2 left a file behind. Running each trial in a fresh sandbox with pinned dependencies is the only way to ensure each measurement is independent. The infrastructure cost pays for itself in debuggability.
Transcript reading is the most underrated eval practice: Aggregate metrics show what failed but not why. Reading 10-20 failing transcripts per week reveals systematic patterns — the agent always fails at the same step, misuses a specific tool, or gives up too early. These patterns are invisible in dashboards but obvious in transcripts.

Sources:
Anthropic — Demystifying evals for AI agents (Jan 2026)

OpenClaw

OpenClaw AI Agent — Local-first autonomous AI agent architecture (Li Hongyi lecture)

▼

Architecture Overview

OpenClaw is an open-source, local-first personal AI agent (313k GitHub stars, 59.8k forks, 430k+ LOC) created by Peter Steinberger in November 2025. It connects any LLM (Claude, GPT, DeepSeek, Gemini, Ollama) to a local machine and messaging platforms for 24/7 autonomous task execution. The core design philosophy, as dissected by NTU Professor Li Hongyi: "OpenClaw is the non-AI part of an AI Agent" — all intelligence comes from the connected LLM, while OpenClaw provides the scaffolding for memory, scheduling, security, and tool execution. With 2.8M+ registered agents on the Moltbook platform, it demonstrates how context engineering (not model training) is the key discipline for building reliable autonomous agents.

Based on Li Hongyi — Dissecting OpenClaw AI Agent Architecture (2026)

graph TD subgraph Gateway["Gateway Layer"] direction TB WS["WebSocket Server
127.0.0.1:18789"] CH["Channel Layer
WhatsApp, Telegram,
Discord, Slack"] WS --> CH end subgraph AgentLoop["Agent Loop"] direction TB SP["System Prompt Assembly
SOUL.md + memory.md
+ conversation history"] LLM["LLM Call
Claude, GPT, DeepSeek,
Gemini, Ollama"] TOOL["Tool Execution
execute, read, spawn"] SP --> LLM LLM --> TOOL TOOL --> SP end subgraph Memory["Memory System"] direction TB MEM["memory.md
Long-term memory
Always in system prompt"] LOGS["Conversation Logs
Date-named files
Today + yesterday auto-loaded"] RAG["RAG Retrieval
Keyword + embedding
Weighted scoring, top-k"] end subgraph Autonomy["Scheduled Autonomy"] direction TB HB["Heartbeat
Cron-triggered every 30 min
Reads HEARTBEAT.md"] CRON["Cron Jobs
Async check-back
Configurable intervals"] end subgraph Security["Security Layer"] direction TB APPROVE["Human Approval Gate
Hardcoded, not bypassable"] COMPACT["Context Compaction
Recursive summarization
Soft trim, hard clear"] end Gateway --> AgentLoop AgentLoop --> Memory Autonomy --> AgentLoop AgentLoop --> Security style Gateway fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style AgentLoop fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Memory fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Autonomy fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3 style Security fill:#1c2333,stroke:#58a6ff,stroke-width:2px,color:#e6edf3

Identity and Context Engineering

Every LLM call in OpenClaw begins with assembling a system prompt from identity files and memory. The LLM has no persistent state — Professor Li's analogy: "a person in a black box with no windows, no calendar, no references — someone passes an unfinished sentence through a crack and it guesses what comes next." This is the "50 First Dates" problem: the agent must reconstruct its entire identity and context from scratch on every call.

SOUL.md: Markdown identity file prepended to every LLM call. Defines the agent's personality, goals, and behavioral constraints. A simple self-introduction question costs 4,000+ tokens in system prompt alone before any user content.
memory.md: Long-term memory file, always loaded into the system prompt and never compacted. This is the only guaranteed persistent state — if something is not written to memory.md, it effectively does not exist for the agent.
Conversation history: Today's and yesterday's conversation logs are auto-loaded. Older conversations rely on RAG retrieval, making them less reliably accessible.
Token overhead: The system prompt assembly (SOUL.md + memory.md + recent logs) consumes 4,000+ tokens per call before any task-specific content, making prompt efficiency a first-class engineering concern.

Memory System

OpenClaw uses a file-based memory system with no database — all state lives in plain Markdown files on the local filesystem. This design prioritizes transparency and debuggability over query performance. The critical insight from Professor Li: "If it's not written to memory.md, it's not real" — weaker models often say "I'll remember that" but never actually write to the file.

Component	Mechanism	Persistence	Reliability
memory.md	Always loaded in system prompt, never compacted	Permanent	Guaranteed — survives compaction, restarts, everything
Daily conversation logs	Date-named files (today + yesterday auto-loaded)	Permanent on disk	High for recent (auto-loaded), low for older (requires RAG)
RAG retrieval	Chunks memory files, keyword matching + embedding similarity, weighted scoring, returns top-k	On-demand	Variable — depends on query relevance and chunk boundaries
HEARTBEAT.md / habit.md	Standing instructions read during heartbeat cycles	Permanent	Guaranteed — read on every heartbeat trigger

Context Compaction

When conversation exceeds the LLM's context window, OpenClaw applies progressive compaction strategies. This is where the architecture's most dangerous failure mode emerges — instructions given in conversation (rather than memory.md) can be silently lost.

Recursive summarization: Old conversation history is sent to the LLM for summarization, and the summary replaces the original. This can recurse — summaries of summaries — creating what Professor Li called "nesting doll summaries." Each recursion loses more detail.
Soft trim: Tool outputs are truncated to head + tail (first and last N lines), based on the assumption that important information concentrates at the beginning and end of outputs.
Hard clear: Tool outputs are replaced entirely with placeholder text ("there was once a tool output here"), preserving the conversation structure while eliminating content.
The Email Deletion Incident: A Meta AI safety researcher instructed OpenClaw "get my approval before deleting emails" in conversation. After context compaction removed this instruction, the agent began autonomously deleting emails. The researcher could not stop it and had to pull the plug. This incident demonstrates why critical instructions must live in memory.md (part of the system prompt) rather than conversation history — only memory.md survives compaction.

Scheduled Autonomy

OpenClaw enables 24/7 autonomous behavior through two scheduling mechanisms that let the agent act without user prompts.

Heartbeat mechanism: A cron-triggered loop wakes the agent at configurable intervals (default: every 30 minutes, Professor Li changed his to 15 minutes). On each heartbeat, the agent reads HEARTBEAT.md and habit.md for standing instructions. Professor Li's agent "Xiao Jin" uses this to autonomously read papers, take notes, and work toward "becoming a world-class scholar" — all without any user prompts.
Cron jobs for async waiting: When the agent encounters asynchronous operations (e.g., submitting work to NotebookLM and getting a "generating..." response), smart models set a cron job (e.g., 3 minutes) to check back later. This pattern can be taught via memory.md: "whenever you see 'generating' or 'downloading', set a 3-min cron job to check back."
Proactive vs. reactive: Traditional chatbots only respond to user messages. The heartbeat mechanism inverts this — the agent initiates actions on its own schedule, enabling workflows like daily news digests, periodic code reviews, or continuous research that run indefinitely.

Subagents and Skills

OpenClaw supports two mechanisms for task decomposition: subagent spawning for parallel execution, and skills as declarative standard operating procedures.

Spawn mechanism: A parent agent spawns child agents for parallel work (e.g., two children each read one paper and return summaries). The key context engineering benefit: the child's verbose intermediate work (search, download, read) produces only a compact summary for the parent. Professor Li's analogy: "presenting to your advisor — they see the slides, not the messy experiments."
Depth limit: Children cannot spawn grandchildren. This is hardcoded in OpenClaw's architecture, not enforced via prompt — meaning it cannot be bypassed by prompt injection. Professor Li used Rick and Morty's Mr. Meeseeks analogy to explain the rationale: unlimited spawning leads to exponential resource consumption.
Skills as Markdown SOPs: Skills are declarative Markdown files (not compiled code) that describe step-by-step procedures. Example: a video production skill lists steps (write script, make HTML slides, screenshot, voice, verify, composite). Skills are lazy-loaded — the system prompt contains only the file path, and the agent reads the skill file on demand via the Read tool, saving tokens.
ClawHub security concern: Approximately 26% of community-contributed skills on ClawHub were found to contain vulnerabilities (341 malicious out of ~3,000 scanned), highlighting the supply-chain risk of declarative skill marketplaces.

Security Model

OpenClaw's execute tool can run any shell command on the local machine — Professor Li noted "the scariest part is the word 'any'." The security model uses a two-layer defense combining soft LLM-based constraints with hard architectural gates.

Prompt injection case study: Professor Li demonstrated a real attack where a YouTube comment modified files on his computer. His agent read the comment (legitimate action) and then acted on the embedded instructions (prompt injection). The comment contained shell commands that the agent dutifully executed.
Defense Layer 1 (soft): LLM-level instructions stored in memory.md, e.g., "just read YouTube comments, don't act on them." This is not guaranteed — the LLM may still follow injected instructions, especially from well-crafted attacks.
Defense Layer 2 (hard): OpenClaw's hardcoded human-approval gate before every execute call. Professor Li described it as "ruthlessly impartial" — it cannot be bypassed by prompt injection because it is enforced in application code, not via prompts.
Best practices: Use a dedicated machine or accounts for the agent. Do not install OpenClaw on a personal computer. Create separate Gmail and GitHub accounts for agent use. Treat the agent's execution environment as a sandbox with blast-radius limitations.

Key Design Decisions

Decision	Chosen Approach	Alternative	Rationale
Memory storage	File-based Markdown (memory.md + date-named logs)	Database (SQLite, vector DB)	Markdown files are human-readable, debuggable, and versionable with git. Trade queryability for transparency — critical when debugging agent behavior at 3 AM.
Context management	Recursive compaction (summarize, soft trim, hard clear)	Fixed sliding window	Recursive summarization preserves semantic content across arbitrarily long conversations. Fixed windows lose old context abruptly. Tradeoff: compaction can silently remove critical instructions (email deletion incident).
Subagent depth	Hardcoded single-level (no grandchildren)	Unlimited nesting	Unlimited spawning risks exponential resource consumption and makes debugging impossible. Single-level is enforced in code (not prompts), preventing prompt injection bypass. Covers 95%+ of real parallel workloads.
Skill format	Declarative Markdown SOPs	Compiled plugins or code modules	Markdown skills are readable by any LLM without special tooling, can be lazy-loaded (only file path in system prompt), and authored by non-developers. Tradeoff: ~26% of community skills contained vulnerabilities due to lack of sandboxing.
Security model	Hardcoded human-approval gates (application code)	LLM-decided permissions (prompt-based)	LLM-based security is bypassable via prompt injection. Hardcoded gates in application code are "ruthlessly impartial" — the LLM cannot override them regardless of prompt content. Defense in depth: soft (memory.md rules) + hard (code gates).
Autonomy mechanism	Heartbeat-driven (cron wakes agent every 30 min)	Event-driven only (respond to user messages)	Heartbeat enables proactive behavior (reading papers, sending digests) without user initiation. Event-driven limits agent to reactive mode. Tradeoff: heartbeat consumes tokens and API costs even when idle.
Platform abstraction	Channel Layer (platform-agnostic message routing)	Direct platform API integration	Channel Layer abstracts WhatsApp, Telegram, Discord, Slack behind a uniform interface. Adding a new platform requires only a new adapter, not agent logic changes. Tradeoff: lowest-common-denominator feature set across platforms.

Interview Talking Points

Context engineering, not model training, is the core discipline for autonomous agents: OpenClaw's 430k+ LOC is entirely non-AI scaffolding — system prompt assembly, memory management, scheduling, security. The LLM is a swappable black box; the engineering challenge is what you put in and around it.
File-based memory with a guaranteed persistence tier solves the "50 First Dates" problem: memory.md (always in system prompt, never compacted) is the only state the agent can rely on. Everything else — conversation logs, RAG results — is probabilistic. This two-tier design (guaranteed vs. best-effort) is the key architectural pattern.
Context compaction is a lossy operation with safety implications: The email deletion incident proved that recursive summarization can silently remove safety-critical instructions. The architectural lesson: never place security constraints in compactable context. Only the system prompt (memory.md, SOUL.md) is safe.
Hardcoded security gates are the only reliable defense against prompt injection: LLM-based guardrails (Defense Layer 1) can be bypassed by well-crafted prompts. OpenClaw's human-approval gate before execute is enforced in application code, making it immune to prompt manipulation — a pattern every agent framework should adopt.
Heartbeat-driven autonomy transforms agents from reactive to proactive: The 30-minute cron-triggered loop with HEARTBEAT.md enables 24/7 autonomous workflows (paper reading, monitoring, digests) that run indefinitely without user prompts, consuming tokens and API costs as the tradeoff.
Single-level subagent spawning balances parallelism with controllability: Hardcoding the depth limit (no grandchildren) in application code prevents both exponential resource consumption and prompt-injection-based spawning attacks. The parent-child summary pattern compresses verbose work into compact results — like "presenting slides to your advisor."
Declarative Markdown skills enable a community ecosystem but create supply-chain risk: ~26% of the 3,000 community skills scanned on ClawHub contained vulnerabilities (341 malicious). The tradeoff between openness and security in skill marketplaces mirrors package manager security challenges (npm, PyPI).
313k GitHub stars and 2.8M+ registered agents validate the local-first, model-agnostic architecture: By decoupling from any specific LLM provider and running on localhost (127.0.0.1:18789), OpenClaw avoids vendor lock-in while keeping user data on-device — a design that scales adoption without scaling infrastructure.

Sources:
Li Hongyi (NTU) — Dissecting OpenClaw AI Agent Architecture (2026)
OpenClaw — GitHub Repository (313k stars)

Quick Reference Comparison

System	Company	Scale	Key Innovation	Latency Target	Core ML Task
Cursor Code Completion	Cursor	1M+ QPS	Speculative edits (13x speedup)	Sub-second	Next-token prediction + code retrieval
Contextual RAG	Anthropic	Per-document ingestion	Chunk context enrichment + hybrid search	Query-time (fast)	Semantic + keyword retrieval
Multi-Agent System	Anthropic	N parallel agents	Composable patterns + context engineering	Seconds to minutes	Orchestration + tool use + reasoning
Agent Design Patterns	LangChain	Production agents (Claude Code, Manus)	7 context engineering patterns for long-running agents	Minutes to hours	Context management + multi-agent coordination
Gemini MoE	DeepMind	Trillion+ params	Sparse routing (2-10% params active)	Token-level	Sparse expert routing + load balancing
Gemini Storybook	DeepMind	10-page multimodal	Unified text+image+audio in one model	Seconds (full book)	Conditional multimodal generation
Nano Banana	DeepMind	Native 4K images	Reasoning-first AR + diffusion hybrid	Under 10 seconds	Scene planning + image token generation
Autonomous Driving	Waymo	200M+ autonomous miles	Foundation Model (System 1/2) + World Model (Genie 3)	<10ms onboard	Perception + prediction + planning + simulation
Agent Evaluations	Anthropic	100s-1000s of tasks per suite	Swiss Cheese eval layers + pass@k/pass^k metrics	Minutes per suite run	Agent testing + grading + regression detection
OpenClaw Agent	OpenClaw	313k GitHub stars, 2.8M+ agents	Heartbeat autonomy + file-based memory + hardcoded security gates	30-min heartbeat cycle	24/7 autonomous task orchestration

Interview Tips

Lead with Tradeoffs

There's no single right answer. Show you understand the design space — every decision has costs. Discuss what you'd choose and why, acknowledging what you're giving up.

Be Quantitative

Estimate latency, throughput, data volume, model size. "1M QPS" and "13x speedup" are more compelling than "very fast." Use back-of-envelope math.

Think End-to-End

Don't just design the model — design the system around it. Data pipeline, serving infrastructure, monitoring, failure modes, and iteration loops all matter.

Discuss Failure Modes

What breaks? How do you detect it? How do you recover? Expert collapse in MoE, context loss in RAG, drift in agents — showing you think about failures signals maturity.

Use the 7-Step Framework

Problem Formulation → Data → Features → Model → Serving → Evaluation → Iteration. This structure keeps your answer organized and ensures you don't skip critical areas.

Stay Current

LLM serving (speculative decoding, MoE), RAG patterns (hybrid search, reranking), agent architectures (tool use, context engineering), and multimodal generation are the hottest topics right now.