How do you design an AI agent system design?

01Interview Context

This question is about system discipline, not prompt tactics. The interviewer wants to hear the boundaries you set, the control plane you build, and the operational failure modes you expect. Weak answers say "LLM plus tools" and then stop.

02What to open with

Open by naming the task shape and the risk boundary. I would start with the user, the goal, and the autonomy level before I sketch any architecture.

• Is this a customer support, coding, research, automation, or internal operations agent?
• Who uses it: external customers, support staff, developers, or ops?
• What should it accomplish: answer, escalate, automate, or take action?
• Should it act automatically, or only suggest actions?
• What systems can it access, and what must stay off limits?
• What happens if it makes a mistake?
• Is the experience interactive, batch, or background?
• How much volume and concurrency should it handle?

If you skip that, you are designing the wrong architecture.

03What the agent loop should look like

The selling point is not the model; it is the loop. I would describe the loop as the thing that makes the system observable and recoverable.

User request → Understand intent → Plan steps → Choose tool → Call tool → Observe result → Update state → Decide next action → Final response.

Call out the patterns interviewers are listening for:

• ReAct: reason, act, observe when tool choice is uncertain.
• Plan-and-execute: separate planning from execution for complex multi-step work.
• Router agent: select a specialized workflow or tool instead of one giant policy.
• Reflection: review progress before closing the task.
• Human approval: require it for destructive or high-risk actions.
• Workflow-first: keep deterministic steps and use the model only where flexibility matters.

Strong phrasing: “The orchestrator owns the control flow. The LLM is a decision engine inside a bounded loop, not the boss of the system.” I would always say that before I describe the components.

04What the architecture should include

Describe a layered system, not a single box.

Frontend / API → Auth & rate limit → Agent orchestrator → Prompt/policy manager → Memory/state store → Tool registry → Model gateway → Eval/guardrails → Observability → Approval queue.

The important components:

• Agent orchestrator: step budgets, retries, stop conditions, and flow control.
• Tool registry: schemas, permissions, and allowed actions.
• Model gateway: model selection, fallback, and cost tracking.
• State store: separate conversation context from execution state.
• Policy/eval: block unsafe decisions before they execute.
• Observability: trace prompts, tool calls, and intermediate decisions.
• Approval queue: human sign-off for sensitive side effects.

Say this before you mention any vendor or library.

05What tool design must cover

This is where senior answers separate from generic ones.

• Strict tool schemas for inputs and outputs.
• Tool-level authorization tied to user and task scope.
• Idempotency for all side-effecting calls.
• Argument validation before execution.
• Timeouts and retry policy for every external call.
• Explicit confirmation for destructive actions.
• Audit logging for every tool call and result.

For read-only tools, the edge is lower. For writes, I would require a separate approval step and show the exact payload.

A good line: “The final answer should cite tool results, not rely on the model’s memory of what it did.”

06What memory and state really mean

Do not answer “vector DB” and stop.

Use distinct memory types:

• Short-term memory: current conversation context.
• Working memory: current plan, tool outputs, and scratchpad.
• Long-term memory: user preferences and past tasks.
• Retrieval memory: documents, tickets, knowledge bases, with ACL filtering.
• Execution state: task status, current step, retries, approval state.

For long-running agents, I suggest to store:

task_id
status
current_step
plan
tool_calls
observations
retry_count
approval_status
final_result

Important distinction: chat history is language context; execution state is workflow control.

07What control must be enforced

Agent systems fail when they never stop.

Cover these controls:

• Max steps: hard limit on loop iterations.
• Budget limits: caps for tokens and tool spend.
• Tool allowlist: only authorized actions are available.
• Confidence checks: ask for clarification or fallback when uncertain.
• Required confirmations: humans approve high-risk changes.
• Checkpoints: resume failed workflows safely.
• Fallback path: escalate to human or simpler workflow.

Again: the orchestrator, not the LLM, owns the control flow.

08What RAG/grounding should look like

If the agent uses external knowledge, make retrieval an explicit step.

User query → rewrite/search query → retrieve docs → rerank → inject relevant context → generate answer → cite sources.

Failure modes to name:

• bad retrieval: rerank or hybrid search.
• stale data: freshness ranking and timestamps.
• hallucination: require citations and abstain when unsupported.
• context overload: chunk and limit injected context.
• wrong permissions: retrieval must be ACL-aware.

Strong line: “Retrieval must be permission-aware. The agent must never surface data the user is not allowed to see.”

09What safety and guardrails should include

This question is more about what you block than what you permit.

Cover both input and output safety:

• Prompt injection: treat retrieved text as untrusted.
• Tool misuse: enforce authorization and schema checks.
• Data leakage: redact sensitive outputs.
• Hallucination: ground everything in evidence.
• Unsafe actions: require human approval for risky steps.
• Sensitive data: detect PII and limit access.

A strong warning: “Prompt injection is especially dangerous for agents because the model can trick itself into requesting unsafe tool execution.”

10What evaluation and observability must cover

Good answers show measurable discipline.

Evaluate multiple layers:

• Final answer: correctness, usefulness, groundedness.
• Tool selection: did it choose the right action?
• Tool arguments: were they valid?
• Plan quality: was it reasonable?
• Task success: did it complete.
• Safety: did it avoid unsafe actions?
• Cost and latency: was it efficient?

Metrics to call out:

task success rate
tool call accuracy
hallucination rate
groundedness rate
latency p50/p95
cost per task
human escalation rate
retry rate
user satisfaction

Add this: unit tests are not enough. Run end-to-end task simulations, because agent failures often unfold across steps.

11What production discipline matters

Finish with reliability, cost, and deployment discipline.

Reliability:

• model timeout: retry or fallback.
• tool failure: retry with backoff.
• bad structured output: validate and repair before execution.
• duplicate action: use idempotency.
• long task: run outside request scope.
• ambiguous request: ask for clarification.

Cost and latency:

• use smaller models for routing.
• cache retrieval and tool results.
• limit max steps.
• batch calls where possible.
• only escalate to stronger models when needed.

Deployment:

• version prompts, models, and schemas.
• use feature flags and canaries.
• rollback bad agent behavior quickly.
• treat prompt changes like code changes: review, test, stage.

Strong closing line: “Design the agent as a disciplined system, not a shortcut for hand-wavy reasoning. The model is a component; the orchestrator, tooling, guardrails, and observability are what make it production-ready.”