Agents Last: A Tiered Architecture for Production Enterprise AI

AI agents are powerful because they can reason across ambiguous information, select tools and adapt their execution plan as conditions change.

Those same characteristics also make agents slower, more expensive and less predictable than conventional software.

That creates a straightforward architectural principle:

Use agents for work that requires agentic behavior—not as the default execution method for every task.

Most enterprise work does not require an autonomous system to reason through every step. Many cases can be resolved with deterministic validation, business rules, specialized machine learning or a single large language model call.

A production enterprise AI architecture should therefore route work through progressively more flexible—and progressively less deterministic—execution tiers:

Deterministic validation → rules → specialized ML → LLM → agent → human exception handling

This does not mean agents are unimportant. It means agents should be positioned where their flexibility creates enough value to justify their additional cost, latency and risk.

Agents belong near the end of the execution hierarchy.

A Production Security System Demonstrates “Agents Last”

An AWS case study describing Abnormal AI’s email-threat-detection architecture provides a useful production example.

According to the case study, Abnormal AI processes billions of email messages using three primary detection tiers:

1. Heuristics, small models and lightweight classifiers process billions of messages per day.
2. Larger machine-learning models evaluate millions of uncertain cases.
3. Agents equipped with code-interpreter capabilities analyze tens of thousands of the most difficult cases.

The overwhelming majority of messages never reach an agent.

That is not a limitation of the architecture. It is the reason the architecture can operate at scale.

Running a full agent against every message would be unnecessarily expensive and computationally wasteful. Most messages can be evaluated using faster, cheaper and more predictable methods.

The agentic tier is reserved for cases requiring deeper analysis. These agents receive threat-intelligence data, use a sandbox to perform computational analysis and make a determination based on the broader behavioral model.

Abnormal AI also uses a separate analyst agent to study misclassifications and draft candidate improvements for the cheaper upstream tiers. In other words, the agent does not merely resolve difficult cases. It helps improve the rules and models that prevent future cases from escalating.

The architecture creates a feedback loop:

Difficult case → agent analysis → identified pattern → improved heuristic or model → more cases resolved upstream

That is a much stronger production pattern than continuously expanding the agent’s responsibilities.

The scale and performance claims in the AWS publication are supplied by AWS and its customer, Abnormal AI, and have not been independently audited. However, the underlying architectural lesson applies broadly:

The most expensive and flexible execution method should not process work that a cheaper and more deterministic method can resolve correctly.

Agent-First Architecture Reverses the Proper Design Sequence

Many organizations currently begin with an agent platform and ask what tools they can connect to it.

That reverses the design sequence.

The correct sequence begins with the work:

1. Define the business outcome.
2. Decompose the workflow into unit tasks and decisions.
3. Define the input, output and validation contract for each capability.
4. Determine the least complex execution method that can perform each capability reliably.
5. Introduce adaptive agent orchestration only where the workflow genuinely requires it.

An agent should consume stable, governed capabilities. It should not become the location where business rules, permissions, integrations and exception behavior are improvised inside a prompt.

The agent is an execution option—not the architecture.

The Tiered Enterprise AI Execution Router

A tiered enterprise AI architecture evaluates each request and routes it to the lowest-cost, lowest-risk execution method capable of resolving it.

This is not necessarily a rigid pipeline through which every case passes sequentially. It is an execution router.

Some cases may be rejected immediately by deterministic validation. Others may be resolved through a rule. A predictive model might classify another case with sufficient confidence. An LLM may interpret an ambiguous document. Only the most variable multistep cases may justify agentic execution.

The generalized routing hierarchy is:

Deterministic validation → rules → specialized ML → LLM → agent → human exception handling

Each tier has a different purpose.

Tier 1: Deterministic Validation

Validation should occur before probabilistic reasoning.

It confirms that the request is sufficiently complete, properly formed and authorized for processing.

Typical validation includes:

• Required fields
• Data types and permitted values
• Schema compliance
• Identity and authorization
• File type and size
• Duplicate detection
• Known data-quality constraints
• Workflow state
• Regulatory or policy prerequisites

Suppose a healthcare document-processing system receives a prior-authorization packet without the patient identifier required to match it to the correct record.

An LLM or agent should not guess.

The deterministic validation tier should reject or route the incomplete packet for correction.

This tier is fast, inexpensive and explainable. It also protects every downstream component from avoidable work.

A surprising amount of supposed “AI reasoning” is actually input validation that should have been written in conventional code.

Tier 2: Business Rules and Heuristics

Rules handle decisions that can be expressed explicitly and must behave consistently.

Examples include:

• Approval thresholds
• Eligibility requirements
• Routing criteria
• Required documentation
• Known fraud indicators
• Escalation conditions
• Security policies
• Regulatory exclusions
• Service-level priorities

Rules are not obsolete because AI exists.

They remain valuable because they are predictable, auditable and inexpensive. They can also enforce boundaries that an AI component is not authorized to reinterpret.

For example, an agent may recommend canceling a purchase order. A deterministic rule should still verify whether the purchase amount falls within the agent’s authorization limit.

Where policy is explicit, enforce it explicitly.

Do not ask an LLM to approximate a business rule that the organization can state precisely.

Tier 3: Specialized Machine Learning

Specialized machine-learning models are appropriate when the task involves classification, prediction, scoring, forecasting or anomaly detection based on historical patterns.

Examples include:

• Fraud detection
• Churn prediction
• Demand forecasting
• Risk scoring
• Ticket classification
• Predictive maintenance
• Anomaly detection
• Document classification

These models can often process high volumes with lower latency and cost than general-purpose LLMs or agents.

They also provide task-specific evaluation metrics. Depending on the use case, teams can measure precision, recall, false-positive rates, false-negative rates, calibration and threshold behavior.

A fraud model, for example, might assign a risk score. A deterministic policy can then decide whether to approve the transaction, request additional verification or escalate it for investigation.

The model predicts. The surrounding architecture decides what to do with the prediction.

Tier 4: Large Language Model

An LLM is useful when a task requires language interpretation or generation but does not require adaptive multistep autonomy.

Examples include:

• Summarizing a document
• Extracting meaning from unstructured text
• Classifying an unusual request
• Drafting a response
• Comparing narrative evidence
• Generating a structured explanation
• Answering a grounded question

A single bounded LLM call is usually easier to test, monitor and control than a general-purpose agent.

The capability should still have a defined contract:

• What context is supplied?
• What output schema is required?
• Which sources may be used?
• What confidence or evidence must be returned?
• What validation occurs after generation?
• What happens when the output fails validation?
• Which downstream actions are permitted?

If the task can be completed through one constrained model interaction, it does not need an agent.

Calling every LLM-powered feature an agent may improve the marketing language. It does not improve the architecture.

Tier 5: AI Agent

An agent becomes appropriate when the work genuinely requires adaptive execution.

That may include:

• Selecting among multiple approved capabilities
• Planning a variable sequence of steps
• Investigating an unfamiliar exception
• Revising an approach after intermediate results
• Maintaining task state across multiple actions
• Comparing several possible courses of action
• Using computational tools to verify conclusions

The agent should operate within explicit boundaries.

Those boundaries may include:

• Whitelisted capabilities
• Scoped workload identity
• Defined data access
• Transaction limits
• Maximum execution time
• Token and cost budgets
• Network restrictions
• Approval thresholds
• Complete action logging
• Kill switches
• Human escalation requirements

In the Abnormal AI example, agents execute code inside ephemeral sandboxes. The case study describes a no-egress configuration intended to prevent outside influences during execution and reduce the risk of data exfiltration.

The agents also use programmatic verifiers—including testing and validation tools—to check their work.

This reflects another important principle:

Trust an agent’s verified results more than its assertions.

Agents should not receive unlimited authority merely because they can reason through difficult cases.

Flexibility must be paired with constrained access, verification and accountability.

Tier 6: Human Exception Handling

Human review remains necessary when:

• The evidence is insufficient
• Models or rules disagree
• Confidence falls below an acceptable threshold
• The action exceeds the system’s authority
• The potential consequence is too severe
• Policy requires human authorization
• A novel case falls outside tested behavior
• The affected person has a right to challenge the decision

Human involvement should not be treated as evidence that the AI system failed.

In a tiered architecture, human exception handling is an intentional execution tier. It absorbs cases that should not be automated under current evidence, authority or risk tolerance.

However, a person clicking “Approve” does not automatically create meaningful human oversight.

The reviewer needs:

• The relevant evidence
• The system’s recommendation
• Visible uncertainty
• Conflicting information
• The reason for escalation
• Enough time to evaluate the case
• Authority to reject or override the recommendation
• A clear record of the final decision

Human review should be designed as a real capability, not as a liability shield attached to the end of the workflow.

The Execution Router Requires Capability Contracts

A tiered architecture depends on clearly defined capabilities.

For every unit of work, the organization should define:

• Input contract
• Output contract
• Validation requirements
• Permitted execution methods
• Confidence thresholds
• Authorization boundaries
• Failure behavior
• Escalation conditions
• Observability requirements
• Cost constraints
• Ownership
• Versioning

Consider an IT support-routing capability.

The contract might accept a ticket title, description, affected system, requester identity and urgency indicators. It might return a recommended support queue, confidence score, evidence and escalation status.

The execution router could then behave like this:

• Reject the request if required fields are missing.
• Apply deterministic routing rules for known systems or security incidents.
• Use a specialized classification model for common tickets.
• Use an LLM when language ambiguity requires semantic interpretation.
• Invoke an agent only when the case requires investigating multiple systems or selecting several capabilities.
• Escalate to a human when confidence is insufficient or the requested action exceeds automated authority.

The business capability remains stable even if the implementation changes.

A C# rule, ML.NET model, Azure-hosted model, local ONNX model, LLM or agent can be replaced without forcing every consuming workflow or interface to change.

That is one of the major benefits of capability-first architecture.

How to Measure a Tiered AI Architecture

A tiered execution router cannot be evaluated with one overall accuracy number.

Each tier serves different cases, operates at a different cost and introduces different consequences.

At minimum, measure the following.

Resolution Rate by Tier

Track the percentage and number of cases resolved at each execution tier.

For example:

• Percentage rejected by validation
• Percentage resolved through rules
• Percentage resolved by specialized ML
• Percentage resolved through an LLM
• Percentage requiring an agent
• Percentage escalated to a human

This reveals whether inexpensive upstream tiers are doing useful work or whether too many cases are falling into costly downstream execution.

Escalation Rate

Measure how frequently each tier escalates a case and why.

A high escalation rate might indicate:

• Weak validation
• Incomplete rules
• Poor model performance
• Bad confidence thresholds
• Missing data
• Overly narrow capability contracts
• Legitimately complex cases

The goal is not to eliminate escalation. The goal is to make escalation accurate and intentional.

Escalation Accuracy

Did the system escalate the cases that genuinely required more capable processing?

Two errors matter:

• Under-escalation: A difficult or consequential case is incorrectly resolved by a lower tier.
• Over-escalation: A routine case is unnecessarily sent to an LLM, agent or human.

Under-escalation can create incorrect decisions and business harm. Over-escalation increases cost, latency and operational workload.

End-to-End Latency

Measure latency at each tier and across the entire workflow.

Important measurements include:

• Processing time per tier
• Queue time between tiers
• Total time to accepted resolution
• Tail latency for difficult cases
• Human-review wait time

An agent may complete its own analysis quickly while the resulting case waits hours for approval. The workflow metric matters more than the component metric.

Cost per Resolved Case

Measure the complete cost of resolving a case, including:

• Compute
• Model or token charges
• Agent execution
• Tool calls
• Data retrieval
• Human review
• Rework
• Infrastructure
• Monitoring and support

Average cost alone can conceal problems. Track cost by tier, case type and final outcome.

The important question is not simply, “What did the model call cost?”

It is:

What did it cost to produce one correct, accepted business result?

Error Consequences

Not all errors have equal consequences.

A false positive that sends a routine email for secondary review is different from a false negative that allows a sophisticated phishing attack to reach an employee.

For each capability, document:

• False-positive consequence
• False-negative consequence
• Reversal cost
• Customer or employee impact
• Financial exposure
• Security impact
• Regulatory impact
• Maximum acceptable error rate

Routing thresholds should reflect these consequences.

Human Review Burden

Track:

• Number of escalated cases
• Review time per case
• Percentage approved without meaningful change
• Percentage corrected or rejected
• Reviewer disagreement
• Queue length
• Reviewer fatigue
• Cases returned for additional evidence

If nearly every agent decision is approved without examination, the organization may have automation bias rather than effective human oversight.

If humans frequently reverse the agent, the system may not be ready for its current authority.

Business Outcome

Finally, measure the result the workflow exists to produce.

Examples include:

• Threats blocked
• Claims correctly processed
• Tickets routed and resolved
• Documents accepted
• Equipment failures prevented
• Orders fulfilled
• Review backlogs reduced
• Customer issues resolved

A technically sophisticated agent is irrelevant if the complete workflow does not improve.

Promote Repeated Agent Work Upstream

A mature tiered architecture should learn from its difficult cases.

When an agent repeatedly solves the same pattern, ask whether that behavior can be promoted into a cheaper and more deterministic tier.

For example:

• A repeated validation pattern may become deterministic code.
• A recurring decision may become a business rule.
• A stable classification pattern may become a specialized model.
• A repeatable language transformation may become a bounded LLM capability.
• Only novel and variable cases remain with the agent.

This creates an architectural ratchet:

Agent discovery → validated pattern → upstream implementation → lower cost and greater predictability

The agent becomes a source of learning, not an endlessly expanding center of responsibility.

Abnormal AI’s analyst-agent pattern illustrates this idea. The agent studies misclassifications and generates candidate improvements for upstream heuristics and models.

The strongest production agent may be the one that gradually reduces how often other agents are needed.

When an Agent Should Not Be Used

Do not use an agent merely because:

• The vendor labels the feature “agentic”
• The workflow has multiple steps
• An LLM can call a tool
• Executives want an agent demonstration
• The organization already purchased an agent platform
• A deterministic workflow appears less innovative
• The use case can be described conversationally

Use an agent when the workflow requires adaptive planning and the organization can demonstrate that this flexibility provides enough value to justify its cost and risk.

If the sequence is known, implement the sequence.

If the rule is known, encode the rule.

If the task is predictive, evaluate a predictive model.

If one bounded LLM call can complete the work, do not create an autonomous loop.

Agents should earn their place in the architecture.

Agents Last Does Not Mean Agents Never

“Agents last” is not an argument against agents.

It is an argument for production engineering.

Agents can solve cases that are too variable for rigid workflows and too complex for a single model invocation. They can investigate, plan, use tools, evaluate intermediate results and revise their approach.

But they should operate on top of defined work, stable capabilities, controlled interfaces and explicit authorization.

The construction order is:

Strategy → work definition → capability realization → reusable services → interfaces → agents

The execution order is:

Validation → rules → specialized ML → LLM → agent → human exception handling

Together, these patterns prevent organizations from building expensive autonomous systems around work they have not defined and decisions they could have implemented more reliably.

The objective is not to route everything through the most intelligent component.

The objective is to resolve each case with the least complex method that can produce a correct, secure and economically sensible outcome.

That is how enterprise AI moves from an impressive demonstration to a scalable production system.

Source

AWS and Abnormal AI, “Abnormal AI: Amazon Bedrock AgentCore for Agentic Email Security at Scale,” September 14, 2026.

Related AInDotNet Resources

• Enterprise AI Agents: Pillar 6 of Enterprise AI Architecture
• Capability Realization for Enterprise AI
• Enterprise AI Architecture
• Interfaces: Pillar 5 of Enterprise AI Architecture
• AI Engineering Discipline and Delivery Lifecycle

Frequently Asked Questions