Your AI Gateway Is a Tier-0 System: What Enterprise Architects Must Change

AI security conversations often focus on prompt injection, hallucinations and data leakage. Those risks are real, but attackers do not need to manipulate a model if they can compromise the infrastructure surrounding it.

Microsoft recently documented compromises involving a LiteLLM gateway, a RAGFlow deployment and a Kestra workflow environment. The affected products perform different functions, but the attackers pursued a familiar set of objectives: steal credentials, establish persistence, access connected systems and monetize compromised computing resources.

The larger lesson is more important than any individual vulnerability. AI gateways, retrieval platforms and orchestration engines are becoming privileged enterprise control points. They can sit between users, applications, models, databases and internal services while holding credentials and execution rights across all of them.

Microsoft’s recommendation is blunt: treat AI gateways as Tier-0 secrets stores.

That does not necessarily make every AI gateway part of the traditional identity Tier-0 used for domain controllers and other identity control-plane systems. It means the gateway’s concentration of secrets, connectivity and authority can create a comparable blast-radius problem. Enterprise architects should classify and secure it according to what it can reach—not according to the fact that it happens to route AI requests.

What Is an AI Gateway?

An AI gateway is an intermediary between enterprise applications and one or more AI model providers. Depending on the implementation, it may provide:

• Centralized access to Azure OpenAI, OpenAI and other model providers
• Model selection and routing
• Authentication and authorization
• Rate limits, quotas and spending controls
• Prompt and response logging
• Content filtering and policy enforcement
• Tenant or department isolation
• Virtual API keys
• Failover between models or providers

This sounds similar to a conventional API gateway, but an AI gateway frequently accumulates additional authority. It may store or retrieve model-provider keys, database connection strings, proxy master keys, tenant configuration and routing policies. It may also connect to retrieval systems, workflow engines and tools capable of acting on enterprise systems.

The gateway is therefore not merely forwarding prompts. It can become the trust broker for the enterprise AI environment.

What Microsoft Observed

Microsoft’s security research described three different compromises that illustrate the broader attack surface.

LiteLLM: The Gateway Became the Attack Origin

LiteLLM is commonly used as a proxy between applications and multiple model providers. In the incident Microsoft investigated, attackers executed commands in the gateway process context and harvested secrets from the runtime environment.

The observed activity included collection of model-provider API keys, a LiteLLM master key, database connection strings, tokens and other credentials. Attackers then accessed the gateway’s PostgreSQL database, collected model and virtual-key records, installed cryptocurrency-mining software and established persistent access.

This was not simply the compromise of an isolated container. The gateway’s position gave the attacker access to credentials and configuration associated with downstream AI services.

RAGFlow: The Retrieval Platform Was Modified to Steal Future Credentials

RAGFlow supports retrieval-augmented generation and document-processing workflows. Microsoft observed attackers modifying the application runtime so that a hidden hook loaded when the service started.

The hook targeted the tenant model-configuration process. When users later supplied model-provider credentials, the modified application could intercept and exfiltrate the provider type, model name, API key and endpoint information.

This is especially important for incident response. Rotating the credentials that existed when the intrusion was discovered would not be sufficient if the compromised platform remained capable of stealing their replacements.

Kestra: The Orchestrator Provided a Path to Shell and Container Access

Kestra is a workflow-orchestration environment. Workflow engines are designed to execute tasks and interact with other systems, so compromise can turn legitimate automation functionality into an attack mechanism.

Microsoft observed workflow-originated shell execution, Docker socket access, inspection of container environments, deployment of cryptocurrency-mining software and follow-on data collection. Access to the Docker socket allowed the attackers to inspect environment-backed values from other reachable containers.

The lesson is straightforward: an orchestrator with access to the container runtime may have authority far beyond its own application boundary.

Why AI Infrastructure Creates an Unusual Blast Radius

Traditional applications usually have a defined relationship with a limited set of dependencies. Enterprise AI platforms are often intentionally designed to span boundaries.

An AI gateway or orchestration platform may connect to:

• Multiple model providers
• Vector databases and search indexes
• SQL databases and data lakes
• SharePoint, Microsoft Graph and Microsoft 365
• Customer relationship management and enterprise resource planning systems
• Internal APIs and line-of-business applications
• Agent tools and MCP servers
• Workflow and automation platforms
• Logging, evaluation and observability systems

This concentration is useful because it allows the organization to apply governance, security and cost controls in one place. It is dangerous for exactly the same reason.

If the platform uses shared master credentials or overly broad service identities, one compromised component can become a bridge to many downstream systems. The gateway may also log sensitive prompts, responses and metadata, creating another valuable collection point.

The right security classification therefore depends on accumulated privilege. Ask what the component can authenticate to, what secrets it can retrieve, what data it can observe and what actions it can initiate. The product label—gateway, RAG platform, agent framework or orchestrator—is secondary.

Prompt Injection Is Not the Only Enterprise AI Security Problem

Prompt injection receives enormous attention because it is specific to generative AI. The Microsoft incidents are a useful correction: conventional security failures remain capable of compromising the entire AI stack.

The observed patterns included exposed management surfaces, remote-code-execution paths, credential harvesting, excessive container privileges, unrestricted outbound connections and persistence through application or host modification.

None of those attack techniques requires the model to be tricked.

Enterprise AI security must therefore operate on two levels:

1. AI-specific controls: prompt-injection defenses, grounding, content safety, tool authorization and model evaluation.
2. Infrastructure controls: patching, identity, secrets management, network isolation, workload hardening, runtime monitoring and incident response.

An organization that implements elaborate prompt defenses while exposing an orchestration console to the internet has optimized the wrong part of the risk model.

Seven Architecture Changes Enterprises Should Make

1. Inventory Every AI Control Point

Maintain an inventory of AI gateways, retrieval platforms, model endpoints, orchestration engines, agent runtimes and MCP servers. Record the owner, environment, public exposure, identities, secrets, connected systems, data classifications and execution rights for each component.

This inventory should feed the enterprise AI capability registry and configuration-management process. If the security team cannot identify all deployed AI middleware, it cannot reliably patch, monitor or contain it.

2. Remove Management Surfaces From the Public Internet

Administrative interfaces should not be directly internet-accessible. Use private networking, controlled administrative access and explicit authentication across both user interfaces and APIs.

For Microsoft-centric environments, this generally means virtual-network integration, private endpoints where supported, restrictive network security rules and carefully controlled administrative paths. Public access should be an explicit exception supported by a documented requirement and compensating controls.

3. Replace Shared Secrets With Scoped Workload Identities

Use Microsoft Entra workload identities and managed identities where supported. Assign each workload only the permissions required for its function.

When provider keys remain necessary, issue scoped keys by application, team or environment. Apply spending limits and usage restrictions instead of distributing a single master key across workloads. Separate development, test and production credentials.

Least privilege reduces the blast radius, but it also improves attribution. A credential used by one workload is easier to monitor and revoke than a credential shared by twenty applications.

4. Move Secrets Out of Process Environment Variables

Environment variables are convenient, but a compromised process or container can often read them. In the LiteLLM case, Microsoft observed attackers reading the gateway process environment and extracting credential-related values.

Store secrets in a managed service such as Azure Key Vault and retrieve them through a managed identity. Restrict vault access with role-based access control, private endpoints and logging. Automate rotation and design the application so credentials can be replaced without a prolonged outage.

This does not make secrets invulnerable. A compromised workload may still request a secret that it is authorized to use. It does, however, remove broad static exposure and provides a central control and audit point.

5. Restrict Outbound Traffic

AI workloads need outbound access to model endpoints and other services, but they rarely need unrestricted internet access.

Use deny-by-default egress rules and allow only approved destinations. Route outbound traffic through controls capable of FQDN filtering and logging. Monitor raw-IP connections, unusual ports, unexpected DNS activity and callbacks to unapproved domains.

Restricted egress can interrupt credential exfiltration, payload downloads and cryptocurrency-mining traffic even after an attacker obtains code execution.

6. Isolate the Runtime and Protect the Container Boundary

Run the service under a dedicated, nonprivileged identity. Use minimal container images, scan them for vulnerabilities and remove unnecessary shells, downloaders and utilities where practical.

Do not mount the Docker socket into an AI gateway or orchestrator unless the business requirement is unavoidable and the risk is explicitly accepted. Docker socket access can effectively grant control over other containers and expose their configuration and credentials.

Separate gateways, retrieval services and workflow workers when they have different privilege requirements. A public-facing routing component should not automatically share a runtime boundary with a worker capable of executing enterprise workflows.

7. Monitor Behavior, Not Merely Availability

Infrastructure monitoring that reports only CPU, memory, latency and uptime will miss the most important security signals.

High-value detections include:

• An AI gateway spawning a shell, Python interpreter, downloader or system utility
• Reads of process environments or credential files
• Unexpected access to provider-key or virtual-key records
• Application files modified inside a running container
• Access to the Docker socket
• New SSH keys, scheduled tasks or persistence mechanisms
• Outbound callbacks to raw IP addresses or unfamiliar domains
• Sudden cryptocurrency-mining behavior or unexplained compute consumption

Correlate these signals as a possible attack chain. A Python process may be normal in isolation; a gateway spawning Python to read environment variables and contact an unknown host is not.

Incident Response Must Include Downstream Credential Rotation

Rebuilding the affected container is only one part of recovery. When an AI control point is compromised, incident responders should assume that every secret available to its runtime may have been exposed.

The response plan should identify and rotate:

• Model-provider keys
• Gateway master and virtual keys
• Database credentials
• Service-principal credentials
• API tokens for connected applications
• Signing keys and certificates where exposure is plausible
• Credentials visible through connected container environments

Teams must also inspect databases, application startup paths, mounted volumes, images and deployment pipelines for persistence. The RAGFlow incident demonstrates why rotating credentials before removing the interception mechanism can simply deliver fresh credentials to the attacker.

The organization should be able to revoke the gateway’s identity and isolate the workload without first designing the procedure during an active incident.

This Is Another Reason to Put Agents Last

Agentic AI increases the importance of the control plane. A conventional assistant may retrieve information and generate a response. An agentic system may select tools, execute workflows, modify records or interact with multiple enterprise systems.

That authority should be introduced only after the organization has established:

• A governed capability catalog
• Explicit input and output contracts
• Scoped workload identities
• Tool-level authorization
• Human approval for consequential actions
• Central logging and evaluation
• Cost and rate controls
• Revocation and kill-switch procedures

Starting with autonomous behavior before building these foundations does not accelerate enterprise AI maturity. It accelerates the accumulation of unmanaged privilege.

The Enterprise Architecture Decision

Centralized AI gateways and orchestration platforms are not inherently a mistake. In fact, they can improve governance by consolidating authentication, policy enforcement, model routing, telemetry and spending controls.

But centralization creates a security obligation. The more trust the organization concentrates in the platform, the more deliberately it must isolate, monitor and govern that platform.

The decisive architecture question is not:

Which AI gateway product should we use?

It is:

What authority will this gateway accumulate, and what happens if an attacker gains that authority?

Microsoft’s findings show that attackers are already treating AI infrastructure as a control plane. Enterprise architects should do the same.

Enterprise AI Gateway Security Checklist

Before approving an AI gateway, retrieval platform or orchestration engine for production, verify that:

• The component and its dependencies are inventoried and owned.
• Administrative surfaces are not exposed directly to the internet.
• Supported versions and security patches are enforced.
• Authentication is required across management, UI and API surfaces.
• Workload identities and provider credentials are scoped by application and environment.
• Master credentials are not unnecessarily shared.
• Secrets are managed centrally and can be rotated quickly.
• Databases and supporting services use private connectivity and restrictive firewall rules.
• Outbound traffic is restricted to approved destinations.
• Containers run with minimal privileges and without unnecessary runtime-socket access.
• Security monitoring detects application-originated shells, secret access, persistence and suspicious egress.
• Incident-response procedures include containment, persistence removal and downstream credential rotation.
• The system has a tested revocation path and operational kill switch.

AI infrastructure is becoming critical enterprise infrastructure. Its security classification should reflect the credentials, data and execution authority it concentrates—not the experimental label under which it was originally deployed.

More Information

Check out our Enterprise AI Architecture (EAA)

Our Capability First framework describes a safer and better implementation

Frequently Asked Questions

Sources

• Microsoft Security Research: When AI infrastructure becomes the target
• Microsoft Learn: Secure AI-ready infrastructure
• Microsoft Learn: Configure workload identities and permissions for AI environments
• Microsoft Cloud Adoption Framework: Configure secure networking for Azure AI platform services