# The Identity Chain: A Foundry Hosted Agent, an AKS MCP Gateway, and Agent 365 Governance

A proof of concept that carries one signed-in user's identity through two OAuth On-Behalf-Of (OBO) exchanges — from a client, through a trusted middleware, a Microsoft Foundry hosted agent, and a LiteLLM MCP gateway on Azure Kubernetes Service (AKS), to a Microsoft Agent 365 allow/block decision — while every workload authenticates under its own, separate identity. This article explains the design, the controls, the tests, and what the build taught me. All identifiers are replaced with generic names, and the diagrams are sanitized.

> Preserving *who the user is* and proving *what the workload is* are two different jobs. Mixing them in one credential is the most common way agent identity chains go wrong. This reference keeps them in separate identities, separate tokens, and separate code boundaries — and puts a governance decision in front of every tool call.

> **Rendering note.** Diagrams are provided in both a Mermaid block and an ASCII block. Where they disagree, the ASCII form is authoritative.

## What this is, and is not

This is a **working proof of concept**, deployed in a single Azure subscription and exercised end to end with a real signed-in user. It is a teaching reference, deliberately not production-ready, and it makes no claim that it is. The tool at the end of the chain is [DeepWiki](https://deepwiki.com), a public MCP server, queried about one fixed public repository — so no private data is involved.

| Claim | Status |
|---|---|
| User → middleware → Foundry hosted agent → LiteLLM → Agent 365 allow → DeepWiki, with a real user | **Verified end to end** |
| Request without a delegated user token is refused before any tool runs | **Verified** (HTTP 401 at the gateway) |
| Malformed, wrong-issuer, wrong-audience, wrong-tenant, and missing-scope tokens are rejected by the middleware | **Verified by automated tests** |
| An administrator-configured Defender policy *blocks* a tool call and DeepWiki is never contacted | **Designed, not yet exercised live** — the allow path is proven; the block path awaits an approved blocking policy |
| LiteLLM-native JWT admission of the user token | **Not part of the verified path** — it requires an Enterprise license; admission uses a virtual key instead (see [Lessons learned](#lessons-learned)) |

The repository contains Bicep infrastructure, Kubernetes manifests, a Python middleware, a Foundry hosted agent, and operational documentation. This article includes only the excerpts needed to explain decisions, with identifiers replaced by placeholders.

## The problem: one user, five trust domains

A user asks an agent a question. Before a tool runs, the request crosses five trust domains — a client, a middleware, an agent platform, a tool gateway, and a governance service — and each one needs a different answer to a different question:

| Domain | Question it must answer |
|---|---|
| Middleware | Is this token really from my tenant, for my API, with the delegated scope? |
| Agent platform (Foundry) | Is the caller a workload I trust, and may it act for this user? |
| Tool gateway (LiteLLM) | Is this application allowed to use this MCP server, and is a user present? |
| Governance (Agent 365 / Defender) | Should *this user* be allowed to run *this tool call* right now? |
| Tool (DeepWiki) | Has everything upstream said yes? |

If a single bearer token or a single shared secret answers all of those, any compromise at any hop becomes a compromise of the whole chain, and the audit trail cannot say who asked. The design goal is the opposite: **each hop receives a token whose audience is that hop, and each workload proves itself with a credential that is not the user's.**

## Architecture at a glance

```text
User
  |  interactive sign-in (public client, no secret)
  v
+---------------------+
| Client (CLI / UI)   |   token: aud = middleware API, scp = access_as_user
+---------------------+
  |  HTTPS, rate-limited, exact-path ingress
  v
+---------------------------------------------+
| AKS: trusted middleware (2 replicas)        |   validates user token
|  - app registration #1 (OBO confidential)   |   OBO #1 -> token for LiteLLM API
|  - workload identity (federated, no secret) |   workload token for Foundry
+---------------------------------------------+
  |  Authorization: workload token (not forwarded)
  |  x-client-*: delegated token   x-ms-user-identity: user object ID
  v
+---------------------------------------------+
| Microsoft Foundry                           |   authenticates the workload,
|  hosted agent (fixed capability)            |   authorizes impersonation
+---------------------------------------------+
  |  Authorization: Bearer <delegated token>
  |  x-litellm-api-key: <tool-scoped virtual key>
  v
+---------------------------------------------+
| AKS: LiteLLM MCP gateway                    |   virtual key -> MCP server permission
|  - app registration #2 (OBO confidential)   |   OBO #2 -> Agent Tools token
|  - pre_mcp_call guardrail, fail closed      |
+---------------------------------------------+
  |  evaluate pending tool call (server, tool, arguments)
  v
+---------------------------------------------+
| Microsoft Agent 365 / Defender              |   Allow or Block
+---------------------------------------------+
  |  only on Allow
  v
DeepWiki MCP server
```

```mermaid
flowchart TD
    U["User"] --> C["Client (public, no secret)"]
    C -->|"token aud = middleware"| MW["AKS middleware<br/>validate + OBO #1"]
    MW -->|"workload token + delegated token<br/>+ validated user object ID"| F["Foundry hosted agent<br/>fixed capability"]
    F -->|"delegated token + virtual key"| G["LiteLLM MCP gateway on AKS<br/>OBO #2, pre_mcp_call"]
    G -->|"evaluate tool call"| A["Agent 365 / Defender"]
    A -->|"Allow"| T["DeepWiki MCP"]
    A -->|"Block"| X["Refused, tool not called"]
```

Two workloads run in the same AKS cluster — the middleware and the gateway — but they are separate Deployments with separate service accounts and separate Entra identities. Co-locating them reuses ingress, observability, and deployment controls; it does not merge their trust. Note too that the Foundry-managed compute is **not** inside the cluster network: it reaches the gateway over a public HTTPS endpoint, which is why the gateway's admission controls matter.

## Identity separation: four credentials, four jobs

The most useful table in this project is the one that says what each credential is *for* and what it must never be used for.

| Credential | Represents | Authenticates to | Must never be |
|---|---|---|---|
| **User token** (aud = middleware API) | The signed-in user | The middleware | Forwarded downstream as-is |
| **Delegated token** (aud = LiteLLM API) | The same user, re-issued for the next hop | LiteLLM, and the input to OBO #2 | Written to a prompt, log, or response |
| **Workload token** (Foundry scope) | The middleware *workload* | Foundry | Used to carry user identity, or forwarded to the agent container |
| **Virtual key** | The *application* | LiteLLM | Supplied by the user or the middleware |

Four identities sit behind those credentials, and each is a different Entra object:

| Entra object | Role |
|---|---|
| Client app registration | Public client. Interactive sign-in only. **No credential.** |
| Middleware app registration | Exposes the middleware API *and* acts as the confidential client for OBO #1. |
| Gateway app registration | Exposes the LiteLLM API *and* acts as the confidential client for OBO #2. |
| User-assigned managed identity | The middleware's Azure workload identity for calling Foundry. Federated to one Kubernetes service account. |

Keeping the OBO credential separate from the Azure workload identity is deliberate: the identity that can *represent a user* is not the identity that can *invoke Foundry*, so neither compromise alone yields both abilities.

## Zero-secret AKS-to-Foundry authentication

The middleware authenticates to Foundry with **no stored secret**. AKS Workload Identity federates a Kubernetes service account to a managed identity:

```text
Middleware pod
  | uses one Kubernetes ServiceAccount
  v
AKS OIDC issuer
  | signs a short-lived projected ServiceAccount token
  v
Entra federated identity credential
  | verifies issuer + subject + audience
  v
User-assigned managed identity
  | requests the Foundry resource scope
  v
Foundry workload access token
```

The federated credential is three facts and nothing else:

| Federation field | Meaning |
|---|---|
| Issuer | The one token issuer Entra will trust — this cluster's OIDC issuer |
| Subject | The exact namespace and service account permitted to act as the identity |
| Audience | `api://AzureADTokenExchange`, marking the projected token as intended for Entra |

It is expressed as a small reusable Bicep module:

```bicep
resource federatedCredential 'Microsoft.ManagedIdentity/userAssignedIdentities/federatedIdentityCredentials@2024-11-30' = {
  parent: identity
  name: federatedCredentialName
  properties: {
    audiences: [ 'api://AzureADTokenExchange' ]
    issuer: oidcIssuerUrl
    subject: 'system:serviceaccount:${kubernetesNamespace}:${serviceAccountName}'
  }
}
```

In the pod, the service account carries a client-ID annotation and the pod carries one opt-in label (`azure.workload.identity/use: "true"`). The webhook injects the projected token file, and `DefaultAzureCredential` exchanges it. There is no client secret, certificate, or key in the cluster for this leg.

### Federation answers one question

Federation is authentication only: *may this exact service account authenticate as this managed identity?* It grants nothing. Four separate systems authorize four separate things:

| Control system | What it authorizes |
|---|---|
| Kubernetes RBAC | Access to Kubernetes API objects — nothing in Azure |
| Workload Identity federation | Service account → managed identity authentication — no permissions by itself |
| Azure RBAC | The managed identity's access to the Foundry project |
| Entra delegated permissions | The user-level OBO path between the app registrations |

Azure RBAC grants the managed identity exactly two project-scoped permissions: the **Foundry Agent Consumer** role to invoke the hosted agent, and the user-identity-impersonation permission that lets a *trusted* middleware assert a validated user's object ID. Both are in a short Bicep module scoped to the project, not the account.

### The honest accounting: which secrets remain

"Zero-secret" is true for this leg, not for the whole chain. A complete accounting looks like this:

| Secret | Where it lives | Why it exists | Production direction |
|---|---|---|---|
| Middleware OBO client secret | Kubernetes Secret, injected through a hidden-prompt helper | Entra requires a confidential-client credential to perform OBO #1 | Replace with a certificate or a federated credential on the app registration |
| Gateway OBO client secret | Kubernetes Secret, same helper pattern | Same, for OBO #2 | Same |
| Tool-scoped virtual key | Foundry project connection (protected custom credential) | LiteLLM's application-level credential | Short-lived or rotated keys, per-agent aliases |
| Gateway master key and bundled database credentials | Kubernetes Secrets | Gateway administration and persistence for the POC | Secret manager, managed database with Entra authentication |

Naming the residual secrets is part of the design. A reference that claims "no secrets" and quietly keeps three is worse than one that lists them and says where they are going.

## The delegated OBO chain

OBO lets a middle-tier API exchange the token it received for a new token — for a different audience, for the *same user*. This project chains it twice, so the user's identity survives two re-issuances, and each one narrows scope to the next hop.

```mermaid
sequenceDiagram
    actor User
    participant Client
    participant Entra as Microsoft Entra ID
    participant MW as AKS middleware
    participant Foundry as Foundry hosted agent
    participant LiteLLM as LiteLLM gateway
    participant A365 as Agent 365 / Defender
    participant Tool as DeepWiki MCP

    User->>Client: Sign in and ask a question
    Client->>Entra: Request token for middleware access_as_user
    Entra-->>Client: User token with audience middleware
    Client->>MW: Question plus user token
    Note over MW: Validate signature, issuer, audience, lifetime, tenant, oid, scope
    MW->>Entra: OBO 1 - validated assertion plus middleware credential
    Entra-->>MW: Delegated token with audience LiteLLM
    MW->>Entra: Workload token via federated identity, no secret
    Entra-->>MW: Workload token for Foundry
    MW->>Foundry: Workload token plus delegated token header plus validated user ID
    Foundry->>LiteLLM: Delegated token plus tool-scoped virtual key
    Note over LiteLLM: Virtual key must permit this MCP server
    LiteLLM->>Entra: OBO 2 - delegated token plus gateway credential
    Entra-->>LiteLLM: Delegated Agent Tools token
    LiteLLM->>A365: Evaluate pending tool call
    alt Allow
        A365-->>LiteLLM: Allow
        LiteLLM->>Tool: Invoke tool
        Tool-->>LiteLLM: Result
    else Block
        A365-->>LiteLLM: Block
        LiteLLM-->>Foundry: Tool error, tool not called
    end
    LiteLLM-->>Foundry: MCP result
    Foundry-->>Client: Answer
```

### Three delegated token acquisitions

In the Entra sign-in logs, the user-delegated part of the flow appears as three token acquisitions. The audience changes at each one; the user does not.

| # | Requesting app | Target resource | What changes in the token |
|---|---|---|---|
| 1 | Client app | Middleware API | `aud` = middleware API, `scp` = `access_as_user`, `oid` = user |
| 2 | Middleware app | Gateway API | New `aud` = gateway API, `scp` = `access_as_user`, **same `oid`** |
| 3 | Gateway app | Agent Tools | New `aud` = Agent Tools, `scp` includes `ThreatProtection.Evaluate.All`, **same `oid`** |

A token for one audience cannot be replayed at the next hop: the gateway-audience token would be rejected by Agent Tools, because its `aud` names the gateway. That is the property that makes each exchange a real boundary rather than a pass-through.

### The claims that carry the chain

| Claim | Role |
|---|---|
| `aud` | The one API allowed to accept this token — changes at every OBO boundary |
| `scp` | Delegated operations the client may perform as the user |
| `oid` | Stable user object ID — the thread that persists across every hop |
| `tid` | Issuing tenant, checked against configuration |
| `iss` | Exact Entra v2 issuer, checked against configuration |
| `sub` | User subject, scoped to the target application |
| `appid` / `azp` | The client application that requested the token |
| `exp`, `nbf` | Validity window, with bounded clock skew |

Two things in this table are easy to miss. First, the sign-in log classifies these flows as *not agentic* — expected, because the client, middleware, and gateway are conventional applications performing delegated OBO, and no code reads that field. The evidence that the Microsoft governance boundary participated is the Agent Tools **audience** and the `ThreatProtection.Evaluate.All` scope, not the classification. Second, those log rows record token *acquisitions*, not tool calls: token caches mean row counts and timestamps do not line up one-to-one with requests.

### Configured, consented, issued

For OBO #2 to succeed, three separate things must be true, and each is a different control surface:

1. **Declared** — the gateway app registration lists the delegated Agent Tools permissions it may request.
2. **Consented** — an administrator has granted tenant-wide consent for those delegated scopes.
3. **Issued** — Entra issues the token *per user assertion*, subject to Conditional Access and tenant policy.

Only `ThreatProtection.Evaluate.All` is needed for the runtime allow/block evaluation. The registration may declare more Agent Tools scopes (for discovery and publishing features), but the presence of a scope in a token does not mean the request exercised it.

## Carrying identity through Foundry

Foundry does not perform OBO. The trusted middleware does, then invokes the hosted agent with three values that look similar and are not interchangeable:

| Header | Value | Reaches the agent container? | Purpose |
|---|---|---|---|
| `Authorization` | Foundry **workload** token | **No** | Authenticates the *middleware* to Foundry |
| `x-client-<name>` | **Delegated** token, raw (no `Bearer` prefix) | **Yes** | Carries the request-scoped user token to agent code; the `x-client-` prefix is what enables forwarding |
| `x-ms-user-identity` | Validated user `oid` | Resolved into trusted request context | Scopes Foundry's per-user context; it is **not** a token and grants nothing downstream |

The middleware derives both the user ID and the delegated token itself; it never trusts either value from the caller. Tokens are request-scoped and never written into prompts, logs, exceptions, conversation history, or responses.

## Agent 365 governance at the tool-call boundary

The decision that matters most in an agent system is not "may this agent reach the model?" but "may this agent run *this tool call*, for *this user*, now?" The gateway answers it with a `pre_mcp_call` guardrail that is on by default and fail-closed:

```yaml
guardrails:
  - guardrail_name: agent365-mcp
    litellm_params:
      guardrail: agent_365
      mode: pre_mcp_call
      default_on: true
      tenant_id: os.environ/AGENT365_TENANT_ID
      client_id: os.environ/AGENT365_CLIENT_ID
      client_secret: os.environ/AGENT365_CLIENT_SECRET

mcp_servers:
  deepwiki:
    transport: http
    url: https://mcp.deepwiki.com/mcp
```

On each MCP tool call, the guardrail exchanges the user's delegated token (OBO #2), then sends the *pending* call to the Agent 365 evaluation API. The tool runs only on an explicit **Allow**. This is a synchronous security decision before execution — not telemetry after the fact.

| Sent to the evaluation (per LiteLLM's documentation) | Not sent |
|---|---|
| Tool name and arguments | The user's prompt |
| MCP server name | The user's token to the tool |
| Conversation ID | |
| Tool description and input schema, when known | |
| Virtual-key alias, as the agent ID | |

Three properties make this a control rather than a log line:

- **User-context.** The evaluation is made with the represented user's delegated token, so the decision is taken in that user's context rather than the application's.
- **Fail closed.** If the delegated token is absent, the call is refused at the gateway before any tool runs. If the evaluation cannot be completed, the call is not evaluated and not executed.
- **Orthogonal to application permission.** The virtual key decides whether this *application* may use the MCP server at all; Agent 365 decides whether this *user's* call may proceed. A request needs both.

The key alias is correlation metadata, not proof of agent identity: LiteLLM sends it as the agent ID, but nothing verifies that the alias matches a registered agent. Use aliases that map cleanly to real agents, and document the mapping.

Separately, the Microsoft 365 admin center can connect to the gateway as a **Tools Gateway** (currently a preview). That integration is for discovery and administrative allow/block of registered MCP servers. It does not replace the runtime guardrail, and it uses its own dedicated, purpose-scoped key — never the master key.

## Mapping to the control-plane boundaries

Mapped to the [six boundaries](/content/pattern.md), this reference exercises four, and is explicit about the rest:

| Boundary | How this reference enforces it |
|---|---|
| 1. Agent admission | Middleware validates signature, issuer, audience, lifetime, tenant, user `oid`, and delegated scope before anything else; ingress is exact-path, size-limited, and rate-limited |
| 2. Retrieval | Not exercised — the tool reads a public repository |
| 3. Model invocation | Out of scope — the agent is deterministic and invokes no model |
| 4. Tool proposal | The hosted agent has one fixed tool; the virtual key is limited to one MCP server; Agent 365 evaluates every call |
| 5. Resource-side authorization | Foundry RBAC scopes what the workload may invoke; Entra consent scopes what each app may request; the tool is public, so there is no data-side check to show |
| 6. Side effect | Not exercised — the tool is read-only |

Because the agent is deterministic rather than model-driven, boundary 4 here is about a *fixed capability plus a governance check*, not about validating a model's free-form proposal. The [agentic harness series](/content/agent-harness.md) covers the model-driven case, and [Part 3](/content/agent-harness-03-gateway-and-harness.md) explains why a gateway and an application tier are complementary. This project adds a third layer to that picture: a governance service the gateway consults.

## Security controls

### Middleware: fail closed, in stages

The middleware's core is one linear, staged validation. Nothing downstream runs unless every stage passes:

```text
1. Inspect the untrusted JWT header       -> pin alg = RS256; require a string kid
2. Verify signature + standard claims     -> aud, iss, exp, nbf, iat, sub, tid all required
3. Apply application rules                -> tenant matches, oid present, scope present
4. Only then: OBO, then Foundry invocation
```

Design choices worth calling out:

- **Algorithm pinning.** The accepted algorithm is fixed in code, never read from the token, preventing algorithm-confusion attacks.
- **Key rotation without a flood.** Signing keys come from tenant-specific OpenID discovery and are cached. An unfamiliar `kid` triggers a bounded refresh, so an attacker cannot turn arbitrary key IDs into one outbound call per request.
- **Signature validity is not authorization.** A correctly signed token still fails if the tenant differs, the user ID is absent, or `access_as_user` is missing.
- **Conditional Access is preserved, not bypassed.** If OBO fails with a claims challenge, the middleware returns it to the interactive client so the user can satisfy policy and retry. It never falls back to app-only access.
- **Sanitized failures.** Upstream errors become stable responses — 401, 502, or 504 with a short code — and never reflect upstream bodies, URLs, token details, or raw exception text. A correlation ID is the only upstream detail passed through.
- **No silent success.** An OBO result without a real access token, or a Foundry response that is not a JSON object, is an error — never a success-shaped value.
- **Strict request shape.** The request schema rejects unknown fields and streaming; the caller supplies a question and nothing else.

### Hosted agent: a capability that cannot be redirected

The agent accepts one thing from the caller — a question — and everything else is fixed at deployment:

```python
# Fixed at startup: the caller cannot choose any of these.
_TOOL_NAME = "deepwiki-ask_wiki_question"

headers = {
    "Authorization": f"Bearer {delegated_token}",
    "x-litellm-api-key": settings.litellm_virtual_key,
}
arguments = {"repoName": settings.deepwiki_repository, "question": question}
```

The destination must be HTTPS and end in `/mcp`; the repository is validated as `owner/repository`; the delegated token must arrive in the one trusted header, as a raw token with no whitespace or `Bearer` prefix. Each request opens a short-lived MCP session that is closed before the function returns, so credentials do not outlive the call. Transport errors are translated into messages that never include exception text, because third-party exceptions can embed headers. The agent returns failure for a tool error or empty output rather than inventing an answer.

### Kubernetes: hardened by default

| Control | Setting |
|---|---|
| Pod identity | Dedicated service account; Workload Identity opt-in label; no ambient cluster credentials |
| Process | Non-root user, `seccomp: RuntimeDefault`, read-only root filesystem, all Linux capabilities dropped, privilege escalation disabled |
| Resources | CPU and memory requests and limits on every container |
| Availability | Two replicas plus a PodDisruptionBudget (`minAvailable: 1`) |
| Network | NetworkPolicy for ingress and egress — ingress from the ingress controller only; egress limited to cluster DNS and outbound HTTPS |
| Exposure | Only the runtime MCP route and the agent route are public; the admin UI and model API return 404 at the ingress |
| Ingress | TLS enforced, request body capped, requests-per-second and connections limited |
| Secrets | Real values enter only through hidden-prompt helpers that pipe to `kubectl`; tracked secret manifests contain a placeholder and are non-deployable |

The NetworkPolicy's egress allowance is honest, not perfect: outbound HTTPS is open because the middleware must reach Entra and Foundry. A production version would narrow it with an egress proxy or firewall allow-list.

### Immutable deployments

The middleware image is deployed **by digest**, not by tag. A new build produces a new digest, the Deployment is updated to it, and rollout history becomes an exact record of what ran. The gateway runs a pinned chart and image version, with Helm revisions kept as explicit rollback points, and the same pinning applies to the database image.

### Infrastructure as reusable modules

The Bicep is split into small modules — network, identity, federation, AKS, monitoring, alerts, model-role assignment, Foundry access, container registry — so a customer supplies parameters rather than rewriting the deployment. Azure RBAC assignments are scoped as narrowly as the resource allows, and the cluster itself has local accounts disabled, Entra authentication with Azure RBAC, and Workload Identity enabled.

## Testing

The two Python components have **28 automated tests** (13 for the middleware, 15 for the hosted agent), plus lint, format, and type checks. The tests are organized around trust boundaries, not around functions:

| Boundary | What the tests prove |
|---|---|
| Token validation | A real RSA-signed token yields a trusted identity; invalid signature, issuer, audience, tenant, and missing scope are all rejected; JWKS key rotation refreshes the cache for an unknown `kid` |
| OBO | The incoming assertion and only the configured scope are submitted; a Conditional Access claims challenge is preserved without leaking token details |
| Foundry invocation | A read timeout becomes a sanitized gateway-timeout; the validated identity and OBO token — not caller-supplied values — reach the client |
| Request shape | Missing bearer token, streaming requests, and unknown fields are rejected |
| Hosted-agent configuration | Unsafe configuration is rejected at startup, and the error never includes the virtual key |
| Hosted-agent calls | Only the fixed destination, repository, and tool are used; multiple text parts are joined; credentials never appear in transport errors |
| Protocol boundary | A real in-process server route returns a completed response when a delegated token is present, and fails closed with a generic error when it is absent |

Tests pass injected fakes — protocols for the validator, exchanger, and Foundry client — so security logic is tested without network access. The tests deliberately do not prove what only a live tenant can prove, so live validation covered:

- a real user signing in through the registered client and completing the full chain;
- a request without a delegated token refused with 401 *before* tool execution;
- the guardrail initializing as default-on, fail-closed `pre_mcp_call`, and the guarded call recorded as successful;
- a temporary tool-scoped key listing and calling tools, then being deleted;
- the gateway's pre-existing model route still working after each change (a regression probe);
- public routes: the MCP route returns an authentication challenge, and the admin UI and model API return 404 at the ingress.

## Lessons learned

**1. Identity design is mostly about *where one credential ends*.** Every hop in this chain exists because a single credential could not honestly serve two purposes. The table of four credentials is the most reused artifact in the project, and it was the clearest way to explain the design to others.

**2. A client that can send only one credential forces an architecture change.** The first target was Copilot Studio calling the gateway directly. Its MCP onboarding offers one authentication mode — none, API key, or OAuth — not both. The gateway needs a user token for governance *and* a key for application admission, and the portal cannot send both. Rather than weaken the control, the design moved the call into a trusted backend path (middleware plus hosted agent) that can send both.

**3. LiteLLM-native JWT admission is a licensing decision, not a configuration one.** A correctly signed, correctly scoped user token reached the gateway's JWT path and was rejected because that feature needs an Enterprise license. The virtual key remains the admission credential, and the user token is validated by Entra when it is accepted as an OBO assertion. Worth knowing before you plan around it.

**4. Foundry does not do OBO, and does not forward `Authorization`.** The trusted-backend pattern is deliberate: you do the exchange, authenticate separately, and pass the user token in an `x-client-` header. Reading the platform's header-forwarding rules early would have saved a design iteration.

**5. Compose from small, honest boundaries.** The middleware is six small modules — config, models, auth, OBO, Foundry, app — each with a stated purpose, with the trust boundary commented rather than the syntax. That made reviewing and testing a security chain tractable.

**6. Plan for secret exposure, not just secret storage.** During the build, credentials were inadvertently exposed outside their intended store. Each was treated as compromised and rotated, and the delivery path was redesigned so a real value is only ever entered at a hidden prompt and piped directly to its destination — never written to a file, a command argument, or a shared surface. A good secret process assumes a mistake will happen and makes recovery cheap.

**7. The network is part of the identity story.** The first certificate order failed because a subnet network security group denied inbound web traffic, even though the node-level group allowed it. Later, a hostname was blocked by a tenant web-content filter before it ever reached the cluster, which looked like an authentication failure. Both were fixed in infrastructure and DNS, not in the identity chain — but they presented as identity failures.

**8. Restricted build environments change the deployment shape.** With package access limited to an approved proxy, the hosted agent's remote build could not resolve dependencies. The fix was to bundle pre-built wheels into the deployment, with fallback indexes disabled. A "secure" supply chain also means the build has to work inside it.

**9. Timeouts must agree across layers.** Hosted agents can cold-start. The ingress, the client, and the middleware share one timeout budget, and an exhausted wait becomes a sanitized 504 instead of an unhandled 500.

**10. A security demonstration is only convincing while it is running.** The cluster's nodes were deallocated several times by something outside the application. The response was ordinary operations work — control-plane diagnostics, alerting on node loss, and a guarded, single-attempt recovery workflow — and it deserves a place in any reference architecture.

## What I would do before production

- **Replace the OBO client secrets** with certificates or federated credentials on the app registrations, so the whole chain uses short-lived or secretless credentials.
- **Separate registrations by role.** The gateway registration currently serves as both resource API and confidential client. Production should isolate client, API, and OBO identities so revocation and audit are independent.
- **Move secrets into a secret manager** with the Secrets Store CSI driver and workload identity, and remove them from Helm release state.
- **Exercise the block path** with an approved Defender policy, and add that as an automated regression.
- **Narrow egress** with an allow-list, and move the Foundry-to-gateway hop onto a private path where the platform supports it.
- **Run the gateway highly available** with a managed database, instead of a single replica and bundled PostgreSQL.
- **Make the agent identity first-class**, so the governance service sees a registered agent rather than a key alias.

## Takeaway

> Identity does not survive a multi-hop agent chain by accident. It survives when every hop receives a token intended for it, every workload proves itself with a credential that is not the user's, the secret inventory is stated rather than assumed, and a governance decision sits in front of every tool call — failing closed when it cannot be made.

## Where to go next

- [The AI Control-Plane Pattern](/content/pattern.md) — the six-boundary model this reference maps onto.
- [The Secure Agent Lifecycle, Part 2: Identify](/content/agent-lifecycle-02-identify.md) — the agent identity model behind the "make the agent first-class" step.
- [The Secure Agent Lifecycle, Part 5: Authorize](/content/agent-lifecycle-05-authorize.md) — Agent 365 as a pre-approval check, and resource-side authorization.
- [Inside the Agentic Harness, Part 3: The Harness and the AI Gateway](/content/agent-harness-03-gateway-and-harness.md) — why gateways and application tiers are complementary.
- [OpenTelemetry, Agent 365, and Security Observability](/content/otel-agent-365-security-observability.md) — the telemetry and Defender view of the same chain.