What I Look for When Reviewing an AI Platform Architecture

An AI platform review is not a code review. It is not a vendor evaluation either. It is two days of asking where things live, who can reach them, and what happens when they fall over.

Six areas decide whether a platform survives contact with production: tenancy, egress control, model routing, observability, kill-switches, and IaC coverage. Below, for each one, the questions I ask, what good looks like, and what I usually find instead. That last part is where the truth lives.

1. Tenancy

How is one tenant isolated from the next: namespace, account, VPC, or a row filter? Who shares GPU capacity, and what happens when one tenant's batch job lands in the middle of another's latency-sensitive window? Can a tenant's plugin read a neighbor's data through a shared cache or a shared vector index?

Good looks like this: isolation boundaries are infrastructure, not application logic. Namespaces or accounts per tenant tier, network policy default-deny between them, per-tenant encryption keys for anything regulated, and an explicit answer for noisy neighbors on shared accelerators.

What I usually find: a shared cluster with per-row tenant IDs and a sincere belief that the ORM enforces them. The vector database is shared too, namespaced by a metadata field. One malformed filter and you have a cross-tenant data leak, served quietly, with no error message.

2. Egress control

Where can a workload send data, physically? Is model traffic forced through a single gateway, or can any pod holding the API key call the provider directly? What does DNS resolve from the AI subnets? And if an agent gets prompt-injected tomorrow, which channels can it use to exfiltrate?

Good: the AI subnets have no public route. Model APIs resolve to private endpoints, DNS is allowlisted, and the only permitted destination is a gateway that authenticates workloads, redacts, logs, and calls providers over pinned paths. The control is a property of the network, not a promise in a policy document.

What I usually find: an acceptable-use policy and a NAT gateway. The routing table disagrees with the policy, and the routing table wins. It always wins.

3. Model routing

Where does the decision of which model serves a request live? Is there a routing table, and does it encode data sensitivity as well as task difficulty? What is the fallback when a provider degrades? Who changed the routing last, and how would you even know?

Good: routing lives in the gateway, not in application code. The table maps task types to model tiers with an eval gate behind each tier change. Data classification is a routing input. Which model may see which data is encoded in the router itself, not on a wiki page someone wrote in 2024.

What I usually find: model names hardcoded in application code, three services each calling the provider SDK directly, and a fourth service nobody remembers deploying. That fourth one is usually the interesting one.

4. Observability

Can you state cost per successful request by feature? Do traces connect the user action to the model call to the provider response? Where are prompts and completions logged, and who can read them? What evals run continuously, and what actually happens when they regress?

Good: every model call is logged at the gateway with workload identity, token counts, cost, and latency. Traces carry the feature and tenant. Eval suites run on a schedule and gate the routing table. Prompt logs get their own access policy, because they are the most sensitive data in the system. Full stop.

What I usually find: request logs without token counts, so cost attribution is archaeology. No evals in production at all. Prompts sitting in a general-purpose log bucket readable by half the engineering org.

5. Kill-switches

Can you disable one model, one feature, or one tenant in minutes, without a deploy? When did someone last actually do it? If a provider has an incident at 2am, what routes around it, and has that path ever been exercised under any load at all?

Good: flags at the gateway for model, feature, and tenant, each tested within the last quarter. Provider failover is configured and has carried traffic at least once, even in a drill. A kill-switch that has never been pulled is not a control. It is a hypothesis.

What I usually find: disabling a model means a code change and a pipeline run. The failover path exists in a diagram. Diagrams do not route traffic.

6. IaC coverage

What percentage of this platform could you rebuild in a fresh account from the repo? Which resources were created in a console, by hand, at some point nobody can date? Where is drift detected? Are the IAM policies and endpoint policies in version control, or only the compute?

Good: everything except a documented break-glass list is in Terraform or Pulumi, drift detection runs nightly, and the answer to "rebuild it" is measured in hours. Policies, the things a reviewer actually asks about, are code.

What I usually find: compute is in Terraform; IAM, endpoints, DNS, and budgets are console archaeology. The platform can be redeployed but cannot be re-secured.

How I score it

Each area gets a rung, not a grade: what is true today, with evidence. Not what the design doc says. Not what is planned for Q4. What is true.

The platform's effective maturity is capped by its weakest critical area. For regulated industries the critical ones are egress control and observability, because those are what an examiner or a breach investigation will ask about first, and they will not accept a roadmap as an answer. A platform with excellent MLOps and open egress is not mature. It is fast and exposed.

Sequencing the fixes matters more than the scores. Kill-switches and egress first: both are days of work, and together they bound the blast radius of everything else. Routing and observability next, because they turn cost and quality from mysteries into measurements. Tenancy work is usually the most expensive item on the list and should be planned, not panicked into. IaC coverage is continuous. Every fix lands as code or it did not land.

One more thing, and it is the most reliable signal in the whole review. The tell of a healthy platform is not any single answer. It is how fast the team can produce evidence. If every question takes a day of archaeology, the platform runs on memory, and memory does not survive the next reorg.