Why I moved AI access decisions to the gateway instead of the application

Here’s a pattern I keep seeing across the enterprises we work with at Ashnik. A team needs AI in their product, so they call a model provider straight from the application. Works fine. Then another team needs AI too, picks a different model, wires it up the same way. Six months in, you’ve got model calls scattered across half a dozen services, each with its own rules baked into the code, and nobody who can tell you which team is talking to which model, or why.

Nobody planned it this way. It’s just what happens when AI gets bolted onto applications one at a time instead of being routed through something that actually governs it.

So I built the governance layer where it belonged: at the gateway, not inside the application.

NGINX was already the right place for this

NGINX Plus has been sitting between applications and their backends for years. Turns out an AI model is just another backend. The only real change is treating it like one instead of letting applications reach out to model providers directly.

Requests come in through a single port, pass an auth check, then hit the gateway logic, which handles policy, identity, model selection, rate limits, token budgets, and security checks in one pass, before anything reaches Gemma, Mistral, Anthropic, or a ChatGPT-style backend. Add a new model, and it’s a line in a backend registry. The application never finds out.

Here’s what that path actually looks like end to end:

Everything from the auth check to the model call happens in that one pass down through NJS. The side channel running alongside it, audit events flowing into PostgreSQL and back out to the dashboards, is what makes every routing decision reviewable after the fact, not just enforced in the moment.

NJS is doing the actual thinking

NGINX on its own is a very good traffic cop. But it’s not going to read a prompt, check whether the request came from the finance team, or pick between four models based on either. For that, you need NJS, the JavaScript module built for NGINX and loaded into it explicitly, not something that runs by default on a stock install.

NJS is where the enforcement lives, API key policy, roles, model access, rate limits, token budgets, prompt security, PII checks. If NGINX is the road, NJS is the traffic signal deciding who goes where.

The routing isn’t arbitrary

Short, simple prompt? Gemma handles it. Something that needs balanced analysis? Mistral. Coding work goes to a ChatGPT-style backend. Long reasoning or a policy review that needs real depth goes to Anthropic.

Identity plays into it too. A request tagged as coming from a business team defaults to Gemma. Technical teams route to Mistral or ChatGPT depending on policy. Support, security, and operations teams each get their own model class the same way, one identity header, one policy lookup, no application anywhere in that decision. Admin keys get the full menu. Standard keys get whatever’s on their approved list, nothing more.

None of that decision-making happens in application code. And it’s not a black box either, every response comes back with an X-Route-Reason header, so if you’re debugging why a request went where it went, the answer’s right there.

What used to be code is now a config file

The rules driving all of this live in a handful of JSON files the gateway reads on every request: policies.json for key roles and limits, backends.json for the model registry, routes.json for API preferences, model_routing_rules.json for the actual routing logic.

Need to lock a model to one department? Edit a policy file. Onboarding a new provider? Add it to the registry. No pull request against the application, no application rebuild, no waiting for its next release window, just a config change and an NGINX reload on the gateway side.

The same pattern holds outside the org chart too. A client registry carries per-customer policy, which PII rules apply, which models a customer’s traffic is allowed to touch, how their requests get routed, so onboarding a new customer with different requirements is a registry entry, not a fork in the codebase.

And internally, every team hits the same door. One /api/chat endpoint, exposed once, with limits and model permissions enforced centrally instead of trusted to whoever wrote each client. That’s really what an internal AI platform looks like in practice, not a new system every team builds for itself, just one endpoint everyone already knows how to call.

Why any of this is worth the trouble

At low volume, you could get away without this. Nobody notices a handful of stray API calls. At enterprise scale, across departments and customers with real compliance obligations, that stops being true fast.

What you get instead: cheaper bills, because simple requests stop hitting your most expensive model by default. Better answers, because the hard stuff actually goes to a model built to handle it. Access control that holds, because an unauthorized model request gets stopped at the gateway, not discovered after the fact. And a paper trail that actually holds up, who called what, from where, which model handled it, why, how long it took, and how many tokens it burned, all captured without anyone having to remember to log it.

Two teams, same gateway, different rules

Access is usually the part that gets skipped when enterprises rush into AI. Everyone wants the model working first and figures out who’s allowed to use it later, if at all. We’ve found the opposite order works better: figure out who needs what, at what level of trust, before a single prompt goes anywhere. That matters most for regulated teams, BFSI, telecom, healthcare, anyone whose support or compliance function can’t afford a prompt shield and a PII check being optional. The routing and the cost savings are real, but access is the problem that actually breaks things when nobody’s solved it upfront.

A BFSI support team’s prompts get PII-masked automatically, routine summaries go to Gemma, anything that looks like complaint analysis goes to Mistral, and the whole thing gets audited without anyone lifting a finger.

An engineering team gets locked into ChatGPT or Mistral, a token budget tied to their role, and a route-reason header on every call so whoever’s debugging doesn’t have to guess.

Same infrastructure underneath. Completely different rules on top, because that’s how policy should work.

The gateway is what makes this work

Take NGINX out of this picture and you’re back to square one: applications calling models directly, each one making up its own rules as it goes. Nobody’s checking a policy before a request goes out. Nobody’s masking PII. Nobody knows which model handled what, or why.

Put NGINX back in, with NJS running the actual decision-making, and all of that changes without a single application needing to know it happened. The gateway checks the policy. NJS picks the model. The header explains why. Gateway, prompt protection, PII control, rate limiting, model routing- all of it sits in one place instead of eight different tools someone has to stitch together, and hope stays in sync. None of that is new infrastructure showing up to solve a new problem; it’s the same reverse proxy that’s been sitting in front of everything else an enterprise runs, now doing it for AI too.

That’s the part worth remembering next time someone pitches a brand-new platform for this. The control point probably already exists. It just hasn’t been asked to do this job yet. Start with who gets access to what, and the rest of the AI platform tends to fall into place around it, not the other way round.