The AI stack changes weekly. A model that was the obvious choice in January is the wrong one by March, and the framework built around it gets replaced before most teams finish documenting it. That pace isn’t optional and it isn’t slowing down. It forces one question onto every team building on top of it, whether they’ve asked it deliberately or not: what’s holding steady enough underneath to keep shipping while everything above it keeps changing.

This is how we answered that question, and what we got wrong along the way.

Before any of this existed

Most teams arrive at a setup like this the same way, not by planning it from day one, but because the alternative eventually stops working. Changes pushed straight into a live cluster hold up fine, until a fix made under pressure never makes it back into the actual config, environments quietly drift apart from being set up by hand at different times, and the honest answer to “what’s actually running right now” starts taking longer to produce than it should.

That’s not a story about carelessness. It’s the default outcome of infrastructure changes going in one command at a time, with nothing recording what should be true. It’s exactly the situation this pipeline exists to prevent.

The decision that changed how we work

We moved the desired state of every environment into git, and put a controller, ArgoCD, in charge of continuously making the cluster match it. Not applying a change once and walking away. Watching, on a loop, and correcting the moment reality drifted from what the repository said should be true.

Two repositories, two jobs. One holds application code. A second, separate repo holds the Helm charts and per-environment values, dev, uat, prod, and that second repo became the actual source of truth. CI’s job narrowed to almost nothing: build an image, push it, update one field in that second repo. The rest happens without anyone touching the cluster.

Inside that second repo, we split things further, on purpose. The chart itself is generic and barely changes once it’s written. The values are what change, constantly, because that’s where the image tag lives. Each environment’s ArgoCD Application points at the same chart and layers a shared values.yaml with an environment-specific one on top. Keeping the rarely-changing part separate from the constantly-changing part meant we stopped touching the chart every time we just needed to ship a new build.

The environments themselves aren’t identical either. All three, dev, uat, prod, run in one cluster, in three separate namespaces. Dev runs a single replica with no autoscaling, because it doesn’t need more. Uat runs two replicas, still no autoscaling. Production runs two replicas as a floor, with autoscaling configured to scale out to six under load. The shape of each environment matches what it’s actually for, not a single template copied three times.

The decision that mattered more than the rest

Build the image once. Promote that exact artifact everywhere.

build.yml builds and tags on every push to develop, a commit SHA, sha-e334ee0, never latest. That tag means exactly that code, permanently. It gets written into the dev values file, ArgoCD picks up the change and syncs it. When that build is ready to move forward, promote.yml takes the same tag and writes it into the next environment’s values file. No rebuild. What ran in dev is bit for bit what runs in uat, and then production.

This single decision removed an entire category of failure we used to accept as normal, the version that behaves differently between environments because it happened to compile twice.

Where we chose to slow down on purpose

Dev and uat auto-sync, because iteration speed matters there. Production doesn’t. ArgoCD marks a production change OutOfSync and stops, waiting for someone to open the diff and click Sync. Promoting a tag and syncing it are two separate actions, done by potentially two different people, and either one can hold the other back on its own.

Credentials follow the same rule as code. Sealed and encrypted, they sit in git, reviewable and versioned, readable only by the specific cluster meant to decrypt them. Nothing in this system gets a quiet exception.

What we got wrong before we got this right

GitHub’s own enforced branch protection isn’t available to us on a private repo without paying for a higher tier. We didn’t route around that by pretending it wasn’t a gap. We built the actual gate ourselves: production stays on manual sync, and access to actually click that Sync button is restricted through an ArgoCD AppProject, scoped so that only approvers hold the sync permission on it at all. Not everyone with cluster access can move production. Only the people explicitly granted that one permission can.

We also learned to distrust a specific error the hard way. A token missing the right permissions doesn’t fail with a clear message. GitHub returns “Not Found,” identical to what you’d see if the repository simply didn’t exist. It’s an easy mistake to make once, reading that as a vanished repository instead of a permission nobody granted. Now it’s the first thing we check.

What this journey actually bought us

The structure we ended up with, one chart per project, one set of environment values per project, one pipeline, wasn’t built for a single system. A new project onto this platform inherits the same promotion path, the same manual gate on production, the same secrets handling, by default. Nobody has to rebuild this discipline from scratch for the next thing we take on.

That’s the part that compounds. We’re not the team trying to keep up with every model release the week it drops. We’re the team that made trying the next one cost almost nothing, because the layer that ships it stopped being something anyone has to think about. In an era where the model changes constantly, that’s the layer that was actually worth getting right.