Ask most engineering teams how long it takes to get a new environment and you will get a number in weeks. Ask why, and you will get a shrug. Nobody designed it that way. It accumulated.
On one estate we worked across, that number was about six weeks. By the end it was under a day. The interesting part is not the number. It is that almost none of the work that closed the gap was the work we expected to be doing.
Six weeks was never a ticketing problem
The obvious diagnosis is that requests sat in a queue, so the obvious fix is a faster queue. That diagnosis is wrong often enough to be worth stating plainly.
When we traced where the six weeks actually went, very little of it was waiting for someone to pick up a ticket. It was spent on questions that had to be answered afresh every time, because no answer had ever been written down in a form a machine could act on:
- Which account does this belong in, and who owns the bill?
- What network segment, and does this tenant share anything with that one?
- Which policies apply, and does a regulator care about this workload?
- Who gets access, and who approves that access?
Every one of those is a design decision. Answering them by hand each time is not a queue problem, it is a missing abstraction. A faster queue would have produced the same answers slightly sooner, and the estate would have kept drifting apart at exactly the same rate.
What we actually built
Three layers, in the order we built them. The order matters more than the tool choices.
First, the resource layer as code. Terraform modules for the things that already existed, written so the same module produced a development environment and a production one from different inputs. Nothing self-service yet. The goal was only that two environments claiming to be the same actually were.
Second, the provisioning layer. Crossplane compositions that turned a request for a service into the full set of resources it needs: namespace, IAM role, secrets, pipeline, dashboards, alert routing. This is the layer that removes the six weeks, because it encodes the answers to those four questions once instead of every time.
Third, the interface. Backstage, so a developer asks for what they want in the vocabulary of their own job rather than in the vocabulary of the infrastructure underneath it.
If you build the portal first, you get a nice front door onto the same six weeks.
We have seen teams start at layer three because it demos well. It does demo well. It also produces a portal that files tickets, which is a worse version of what you already had, with a maintenance burden attached.
Crossplane or Terraform modules
We ran both, so this is a decision record rather than a comparison.
Terraform kept the things that change rarely and are owned by one team: landing zones, account structure, network topology, the base of each cluster. These benefit from a plan you read carefully before applying, and they do not need to be self-service because nobody provisions a new landing zone on a Tuesday afternoon.
Crossplane took everything a product team should be able to ask for without us in the room. The advantage is not that it is better infrastructure as code. It is that the desired state lives in the cluster and gets continuously reconciled, so a developer request and a drift correction are the same operation. That property is what makes self-service safe rather than merely fast.
The cost is real and worth stating: you are now operating a control plane, and its failure modes are yours. Budget for that or do not do it.
Multi-tenancy made the guardrails non-negotiable
This particular estate ran a portfolio of operating companies, each answering to a different regulator. That constraint shaped everything.
You cannot hand a product team a self-service button if the button might place a regulated workload in the wrong segment. So the compositions carry the constraints with them: a request for a service in one tenant produces different network placement, different policy attachment and different data residency than the same request in another, without the person asking needing to know that.
Policy as code did the enforcement, so the constraint held whether or not anyone remembered it. That is the whole point. A guardrail that depends on a human remembering is a suggestion.
The part that took longest was not the platform
The provisioning worked earlier than the platform was genuinely adopted. Those are different milestones and it is worth tracking them separately.
A golden path only works if it is genuinely the easiest route. The moment it is missing something a team needs, that team goes around it, and once one team has gone around it you have two platforms to maintain and the newer one is losing. Most of the effort after the first working composition went into closing those gaps: the unusual database, the legacy service that could not be expressed in the model, the team whose compliance requirement did not fit the template.
Our rule became: when someone routes around the platform, that is a bug report about the platform, not a discipline problem.
What we would do differently
Instrument adoption from day one. Time to first deploy, how many services come through the platform versus around it, how often people fall back to raw access. We ended up building that later, and for a while we were guessing about how well it was working.
Pick the first two consumer teams deliberately. One that fits the model cleanly to prove it, and one that does not, to find the gaps while the abstraction is still cheap to change.
And resist the temptation to model everything. The platform should cover the common path extremely well and have an honest escape hatch for the rest. Trying to express every workload in the abstraction is how a platform team ends up maintaining a worse version of the cloud provider API.
