Building a Real-World GitOps Setup, The Nightmare Way vs Using ArgoCD & Kro

If you’ve ever felt like managing Kubernetes manifests is a part-time job, or if you’ve groaned at the thought of submitting three pull requests just to tweak a single port, this series is for you. I’m going to walk you through how I started with Argo CD, hit scaling pains, and ended up with a setup using Kro and Kargo that’s clean, scalable, and—dare I say it—kinda fun to work with.

Why GitOps and Argo CD?

Let’s set the scene. My team builds and deploys a bunch of services, and like most folks, we use GitLab to manage our code and CI/CD pipelines. Everything lives in Git—our source code, our configs, our dreams of a bug-free production environment (ha!). So when we decided to embrace GitOps, it made total sense to pick a tool that treats Git as the single source of truth.

Enter Argo CD. If you haven’t used it, Argo CD is a GitOps tool that syncs your Kubernetes cluster with what’s in your Git repo. You define your desired state in Git (usually a pile of YAML files), and Argo CD makes sure your cluster matches that state. It’s stable, it’s got a nice UI for visualizing your apps, and it’s pretty straightforward to set up—at least when you’re just getting started.

We adopted Argo CD early on, and at first, it felt like a dream. We’d commit our Kubernetes manifests to Git, Argo CD would pick them up, and boom—our services were running in the cluster. No more SSH-ing into servers or running kubectl apply like cavemen. Life was good.

But then, as our project grew, things started to get… complicated.

The Problem: YAML Overload and Maintenance Nightmares

Picture this: our app starts as a simple setup with a frontend, a backend, and maybe a database. Each service has a few Kubernetes resources—a Deployment to run the pods, a Service to expose it, and a ConfigMap for environment variables. For each service, that’s three YAML files sitting in our Git repo. No big deal, right?

Fast forward a few months. We’ve added more services—maybe a payment processor, a notification system, and some internal tools. Suddenly, our repo looks like a YAML explosion. Every service has its own folder, and each folder has at least three files: deployment.yaml, service.yaml, and configmap.yaml. Want to update the image tag for the frontend? Cool, go edit frontend/deployment.yaml. Need to tweak the backend’s log level? Head to backend/configmap.yaml. Oh, and don’t forget to update the port in service.yaml if you’re exposing a new endpoint.

It didn’t take long for me to realize this was unsustainable. Updating a single service often meant touching multiple files and submitting multiple pull requests (PRs). For example, changing a port might require:

1. Updating the port in service.yaml.
2. Tweaking the container port in deployment.yaml.
3. Maybe adjusting an environment variable in configmap.yaml to match.

Three files, three PRs, and a whole lot of mental overhead just to make one logical change. It felt like I was spending more time wrangling YAML than actually building features.

And it wasn’t just me. My teammates were starting to grumble too. One day, someone jokingly said, “I didn’t sign up to be a YAML engineer.” That hit hard because it was true. We were developers, not manifest babysitters.

I started asking myself: Isn’t there a better way? Couldn’t we just define what we want a service to do in one place and have everything else flow from there? I didn’t want to maintain three YAML files just to say, “Hey, this service is now running version 1.2.3.” I wanted a setup where I could:

• Define a service in a single file (let’s call it instance.yaml).
• Specify things like the image tag, port, and environment variables in one spot.
• Let some tool generate the actual Kubernetes manifests (Deployment, Service, ConfigMap) automatically.
• Keep using Argo CD to sync those manifests to the cluster without cluttering our repo.

That was my dream. But at the time, it felt like chasing a unicorn.

The Breaking Point: When Scaling Hurt

The tipping point came when we onboarded a new service that had a more complex setup. It wasn’t just a Deployment and a Service—it needed multiple ConfigMaps, a couple of Secrets, and some custom annotations. Writing all those manifests by hand was painful, and keeping them in sync was even worse. I’d update the image tag in the Deployment, forget to update the ConfigMap, and then spend an hour debugging why the service was misbehaving.

On top of that, our repo was starting to look like a maze. With dozens of services, navigating the folder structure felt like a treasure hunt. Want to find the backend’s config? Good luck digging through backend/configmap.yaml buried under layers of folders. And don’t even get me started on reviewing PRs—every change meant sifting through walls of YAML to make sure nothing broke.

I knew we needed a better system. I didn’t want to abandon GitOps or Argo CD—they were still awesome for keeping our cluster in sync. But I needed a way to simplify how we defined and managed our services. That’s when I started hunting for tools that could help.

Introducing Kro: A Light at the End of the YAML Problem

After some late-night Googling and a few rabbit holes on Reddit and GitHub, I stumbled across Kro. If you haven’t heard of it, Kro is a tool that lets you define your services at a high level and then generates the low-level Kubernetes manifests for you. Think of it like a translator: you write a simple, human-friendly description of your service, and Kro turns it into the YAML that Kubernetes understands.

The first time I saw an example of Kro in action, I got excited. Here’s what our setup looked like before Kro:

frontend/ ├── deployment.yaml ├── service.yaml └── configmap.yaml backend/ ├── deployment.yaml ├── service.yaml └── configmap.yaml

Each folder had multiple files, and every change meant editing at least one of them (usually more). Now, here’s what Kro promised:

frontend/ └── instance.yaml backend/ └── instance.yaml

One file per service. That’s it. The instance.yaml file would define everything about the service—image tag, ports, environment variables, you name it. Kro would take that file and generate the Deployment, Service, and ConfigMap automatically. Then Argo CD could sync those generated manifests to the cluster, just like before.

Here’s an example of what an instance.yaml might look like:

spec: values: deployment: tag: 0320.1 port: 3000 image: frontend config: LOG_LEVEL: debug API_URL: https://api.example.com

With this one file, I could define the entire setup for the frontend service. Want to bump the image tag? Just change tag: 0320.1 to tag: 0320.2. Need to tweak the log level? Update LOG_LEVEL: debug to LOG_LEVEL: info. One file, one PR, done.

It felt clean. It felt right. For the first time in months, I wasn’t dreading config changes.

Why Not Helm or Kustomize?

Now, you might be thinking, “Wait a minute—why not just use Helm or Kustomize? Aren’t those the go-to tools for managing Kubernetes manifests?” That’s a fair question, and I’ll be honest: I didn’t pick Kro because I was clueless about Helm or Kustomize. In fact, I’ve used Helm plenty of times. If I need to deploy something like a PostgreSQL database or an Nginx ingress controller, I’ll run helm install with a custom values.yaml and call it a day. It’s great for off-the-shelf charts.

But here’s the thing: I’ve never written my own Helm charts from scratch. I’ve seen teammates do it, and let me tell you, it’s not exactly a walk in the park. Helm templates are powerful, but they can get messy. You’ve got values.yaml, _helpers.tpl, and a bunch of template files that all work together to render your manifests. If you need to change something simple, like an environment variable, you might end up digging through three different files to figure out what’s going on. I’ve watched colleagues curse at nested values and cryptic helper functions, and I wasn’t eager to join that club.

As for Kustomize, I gave it a shot early on. It’s built into kubectl, which is nice, and it’s great for applying small patches to existing manifests. But as our setup grew, Kustomize started to feel like a house of cards. We ended up with a three-layer stack of overlays for staging, production, and dev environments. Debugging meant tracing through a pile of patches to figure out which one was overriding what. It was like trying to solve a puzzle while someone kept rearranging the pieces.

Here’s what bugged me about both tools:

• Helm: It’s awesome for pre-built charts, but writing custom charts felt like overkill for our needs. Plus, the complexity of templates made it hard to reason about what was actually getting deployed.
• Kustomize: It’s lightweight, but managing overlays and patches got chaotic fast. I didn’t want to spend my days playing YAML detective.

What I really wanted was something simpler. I wanted a single file that described the intent of a service—what image it uses, what ports it exposes, what configs it needs. I didn’t want to care about the nitty-gritty details of how that translated into a Deployment or a ConfigMap. I just wanted to say, “Here’s my service, make it happen,” and let a tool handle the rest.

That’s where Kro came in. It gave me:

• A single instance.yaml file as the source of truth for each service.
• A clear, declarative structure that was easy to read and reason about.
• Generated manifests that were predictable and scalable.
• Seamless integration with Argo CD, so I didn’t have to ditch my existing GitOps workflow.

It wasn’t perfect (no tool is), but it felt like a massive step forward.

The Bigger Picture: How It All Fits Together

At this point, I was sold on Kro, but I needed to figure out how it would fit into our broader GitOps workflow. Here’s the high-level picture of what I was aiming for:

1. Service Definitions: Each service has an instance.yaml file that describes its setup—image tag, ports, configs, etc.
2. Kro: Takes those instance.yaml files and generates the Kubernetes manifests (Deployment, Service, ConfigMap).
3. Argo CD: Syncs the generated manifests to the cluster, keeping everything in line with Git.
4. Kargo (we’ll get to this later): Handles the orchestration of updates, like triggering rollouts when image tags change.

The beauty of this setup is that it’s driven by those instance.yaml files. Change one file, and everything else flows from there. Kro generates the manifests, Argo CD applies them, and Kargo makes sure updates happen smoothly. It’s like a well-oiled machine—at least, that was the goal.

Here’s a quick mental image of the flow:

• You update frontend/instance.yaml to bump the image tag.
• Kro sees the change and regenerates frontend/deployment.yaml, frontend/service.yaml, and frontend/configmap.yaml.
• Argo CD notices the updated manifests in Git and syncs them to the cluster.
• Kargo (if you’re using it) orchestrates the rollout, making sure the new version goes live without downtime.

It’s clean, it’s declarative, and it scales without turning your repo into a YAML landfill.

What Kro Actually Does: A Deeper Look

Let’s zoom in on Kro for a second because it’s the heart of this setup. At its core, Kro is about simplifying how you define Kubernetes resources. Instead of writing low-level manifests by hand, you create a higher-level description of your service—what Kro calls a ResourceGraphDefinition (RGD). The RGD is like a blueprint that tells Kro how to generate your Deployment, Service, ConfigMap, and anything else you need.

Here’s a super simplified example of how it works. Let’s say I want to define a frontend service. My instance.yaml might look like this:

spec: values: deployment: tag: 0320.1 port: 3000 image: frontend config: LOG_LEVEL: debug API_URL: https://api.example.com

Behind the scenes, I’d also have an RGD (usually defined separately) that tells Kro how to turn this instance.yaml into actual manifests. The RGD might say something like:

• For the Deployment: Create a pod with the frontend image, use the tag from instance.yaml, and expose the port.
• For the Service: Expose the port internally and route traffic to the Deployment.
• For the ConfigMap: Take the config values (like LOG_LEVEL and API_URL) and create key-value pairs.

When I run Kro, it reads the instance.yaml and the RGD, then spits out the manifests:

# Generated deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: frontend spec: replicas: 1 template: spec: containers: - name: frontend image: frontend:0320.1 ports: - containerPort: 3000 ...

# Generated service.yaml apiVersion: v1 kind: Service metadata: name: frontend spec: ports: - port: 3000 targetPort: 3000 ...

# Generated configmap.yaml apiVersion: v1 kind: ConfigMap metadata: name: frontend data: LOG_LEVEL: debug API_URL: https://api.example.com

These manifests go into a folder that Argo CD watches, and from there, it’s business as usual—Argo CD syncs them to the cluster, and my service is up and running.

The magic is that I only had to edit one file (instance.yaml). Kro handled the rest, and I didn’t have to touch a single line of low-level YAML. That’s the kind of simplicity I was craving.

The Reality Check: It Wasn’t All Smooth Sailing

Now, I don’t want to paint this as a fairy tale where Kro solved all my problems overnight. There was a learning curve. Writing my first RGD was… let’s just say it wasn’t love at first sight. I had to figure out how to structure it, what fields were required, and how to avoid common pitfalls (like forgetting to link the Service to the Deployment properly).

I also ran into a few gotchas:

• Validation: Early on, I made some typos in instance.yaml (like forgetting a field), and Kro’s error messages weren’t always crystal clear. I spent a frustrating hour debugging a “missing field” error that turned out to be a misplaced colon.
• Customization: While Kro’s defaults were great for simple services, our more complex services needed custom annotations and labels. Figuring out how to add those to the RGD took some trial and error.
• Team Buy-In: Not everyone on my team was thrilled about learning a new tool. A couple of folks were comfortable with raw YAML and didn’t see the point of Kro at first. It took some convincing (and a few demos) to get everyone on board.

But even with those hiccups, the benefits were undeniable. Once I got the hang of Kro, our workflow felt so much smoother. PRs were smaller, reviews were faster, and I wasn’t spending my days copy-pasting YAML like a robot.

What’s Next in the Series?

This post was all about the why—why I started down this path, why YAML was driving me nuts, and why Kro felt like the right fit. But we’re just getting started. In the next post, I’m going to dive into the nitty-gritty of writing my first ResourceGraphDefinition (RGD). I’ll walk you through:

• The structure of an RGD and how it ties into instance.yaml.
• The mistakes I made (so you can avoid them).
• How I got Kro to generate production-ready manifests that worked with Argo CD.
• Some real-world examples from our setup.

If you’ve ever patched the same deployment.yaml three times just to bump an image tag, you’ll know the pain I’m talking about. This series is for anyone who’s tired of fighting their tools and wants a GitOps workflow that actually works.

So stick around—we’re going to build something awesome together.