Parameterizing Configurations for reusable YAML

In a previous article we learned how to deploy an OpenTelemetry collector on a Linux server to move logs from our log-generator-dev server into our observability-dev server. To acheive this we defined an OpenTelemetry configuration in the telflo editor which contained a hard coded endpoint to our Loki receiver IP.

This all worked (Yay!), but what if we wanted to deploy the same configuration to our prod servers? There are a couple of ways we could achieve this (spoiler: none of these are great)

• Duplicate the Configuration: We could navigate to the Configuration page in telflo, open the "starfeuds logging" configuration, copy the YAML, then create a new Configuration named "starfeuds logging prod", paste the configuration in the YAML editor, and finally create a new deployment setting "starfeuds logging prod" as the configuration to serve. Look, we're all professionals here so I don't need to explain why that's a bad idea. Everyone reading this should have uncomfortably shaken a little.
• Publish a Duplicate Version: Similarly, we could have opened the "starfeuds logging" Configuration, clicked on the Version dropdown, selected "Create new Version" to create a copy of Version 1 (the one we published in the previous article), named it Version 2, published Version 2 to the org, and then created a new deployment specifying Version 2 as the prod configuration to serve.

Both solutions suffer the same issue: two copies of YAML being served by two different deployments to different servers. This might not be extremely offensive at first, because the current YAML is pretty simple at this point but imagine you add Filters, Redactors, Tail Sampling, and other processors to your pipeline...you've duplicated all of that and now imagine you need to adjust and deploy a new version of that pipeline to both dev and prod. It's a mess to say the least, and ripe for errors and inconsistent behaviors that increase your risk of data leaks, broken pipelines, and more.

But wait...what if I told you you could use the same Configuration definition in multiple deployments!

Cue angelic music!

Let's start by spinning up 2 additional VMs that are exact copies of the dev machines in our previous article, however I'll name them with the "prod" suffix. I can do this by following the exact same instructions in steps 1 and 2, with two minor exceptions:

1. We will name our machines "observability-prod" and "log-generator-prod".
2. We will map the Grafana forwarded port to 3002

   ssh -L 3002:localhost:3000 login_name@<IP of observability-prod>

When finished we will have 4 VMs. These are (with my IP addresses as examples)

1. observability-dev: 10.211.55.73
2. log-generator-dev: 10.211.55.74
3. observability-prod: 10.211.55.75
4. log-generator-prod: 10.211.55.76

And if you followed the previous article and this article, up to this point, you should be able to open two browsers and navigate to "http://localhost:3001" (dev Grafana), and "http://localhost:3002" (prod Grafana).

If you recall in the previous article we moved our starfeud trooper's night out logs from log-generator-dev to observability-dev with the following configuration

Notice that the exporters.otlphttp.endpoint value includes "10.211.55.73:3100", which is the Loki instance listening on our observability-dev platform. This won't work for our production logs because it requires a different destination IP address. But let's do something fun. Let's parameterize the YAML.

Step 1: Create the Org Variable

The first thing we need to do is define an Org Variable to represent a generic Loki Receiver endpoint. We can do that as follows.

1. In your telflo account, click on the "Vault" tab in the left menu, and then select the "Org Variables" submenu. In this tab we can define organization variables that can be used as parameters within a Configuration YAML definition.

2. Click the "+ Add Org Variable" button, and create a "Loki Receiver" variable as follows.

   

   Click the "Create Org Variable" button when you're done to finish creating this variable.

The Org Variables defined on this page are the default values. They can be used as parameters in any YAML and the default value will substitute in. That will become important later. But we also need to be able to override those values. That's where "Org Variable Groups" come into play.

Step 2: Define an Org Variable Group (default value override)

Org Variable Groups allow you to define a group of Org Variables, and override their default values. This is where the real power of variables begins to show. Let's demonstrate.

1. Click on the "Org Variable Group" submenu.

2. Click on the "+ Group". Let's create a group of overrides for production, so we'll name our group "Prod".

   

3. Click the "Create Group" button to confirm, and the group will be created.

4. With the group defined, we have an empty bucket that we can add parameters overrides to. Click on the "Prod" name in the UI to open the "Prod" variable details page. Currently it's empty meaning that there are no value overrides. Let's select a variable to override the value on. Click the "+ Select Variable". Select the "Loki Receiver" variable.

   

5. Click the "Add" button to confirm your choice and the Loki Receiver should appear in your list.

6. Now click the edit button to open the "Edit Override" dialog. This is where we can override the default dev Loki Receiver value (10.211.55.73:3100) with our prod value (10.211.55.75:3100).

Fantastic. Now we have 2 different values assigned to 1 variable. But we haven't actually assigned the variable to anything. Let's fix that right now.

Step 3: Parameterize the YAML

Let's return to the configuration visual editor.

1. In the left side menu, click "Configurations" and then click "starfeuds logging". You should navigate to the simple pipeline we defined in the previous article.

2. Click on the OTLP/HTTP exporter. Now, in the "Endpoint" property clear out the old value and press the "@" key and now you'll be prompted with all of the org variables. Select the LOKI_RECEIVER parameter.

   

3. When you choose the "LOKI_RECEIVER" variable, you should see "$orgvar:LOKI_RECEIVER" appear in the Endpoint property, and if you select the YAML tab, you should see it injected in the YAML.

   

4. Click the save button and you'll be prompted to create a new version. That's because @V1 is already published and cannot be modified, but we can create a new version (@V2) and publish it. Confirm the creation of a new version, and then publish it.

Congrats, you've now parameterized your YAML endpoint and published the Configuration to V2. Additionally, we have an Org Variable defined with 2 different values:

• The default value assigned to the variable: http://10.211.55.73:3100/otlp
• The Prod group override: http://10.211.55.75:3100/otlp

Now let's assign this parameterized YAML to two different deployments.

Step 4: Deploy the Parameterized YAML to Dev

Return to the Fleet page and click on the "My Dev Fleet". You should see the previously executing log-generator-dev instance polling as it did before. Notice that it has the effective version of @V1. We've just parameterized our version as @V2, let's go ahead and swap that out.

1. Click on the "Deploy Configurations" tab to find your existing deployment.

2. Click the "Edit" button.

3. Change the selected Version to @V2.

4. Leave the assigned variable group to "None (org defaults only)". By leaving this unselected we're telling telflo "Use the default values on any parameters" and because we set the default value to point to our dev Loki receiver, it should still align and work as expected.

   

   Hop back over to your client instances and you should see that your instance is pending pickup.

   

   The OpAMP message exchange is underway and soon you're collector will be running the new configuration. In just a minute that status should resolve and switch back to green and report as "In Sync".

5. Once the status is "In Sync", refresh your "http://localhost:3001" Dev Grafana instance. You should see logs still flowing in which means that the parameterized YAML worked (or at least didn't break anything). Now the really exciting part. Let's deploy our Prod instance.

Step 5: Deploy the Parameterized YAML to Prod

We've already deployed to Dev and everything is working as expected. Let's create a prod deployment.

1. Return to the "Deploy Configurations" tab.
2. Click the "+ Add deploy configuration" button.
3. Choose the "starfeuds logging" Configuration and select @V2.
4. Choose the "Prod" option in the "Assign variable group". This is a big step; it's telling telflo, "Hey, when a collector phones home override any parameters in the YAML you return with the Prod group values!".

   

5. When done, you should see the new deployment. Go ahead and click "Deploy" and choose the "All In One" option matching your chip architecture, and click "Quick Deploy".
6. Just like we deployed in the previous article, copy the Quick Deploy instructions, locate your log-generator-prod ssh, paste the copied command, and press enter.
7. Return to your "Client Instances" tab and you should see the new deployment register.

And now...for the moment of truth! Refresh your "http://localhost:3002" Grafana instance.

Logs...Logs...Glorious Logs!

Conclusion

That's it! You now have a single YAML configured with parameters that can be swapped into multiple deployments. Think about the power of that. You can test one Configuration Version, get it all cleaned up and ready the way that you want it, and then deploy the exact same YAML to different machines without worrying that you've introduced drift. You can experiment with parameterizing other properties as well: cache sizes, timeouts, and whatever your heart desires.

And just like in the previous article this is a playground to learn so get your hands dirty, break things, fix things, and enjoy learning the power that OpenTelemetry + telflo brings to your organization!

Thanks, see you soon!