Managing a Pipeline on a Server with Telflo

Managing OpenTelemetry collectors using an OpAMP client/server solution seems a bit daunting at first. The big box observability platforms have OpAMP servers that are confusing, and feel like the adoption of a large expensive platform that is beyond the scope of your needs, while implementing an OpAMP server yourself seems unnecessarily complex. So what's an SRE to do? Well, fortunately telflo sits right in that sweet spot between the two and today we'll walk through a simple implementation that shows how easy it can be to manage your collectors in telflo. Let's start by understanding the demo design.

Before you begin

• A computer with virtualization software. We'll create 2 Ubuntu VMs for this demo.
• A Telflo account. Sign in at telflo.com.

Initial Architecture

In the first part of this tutorial we're going to configure two VMs: one VM will be an observability platform that is ready and waiting to receive log data. The other VM will simulate an application producing real-time logs that you want to send to an observability platform (guess which one). Visually, the setup resembles this.

Each of my VMs is a "Ubuntu 24.04.3 Server (minimum)" instance deployed using Parallels. Of course the virtualization software shouldn't matter, but should any discrepancies arise you know what these instructions are based on. I also named my machines "observability-dev" and "log-generator-dev", and I'll refer to them that way. I'm not trying to tell you how to live your life, but it might make things easier if you did the same. Lastly, you'll notice the "-dev" suffixes. That's because an upcoming blog will demonstrate the awesome power (I'm not joking) of parameterizing the YAML and we'll need to distinguish between the VMs. Ok, onward and upward!

Step 1: Create the Observability Platform

Let's get the observability platform set up!

1. Log in to telflo, click the "Learning" button in the left menu bar, and type "Setup Loki" in the filter. This should reveal a "Setup Loki + Grafana Observability" option. Click the ">_ Curl" button to copy a Curl command to your clipboard.

   

2. SSH into your observability-dev machine, paste the command and press enter. The "setup-observability-server.sh" script is downloaded. Run it with the command

   sudo chmod +x setup-observability-server.sh && sudo ./setup-observability-server.sh

   That's it! When the script is finished you have a Loki instance awaiting OTLP/HTTP data

   and a Grafana instance ready to display the data. Of course the server doesn't have a browser so let's set up port forwarding on the host machine so that we can view Grafana on it.

3. Open a new terminal on your host machine and run the following

   ssh -L 3001:localhost:3000 login_name@<IP of observability-dev>

   For example, my login was "jason" and the ip of my machine was 10.211.55.73, and so my command looks like

   ssh -L 3001:localhost:3000 [email protected]

   Now open a browser and navigate to http://localhost:3001/. At the login, input the default username/password as admin/admin and press enter. Enter a new password and continue. You'll be presented with the Grafana interface.

   This demo runs your observability platform locally, but you could just as well set up a cloud instance of Grafana, DataDog, Sematext...it doesn't matter. In the end you've got an observability platform listening for data and in this instance it's inexpensive, easy, and a great way to learn. Now that you can see the Grafana UI, go ahead and click "Drilldown" -> "Logs" in the left navbar. You should see no logs as expected since all we've done is set up a listener. We haven't even generated log data yet...yet...let's do that now.

Step 2: Set Up the Logging Test Fixtures

Next we'll set up a test fixture that will generate logging data. This data will capture the evening exploits of StarFeuds troopers enjoying a night on the town, and will land in the "/var/log/starfeuds/cantina.log" file. Let's go!

1. Return to telflo, and type "log generator" in the learning page filter to reveal the "StarFeuds - Log Generator for Server" option. Click the ">_ Curl" button to copy a Curl command to your clipboard.

2. SSH into your "log-generator-dev", paste the command and hit enter. This will download a file named "log-generator.tar.gz". Run the following command to extract it.

   tar -xvf log-generator.tar.gz && cd log-generator

   Logging is now underway. You can see it by running

   tail -f /var/log/starfeuds/cantina.log

   Note: This starts a systemd service called starfeuds-generator.service which will restart on reboot. For completeness, here are the commands to stop and remove the service

   # Stop the service
   sudo systemctl stop starfeuds-generator

   # Remove the service to prevent it from starting again on boot
   sudo systemctl disable starfeuds-generator

   Ok, now we have the observability platform listening for data, and the generator generating data. We just need to connect the two, and here's where the magic of the OpenTelemetry collector and telflo come in.

Step 3: Set Up the Collector Pipeline

1. In telflo, click the "Configurations" option in the left menu bar, then click the "+ New Configuration" button. Let's name the configuration "starfeuds logging" and press enter.

   You'll navigate to your new pipeline in the pipeline editor. Select the Traces pipeline and click your delete button. Do the same to the Metrics pipeline, and all that should remain is the "Logs".

   

2. In the left side, you'll find many of the most commonly used pipeline components. In our case, we want to pull logs into the receiver, so in the Receivers section find the "File Log" option and drag an instance into the editor. In the "Include Paths" section of the properties editor paste "/var/log/starfeuds/cantina.log", the path where the Filelog receiver will be checking. You could also click the "+ Add Path" button for more paths, but for now this path is sufficient.

   Ok, now you've wired up your receiver, we need to wire up the exporter to push the received data to the observability platform. In future learnings, we'll learn about the processors that you can inject in the pipeline to transform your data, but for now let's just export the raw data. In the "Exporters" section of the components (on the left side of the editor canvas), find the "OTLP/HTTP" exporter. Drag an instance into the "Logs" section, and connect the two components by clicking on the connector in the Filelog component, and dragging onto the OTLP/HTTP exporter.

3. Click on the OTLP/HTTP exporter component and set the endpoint as your Loki listening endpoint on your observability-dev server. This will be "http://<IP of observability-dev>/otlp". For example, for me this was "http://10.211.55.73/otlp"

   

   Note: You could also edit the YAML directly. In the tabbed panels on the right side of the screen, click the "YAML" tab to open the YAML editor. All edits sync between the visual components and the YAML tab, so you could edit directly but since we used the visual editor everything is up to date, but just to be sure it should look something like this.

   

Step 4: Publish the Pipeline

And now that we have our pipeline defined, how do we get it onto log-generator-dev so that it starts pushing those logs over to our observability platform? We start by publishing the pipeline to the organization. This creates a snapshot of the pipeline at this point in time and makes it available for fleet deployment (pretty cool, huh). And it's super simple, just click the Publish button and confirm your choice (don't worry, we can create new versions to make changes if needed).

That's it, the pipeline is now defined and ready to be deployed. We're almost there.

Step 5: Deploy the Fleet

It's finally time to connect the dots here. At this point you should have 2 VMs (observability-dev and log-generator-dev) and 3 terminals (one to each of the VMs and a third forwarding port 3001 traffic to your observability platform). Note, your port forwarding might disconnect occasionally, just re-run the command to re-establish the connection.

1. Return to telflo and click on the "Fleet" option on the left hand side. A fleet is a collection of machines that you want to manage as a unit, or as multiple units. In either case think of it as something you want grouped together such as "all my dev machines". In the Active Fleets page, click on the "+ New Fleet" button and select the "Create a Supervisor Managed Fleet" option. Give your fleet a name such as "My Dev Fleet" and click the "Create Fleet" button. You should see the new fleet defined.

   

2. Click on the new fleet name to navigate to the fleet details. You'll notice the set of tabs

   • Client Instances: These are the collectors that have "phoned home" to register and pick up their assigned pipeline configurations.
   • Deploy Configurations: These are the different deployment configurations defined for this fleet.
   • Client Tokens: These are the security tokens that have been assigned to this fleet.
   • Deployment Tokens: These allow you to disable a download link when you're done using it.

   We need to start in the "Deploy Configurations" tab. When we created the fleet, a default deployment configuration was created, but not configured. Click the Edit button to open the "Edit Default" dialog. Select the "starfeuds logging" configuration and @v1 for the version (only published configurations are available here). Leave the Assigned variable group as "None" and click "Save".

   

   Congrats, you've now defined your deployment. So now use it! Deploy an instance by clicking on the "Deploy" button. Choose one of the "All In One" selections, selecting according to your chipset architecture. Leave all other defaults, and then click "Quick Deploy". This will open the Quick Deploy instructions.

   

   Click the Copy button and return to the log-generator-dev ssh, paste the command, and press enter.

   Now quick! Return to the "Client Instances" page to see the new row appear with a status of "Applying."

   

   Ok, I missed it but after applying (meaning that the assigned starfeuds-logging YAML was picked up by the collector), the Status should show as "In Sync" with a green circle. That means your collector is running, and has registered itself within your account in telflo.

   Return to your browser, the one with the empty Grafana, and refresh the logs view.

Step 6: Celebrate

Look at what you've done in almost no time. You've set up an observability server, you've deployed a test fixture generating logs of a raucous night out on the town, you've defined a pipeline to tell a collector how to move the logs into Loki, and you've deployed a small collector fleet that moved the data. And you can see the results in Grafana. I mean, come on! Hip hip!

What's great about these kinds of environments is that you can easily test, tinker, and learn without fear of destroying things. So experiment and learn, and keep an eye out for upcoming learnings that can propel your OpenTelemetry skills to new levels.