Collecting Linux Logs using Open Telemetry Collectors

Managing OpenTelemetry collectors using an OpAMP client/server solution seems a bit daunting at first. The big box observability platforms have OpAMP solutions that are confusing, and adopting them feels like you're joining a large expensive platform that is beyond the scope of your needs; implementing an OpAMP server yourself, however, seems unnecessarily complex. So what's an SRE to do? Well, fortunately telflo sits right in that sweet spot between the two and today we'll walk through a simple implementation that shows how easy it can be to manage your collectors in telflo. Let's start by understanding the demo setup.

Before you begin

To follow along you'll need

• A computer with virtualization software. We'll create 2 Ubuntu VMs for this demo.
• A Telflo account. Sign in at telflo.com.

Initial Architecture

In the first part of this tutorial we're going to configure two VMs: one will be an observability platform that will receive and store log data using Loki, and report it using Grafana. The other VM will simulate an application producing real-time logs that you want to send to an observability platform (guess which one). Visually, the setup resembles this.

Each of my VMs is a "Ubuntu 24.04.3 Server (minimum)" instance deployed using Parallels. Of course the virtualization software shouldn't matter, but should any discrepancies arise you know what these instructions are based on. I also named my machines "observability-dev" and "log-generator-dev", and I'll refer to them that way. I'm not trying to tell you how to live your life, but it might make things easier if you did the same. Lastly, you'll notice the "-dev" suffixes. That's because an upcoming blog will demonstrate the awesome power (I'm not joking) of parameterizing the YAML and we'll need to distinguish between the VMs. Ok, onward and upward!

Step 1: Create the Observability Platform

Let's get the observability platform set up!

1. Create the first Ubuntu VM and name it observability-dev. Then SSH into the machine (copy/paste commands work better in my SSH terminal rather than the root terminal).

2. Now log in to telflo, click the "Learning" button in the left menu bar, and type "Setup Loki" in the filter. This should reveal a "Setup Loki + Grafana Observability" option. Click the ">_ Curl" button to copy a Curl command to your clipboard.

   

3. Return to the observability-dev ssh, paste the command and press enter. The "setup-observability-server.sh" script is downloaded. Run it with the command

   sudo chmod +x setup-observability-server.sh && sudo ./setup-observability-server.sh

   That's it! When the script is finished you have a Loki instance awaiting OTLP/HTTP log data and a Grafana instance ready to report it. Of course the server doesn't have a browser so let's set up port forwarding on the host machine so that we can view Grafana on it.

4. Open a new terminal on your host machine and run the following

   ssh -L 3001:localhost:3000 login_name@<IP of observability-dev>

   For example, my login was "jason" and the ip of my machine was 10.211.55.73, and so my command looks like

   ssh -L 3001:localhost:3000 [email protected]

   Now open a browser and navigate to http://localhost:3001/; the Grafana landing page should appear. At the login, input the default username/password as admin/admin and press enter. Enter a new password and continue. You'll be presented with the Grafana interface.

   While this demo runs your observability platform locally, you could have just as well set up a cloud instance of Grafana, DataDog, Sematext...it doesn't matter. In the end you've got an observability platform listening for data and in this instance it's inexpensive, easy, and a great way to learn. Now that you can see the Grafana UI, go ahead and click "Drilldown" -> "Logs" in the left navbar. You should see no logs (as expected) since all we've done is set up a listener. We haven't generated any log data yet...yet...let's do that now.

Step 2: Set Up the Logging Test Fixtures

Next we'll set up a test fixture to generate realistic logging data, and save the logs to a file just like an application would. The logs will be stored at "/var/log/starfeuds/cantina.log" (make note of this path, we'll need it in a few), and will capture the evening exploits of StarFeuds troopers enjoying a night on the town. The simulation will include sensitive data such as credit card numbers (fake) and PII data (all made up), giving us a nice playground to learn about OpenTelemetry processors in the future. Let's go!

1. Create your second Ubuntu VM and name it "log-generator-dev". SSH into the machine.

2. Return to telflo, and type "log generator" in the learning page filter to reveal the "StarFeuds - Log Generator for Server" option. Click the ">_ Curl" button to copy a Curl command to your clipboard.

3. Return to your "log-generator-dev" ssh, paste the command and hit enter. This will download a file named "log-generator.tar.gz". Run the following command to extract it.

   tar -xvf log-generator.tar.gz && cd log-generator

   Logging is now underway. You can see it by running a tail command on that file we noted earlier.

   tail -f /var/log/starfeuds/cantina.log

   Note: This configures a systemd service called starfeuds-generator.service which will restart on reboot. For completeness, here are the commands to stop and remove the service

   # Stop the service immediately (will restart on reboot)
   sudo systemctl stop starfeuds-generator

   # Disable the service to prevent it from starting again on boot
   sudo systemctl disable starfeuds-generator

   Ok, now we have the observability platform listening for data, and the generator generating data. If you were to refresh your Grafana instance there would still be no logs displayed. That's because we still need to connect the two, and here's where the magic of the OpenTelemetry collector and telflo come in. We can use the collector to grab those logs off of disk and send them to Loki. Once Loki has saved them to its database, Grafana can query it, and render the logs. But first we have to create a pipeline that defines the behavior of a collector.

Step 3: Set Up the Collector Configuration

The collector configuration is a YAML file that defines the collector's behavior, which we need to do in order to tell it "grab this data from here and send it over there". Keep in mind that the collector has a LOT of processing capabilities which allow you to curate the data as it flows through a "pipeline" of transformations, but for this demo we'll keep it simple: Grab the data off of the disk and send it to Loki. Let's define the configuration.

1. In telflo, click the "Configurations" option in the left menu bar, then click the "+ New Configuration" button. Name the configuration "starfeuds logging" and press enter. This will create your pipeline and automatically navigate to it in the pipeline editor.

   Delete the Traces and Metrics pipelines by clicking on them, and then pressing your Delete button. All that should remain is the "Logs" pipeline.

   

2. In the left "Components" list, you'll find many of the most commonly used pipeline components. In our case, we want to pull logs off of disk and into the receiver, so in the Receivers section find the "File Log" option and drag an instance into the editor. In the "Include Paths" section of the properties editor paste "/var/log/starfeuds/cantina.log", the path where the test fixture is writing and the source of the Filelog receiver data. Notice the "+ Add Path" button. You could add additional paths, but for now this path is sufficient.

   

   Ok, now you've wired up your receiver, we need to wire up the exporter to push the received data to the observability platform. In future learnings, we'll learn about the processors that you can inject in the pipeline to transform your data, but for now let's just export the raw data. In the "Exporters" section of the components (on the left side of the editor canvas), find the "OTLP/HTTP" exporter. Drag an instance into the "Logs" section, and connect the two components by clicking on the connector in the Filelog component, and dragging onto the OTLP/HTTP exporter.

   

3. Click on the OTLP/HTTP exporter component and set the endpoint as your Loki listening endpoint on your observability-dev server (the ip:port that your other VM is listening for log traffic at). This will be "http://<IP of observability-dev>/otlp". For example, for me this was http://10.211.55.73/otlp

   

   Note: You could also edit the YAML directly. In the tabbed panels on the right side of the screen, click the "YAML" tab to open the YAML editor. All edits sync between the visual components and the YAML tab, so you could edit directly but since we used the visual editor everything is up to date, but just to be sure it should look something like this.

   

Step 4: Publish the Pipeline

And now that we have our pipeline defined, how do we get it onto log-generator-dev so that it starts pushing those logs over to our observability platform? We start by publishing the pipeline to the organization. This creates a snapshot of the pipeline at this point in time and makes it available for fleet deployment (pretty cool, I have to say). And it's super simple, just click the Publish button and confirm your choice (don't worry, we can create new versions to make changes if needed).

That's it, the pipeline is now defined and ready to be deployed. We're almost there.

Step 5: Deploy the Fleet

It's finally time to connect the dots here. At this point you should have 2 VMs (observability-dev and log-generator-dev) and 3 terminals (one to each of the VMs and a third forwarding port 3001 traffic to your observability platform). Now we deploy the collector.

Note, your port forwarding might disconnect occasionally, just re-run the command to re-establish the connection.

1. Return to telflo and click on the "Fleet" option on the left hand side. A fleet is a collection of machines that you want to manage as a unit, or as multiple units. In either case think of it as something you want grouped together such as "all my dev machines". In the Active Fleets page, click on the "+ New Fleet" button and select the "Create a Supervisor Managed Fleet" option. Give your fleet a name such as "My Dev Fleet" and click the "Create Fleet" button. You should see the new fleet defined.

   

2. Click on the fleet name (My Dev Fleet) to navigate to the fleet details. You'll notice a set of tabs:

   • Client Instances: These are the collectors that have "phoned home" to register with telflo for fleet management and pick up their assigned pipeline configurations.
   • Deploy Configurations: These are the different deployment configurations defined for this fleet. You can have many deployment configurations in a single fleet.
   • Client Tokens: Here you can manage the security tokens that have been assigned to this fleet.
   • Deployment Tokens: This allows you to disable download tokens when you're done with them.

   We need to start in the "Deploy Configurations" tab, go ahead and click on it.

   When we created the fleet, a "Default" deployment configuration was created, but not configured; we need to set that up. Click the Edit button to open the edit dialog. Select the "starfeuds logging" configuration and @v1 for the version (only published configurations are available here). This links the configuration version we set up in Step 3 with this deployment so that when a collector phones home we respond with "Hey, this is the configuration you should be running. Now have a nice day!" Click "Save".

   

   Congrats, you've now defined your deployment and all that's left is to use it!

   Click the "Deploy" button to begin a collector deployment. Choose the "All In One" selection that matches your chipset architecture and leave all other defaults. Click "Quick Deploy". This will open the Quick Deploy instructions.

   

   Click the Copy button next to the "Install Command" and return to the log-generator-dev ssh; paste the command and press enter.

   Now quick! Return to the "Client Instances" page to see the new row appear with a status of "Applying."

   

   Ok, I missed it but after Applying (meaning that the assigned starfeuds-logging YAML was picked up by the collector), the Status should show as "In Sync" with a green circle (see image above). That means your collector is running, has registered itself within your account in telflo, and is running your "starfeuds logging" configuration.

   Return to your browser, the one with the empty Grafana, and refresh the logs view. Now you'll see your logs! That's it, you've pushed an application's log files into an observability platform. And while this demo may seem simple, it is a really big step in learning how to manage your fleet of OpenTelemetry deployments.

   Note: As with the log generator, this configures a systemd service. A couple of useful commands are

   # Stop the service immediately (will restart on reboot)
   sudo systemctl stop telflo-supervisor.service

   # Disable the service to prevent it from starting again on boot
   sudo systemctl disable --now telflo-supervisor.service

Step 6: Celebrate

Look at what you've accomplished in almost no time. You've set up an observability server, you've deployed a test fixture generating logs of a raucous StarFeuds trooper's night out on the town, you've defined a pipeline, and you've deployed a small collector fleet that moved the data into your observability platform. And you can now see the results in Grafana. I don't know about you, but I think that deserves a cold frothy Bantha Brew. Hip Hip!

What's great about these kinds of environments is that you can easily test, tinker, and learn without fear of destroying things. So experiment and learn, read the telflo docs to learn how to modify and publish new versions of your Configuration to your fleet, and keep an eye out for upcoming learnings that can propel your OpenTelemetry skills to new levels.

Thanks, see you soon!