#fluentbit — Public Fediverse posts

Have been experimenting with #signoz in the #homelab as a possible replacement for #grafana #prometheus #loki #fluentbit. Early results across the #kubernetes cluster are positive - metrics, logs, dashboards and alerts tested. Before I start a more thorough test ahead of migration, does anyone have any experiences to share? Really like the idea of queries and logs with SQL, are there any other tools I should be considering?

Have been experimenting with #signoz in the #homelab as a possible replacement for #grafana #prometheus #loki #fluentbit. Early results across the #kubernetes cluster are positive - metrics, logs, dashboards and alerts tested. Before I start a more thorough test ahead of migration, does anyone have any experiences to share? Really like the idea of queries and logs with SQL, are there any other tools I should be considering?

Have been experimenting with #signoz in the #homelab as a possible replacement for #grafana #prometheus #loki #fluentbit. Early results across the #kubernetes cluster are positive - metrics, logs, dashboards and alerts tested. Before I start a more thorough test ahead of migration, does anyone have any experiences to share? Really like the idea of queries and logs with SQL, are there any other tools I should be considering?

Have been experimenting with #signoz in the #homelab as a possible replacement for #grafana #prometheus #loki #fluentbit. Early results across the #kubernetes cluster are positive - metrics, logs, dashboards and alerts tested. Before I start a more thorough test ahead of migration, does anyone have any experiences to share? Really like the idea of queries and logs with SQL, are there any other tools I should be considering?

Have been experimenting with #signoz in the #homelab as a possible replacement for #grafana #prometheus #loki #fluentbit. Early results across the #kubernetes cluster are positive - metrics, logs, dashboards and alerts tested. Before I start a more thorough test ahead of migration, does anyone have any experiences to share? Really like the idea of queries and logs with SQL, are there any other tools I should be considering?

Fluent Bit configuration quick reference and editor

Fluent Bit’s documentation is spread across many static web pages; in some cases, attributes allowed for a plugin are documented across several pages. There is absolutely nothing wrong with this. Having written a book on Fluent Bit, I can say it makes sense, and if the standard docs went into further detail, it would increase the spread of content.

The problem is, once you’ve got a grip on Fluent Bit, you want a quick reference just to check the attribute names or confirm that an attribute does what you expect.

The good news is that’s what we have created …

• Quick references
  • 3.2.10 HTML, Markdown
  • 4.2.4 HTML , Markdown
  • 5.0.4 HTML , Markdown

Each quick reference page has a section at the top that provides a comma-separated list of all the plugins for the different operations, inputs, outputs, etc., with anchor links to that section. Each plugin has a table that lists all configuration attributes, including those inherited from the Fluent Bit kernel and those introduced by extending another plugin.

Why and how …

The “why” may prove even more appealing. As part of our OpAMP project, we wanted to make it easy to edit and validate Fluent Bit and Fluentd configurations before deployment using the OpAMP tooling. That tool became the config-service part of the OpAMP repo, and can be independently deployed as well as function as part of the main OpAMP service. As a result, the UI offers the same authentication and authorisation options, ranging from running without authentication to using OAuth.

The key to both the UI and documentation is the use of JSON Schemas, as they contain all the information needed to create documentation just as easily as they power the UI. We have therefore generated a markdown page using a Python script. This means the docs are easy to check (compared to an HTML file) and can be rendered within GitHub. We could extend the script to generate HTML. But rather than trying to remember to keep both document types in sync (and double-check formatting), we found a JavaScript tool (marked.js) that performs an on-the-fly transformation that, as long as we stay within the core Markdown syntax, won’t cause any issues. We’ve then enriched that output a bit by applying stylesheets.

To come…

At the moment, we have only covered a subset of Fluent Bit versions, so we will, in due course, expand to cover more versions. Today, this is just the last version from each major release. We haven’t done every version to date as it does take a lot of effort to go through the documentation to generate and check the schema, and we’re still refining things as we enhance the UI.

We are also looking to do the same for Fluentd, though this is much trickier, as the portfolio of plugins that make up the core is smaller than those incorporated into the core of Fluent Bit, and the leveraging of Ruby’s dynamic behaviour makes it very easy for people to offer plugins separately. Then, of course, there is the task of collating all that information.

Resources

• Quick references
  • 3.2.10 HTML, Markdown
  • 4.2.4 HTML, Markdown
  • 5.0.4 HTML, Markdown
• OpAMP landing page
• Fluent Bit
• Fluentd

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

OpAMP with Fluent Bit – Observability and ChatOps

With KubeCon Europe happening this week, it felt like a good moment to break cover on this pet project.

If you are working with Fluent Bit at any scale, one question keeps coming up: how do we consistently control and observe all those edge agents, especially outside a Kubernetes-only world?

This is exactly the problem the OpAMP specification is trying to solve. At its core, OpAMP defines a standard contract between a central server and distributed agents/supervisors, so status, health, commands, and config-related interactions follow one protocol instead of ad-hoc integration per tool.

That is where this project sits. We’re implementing the OpAMP specification to support Fluent Bit (and later Fluentd).

In this implementation, we have:

• a provider (the OpAMP server), and
• a consumer acting as a supervisor to manage Fluent Bit deployments.

Right now, we are focused on Fluent Bit first. That is deliberate: it keeps scope practical while we validate the framework. The same framework is being shaped so it can evolve to support Fluentd as well.

The repository for the implementation can be found at https://github.com/mp3monster/fluent-opamp

Quick summary

The provider/server is the control plane endpoint. It tracks clients, accepts status, queues commands, and returns instructions using OpAMP payloads over HTTP or WebSocket.

The consumer/supervisor handles the local execution and reporting. It launches Fluent Bit, polls local health/status endpoints, sends heartbeat and metadata to the provider, and handles inbound commands (including custom ones). The server and supervisor can be deployed independently, which is important for real-world rollout patterns.

Because they follow the OpAMP protocol model, clients and servers can be interchanged with other OpAMP-compliant implementations (although we’ve not yet tested this aspect of the development).

Together, they give us a manageable, spec-aligned path to coordinating distributed Fluent Bit nodes without hard-coding one-off control logic into every environment.

Deployment options and scripts

There are a few practical ways to get started quickly:

• Deploy just the server/provider using scripts/run_opamp_server.sh (or scripts/run_opamp_server.cmd on Windows).
• Deploy just the client/supervisor using scripts/run_supervisor.sh (or scripts/run_supervisor.cmd on Windows).
• Run both components either together in a single environment or independently across different hosts.

The scripts will set up a virtual environment and retrieve the necessary dependencies.

If you want an initial MCP client setup as part of your workflow, there are helper scripts for that too:

• mcp/configure-codex-fastmcp.sh and mcp/configure-codex-fastmcp.ps1
• mcp/configure-claude-desktop-fastmcp.sh and mcp/configure-claude-desktop-fastmcp.ps1

Server screenshots

Here is a first server view we can include in the post:

The UI is still evolving, but this gives a concrete picture of the provider side control plane we are discussing.

What the OpAMP server (provider) does

The provider is responsible for the shared view of fleet state and intent.

Today it provides:

• OpAMP transport endpoints (/v1/opamp) over HTTP and WebSocket.
• API and UI endpoints to inspect clients and queue actions.
• In-memory command queueing per client.
• Emission of standard command payloads (for example, restart).
• Emission of custom message payloads for custom capabilities.
• Discovery and publication of custom capabilities supported by the server side command framework.

Operationally, this means we can queue intent once at the server and let the next client poll/connection cycle deliver that action in protocol-native form.

What the supervisor (consumer) does for Fluent Bit

The supervisor is the practical glue between OpAMP and Fluent Bit:

• Starts Fluent Bit as a local child process.
• Parses Fluent Bit config details needed for status polling.
• Polls Fluent Bit local endpoints on a heartbeat loop.
• Builds and sends AgentToServer messages (identity, capabilities, health/status context).
• Receives ServerToAgent responses and dispatches commands.
• Handles custom capabilities and custom messages through a handler registry.

So for Fluent Bit specifically, the supervisor gives us a way to participate in OpAMP now, even before native in-agent OpAMP support is universal.

And to be explicit: this is the current target. Fluentd support is a planned evolution of this same model, not a separate rewrite.

Where ChatOps fits

ChatOps is where this gets interesting for day-2 operations.

In this implementation, ChatOps commands are carried as OpAMP custom messages (custom capability org.mp3monster.opamp_provider.chatopcommand). The provider queues the custom command, and the supervisor’s ChatOps handler executes it by calling a local HTTP endpoint on the configured chat_ops_port.

That gives us a cleaner control path:

• Chat/user intent can go to the central server/API.
• The server routes to the right node through OpAMP.
• The supervisor performs the local action and can return failure context when local execution fails.

This is a stronger pattern than directly letting chat tooling call every node individually, and it opens the door to better auditability and policy controls around who can trigger what.

Reality check: we are still testing

This is important: we are still actively testing functionality.

Current status is intentionally mixed:

• Core identity, sequencing, capabilities, disconnect handling, and heartbeat/status pathways are in place.
• Some protocol fields are partial, todo, or long-term backlog.
• Custom capabilities/message pathways are implemented as a practical extension point and are still being hardened with test coverage and real-world runs.

So treat this as a working framework with proven pieces, not a finished all-capabilities implementation.

What is coming next (based on docs/features.md)

Near-term priorities include:

• stricter header/channel validation,
• heartbeat validation hardening,
• payload validation against declared capabilities,
• server-side duplicate websocket connection control behaviour.

Broader roadmap themes include:

• authentication/security model for APIs and UI,
• persistence in the provider,
• richer UI controls for node/global polling and multi-node config push,
• certificate and signing workflows,
• packaging improvements.

And yes, a key strategic direction is evolving the framework abstraction so it can support Fluentd in due course, not only Fluent Bit. Some feature areas (like package/status richness) make even more sense in that broader collector ecosystem.

Why this matters

OpAMP gives us a standard envelope for control-plane interactions; the server/supervisor split gives us pragmatic deployment flexibility; and ChatOps provides a human-friendly control surface.

Put together, this becomes a useful pattern for managing telemetry agents in real environments where fleets are mixed, rollout velocity matters, and “just redeploy everything” is not always an option.

If you are evaluating this right now, the right mindset is: useful today, promising for tomorrow, and still under active verification as we close feature gaps.

Fluent Bit Vulnerabilities Expose Cloud Services to Takeover https://www.securityweek.com/fluent-bit-vulnerabilities-expose-cloud-services-to-takeover/ #Vulnerabilities #CloudSecurity #cloudsecurity #vulnerability #FluentBit #cloud

Auch in der Firma geht es voran: Für #pcidss (4.x!!!!)sind die unangenehmen Fragen für uns durch. Nur noch Pakete schnüren und alles hochladen zzgl. gefixter Incidents (Icinga Checks für ClamAV Prozess) und dann abwarten. Die Entwickler haben etwas mehr zu tun.

Und hab neues #logsystem fast schon komplett. #fluentbit wird #nxlog ersetzen. #victorialogs wird eine Weile parallel zum #Graylog laufen und die mangelnde Auth Fähigkeit von vmlog wird mittels #Nginx und #oauthproxy kompensiert. Es gibt auch ein schönes Ticket: Feature Request für fluentbit: Parameter für #yaml oder classic. Dann kann man nämlich fluent Config über Graylog ausrollen 😍

Auch in der Firma geht es voran: Für #pcidss (4.x!!!!)sind die unangenehmen Fragen für uns durch. Nur noch Pakete schnüren und alles hochladen zzgl. gefixter Incidents (Icinga Checks für ClamAV Prozess) und dann abwarten. Die Entwickler haben etwas mehr zu tun.

Und hab neues #logsystem fast schon komplett. #fluentbit wird #nxlog ersetzen. #victorialogs wird eine Weile parallel zum #Graylog laufen und die mangelnde Auth Fähigkeit von vmlog wird mittels #Nginx und #oauthproxy kompensiert. Es gibt auch ein schönes Ticket: Feature Request für fluentbit: Parameter für #yaml oder classic. Dann kann man nämlich fluent Config über Graylog ausrollen 😍

Auch in der Firma geht es voran: Für #pcidss (4.x!!!!)sind die unangenehmen Fragen für uns durch. Nur noch Pakete schnüren und alles hochladen zzgl. gefixter Incidents (Icinga Checks für ClamAV Prozess) und dann abwarten. Die Entwickler haben etwas mehr zu tun.

Und hab neues #logsystem fast schon komplett. #fluentbit wird #nxlog ersetzen. #victorialogs wird eine Weile parallel zum #Graylog laufen und die mangelnde Auth Fähigkeit von vmlog wird mittels #Nginx und #oauthproxy kompensiert. Es gibt auch ein schönes Ticket: Feature Request für fluentbit: Parameter für #yaml oder classic. Dann kann man nämlich fluent Config über Graylog ausrollen 😍

Auch in der Firma geht es voran: Für #pcidss (4.x!!!!)sind die unangenehmen Fragen für uns durch. Nur noch Pakete schnüren und alles hochladen zzgl. gefixter Incidents (Icinga Checks für ClamAV Prozess) und dann abwarten. Die Entwickler haben etwas mehr zu tun.

Und hab neues #logsystem fast schon komplett. #fluentbit wird #nxlog ersetzen. #victorialogs wird eine Weile parallel zum #Graylog laufen und die mangelnde Auth Fähigkeit von vmlog wird mittels #Nginx und #oauthproxy kompensiert. Es gibt auch ein schönes Ticket: Feature Request für fluentbit: Parameter für #yaml oder classic. Dann kann man nämlich fluent Config über Graylog ausrollen 😍

#Monitoring #Caddy with #FluentBit and #Prometheus

The most important tool for planning, understanding the impact of changes, and for dealing with the consequence of complexity (← bugs) is the ability to understand and measure what is actually happening. Therefore, what we need is to install some monitoring, so that we can see issues and make plans based on actual data.

https://marctrius.net/monitoring-caddy-with-fluent-bit-and-prometheus/

Please let me know if you have any feedback.

During the #SharkBytes session at #SharkFest conference I had an opportunity to present a lightning talk about my pet project called IDS Lab.
It is a lab infrastructure deployable as docker containers, which simulates the small company network.

The IDS Lab consists of web webserver with #Wordpress, #MySQL database, #Linux desktop with RDP, the #WireGuard VPN for "remote" workers and for connecting another virtual or physical machines into the lab network.
This part of infrastructure can be used for attack simulations.

There are additional components for playing with logs and detections, too: #Fluentbit, #Suricata and #OpenObserve as lightweight SIEM.

In the #SIEM we already have preconfgured dashboards for alerts, netflows, web logs and logs from windows machines, if present.

Using the provided setup script, the whole lab can be up and running in up to 5 minutes. For more info, please check my GitHub repository with the IDS Lab:

https://github.com/SecurityDungeon/ids-lab/

#sf24eu #wireshark @wireshark

Critical Vulnerability in Fluent Bit: CVE-2024-4323

Date: May 20, 2024
CVE: CVE-2024-4323
Vulnerability Type: Memory Corruption
CWE: [[CWE-787]], [[CWE-119]]
Sources: Tenable

Issue Summary

Tenable Research has identified a critical memory corruption vulnerability in Fluent Bit, designated CVE-2024-4323, within its built-in HTTP server. This vulnerability, termed "Linguistic Lumberjack," affects versions 2.0.7 through 3.0.3 and allows potential denial of service, information disclosure, or remote code execution. The issue has been fixed in the main branch and will be included in the upcoming 3.0.4 release.

Technical Key Findings

The vulnerability is rooted in improper handling of data types in the "inputs" array of the /api/v1/traces endpoint. When non-string values, such as integers or negative integers, are passed, it can lead to memory corruption. Exploits could include crashes from wild copies, heap overwrites, and disclosure of adjacent memory, potentially leading to remote code execution under specific conditions.

Vulnerable Products

• Fluent Bit versions 2.0.7 to 3.0.3

Impact Assessment

Exploitation of CVE-2024-4323 can result in significant disruptions such as service crashes, leakage of sensitive information, and in severe cases, remote code execution, which can compromise entire systems relying on Fluent Bit for logging and monitoring.

Patches or Workarounds

Users should upgrade to Fluent Bit version 3.0.4 or later. If upgrading is not immediately feasible, restricting access to the vulnerable endpoints is recommended to mitigate potential exploitation.

Tags

#CVE-2024-4323 #FluentBit #MemoryCorruption #CloudSecurity #RemoteCodeExecution #VulnerabilityManagement

Our simple 'cloud active defense' deception proxy now comes with a #fluentbit config for easy alert forwarding and management (we all know how it's hard to get these config files straight, so we took the effort to build something working)

https://github.com/SAP/cloud-active-defense

Next step: automated response - it's almost there, I hope you'll find it useful!

#infosec #deception #honeytokens #cloudactivedefense #opensource

Our simple 'cloud active defense' deception proxy now comes with a #fluentbit config for easy alert forwarding and management (we all know how it's hard to get these config files straight, so we took the effort to build something working)

https://github.com/SAP/cloud-active-defense

Next step: automated response - it's almost there, I hope you'll find it useful!

#infosec #deception #honeytokens #cloudactivedefense #opensource

Our simple 'cloud active defense' deception proxy now comes with a #fluentbit config for easy alert forwarding and management (we all know how it's hard to get these config files straight, so we took the effort to build something working)

https://github.com/SAP/cloud-active-defense

Next step: automated response - it's almost there, I hope you'll find it useful!

#infosec #deception #honeytokens #cloudactivedefense #opensource

Our simple 'cloud active defense' deception proxy now comes with a #fluentbit config for easy alert forwarding and management (we all know how it's hard to get these config files straight, so we took the effort to build something working)

https://github.com/SAP/cloud-active-defense

Next step: automated response - it's almost there, I hope you'll find it useful!

#infosec #deception #honeytokens #cloudactivedefense #opensource

Our simple 'cloud active defense' deception proxy now comes with a #fluentbit config for easy alert forwarding and management (we all know how it's hard to get these config files straight, so we took the effort to build something working)

https://github.com/SAP/cloud-active-defense

Next step: automated response - it's almost there, I hope you'll find it useful!

#infosec #deception #honeytokens #cloudactivedefense #opensource

I created an addon for #homeassistant last night that allows you to ship logs from your #hassio instance to somewhere else via #fluentbit: https://github.com/ablyler/ha-addon-fluent-bit

I am personally using this to send my logs to a local #openobserve instance: https://openobserve.ai

Days since I build an endless loop in my logging pipeline: 0 🎉

#HomeLab #Fluentd #Fluentbit

Did you ever wish somebody would write a 30 minute Epos on how to deploy a Fluentbit/FluentD/Loki logging stack? Today's your lucky day!
https://blog.mei-home.net/posts/k8s-migration-6-logging/

#HomeLab #blog #GrafanaLoki #Fluentd #Fluentbit

From this week's ADMIN Update newsletter, Artur Skura explores Fluentd and Fluent Bit to help unify data collection and consumption https://www.admin-magazine.com/Archive/2023/77/A-modern-logging-solution #Fluentd #FluentBit #OpenSource #logging #debugging #monitoring #troubleshooting #FOSS #data #LogManagement

Have you missed #ObservabilityDay at #KubeCon yesterday?

#FluentBit creator Eduardo Silva Pereira shared exciting updates on my fireside chat with him at OpenObservability Talks, about the v2.2 release, about a new secret project and cool UI 🤫 and more:
📺 https://www.youtube.com/watch?v=V02Ctv0Rtg8&t=2313s

or check out the TL;DR post: https://lnkd.in/djd_eHjb

#KubeConNA #DevOps #kubecon2023 #kubecon23 #fluentd #opensource #observability #cloudnative

@markstos #fluentd is definitely more powerful, but it's also more complex and it helps to know at least a little #ruby. You run the exact same agent on the client as the aggregation servers and it all depends on the config. #fluentbit is lightweight with a much simpler config. If you're just getting started, I'd definitely recommend trying #fluentbit first at this point.

@markstos Run it on every server, though #fluentbit is now more common than #fluentd in our deployments. It's deployed and managed via #Ansible AWX.