#fluentd — Public Fediverse posts

Fluent Bit configuration quick reference and editor

Fluent Bit’s documentation is spread across many static web pages; in some cases, attributes allowed for a plugin are documented across several pages. There is absolutely nothing wrong with this. Having written a book on Fluent Bit, I can say it makes sense, and if the standard docs went into further detail, it would increase the spread of content.

The problem is, once you’ve got a grip on Fluent Bit, you want a quick reference just to check the attribute names or confirm that an attribute does what you expect.

The good news is that’s what we have created …

• Quick references
  • 3.2.10 HTML, Markdown
  • 4.2.4 HTML , Markdown
  • 5.0.4 HTML , Markdown

Each quick reference page has a section at the top that provides a comma-separated list of all the plugins for the different operations, inputs, outputs, etc., with anchor links to that section. Each plugin has a table that lists all configuration attributes, including those inherited from the Fluent Bit kernel and those introduced by extending another plugin.

Why and how …

The “why” may prove even more appealing. As part of our OpAMP project, we wanted to make it easy to edit and validate Fluent Bit and Fluentd configurations before deployment using the OpAMP tooling. That tool became the config-service part of the OpAMP repo, and can be independently deployed as well as function as part of the main OpAMP service. As a result, the UI offers the same authentication and authorisation options, ranging from running without authentication to using OAuth.

The key to both the UI and documentation is the use of JSON Schemas, as they contain all the information needed to create documentation just as easily as they power the UI. We have therefore generated a markdown page using a Python script. This means the docs are easy to check (compared to an HTML file) and can be rendered within GitHub. We could extend the script to generate HTML. But rather than trying to remember to keep both document types in sync (and double-check formatting), we found a JavaScript tool (marked.js) that performs an on-the-fly transformation that, as long as we stay within the core Markdown syntax, won’t cause any issues. We’ve then enriched that output a bit by applying stylesheets.

To come…

At the moment, we have only covered a subset of Fluent Bit versions, so we will, in due course, expand to cover more versions. Today, this is just the last version from each major release. We haven’t done every version to date as it does take a lot of effort to go through the documentation to generate and check the schema, and we’re still refining things as we enhance the UI.

We are also looking to do the same for Fluentd, though this is much trickier, as the portfolio of plugins that make up the core is smaller than those incorporated into the core of Fluent Bit, and the leveraging of Ruby’s dynamic behaviour makes it very easy for people to offer plugins separately. Then, of course, there is the task of collating all that information.

Resources

• Quick references
  • 3.2.10 HTML, Markdown
  • 4.2.4 HTML, Markdown
  • 5.0.4 HTML, Markdown
• OpAMP landing page
• Fluent Bit
• Fluentd

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)

Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.

Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:

• When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
• OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
• The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
• Occasionally, a little bit of intellectual snobbery can creep in

The common challenge

The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.

AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.

Experimental Solution

Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).

 {    "type": "object",    "title": "Dummy",    "properties": {      "name": {        "type": "string",        "const": "dummy",        "title": "Plugin"      },      "copies": {        "type": "integer",        "description": "Number of messages to generate each time messages are generated.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "integer",        "default": 1      },      "dummy": {        "type": "string",        "description": "Dummy JSON record.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "string",        "default": "{\"message\":\"dummy\"}"      },      "fixed_timestamp": {        "type": "boolean",        "description": "If enabled, use a fixed timestamp.",        "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters",        "x-doc-required": false,        "x-config-data-type": "boolean",        "default": false      }    }  }  

The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.

By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.

We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.

As a result, we very quickly have a UI that can look like this:

A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.

There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.

Outstanding Gaps for Fluent Bit

We still need to address several features that Fluent Bit has, specifically:

• Environment variables
• Includes

These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.

What about Fluentd?

Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.

The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.

Then What?

Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.

As a bonus, perhaps transform the Schema into a quick reference web document?

OpAMP with Fluent Bit – Observability and ChatOps

With KubeCon Europe happening this week, it felt like a good moment to break cover on this pet project.

If you are working with Fluent Bit at any scale, one question keeps coming up: how do we consistently control and observe all those edge agents, especially outside a Kubernetes-only world?

This is exactly the problem the OpAMP specification is trying to solve. At its core, OpAMP defines a standard contract between a central server and distributed agents/supervisors, so status, health, commands, and config-related interactions follow one protocol instead of ad-hoc integration per tool.

That is where this project sits. We’re implementing the OpAMP specification to support Fluent Bit (and later Fluentd).

In this implementation, we have:

• a provider (the OpAMP server), and
• a consumer acting as a supervisor to manage Fluent Bit deployments.

Right now, we are focused on Fluent Bit first. That is deliberate: it keeps scope practical while we validate the framework. The same framework is being shaped so it can evolve to support Fluentd as well.

The repository for the implementation can be found at https://github.com/mp3monster/fluent-opamp

Quick summary

The provider/server is the control plane endpoint. It tracks clients, accepts status, queues commands, and returns instructions using OpAMP payloads over HTTP or WebSocket.

The consumer/supervisor handles the local execution and reporting. It launches Fluent Bit, polls local health/status endpoints, sends heartbeat and metadata to the provider, and handles inbound commands (including custom ones). The server and supervisor can be deployed independently, which is important for real-world rollout patterns.

Because they follow the OpAMP protocol model, clients and servers can be interchanged with other OpAMP-compliant implementations (although we’ve not yet tested this aspect of the development).

Together, they give us a manageable, spec-aligned path to coordinating distributed Fluent Bit nodes without hard-coding one-off control logic into every environment.

Deployment options and scripts

There are a few practical ways to get started quickly:

• Deploy just the server/provider using scripts/run_opamp_server.sh (or scripts/run_opamp_server.cmd on Windows).
• Deploy just the client/supervisor using scripts/run_supervisor.sh (or scripts/run_supervisor.cmd on Windows).
• Run both components either together in a single environment or independently across different hosts.

The scripts will set up a virtual environment and retrieve the necessary dependencies.

If you want an initial MCP client setup as part of your workflow, there are helper scripts for that too:

• mcp/configure-codex-fastmcp.sh and mcp/configure-codex-fastmcp.ps1
• mcp/configure-claude-desktop-fastmcp.sh and mcp/configure-claude-desktop-fastmcp.ps1

Server screenshots

Here is a first server view we can include in the post:

The UI is still evolving, but this gives a concrete picture of the provider side control plane we are discussing.

What the OpAMP server (provider) does

The provider is responsible for the shared view of fleet state and intent.

Today it provides:

• OpAMP transport endpoints (/v1/opamp) over HTTP and WebSocket.
• API and UI endpoints to inspect clients and queue actions.
• In-memory command queueing per client.
• Emission of standard command payloads (for example, restart).
• Emission of custom message payloads for custom capabilities.
• Discovery and publication of custom capabilities supported by the server side command framework.

Operationally, this means we can queue intent once at the server and let the next client poll/connection cycle deliver that action in protocol-native form.

What the supervisor (consumer) does for Fluent Bit

The supervisor is the practical glue between OpAMP and Fluent Bit:

• Starts Fluent Bit as a local child process.
• Parses Fluent Bit config details needed for status polling.
• Polls Fluent Bit local endpoints on a heartbeat loop.
• Builds and sends AgentToServer messages (identity, capabilities, health/status context).
• Receives ServerToAgent responses and dispatches commands.
• Handles custom capabilities and custom messages through a handler registry.

So for Fluent Bit specifically, the supervisor gives us a way to participate in OpAMP now, even before native in-agent OpAMP support is universal.

And to be explicit: this is the current target. Fluentd support is a planned evolution of this same model, not a separate rewrite.

Where ChatOps fits

ChatOps is where this gets interesting for day-2 operations.

In this implementation, ChatOps commands are carried as OpAMP custom messages (custom capability org.mp3monster.opamp_provider.chatopcommand). The provider queues the custom command, and the supervisor’s ChatOps handler executes it by calling a local HTTP endpoint on the configured chat_ops_port.

That gives us a cleaner control path:

• Chat/user intent can go to the central server/API.
• The server routes to the right node through OpAMP.
• The supervisor performs the local action and can return failure context when local execution fails.

This is a stronger pattern than directly letting chat tooling call every node individually, and it opens the door to better auditability and policy controls around who can trigger what.

Reality check: we are still testing

This is important: we are still actively testing functionality.

Current status is intentionally mixed:

• Core identity, sequencing, capabilities, disconnect handling, and heartbeat/status pathways are in place.
• Some protocol fields are partial, todo, or long-term backlog.
• Custom capabilities/message pathways are implemented as a practical extension point and are still being hardened with test coverage and real-world runs.

So treat this as a working framework with proven pieces, not a finished all-capabilities implementation.

What is coming next (based on docs/features.md)

Near-term priorities include:

• stricter header/channel validation,
• heartbeat validation hardening,
• payload validation against declared capabilities,
• server-side duplicate websocket connection control behaviour.

Broader roadmap themes include:

• authentication/security model for APIs and UI,
• persistence in the provider,
• richer UI controls for node/global polling and multi-node config push,
• certificate and signing workflows,
• packaging improvements.

And yes, a key strategic direction is evolving the framework abstraction so it can support Fluentd in due course, not only Fluent Bit. Some feature areas (like package/status richness) make even more sense in that broader collector ecosystem.

Why this matters

OpAMP gives us a standard envelope for control-plane interactions; the server/supervisor split gives us pragmatic deployment flexibility; and ChatOps provides a human-friendly control surface.

Put together, this becomes a useful pattern for managing telemetry agents in real environments where fleets are mixed, rollout velocity matters, and “just redeploy everything” is not always an option.

If you are evaluating this right now, the right mindset is: useful today, promising for tomorrow, and still under active verification as we close feature gaps.

[小ネタ]EC2に導入したFluentdの設定ファイルを変更し時間階層でファイルを送信してみた
https://dev.classmethod.jp/articles/ec2-fluentd-configuration-time-hierarchy-file-transfer/

#dev_classmethod #Fluentd

Fluentd: How to Use a Parser With Regular Expression (regexp) – Source: socprime.com https://ciso2ciso.com/fluentd-how-to-use-a-parser-with-regular-expression-regexp-source-socprime-com/ #rssfeedpostgeneratorecho #CyberSecurityNews #KnowledgeBits #socprimecom #socprime #Fluentd #Blog

Fluentd: How to Use a Parser With Regular Expression (regexp) – Source: socprime.com https://ciso2ciso.com/fluentd-how-to-use-a-parser-with-regular-expression-regexp-source-socprime-com/ #rssfeedpostgeneratorecho #CyberSecurityNews #KnowledgeBits #socprimecom #socprime #Fluentd #Blog

Fluentd: How to Use a Parser With Regular Expression (regexp) – Source: socprime.com https://ciso2ciso.com/fluentd-how-to-use-a-parser-with-regular-expression-regexp-source-socprime-com/ #rssfeedpostgeneratorecho #CyberSecurityNews #KnowledgeBits #socprimecom #socprime #Fluentd #Blog

Are there any #fluentd experts here? I'm trying to have certain events forward to a graylog server, preferably in gelf, but I cannot for the life of me figure this out.

#sysadmin #fedihelp

Amazon Linux 2023 に Fluentd を導入してS3バケットにログを配信
https://dev.classmethod.jp/articles/al2023-fluentd-s3/

#dev_classmethod #Amazon_EC2 #Fluentd #Amazon_S3 #AWS

New blog post: https://blog.mei-home.net/posts/some-k8s-logging-changes/

I made a couple of changes to my initial logging pipeline.

I'm also *trying* to learn to write shorter blog posts on smaller things.

#HomeLab #Blog #FluentD

And I did not find out I made a loop through some superior metrics and alerting - no, I just heard the fans rev up without an obvious reason. 😂

It all started with wanting to massage the cnpg postgres logs a bit. And while doing that, I saw an "issue". There was a wayward "time" field which I had no use for. And now I'm revamping my entire log setup. 🤦

#HomeLab #Fluentd

Days since I build an endless loop in my logging pipeline: 0 🎉

#HomeLab #Fluentd #Fluentbit

Can't figure out how to add mastodon-web.service logging to the Better Stack via Fluentd... Fluentd is completely new thing for me.

I mean this: sudo journalctl -u mastodon-web.service --all -f

I want to add that debug log to log tail. But even though everything is configured, up and running, logtail shows nothing. Nada.

Let's wait for the support to reply for a third time.

#BetterStack #MastoAdmin #Fluentd

Did you ever wish somebody would write a 30 minute Epos on how to deploy a Fluentbit/FluentD/Loki logging stack? Today's your lucky day!
https://blog.mei-home.net/posts/k8s-migration-6-logging/

#HomeLab #blog #GrafanaLoki #Fluentd #Fluentbit

I call it "Oooops", alternative title "I know exactly why this happened. I knew it would happen before it happened, in fact".

Fluentd's stdout going entirely haywire because I feed Fluentd's logs through the log pipeline and because I've got it unparsed right now, it goes to stdout - rinse and repeat. 😅

#HomeLab #Fluentd

Okay, got my logging setup properly primed now - only sending the logs which haven't been properly parsed yet to stdout so I can slowly implement their parse filters, and everything else gets forwarded to Loki for storage.

Ceph is almost done, next is going to be the control plane logs.

#HomeLab #Fluentd

From this week's ADMIN Update newsletter, Artur Skura explores Fluentd and Fluent Bit to help unify data collection and consumption https://www.admin-magazine.com/Archive/2023/77/A-modern-logging-solution #Fluentd #FluentBit #OpenSource #logging #debugging #monitoring #troubleshooting #FOSS #data #LogManagement

@mmeier how are you collecting the logs? We've had good results using SRV records to point to our #fluentd aggregators.

It's nice because the configuration for any server running a local fluentd is pretty straightforward

Have you missed #ObservabilityDay at #KubeCon yesterday?

#FluentBit creator Eduardo Silva Pereira shared exciting updates on my fireside chat with him at OpenObservability Talks, about the v2.2 release, about a new secret project and cool UI 🤫 and more:
📺 https://www.youtube.com/watch?v=V02Ctv0Rtg8&t=2313s

or check out the TL;DR post: https://lnkd.in/djd_eHjb

#KubeConNA #DevOps #kubecon2023 #kubecon23 #fluentd #opensource #observability #cloudnative

Me watching millions of log events per day funnel into an #OpenSearch cluster via #fluentd.

I think I've finally tamed this #OpenSearch setup on this #Rancher #RKE2 cluster. Today's adventure was schema conflicts. Pods labeled with "app" while others are labeled with app.kubernetes.io cause a problem for inputs as it looks to OpenSearch like there's a string where an object should be and the flatten hashes on the #fluentd output wasn't quite enough to cut it, but the dedot filter brought it in the rest of the way there.

@waltertross This all caused me a bit of grief lately for managing/defining #Jupyter lab plugins and #fluentd configurations in #Kubernetes YAML object declarations.

Just closed 176 tabs in #Firefox. The upside is I have #Rancher and #OpenSearch playing well together at the moment via some #fluentd magic. 🥳 :apartyblobcat: 🎉

📣 𝐑𝐞𝐩𝐨𝐬𝐭𝐢𝐧𝐠 𝐦𝐲 𝐨𝐥𝐝 𝐛𝐥𝐨𝐠 𝐩𝐨𝐬𝐭 𝐨𝐧 𝐩𝐫𝐨𝐠𝐫𝐚𝐦𝐦𝐚𝐭𝐢𝐜𝐚𝐥𝐥𝐲 𝐬𝐞𝐧𝐝𝐢𝐧𝐠 𝐥𝐨𝐠𝐬 𝐟𝐫𝐨𝐦 𝐚 𝐑𝐚𝐬𝐩𝐛𝐞𝐫𝐫𝐲 𝐏𝐢 (𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜𝐚𝐥𝐥𝐲 𝐏𝐢𝐡𝐨𝐥𝐞 𝐥𝐨𝐠𝐬) 𝐭𝐨 𝐌𝐢𝐜𝐫𝐨𝐬𝐨𝐟𝐭 𝐒𝐞𝐧𝐭𝐢𝐧𝐞𝐥!

📝 Article link: https://medium.com/p/57da570f1151

🗓️ Originally published in 2019, but still relevant today!

🔍 Over the past few weeks, I have received numerous inquiries from clients and partners regarding the process of programmatically sending logs from a Raspberry Pi, particularly the logs from Pihole, to Microsoft Sentinel. Thant's why I'm reposting my blog post from 2019, which provides a detailed guide on achieving this integration.

📌 Please note that the blog post refers to the older nomenclature of Microsoft Sentinel. While the article may use terms and references from the past, the core concepts and principles remain applicable and can be adapted to the current version of Microsoft Sentinel.

#MicrosoftSentinel #RaspberryPi #Pihole #LogManagement #Security #pihole #dns #log #sentinel #siem #soar #soc #monitoring #fluentd #syslog #microsoft

@markstos #fluentd is definitely more powerful, but it's also more complex and it helps to know at least a little #ruby. You run the exact same agent on the client as the aggregation servers and it all depends on the config. #fluentbit is lightweight with a much simpler config. If you're just getting started, I'd definitely recommend trying #fluentbit first at this point.

@markstos Run it on every server, though #fluentbit is now more common than #fluentd in our deployments. It's deployed and managed via #Ansible AWX.

@vwbusguy Do you run #fluentd on every server or do you use something like systemd-journal-upload to upload to centralized fluentd instances?

https://www.freedesktop.org/software/systemd/man/systemd-journal-upload.html

#systemd

@markstos I'm the mod/owner of the #fluentd channel on #Matrix and @liberachat, by the way, so feel free to reach out there as well. There's also an official Slack channel for the project if you prefer that. To be clear, I don't work for CNCF, TD, etc., I'm just a community member who's been using it for years and wants to help out where I can.

https://www.travnewmatic.com/2022/11/29/proxmoxve-logs-banzai-cloud-logging-operator-elasticsearch/ #kubernetes #proxmox #homelab #fluentd #banzaicloud #elasticsearch #syslog