Search
1000 results for “json”
-
Comparaison de 6 modèles de langages / LLM exécutés en local en utilisant un exercice de code simple en Python (conversion de CSV en JSON) - Un billet intéressant publié chez LinuxFr #IA #LLM #Coding https://linuxfr.org/users/jobpilot/journaux/comparatif-6-llms-locaux-face-a-un-exercice-python-simple
-
Comparaison de 6 modèles de langages / LLM exécutés en local en utilisant un exercice de code simple en Python (conversion de CSV en JSON) - Un billet intéressant publié chez LinuxFr #IA #LLM #Coding https://linuxfr.org/users/jobpilot/journaux/comparatif-6-llms-locaux-face-a-un-exercice-python-simple
-
Comparaison de 6 modèles de langages / LLM exécutés en local en utilisant un exercice de code simple en Python (conversion de CSV en JSON) - Un billet intéressant publié chez LinuxFr #IA #LLM #Coding https://linuxfr.org/users/jobpilot/journaux/comparatif-6-llms-locaux-face-a-un-exercice-python-simple
-
#Development #ReportsAdding author context to RSS · A proposal to bring author identity to web feeds https://ilo.im/16cmp6_____#Business #WebFeeds #RSS #Atom #JSON #Websites #SmallWeb #IndieWeb #Development #WebDev
-
#Development #Reports
Adding author context to RSS · A proposal to bring author identity to web feeds https://ilo.im/16cmp6_____
#Business #WebFeeds #RSS #Atom #JSON #Websites #SmallWeb #IndieWeb #Development #WebDev -
Архитектура автоматической трансформации данных JSON и XML любой структуры унифицированным способом
В современном IT ландшафте широко используютя форматы представления данных JSON и XML, используемые в качестве своеобразного "общего языка", lingua franca для обмене информацией. Данная статья представит архитектуру интеграции данных иерархических форматов, позволяющую кардинально уменьшить трудоемкость процесса до практически полностью универсального пайплайна, обрабатывающего любые виды исходных документов вплоть до автоматического маппинга в табличные структуры данных.
https://habr.com/ru/articles/1034884/
#xml #json #api #nosql #sql #базы_данных #алгоритмы #архитектура #автоматизация
-
RSS, ATOM, and JSON feeds may be getting a much-needed update to the spec with the addition of Byline, which provides structured identity (AKA: author details) for syndication feeds.
The proposal was created by @tg, maker of the popular Current RSS reader.
-
The Delimiter hypothesis: does prompt format actually matter? https://systima.ai/blog/delimiter-hypothesis #AI #Prompting #XML #JSON #Markdown
-
Structured prompting techniques #AI #JSON #XML #Prompting https://codeconductor.ai/blog/structured-prompting-techniques-xml-json/
-
#tersoftware de linha de comando
• #jq - formatação e bonitificação de texto em formato JSON
• #bat - um #cat com asas :)
• #ncdu - interface em ncurses pro #du, muito útil pra encontrar (e apagar) diretórios e arquivos gigantes no disco
• #ranger - navegador de arquivos com atalhos do vim
• #z - acessa pastas frequentemente utilizadas diretamente
• #fzf - backend de fuzzy search compatível com todos os sabores de shells e vim.
• #fd - um #find mais intuitivo
• #rg - um #grep mais intuitivo, com mais funcionalidades (e mais rápido) -
MalChela 3.2: More Cowbell? More Intel!
One of the things I value most about the open-source community is that the best improvements to a tool often don’t come from inside it — they come from outside conversations. A short while back, the author of mlget, xorhex, reached out and suggested I add more malware retrieval sources to FOSSOR, one of my earlier tools for pulling down samples from threat intel repositories. It was a generous nudge, and it landed at exactly the right moment.
FOSSOR started as a simple script. It did one job — grab malware samples from a handful of sources — and it did it well enough. When I wrote it, I already knew it was a candidate for eventual MalChela integration, but “eventually” had stayed firmly in the future tense. The message from xorhex gave me the push to actually sit down and do it properly.
The result is tiquery — and it’s become a new centerpiece to MalChela 3.2.
The Pattern I Keep Repeating (Deliberately)
If you’ve followed this blog or the MalChela project for a while, you might notice a recurring arc in how my tools tend to develop. It goes something like this:
- Step one: write a focused script that solves a specific problem.
- Step two: that script evolves into a standalone tool as the scope grows.
- Step three: the tool finds its permanent home inside MalChela, where it benefits from the broader ecosystem — case management, the GUI, the MCP server integration, the portable workspace.
- Step four: When there’s overlap between tools, follow the KISS principle.
FOSSOR was in step one. The conversation with xorhex accelerated the jump to step three. What emerged was something more ambitious than just a source expansion — it’s a unified threat intelligence query engine, built from the ground up.
If you’re new to MalChela, it’s a Rust-based malware analysis toolkit built for DFIR practitioners — static analysis, string extraction, YARA rule generation, threat intel lookups, network analysis, and now a unified case management layer tying it all together. Free, open-source, and built to run anywhere.
Introducing tiquery
tiquery is now the single threat intel tool in MalChela, replacing the retired malhash. The core idea is straightforward: submit a hash, query multiple sources in parallel, get a clean color-coded summary back. No waiting for one source to finish before the next one starts. No manually juggling browser tabs or API wrappers.
Out of the box, tiquery works with eight confirmed sources:
- VirusTotal
- MalwareBazaar
- AlienVault OTX
- MetaDefender Cloud
- Hybrid Analysis
- FileScan.IO
- Malshare
- ObjectiveSee (no API key required — queries a locally-cached macOS malware catalogue)
Sources are tiered — free sources and registration-required sources are distinguished in the interface. If you haven’t configured an API key for a given source, tiquery skips it gracefully rather than throwing an error. This means you can run it easily with whatever keys you have available.
The ObjectiveSee integration deserves a special mention. It queries the objective-see.org/malware.html catalogue for macOS-specific threats using a locally-cached copy that refreshes every 24 hours, with a stale-cache fallback for offline use. For anyone doing Mac forensics, this is a meaningful addition — a free, no-key-required check specifically against known macOS malware families.
Tiquery, like FOSSOR, supports batch lookups as well — point it to a .csv or .txt file of hashes and they’ll all be checked in parallel. You can also download samples directly, with MalwareBazaar supported in this release and additional sources on the way – (your vote matters).
What It Looks Like in Practice
The screenshots below show tiquery running in both the CLI and GUI. In both cases, for any of the matching sources you get a basic classification (malware family, tags, detections,) and direct links to threat intelligence documents about the samples. It’s the perfect jumping off point when you want to leverage community research.
The CLI output is clean and tabular — source abbreviation, status (color-coded FOUND/NOT FOUND), family and tag information, detection count, and a direct reference URL. Everything you need to make a quick triage decision, no scrolling through API response JSON required. You can run tiquery CLI as a stand-alone, or from within the MalChela CLI menu.In the GUI, the experience is layered a bit more richly. You can toggle individual sources on or off, switch between single-hash and bulk lookup modes, download the sample directly from MalwareBazaar, and export results to CSV — all from one interface. The macOS ObjectiveSee source displays its cache age inline so you always know how fresh the data is.
Both outputs feed into MalChela’s case management system. Check “Save to Case” in the GUI, and tiquery creates a valid case.json automatically — no separate case creation step needed.
Extended Case Management Across the Toolkit
Speaking of case management — 3.2 extends “Save to Case” support across the full GUI. File Analyzer, File Miner, and mStrings, all now include the checkbox. This closes out the last gaps in the case workflow. Whatever tool you’re using for a given task, if you want to preserve the output in a named case, it’s one click. You no longer have to start with the New Case workflow, however it is recommended if that’s the direction you’re going from the start.
The Strings to YARA tool also gains a companion “Save to YARA Library” checkbox. Check it, and the generated rule gets copied directly into the project’s yara_rules/ directory alongside being saved to the case. This will automatically make the rule available when you run fileanalyzer on subsequent files. It’s a small workflow improvement, but one that eliminates a manual copy step I was taking every time anyway. I also added a quick formatter so the special character most often in need of escaping “\” gets handled automatically when the rule is generated.
A Note on malhash
malhash is retired in 3.2. If you’ve been using it in scripts or workflows, tiquery is its direct replacement — it does everything malhash did and then some. This is a breaking change in the sense that the binary is gone, but functionally tiquery is a superset, not a lateral move.
malhash served its purpose well. RIP. tiquery is where that purpose lives now.
Get It
MalChela 3.2 is available now on GitHub. The full release notes are in the repo.
Thanks to xorhex for the nudge. Sometimes the best features start with someone saying “have you thought about…”
#API #DFIR #Forensics #Github #MalChela #Malware #ThreatIntelligence #yara -
🛠️ Tool
===================Opening:
Loki-RS is a Rust-based rewrite of the original Loki scanner that consolidates YARA rule matching and IOC detection into a single high-performance, multi-threaded binary. The project is published as Beta and emphasizes speed, concurrency, and multiple output formats for forensic ingestion.Key Features:
• YARA scanning of files and process memory with the Core YARA Forge rule set as the default detection surface.
• IOC matching covering cryptographic hashes (MD5, SHA1, SHA256), filename patterns and C2 indicators drawn from the signature-base collection.
• Concurrency model permitting configurable thread counts for parallel scanning and CPU-bound tuning.
• Archive handling with ZIP inspection to reach nested artifacts.
• Operational tooling including an interactive TUI for real-time stats and controls, HTML report generation, and JSONL output for SIEM/log pipeline ingestion.
• Remote logging via syslog over UDP/TCP, with both SYSLOG and JSON formats supported.Technical Implementation:
• The codebase leverages Rust for memory safety and performance; multi-threaded scanning suggests internal worker queues and file/process enumeration that avoid scanning virtual filesystems by default (/proc, /sys).
• Signature management integrates signature-base for IOCs and YARA Forge for rule sets; the Core rule set is chosen for accuracy and low false positives, while Extended/Full sets are available for swap-in.
• Output pathways include structured JSONL for ingestion pipelines and HTML for human-readable reporting; remote sinks support syslog framing in both traditional SYSLOG and JSON payload modes.Use Cases:
• Forensic triage on endpoints and mounts where quick identification of known artifacts (hashes, filenames, C2 indicators) is needed.
• Bulk filesystem scans across images or mounted volumes with multi-threaded throughput requirements.
• Integration with logging/monitoring stacks via JSONL or syslog exports.Limitations & Considerations:
• Project is Beta: features and signatures remain under active development.
• Signature freshness depends on external sources; operational users should plan for regular signature updates.
• Default smart filtering skips virtual filesystems and mounted drives; scanning network/cloud mounts requires explicit configuration.References:
• Detection content: signature-base (IOCs) and YARA Forge (YARA rules).🔹 tool #rust #yara #ioctools #forensics
🔗 Source: https://github.com/Neo23x0/Loki-RS
-
🛠️ Tool
===================Opening:
Loki-RS is a Rust-based rewrite of the original Loki scanner that consolidates YARA rule matching and IOC detection into a single high-performance, multi-threaded binary. The project is published as Beta and emphasizes speed, concurrency, and multiple output formats for forensic ingestion.Key Features:
• YARA scanning of files and process memory with the Core YARA Forge rule set as the default detection surface.
• IOC matching covering cryptographic hashes (MD5, SHA1, SHA256), filename patterns and C2 indicators drawn from the signature-base collection.
• Concurrency model permitting configurable thread counts for parallel scanning and CPU-bound tuning.
• Archive handling with ZIP inspection to reach nested artifacts.
• Operational tooling including an interactive TUI for real-time stats and controls, HTML report generation, and JSONL output for SIEM/log pipeline ingestion.
• Remote logging via syslog over UDP/TCP, with both SYSLOG and JSON formats supported.Technical Implementation:
• The codebase leverages Rust for memory safety and performance; multi-threaded scanning suggests internal worker queues and file/process enumeration that avoid scanning virtual filesystems by default (/proc, /sys).
• Signature management integrates signature-base for IOCs and YARA Forge for rule sets; the Core rule set is chosen for accuracy and low false positives, while Extended/Full sets are available for swap-in.
• Output pathways include structured JSONL for ingestion pipelines and HTML for human-readable reporting; remote sinks support syslog framing in both traditional SYSLOG and JSON payload modes.Use Cases:
• Forensic triage on endpoints and mounts where quick identification of known artifacts (hashes, filenames, C2 indicators) is needed.
• Bulk filesystem scans across images or mounted volumes with multi-threaded throughput requirements.
• Integration with logging/monitoring stacks via JSONL or syslog exports.Limitations & Considerations:
• Project is Beta: features and signatures remain under active development.
• Signature freshness depends on external sources; operational users should plan for regular signature updates.
• Default smart filtering skips virtual filesystems and mounted drives; scanning network/cloud mounts requires explicit configuration.References:
• Detection content: signature-base (IOCs) and YARA Forge (YARA rules).🔹 tool #rust #yara #ioctools #forensics
🔗 Source: https://github.com/Neo23x0/Loki-RS
-
🛠️ Tool
===================Opening:
Loki-RS is a Rust-based rewrite of the original Loki scanner that consolidates YARA rule matching and IOC detection into a single high-performance, multi-threaded binary. The project is published as Beta and emphasizes speed, concurrency, and multiple output formats for forensic ingestion.Key Features:
• YARA scanning of files and process memory with the Core YARA Forge rule set as the default detection surface.
• IOC matching covering cryptographic hashes (MD5, SHA1, SHA256), filename patterns and C2 indicators drawn from the signature-base collection.
• Concurrency model permitting configurable thread counts for parallel scanning and CPU-bound tuning.
• Archive handling with ZIP inspection to reach nested artifacts.
• Operational tooling including an interactive TUI for real-time stats and controls, HTML report generation, and JSONL output for SIEM/log pipeline ingestion.
• Remote logging via syslog over UDP/TCP, with both SYSLOG and JSON formats supported.Technical Implementation:
• The codebase leverages Rust for memory safety and performance; multi-threaded scanning suggests internal worker queues and file/process enumeration that avoid scanning virtual filesystems by default (/proc, /sys).
• Signature management integrates signature-base for IOCs and YARA Forge for rule sets; the Core rule set is chosen for accuracy and low false positives, while Extended/Full sets are available for swap-in.
• Output pathways include structured JSONL for ingestion pipelines and HTML for human-readable reporting; remote sinks support syslog framing in both traditional SYSLOG and JSON payload modes.Use Cases:
• Forensic triage on endpoints and mounts where quick identification of known artifacts (hashes, filenames, C2 indicators) is needed.
• Bulk filesystem scans across images or mounted volumes with multi-threaded throughput requirements.
• Integration with logging/monitoring stacks via JSONL or syslog exports.Limitations & Considerations:
• Project is Beta: features and signatures remain under active development.
• Signature freshness depends on external sources; operational users should plan for regular signature updates.
• Default smart filtering skips virtual filesystems and mounted drives; scanning network/cloud mounts requires explicit configuration.References:
• Detection content: signature-base (IOCs) and YARA Forge (YARA rules).🔹 tool #rust #yara #ioctools #forensics
🔗 Source: https://github.com/Neo23x0/Loki-RS
-
La #app #Android fatta l’altro giorno ha svolto il suo lavoro, e programmarla così è stato furbo. 🌚️ Ho usato quindi:
- Quella, per eseguire un piccolo #script #JS per monitorare gli #SMS in entrata, controllare che provengano dal mio numero di #telefono, e siano del tipo giusto (cioè, con il corpo HTML racchiuso all’interno di un elemento XML
<post>che ho inventato di sana pianta), assemblarne multipli in una sola stringa di testo se necessario (per la prova di prima non lo è stato), e inviarli con un’apposita richiesta #HTTP ad… - Un workflow creato al volo su n8n (che gira sul mio #server), che usa la #API semplificata del plugin WP Press This per pubblicare, dato che per qualche motivo il modulo WordPress integrato nella piattaforma non funziona per il mio #sito.
Tutto insieme, questo sistema mi ha permesso di pubblicare su #WordPress effettivamente tramite un messaggio SMS, inviato dal mio #smartphone principale, ricevuto dal secondario che ha #Internet e fa le sue cose. Nel video c’è la dimostrazione di quel post di prima, appunto; di sfondo, l’interfaccia di #n8n che si aggiorna per mostrare la nuova esecuzione avvenuta, e il #MicroBlog con il nuovo post arrivato… 🐵️
Ora, ripeto che questo era un #EsercizioDiStile, e per il momento nell’anno del signore #2023 non ho esigenza di postare così, ma forse il concetto generale potrebbe tornare utile per le emergenze, e in ogni caso certamente fa figo provare sulla mia #piattaforma l’ebrezza di una funzione che solo #Twitter di tutte ha avuto, e solo per poco tempo. 😤️
Se in futuro volessi davvero usarla, innanzitutto implementerei un sistema di #OTP, così che ogni volta che voglio pubblicare devo anche inserire un codice temporaneo (perché SMS è un colabrodo di protocollo, e sarebbe possibile per dei criminali inviare un messaggio che appare come dal mio numero), poi magari implementerei funzioni come la modifica dei post (oltre che qualche feedback per pubblicazione avvenuta con successo o meno, cosa che ho evitato ora perché dalla seconda #SIM non posso inviare messaggi, solo ricevere). Per chi vuole curiosare, il #JavaScript per #DroidScriptin e il JSON per n8n sono qui: https://gitlab.com/octospacc/Snippets/-/tree/main/SmsToWordpress. 🎀️
https://octospacc.altervista.org/2023/12/18/635/
#2023 #Android #API #app #DroidScriptin #EsercizioDiStile #HTTP #Internet #JavaScript #JS #MicroBlog #n8n #OTP #piattaforma #script #server #SIM #sito #smartphone #SMS #telefono #Twitter #WordPress
- Quella, per eseguire un piccolo #script #JS per monitorare gli #SMS in entrata, controllare che provengano dal mio numero di #telefono, e siano del tipo giusto (cioè, con il corpo HTML racchiuso all’interno di un elemento XML
-
From the .NET blog...
In case you missed it earlier...
VSTest is Removing its Newtonsoft.Json Dependency
https://devblogs.microsoft.com/dotnet/vs-test-is-removing-its-newtonsoft-json-dependency/ #dotnet #NETFramework #nuget #testing #visualstudio #vs #vstest -
In ActivityPub, these are all equivalent:
"type":"Banana"
"type":["Banana"]
"type":{"＠id":"Banana"}
"type":[{"＠id":"Banana"}]
"type":{"id":"Banana"}
"type":[{"id":"Banana"}]
"＠type":"Banana"
"＠type":["Banana"]
"＠type":{"＠id":"Banana"}
"＠type":[{"＠id":"Banana"}]
"＠type":{"id":"Banana"}
"＠type":[{"id":"Banana"}]
-
-
-
-
-
-
Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
- When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
- OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
- The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
- Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{ "type": "object", "title": "Dummy", "properties": { "name": { "type": "string", "const": "dummy", "title": "Plugin" }, "copies": { "type": "integer", "description": "Number of messages to generate each time messages are generated.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "integer", "default": 1 }, "dummy": { "type": "string", "description": "Dummy JSON record.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "string", "default": "{\"message\":\"dummy\"}" }, "fixed_timestamp": { "type": "boolean", "description": "If enabled, use a fixed timestamp.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "boolean", "default": false } } }The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
- Environment variables
- Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
#AI #artificialIntelligence #configuration #development #ELK #FluentBit #Fluentd #LLM #observability #OpAMP #Technology -
Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
- When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
- OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
- The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
- Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{ "type": "object", "title": "Dummy", "properties": { "name": { "type": "string", "const": "dummy", "title": "Plugin" }, "copies": { "type": "integer", "description": "Number of messages to generate each time messages are generated.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "integer", "default": 1 }, "dummy": { "type": "string", "description": "Dummy JSON record.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "string", "default": "{\"message\":\"dummy\"}" }, "fixed_timestamp": { "type": "boolean", "description": "If enabled, use a fixed timestamp.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "boolean", "default": false } } }The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
- Environment variables
- Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
#AI #artificialIntelligence #configuration #development #ELK #FluentBit #Fluentd #LLM #observability #OpAMP #Technology -
Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
- When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
- OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
- The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
- Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{ "type": "object", "title": "Dummy", "properties": { "name": { "type": "string", "const": "dummy", "title": "Plugin" }, "copies": { "type": "integer", "description": "Number of messages to generate each time messages are generated.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "integer", "default": 1 }, "dummy": { "type": "string", "description": "Dummy JSON record.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "string", "default": "{\"message\":\"dummy\"}" }, "fixed_timestamp": { "type": "boolean", "description": "If enabled, use a fixed timestamp.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "boolean", "default": false } } }The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
- Environment variables
- Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
#AI #artificialIntelligence #configuration #development #ELK #FluentBit #Fluentd #LLM #observability #OpAMP #Technology -
Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
- When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
- OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
- The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
- Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{ "type": "object", "title": "Dummy", "properties": { "name": { "type": "string", "const": "dummy", "title": "Plugin" }, "copies": { "type": "integer", "description": "Number of messages to generate each time messages are generated.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "integer", "default": 1 }, "dummy": { "type": "string", "description": "Dummy JSON record.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "string", "default": "{\"message\":\"dummy\"}" }, "fixed_timestamp": { "type": "boolean", "description": "If enabled, use a fixed timestamp.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "boolean", "default": false } } }The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
- Environment variables
- Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
#AI #artificialIntelligence #configuration #development #ELK #FluentBit #Fluentd #LLM #observability #OpAMP #Technology -
Reduce developer friction – Configuring tools like Fluent Bit (and Fluentd)
Something that vendors like Microsoft have been really good at is reducing the friction on getting started – from simplifying installations with MSI files and defaulted options through to very informative error messages in Excel when you’ve got a function slightly wrong. Apple is another good example of this; while no two Android phones are the same, my experience is that setting up an iPhone is just so much easier than setting up an Android phone. It is also the setup/configuration where most friction comes from.
Open-Source Software (OSS), as a generalisation, tend to be a bit weaker at minimising friction – this comes from several factors:
- When OSS is part of a business model, vendors can reduce that friction, making their enhanced version more attractive.
- OSS contributors are typically focused on the core problem space and are usually close enough to the fine details to not need those fancy features to keep the rest of us out of trouble.
- The expectation is that tools to make configuration easy are embedded in the application, making it heavier, when the aim is to keep things as light as possible.
- Occasionally, a little bit of intellectual snobbery can creep in
The common challenge
The issue that I have observed is that we often go through cycles of working with a technology. For example, you’re building a microservice. Chances are, you’ll start writing and running it locally, without worrying about containerization. Once you’re pretty happy with things, you’ll Dockerize the service, start testing it locally, and then you’ll be ready to deploy it to a cluster. Now you’ll need your YAML. It may well be weeks since you last looked at Helm charts. You end up cutting and pasting your last configuration. But now you need to use another feature of Helm, can you remember the exact settings for the feature. So now you’re trawling the net for documentation, and then it takes several tries to get it right.
AI may well step in to help developers in this area, where solutions and products are well-documented. But with the wrong model or insufficient detail in the prompt, it’s easy to make a mistake. Personally, I’d turn to AI when it becomes necessary to trawl code to better understand the configuration and its behaviour, and to set options.
Experimental Solution
Solution – well, that depends upon the configuration syntax. We have been experimenting with RJSF (React JSON Schema Form), which provides a React-based UI that can be dynamically driven by a JSON schema and validate data with AJV (an alternative stack considered would have been around JSON Forms).
{ "type": "object", "title": "Dummy", "properties": { "name": { "type": "string", "const": "dummy", "title": "Plugin" }, "copies": { "type": "integer", "description": "Number of messages to generate each time messages are generated.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "integer", "default": 1 }, "dummy": { "type": "string", "description": "Dummy JSON record.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "string", "default": "{\"message\":\"dummy\"}" }, "fixed_timestamp": { "type": "boolean", "description": "If enabled, use a fixed timestamp.", "x-doc-reference": "https://docs.fluentbit.io/manual/data-pipeline/inputs/dummy#configuration-parameters", "x-doc-required": false, "x-config-data-type": "boolean", "default": false } } }The above fragment shows part of the Schema definition for the Dummy plugin for Fluent Bit.
By then creating a schema that defines the different plugins, attributes, etc., we can drive validation and menu items easily in the UI. Admittedly, the config file is significant given all the plugins and configuration options, but it is a fair price to pay for a UI that validates the data. Establishing the schema to start with, we’ve covered it through scripting the retrieval and scraping of the Fluent Bit pages, which are pretty consistent in structure.
We have added some custom elements into the definition, for example, x-doc-reference, which allows us to extend the React components to provide features such as a link back to the original documentation as you select attributes or plugins.
As a result, we very quickly have a UI that can look like this:
A lot easier to view and tweak, with no need to hunt for valid options. Even if we want more information, we’re just a button click away from the open-source data. Perhaps we should provide a version that hyperlinks to the Manning Live Books on Fluent Bit, etc.
There are a few other factors to consider; for example, Fluent Bit configuration is YAML, not JSON, which can be easily resolved given the relationship between the two standards. Then there are processors that can embed Lua code or a SQL-like syntax. As we’ve chosen to provide a Python backend, we’ve addressed this by providing REST endpoints which can query out of the JSON the code or SQL and perform validation using the Python Lua Parser, and the SQL syntax can be addressed using the Lark library for processing the SQL, as the syntax is simple enough to define and maintain the syntax.
Outstanding Gaps for Fluent Bit
We still need to address several features that Fluent Bit has, specifically:
- Environment variables
- Includes
These issues should be straightforward to overcome, although dynamically including the included elements into the UI view elements can be done. The challenge is: if any changes need to go into something that has been included, how do we push them back to the included file? Particularly if there are multiple layers of inclusion.
What about Fluentd?
Fluentd configuration isn’t JSON-based notation, but it is structured. So, to apply the same mechanism, we’ll need to define a schema and a mapping mechanism. The tricky part of the schema is that Fluentd supports nesting plugins, since the way pipelines are defined for routing differs. While JSON schema will enable this with constructs such as anyOf, oneOf, object nesting, and bounded object arrays, the structure will be more complex.
The second challenge will be the transformer/renderer, so we don’t introduce issues from having to escape and unescape characters, since JSON Schema is stricter about character use.
Then What?
Well, if we get this going, we’ll probably incorporate the capability into our OpAMP project and maybe create a build that lets the configuration tool run independently. Lastly, perhaps we should look to see if we can make the different layers a little more abstract, so we can plug in editors for other configurations, such as OTel Collectors or the ELK Stack.
As a bonus, perhaps transform the Schema into a quick reference web document?
#AI #artificialIntelligence #configuration #development #ELK #FluentBit #Fluentd #LLM #observability #OpAMP #Technology -
🎉 Ah, the classic tale of one man's noble quest to overcomplicate downloading #eBooks 🧙♂️. Spoofing browsers, deciphering JSON, and reconstructing SVGs, all to proudly declare that Amazon's app is a bit "meh" 📚🤦♂️. Just what we needed: a 900-word odyssey on how to read books the hard way! 🕵️♂️🔍
https://shkspr.mobi/blog/2025/10/improving-pixelmelts-kindle-web-deobfuscator/ #Overcomplication #TechTales #BrowserSpoofing #JSONDecoding #SVGReconstruction #ReadingChallenge #HackerNews #ngated -
🎉 Wow, they rewrote #JSONata with #AI in a single day and saved a fictional half-million dollars per year! 🙄 Meanwhile, #Reco raised $85M to market their buzzword salad of security features that everyone will forget about by next week. 🥳✨
https://www.reco.ai/blog/we-rewrote-jsonata-with-ai #Rewriting #Funding #TechBuzz #HackerNews #ngated -
Widlet — pet-проект про Server-Driven UI на Dart
Привет, Хабр. Меня зовут Карим, я Flutter разработчик уже 7 лет и последний месяц я делаю фреймворк для server-driven UI на Dart. Зачем еще один SDUI У всех реализаций, которые попадались мне на глаза, есть общая черта - собственный DSL. JSON-схемы, кастомные конфиги. Для каждого решения приходится учить новый язык. При этом Flutter-разработчики уже знают хороший язык описания UI. Он называется Flutter.
https://habr.com/ru/articles/1033978/
#sdui #wasm #flutter #dart #widlet #микрофронтенд #jsinterop