#jsonl — Public Fediverse posts

Declarative all the way down: Building PRONOM signatures with JSONID

PRONOM signatures are a form of declarative language, you describe the anticipated behavior in PRONOM’s regular expression syntax and tools like DROID, FIDO, and Siegfried will interpret those instructions and attempt to match them against different files to return a file format identification.

Normally, you will write PRONOM signatures by hand but doing so for file formats based on other file format building blocks can lead to inconsistencies. Bertrand Caron previously also recognized this in the XML formats that are described with PRONOM signatures on Wikidata.

XML can use single quotes ‘ (hex: 0x27) and double quotes ” (hex: 0x22) for attribute data, and so, do we make a PRONOM signature with multiple sequences anticipating the use of either?

The answer is more often than not likely to be yes, because the appearance of these values are often helpful for identifying boundaries for strings that we know must exist.

But the more file formats that we need to add to PRONOM that are based on foundational formats like XML, or JSON, or similar, the more inconsistencies will creep in, such as sequences that are looking specifically for one byte sequence over another.

The issue extends further if file formats allow data to appear at the beginning of file, or we need to account for a variable amount of white-space, or we want to start thinking about multi-byte character encoding.

We can, and in the XML issue described by myself and Caron, I think the recommendation is very much to create editorial standards for signatures for file formats based on other baseline, structured data formats like XML, JSON, YAML, and so on.

And standards are well and good, but what if tooling could help us?

For JSON this is exactly what I have tried to do in JSONID.

What does this feature look like? And what does it get us? Let’s take a look.

Declarative all the way down: Building PRONOM signatures with JSONID

PRONOM signatures are a form of declarative language, you describe the anticipated behavior in PRONOM’s regular expression syntax and tools like DROID, FIDO, and Siegfried will interpret those instructions and attempt to match them against different files to return a file format identification.

Normally, you will write PRONOM signatures by hand but doing so for file formats based on other file format building blocks can lead to inconsistencies. Bertrand Caron previously also recognized this in the XML formats that are described with PRONOM signatures on Wikidata.

XML can use single quotes ‘ (hex: 0x27) and double quotes ” (hex: 0x22) for attribute data, and so, do we make a PRONOM signature with multiple sequences anticipating the use of either?

The answer is more often than not likely to be yes, because the appearance of these values are often helpful for identifying boundaries for strings that we know must exist.

But the more file formats that we need to add to PRONOM that are based on foundational formats like XML, or JSON, or similar, the more inconsistencies will creep in, such as sequences that are looking specifically for one byte sequence over another.

The issue extends further if file formats allow data to appear at the beginning of file, or we need to account for a variable amount of white-space, or we want to start thinking about multi-byte character encoding.

We can, and in the XML issue described by myself and Caron, I think the recommendation is very much to create editorial standards for signatures for file formats based on other baseline, structured data formats like XML, JSON, YAML, and so on.

And standards are well and good, but what if tooling could help us?

For JSON this is exactly what I have tried to do in JSONID.

What does this feature look like? And what does it get us? Let’s take a look.

I had a bunch of JSONL files with sensor readings and needed a quick way to check field cardinality and fill rates, filter records, and look up specific entries. Sure, I could have used Databricks, but where’s the fun in that? 😉

So I ended up building a TUI for viewing and filtering JSONL data, with some basic statistics included.

https://github.com/jsynowiec/tav

#textual #textualize #python #tui #jsonl #dataEngineering #timeSeries

I had a bunch of JSONL files with sensor readings and needed a quick way to check field cardinality and fill rates, filter records, and look up specific entries. Sure, I could have used Databricks, but where’s the fun in that? 😉

So I ended up building a TUI for viewing and filtering JSONL data, with some basic statistics included.

https://github.com/jsynowiec/tav

#textual #textualize #python #tui #jsonl #dataEngineering #timeSeries

Как я перестал бояться и полюбил MCP или Зачем я написал 4 MCP‑сервера и протокол для их связи

Привет! Вы знаете как это бывает — начинаешь делать одну штуку, а потом просыпаешься через неделю и понимаешь, что написал четыре MCP‑сервера, подключил к ним шедулер, собрал автоматический конвеер для трёх Telegram‑каналов и изобрёл собственную спецификацию для связывания всего этого добра. Классика. Для тех кто не в теме: MCP (Model Context Protocol) — это протокол, через который AI‑ассистенты типа Claude подключаются к внешним сервисам и работают с ними напрямую. По сути это «руки» для нейросетей. Подключил MCP — и ИИ сам ходит в Telegram, ищет лучшие картинки с промптами на Civitai, управляет рекламой в Яндекс.Директе и делает кучу всего полезного. Без костылей, без скриптов‑прослоек, напрямую. В этой статье расскажу почему готовые MCP‑серверы меня не устроили, какие задачи я сейчас решаю с помощью своих, и зачем мне пришлось изобрести целый протокол чтобы эти серверы начали видеть друг друга.

https://habr.com/ru/articles/1019652/

#MCP #Model_Context_Protocol #TRAIL #Claude #AIагенты #автоматизация #оркестрация #нейросети #playwright #JSONL

✨Oh, great! Another revolutionary tool to dig through your digital trash can, aka ~/.claude sessions, to unearth files you probably didn't even want to keep. 🚀 Because nothing says "efficient" like #JSONL transcripts and a #GitHub repo that's as useful as a screen door on a submarine. 🐙🔍
https://github.com/hjtenklooster/claude-file-recovery #digitaltools #innovation #techhumor #dataorganization #HackerNews #ngated

✨Oh, great! Another revolutionary tool to dig through your digital trash can, aka ~/.claude sessions, to unearth files you probably didn't even want to keep. 🚀 Because nothing says "efficient" like #JSONL transcripts and a #GitHub repo that's as useful as a screen door on a submarine. 🐙🔍
https://github.com/hjtenklooster/claude-file-recovery #digitaltools #innovation #techhumor #dataorganization #HackerNews #ngated

#成年後見制度 の闇をGPTデータに加えるため、#OpenAi にフォードバックを送った。#jsonl #ChatGPT

https://chatgpt.com/share/68c87bef-9528-800b-ae36-b8c1a1ee9671

🔍 So, you've built a tool to analyze your own coding insecurities, but disguised it as a #JSONL file inspector? 😂 Bravo! 🚀 Meanwhile, #GitHub is still trying to convince you that #AI will fix your spaghetti code. 🍝🤖
https://github.com/ryoppippi/ccusage #codinginsecurities #spaghettiCode #developerhumor #HackerNews #ngated

🔍 So, you've built a tool to analyze your own coding insecurities, but disguised it as a #JSONL file inspector? 😂 Bravo! 🚀 Meanwhile, #GitHub is still trying to convince you that #AI will fix your spaghetti code. 🍝🤖
https://github.com/ryoppippi/ccusage #codinginsecurities #spaghettiCode #developerhumor #HackerNews #ngated

Ccusage: A CLI tool for analyzing Claude Code usage from local JSONL files

https://github.com/ryoppippi/ccusage

#HackerNews #Ccusage #CLI #tool #JSONL #analysis #ClaudeCode #techtools

Ccusage: A CLI tool for analyzing Claude Code usage from local JSONL files

https://github.com/ryoppippi/ccusage

#HackerNews #Ccusage #CLI #tool #JSONL #analysis #ClaudeCode #techtools

Prepping data for AI feels a bit like meal prep…
Except your dinner is a .jsonl file and your customer is a llama. 🐪

I’m learning the ropes of AI training, starting with how to turn raw .txt into structured data for LLMs.

🎥 How to Feed Your Llama is live:
https://www.youtube.com/watch?v=YB9cVyjV9Bo

#AI #LLM #JSONL #FineTuning #Python #LearningInPublic #LLaMA #Mistral

Prepping data for AI feels a bit like meal prep…
Except your dinner is a .jsonl file and your customer is a llama. 🐪

I’m learning the ropes of AI training, starting with how to turn raw .txt into structured data for LLMs.

🎥 How to Feed Your Llama is live:
https://www.youtube.com/watch?v=YB9cVyjV9Bo

#AI #LLM #JSONL #FineTuning #Python #LearningInPublic #LLaMA #Mistral

@Taffer There used to be #NeDB back when #nosql was all the rage (basically stores data in a plaintext #jsonl file). But it hasn't been updated for 10y. There's a fork though: https://github.com/seald/nedb

#sqlite #mongodb

@Taffer There used to be #NeDB back when #nosql was all the rage (basically stores data in a plaintext #jsonl file). But it hasn't been updated for 10y. There's a fork though: https://github.com/seald/nedb

#sqlite #mongodb

The #wasm build of #sqlite opens up great new possibilities in the #browser, especially when coupled with the Origin Private File System (#opfs). I've used it to implement importing huge #csv or #jsonl #datasets right in the browser. Import, validate, search, edit, close the browser and continue tomorrow, stream the database (with on-the-fly compression!) to the server when ready – it's all possible!

Another interesting use case I came across today: use it in #golang to get rid of #cgo!

[1/2]

The #wasm build of #sqlite opens up great new possibilities in the #browser, especially when coupled with the Origin Private File System (#opfs). I've used it to implement importing huge #csv or #jsonl #datasets right in the browser. Import, validate, search, edit, close the browser and continue tomorrow, stream the database (with on-the-fly compression!) to the server when ready – it's all possible!

Another interesting use case I came across today: use it in #golang to get rid of #cgo!

[1/2]

Fed up with calculating dataset splits (e.g. train, validation, test, dev, silly, etc) for multiple classes to make sure they're balanced? Me too.

I built a tool to help me:

https://sbrl.github.io/research-smflooding/dataset-split-calculator.html

Put 1 integer value per line.

It even spits out shell commands to cut lines-based files (e.g. jsonl, csv, etc) into separate files!

May write a proper blog post soon!

#AI #DataScience #BigData #Automation #JSONL #CSV #Bash / #Shell #Scripts #AreAwesome

Today's winning package of the day is DEFINITELY `jq` for easy conversion of #jsonl to standard #json at the command line.

https://zxvf.org/post/jq-as-grep/

Today's winning package of the day is DEFINITELY `jq` for easy conversion of #jsonl to standard #json at the command line.

https://zxvf.org/post/jq-as-grep/

I just came across a file format called #ndjson - just a few months after I read about #jsonl - and I thought, seriously? I've been using this format for years, and now all of a sudden it has two different names, within months of each other?

Sometimes they say things come in threes...

I just came across a file format called #ndjson - just a few months after I read about #jsonl - and I thought, seriously? I've been using this format for years, and now all of a sudden it has two different names, within months of each other?

Sometimes they say things come in threes...

Q: Can I fine-tune #ChatGPT using #SPARQL such that it basically becomes a #SmartAgent capable of querying across the massive #LODCloud #KnowledgeGraph, using natural language?

Yes!

Read: https://community.openlinksw.com/t/howto-chat-gpt-and-dbpedia-sparql-query-generation-from-natural-language-prompts/3823 -- which covers a basic example using #DBpedia (#Wikidata content in machine-computable and queryable form).

#SemanticWeb #LinkedData #JSONL #RDF #LLMs #DBpedia #VirtuosoRDBMS #JSONL

Q: Can I fine-tune #ChatGPT using #SPARQL such that it basically becomes a #SmartAgent capable of querying across the massive #LODCloud #KnowledgeGraph, using natural language?

Yes!

Read: https://community.openlinksw.com/t/howto-chat-gpt-and-dbpedia-sparql-query-generation-from-natural-language-prompts/3823 -- which covers a basic example using #DBpedia (#Wikidata content in machine-computable and queryable form).

#SemanticWeb #LinkedData #JSONL #RDF #LLMs #DBpedia #VirtuosoRDBMS #JSONL

TIL about https://jsonlines.org/ , a newline-delimited JSON format that _jq_ understands as input. Looks much better than CSV for when you need a line-oriented structured text format.
#json #jq #jsonl

TIL about https://jsonlines.org/ , a newline-delimited JSON format that _jq_ understands as input. Looks much better than CSV for when you need a line-oriented structured text format.
#json #jq #jsonl

I never bothered with optimizing the parsing of #jsonl #ndjson files because in most cases it was an one off task before I put the data into a database or parquet file. But the files got bigger and waiting 20 minutes for the data to load made me reconsider my decision. So tried some different approaches.

Tested with a 1.7 GB file of 300 k Tweets.

jsonlines.reader: 17.2 seconds 100%
orjson: 6.49 seconds 37%
msgspec: 3.06 seconds 17%

I like how orjson cuts the time by two thirds without the need to change anything else. Just use it as a drop in replacement and you are good.

msgspec is twice as fast as orjson or six times as fast as jsonlines if you define the schema of the data that you want. For Tweets that's okay, as I can reuse the schema many times. With data that is used only once, I prefer orjson.

Memory usage was nearly identical across the different solutions. Probably because they all parse the data per line. I restarted the kernel each time to get comparable numbers.

Load time for all 23 million Tweets in the dataset was reduced from 25 to 4 minutes.

This blogposts was useful to me: https://pythonspeed.com/articles/faster-python-json-parsing/ #Python #DataEngineering

I never bothered with optimizing the parsing of #jsonl #ndjson files because in most cases it was an one off task before I put the data into a database or parquet file. But the files got bigger and waiting 20 minutes for the data to load made me reconsider my decision. So tried some different approaches.

Tested with a 1.7 GB file of 300 k Tweets.

jsonlines.reader: 17.2 seconds 100%
orjson: 6.49 seconds 37%
msgspec: 3.06 seconds 17%

I like how orjson cuts the time by two thirds without the need to change anything else. Just use it as a drop in replacement and you are good.

msgspec is twice as fast as orjson or six times as fast as jsonlines if you define the schema of the data that you want. For Tweets that's okay, as I can reuse the schema many times. With data that is used only once, I prefer orjson.

Memory usage was nearly identical across the different solutions. Probably because they all parse the data per line. I restarted the kernel each time to get comparable numbers.

Load time for all 23 million Tweets in the dataset was reduced from 25 to 4 minutes.

This blogposts was useful to me: https://pythonspeed.com/articles/faster-python-json-parsing/ #Python #DataEngineering