#parsing — Public Fediverse posts on home.social

InnocentZero @[email protected] · 2026-07-30 · 13:32 UTC

Sigh... I guess I need to ask this now

So #rust programmers, how does one do errors in lossless parsing?

Context

I'm working on the new nu parser for nushell. The current strategy is to have a giant vector of so called NodeId s which in turn refer other indices within that vector, and ultimately build a parse "tree" using those NodeId (variants in the AstNode may contain other metadata too)

Now, we just created a dummy NodeId for errors and pushed them and continued to parse on our way. We want to have error-resilient parsing, so that we can cover more errors at other locations. If something's badly borked, well, we just barf.

So this worked well for us devs, because it kept things simple easy, but this is not how I'd really like to build the parse tree. There are multiple problems.

Right off the bat, we lose all the type information in one big ball of AstNode enum vector. So even though there are some places where you know that an AstNode would be of a particular kind, you simply cannot do anything about it except explicitly match and check (we want to avoid unsafe memory reinterprets).

As an aside, most of the enum is empty, but due to some variants, the entire enum becomes 48 bytes, so we waste a lot of memory on basically nothing.

My solution to this was to use bumpalo to do allocations and store references, and let go of the vector altogether and build an actual parse tree (like, a node with children, which have further children, and so on)

This is basically doing the vector, but not wasting space on the emptiness of the enums, not lose the types at runtime to the enum, and not have to do hacky index-chasing (which effectively act like pointers now). Instead, we use actual references, which we know will be valid till the end of the arena lifetime (three cheers for bumpalo!)

The Problem

As mentioned before, we had NodeId for errors as well, and those had a dummy span. Now, with the existence of a concrete parse tree, we have... no such thing. So we lose out on the ease of creating an error and attaching it wherever we want. In order to do things the earlier way, we have to make everything into an Result<insert_type_here, Error> which is very, very bad, because with that comes the indirect habit of using ? to bubble upwards, which is a problem because we want to find as big of a parsed expression with as small of an error context (we basically want to minimize how much we weren't able to parse, because that's what any good parser should do)

Right now, I don't really have any concrete ideas to attach errors to the nodes in the parse tree. So finally

Questions

How does one do it? I'm looking for suggestions, although I don't really have anything concrete in my mind
Would this actually lead to better performance? I expect it to be both faster and cheaper resource-wise. I'd argue yes, but I have no idea since it's a massive rewrite and doesn't even compile yet (:

Code snippets

// NOTE(bumpalo_rewrite): This becomes the root of the AST
// TODO(bumpalo_rewrite): Change to the generic enum that block may contain
#[derive(Debug, Clone)]
pub struct Block<'a> {
    pub span_start: usize,
    pub span_end: usize,
    pub nodes: Vec<BlockEntities<'a>>,
}

// TODO(bumpalo_rewrite): Fill with all the possible BlockEntities
// NOTE(bumpalo_rewrite): See Parser::block() for the entity types
#[derive(Debug, Clone)]
pub enum BlockEntities<'a> {
    Def(&'a Def<'a>),
    Let(&'a Let<'a>),
    While(&'a While<'a>),
    For(&'a For<'a>),
    Loop(&'a Loop<'a>),
    Return(&'a Return<'a>),
    Continue(&'a Continue),
    Break(&'a Break),
    Alias(&'a Alias),
    Extern(&'a Extern),
    PipelineOrExprOrAssign(PipelineOrExprOrAssign<'a>),
    Statement(PipelineOrExprOrAssign<'a>),
}

This is kind of what it is like right now. The function that parses a block:

            else if self.is_keyword(b"while") {
                match self.while_statement(arena) {
                    Some(while_) => {
                        code_body.push(BlockEntities::While(while_));
                    }
                    None => {}
                };
            }

has cases like these, where the error context is lost (ignore using Option instead of Result, I'm prototyping null

So really, how do I preserve the error contexts?

Git Repo

Here. You'd want to mostly look through the diff between this commit and the previous one in src/parser.rs

Any help is appreciated because I'm close to losing my mind lol XD

#programming #parsing #rust #errortolerance #compilers #parsers

#rust #programming #parsing #errortolerance #compilers #parsers

InnocentZero @[email protected] · 2026-07-30 · 13:32 UTC

Sigh... I guess I need to ask this now

So #rust programmers, how does one do errors in lossless parsing?

Context

I'm working on the new nu parser for nushell. The current strategy is to have a giant vector of so called NodeId s which in turn refer other indices within that vector, and ultimately build a parse "tree" using those NodeId (variants in the AstNode may contain other metadata too)

Now, we just created a dummy NodeId for errors and pushed them and continued to parse on our way. We want to have error-resilient parsing, so that we can cover more errors at other locations. If something's badly borked, well, we just barf.

So this worked well for us devs, because it kept things simple easy, but this is not how I'd really like to build the parse tree. There are multiple problems.

Right off the bat, we lose all the type information in one big ball of AstNode enum vector. So even though there are some places where you know that an AstNode would be of a particular kind, you simply cannot do anything about it except explicitly match and check (we want to avoid unsafe memory reinterprets).

As an aside, most of the enum is empty, but due to some variants, the entire enum becomes 48 bytes, so we waste a lot of memory on basically nothing.

My solution to this was to use bumpalo to do allocations and store references, and let go of the vector altogether and build an actual parse tree (like, a node with children, which have further children, and so on)

This is basically doing the vector, but not wasting space on the emptiness of the enums, not lose the types at runtime to the enum, and not have to do hacky index-chasing (which effectively act like pointers now). Instead, we use actual references, which we know will be valid till the end of the arena lifetime (three cheers for bumpalo!)

The Problem

As mentioned before, we had NodeId for errors as well, and those had a dummy span. Now, with the existence of a concrete parse tree, we have... no such thing. So we lose out on the ease of creating an error and attaching it wherever we want. In order to do things the earlier way, we have to make everything into an Result<insert_type_here, Error> which is very, very bad, because with that comes the indirect habit of using ? to bubble upwards, which is a problem because we want to find as big of a parsed expression with as small of an error context (we basically want to minimize how much we weren't able to parse, because that's what any good parser should do)

Right now, I don't really have any concrete ideas to attach errors to the nodes in the parse tree. So finally

Questions

How does one do it? I'm looking for suggestions, although I don't really have anything concrete in my mind
Would this actually lead to better performance? I expect it to be both faster and cheaper resource-wise. I'd argue yes, but I have no idea since it's a massive rewrite and doesn't even compile yet (:

Code snippets

// NOTE(bumpalo_rewrite): This becomes the root of the AST
// TODO(bumpalo_rewrite): Change to the generic enum that block may contain
#[derive(Debug, Clone)]
pub struct Block<'a> {
    pub span_start: usize,
    pub span_end: usize,
    pub nodes: Vec<BlockEntities<'a>>,
}

// TODO(bumpalo_rewrite): Fill with all the possible BlockEntities
// NOTE(bumpalo_rewrite): See Parser::block() for the entity types
#[derive(Debug, Clone)]
pub enum BlockEntities<'a> {
    Def(&'a Def<'a>),
    Let(&'a Let<'a>),
    While(&'a While<'a>),
    For(&'a For<'a>),
    Loop(&'a Loop<'a>),
    Return(&'a Return<'a>),
    Continue(&'a Continue),
    Break(&'a Break),
    Alias(&'a Alias),
    Extern(&'a Extern),
    PipelineOrExprOrAssign(PipelineOrExprOrAssign<'a>),
    Statement(PipelineOrExprOrAssign<'a>),
}

This is kind of what it is like right now. The function that parses a block:

            else if self.is_keyword(b"while") {
                match self.while_statement(arena) {
                    Some(while_) => {
                        code_body.push(BlockEntities::While(while_));
                    }
                    None => {}
                };
            }

has cases like these, where the error context is lost (ignore using Option instead of Result, I'm prototyping null

So really, how do I preserve the error contexts?

Git Repo

Here. You'd want to mostly look through the diff between this commit and the previous one in src/parser.rs

Any help is appreciated because I'm close to losing my mind lol XD

#programming #parsing #rust #errortolerance #compilers #parsers

#rust #programming #parsing #errortolerance #compilers #parsers

athmane mokraoui [BoF] ⏚ꝃñ⌁⁂ @[email protected] · 2026-07-23 · 08:21 UTC

PR pour rendre meilleure la prise en charge du kabyle sur ce « number parser » d'Open Voice OS.

Faut savoir que même `libnumbertext` ne prend pas en charge le système de numération en kabyle. J'ai une version localement qui le supporte.

En attendant :

https://github.com/OpenVoiceOS/ovos-number-parser/pull/282

#kabyle #numbers #numerals #parsing

#parsing #numerals #numbers #kabyle

athmane mokraoui [BoF] ⏚ꝃñ⌁⁂ @[email protected] · 2026-07-23 · 08:21 UTC

PR pour rendre meilleure la prise en charge du kabyle sur ce « number parser » d'Open Voice OS.

Faut savoir que même `libnumbertext` ne prend pas en charge le système de numération en kabyle. J'ai une version localement qui le supporte.

En attendant :

https://github.com/OpenVoiceOS/ovos-number-parser/pull/282

#kabyle #numbers #numerals #parsing

#parsing #numerals #numbers #kabyle

Habr @[email protected] · 2026-06-19 · 07:42 UTC

Как я случайно написал что-то быстрое и декларативное (на Rust)

Писал парсер строго под свой проект, а получился быстрый декларативный движок для парсинга текстовых форматов. Как?

https://habr.com/ru/articles/1049432/

#rust #parsing #engine #declarative #fast #парсер #парсинг #работа_с_текстом #работа_с_текстовыми_данными

#работа_с_текстовыми_данными #работа_с_текстом #парсинг #парсер #fast #declarative

Sascha Wolfer @[email protected] · 2026-06-13 · 14:26 UTC

Okay, es ist mal wieder Psycholinguistik-Time, weil #Psycholinguistik geilste #Linguistik!

Lest mal den folgenden Satz:

"Dass Nagelsmann zugunsten von Sané nie etwas unternommen worden wäre, ist eine Lüge."

Ich gebe zu, der ist ein bisschen kompliziert, und vielleicht habt Ihr gedacht: "Müsste da statt 'worden wäre' nicht 'hat' stehen?" - Ja, kann es, aber es ist auch so ein auflösbarer ("richtiger") Satz.

Interessant ist aber, dass es in der Form oben ein #Holzweg-Satz (Engl. #gardenpath) ist. Die heißen deshalb so, weil sie einen bei der Satzinterpretation (Engl. #parsing) in die Irre führen: Die meisten Leute denken beim ersten Lesen, dass Nagelsmann die handelnde Person (Agens) beim "unternehmen" ist. In Wahrheit ist es aber Sané – und das wird klar, wenn man den Satz entsprechend betont. Liegt die Betonung auf Nagelsmann, wird die Interpretation einfacher.

#psycholinguistik #linguistik #holzweg #gardenpath #parsing

Sascha Wolfer @[email protected] · 2026-06-13 · 14:26 UTC

Okay, es ist mal wieder Psycholinguistik-Time, weil #Psycholinguistik geilste #Linguistik!

Lest mal den folgenden Satz:

"Dass Nagelsmann zugunsten von Sané nie etwas unternommen worden wäre, ist eine Lüge."

Ich gebe zu, der ist ein bisschen kompliziert, und vielleicht habt Ihr gedacht: "Müsste da statt 'worden wäre' nicht 'hat' stehen?" - Ja, kann es, aber es ist auch so ein auflösbarer ("richtiger") Satz.

Interessant ist aber, dass es in der Form oben ein #Holzweg-Satz (Engl. #gardenpath) ist. Die heißen deshalb so, weil sie einen bei der Satzinterpretation (Engl. #parsing) in die Irre führen: Die meisten Leute denken beim ersten Lesen, dass Nagelsmann die handelnde Person (Agens) beim "unternehmen" ist. In Wahrheit ist es aber Sané – und das wird klar, wenn man den Satz entsprechend betont. Liegt die Betonung auf Nagelsmann, wird die Interpretation einfacher.

#psycholinguistik #linguistik #holzweg #gardenpath #parsing

Habr @[email protected] · 2026-05-28 · 14:02 UTC

Шифрование на уровне протокола

Как организовать шифрование на уровне протокола? На самом деле тема непростая и пожалуй (имхо) это как раз та самая тема, где прийти к компромиссу почти никогда не получается. Разве что просто не передавать чувствительные данные вовсе. Я расскажу как шифрование можно организовать на уровне протокола brec и ни в коем случае не буду затрагивать те самые принципиальные решения, влияющие на безопасность (как передавать, куда передавать, отправлять ли, и хранить ли чувствительные данные вовсе). Иными словами нас интересует инструментальная сторона вопроса.

https://habr.com/ru/articles/1040298/

#rust #protocols #parsing #binary #communication #no_database #storage #binary_storage #filtering #crypt

#crypt #filtering #binary_storage #storage #no_database #communication

N-gated Hacker News @[email protected] · 2026-05-27 · 09:30 UTC

🎉 Behold the ultimate #toolkit for #nerds who find #parsing MPEG-TS streams exhilarating! #TSDuck presents a dizzying array of standards and protocols, because nothing screams #fun like endless acronyms and a PDF bonanza 📚. Dive in, if you dare, and unleash your inner transport stream aficionado! 😂
https://tsduck.io/ #MPEGTS #TransportStream #HackerNews #ngated

#toolkit #nerds #parsing #tsduck #fun #mpegts

N-gated Hacker News @[email protected] · 2026-05-27 · 09:30 UTC

🎉 Behold the ultimate #toolkit for #nerds who find #parsing MPEG-TS streams exhilarating! #TSDuck presents a dizzying array of standards and protocols, because nothing screams #fun like endless acronyms and a PDF bonanza 📚. Dive in, if you dare, and unleash your inner transport stream aficionado! 😂
https://tsduck.io/ #MPEGTS #TransportStream #HackerNews #ngated

#toolkit #nerds #parsing #tsduck #fun #mpegts

GOMOOT :mastodon: @[email protected] · 2026-05-12 · 12:42 UTC

💡 Docling + Ollama + Qdrant: costruisci una knowledge base privata con Qwen3.6 in locale
https://gomoot.com/docling-la-libreria-open-source-per-il-pdf-parsing-nelle-pipeline-rag-con-ollama-e-qdrant/

#docling #ollama #parsing #pdf #qdrant #rag

Frontend Dogma @[email protected] · 2026-04-23 · 13:15 UTC

Parse, Don’t Validate—in a Language That Doesn’t Want You To, by (not on Mastodon or Bluesky):

https://cekrem.github.io/posts/parse-dont-validate-typescript/

#parsing #validation #programming #typescript #typesafety

#typesafety #typescript #programming #validation #parsing

Frontend Dogma @[email protected] · 2026-04-23 · 13:15 UTC

Parse, Don’t Validate—in a Language That Doesn’t Want You To, by (not on Mastodon or Bluesky):

https://cekrem.github.io/posts/parse-dont-validate-typescript/

#parsing #validation #programming #typescript #typesafety

#typesafety #typescript #programming #validation #parsing

Ivan Enderlin 🦀 @[email protected] · 2026-04-20 · 06:36 UTC

The fastest way to match characters on ARM processors?, https://lemire.me/blog/2026/04/19/the-fastest-way-to-match-characters-on-arm-processors/.

In this article, Lemir talks about two SIMD ARM SVE/SVE2 instructions: `match` and `nmatch`, which fit nicely in the _vectorized classification_ step of `simdjson`. These instructions improve the performance of `simdjson` from 11.4Gb/s to 14.4Gb/s.

#performance #simd #arm #json #parsing

Ivan Enderlin 🦀 @[email protected] · 2026-04-20 · 06:36 UTC

The fastest way to match characters on ARM processors?, https://lemire.me/blog/2026/04/19/the-fastest-way-to-match-characters-on-arm-processors/.

In this article, Lemir talks about two SIMD ARM SVE/SVE2 instructions: `match` and `nmatch`, which fit nicely in the _vectorized classification_ step of `simdjson`. These instructions improve the performance of `simdjson` from 11.4Gb/s to 14.4Gb/s.

#performance #simd #arm #json #parsing

Snapp iOS Weekly @[email protected] · 2026-04-07 · 06:00 UTC

How do you build a complex parser without it becoming a mess? The swift-parsing library from Point-Free lets you compose smaller, focused parsers into powerful pipelines using the Parse block and map modifiers.

🔗: https://swiftdevjournal.com/posts/composing-parsers/ by Mark Szymczyk

#Swift #Parsing #iOSDev

#swift #parsing #iosdev

Barry Schwartz 🫖 @[email protected] · 2026-04-03 · 21:53 UTC

I do not see why a recursive-descent parser cannot have numeric precedences like a Pratt parser. You just move the order of productions freely, and let any production call any other.

Am I missing something?

#parsing #programming

Barry Schwartz 🫖 @[email protected] · 2026-04-03 · 21:53 UTC

I do not see why a recursive-descent parser cannot have numeric precedences like a Pratt parser. You just move the order of productions freely, and let any production call any other.

Am I missing something?

#parsing #programming

Frontend Dogma @[email protected] · 2026-04-03 · 19:15 UTC

Rewriting Our Rust Wasm Parser in TypeScript, by (not on Mastodon or Bluesky):

https://www.openui.com/blog/rust-wasm-parser

#migrating #parsing #rust #typescript

#typescript #rust #parsing #migrating