#ndjson — Public Fediverse posts

Как стримить данные в ASP.NET и как их принять

Недавно мне попалась отличная статья про IAsyncEnumerable и стриминг данных . В ней у автора упал прод, который пытался выдать 500 000 записей разом и упал на вызове ToListAsync() с OOM при 8 ГБ RAM. Далее в статье описывается, как все это стримить с помощью IAsyncEnumerable с примерами кода. В целом после прочтения статьи может сложиться впечатление, что все свои ToListAsync() срочно нужно убрать и заменить на стриминг. Но со времен появления стримингового апи в .NET мне всегда было интересно, не только то как отдавать стримы, но и как это все получать на клиенте?

https://habr.com/ru/articles/1052622/

#IAsyncEnumerable #streaming #SSE #NDJSON #ASPNET_Core #EF_Core #Kestrel #производительность #net_10

Как стримить данные в ASP.NET и как их принять

Недавно мне попалась отличная статья про IAsyncEnumerable и стриминг данных . В ней у автора упал прод, который пытался выдать 500 000 записей разом и упал на вызове ToListAsync() с OOM при 8 ГБ RAM. Далее в статье описывается, как все это стримить с помощью IAsyncEnumerable с примерами кода. В целом после прочтения статьи может сложиться впечатление, что все свои ToListAsync() срочно нужно убрать и заменить на стриминг. Но со времен появления стримингового апи в .NET мне всегда было интересно, не только то как отдавать стримы, но и как это все получать на клиенте?

https://habr.com/ru/articles/1052622/

#IAsyncEnumerable #streaming #SSE #NDJSON #ASPNET_Core #EF_Core #Kestrel #производительность #net_10

Как стримить данные в ASP.NET и как их принять

Недавно мне попалась отличная статья про IAsyncEnumerable и стриминг данных . В ней у автора упал прод, который пытался выдать 500 000 записей разом и упал на вызове ToListAsync() с OOM при 8 ГБ RAM. Далее в статье описывается, как все это стримить с помощью IAsyncEnumerable с примерами кода. В целом после прочтения статьи может сложиться впечатление, что все свои ToListAsync() срочно нужно убрать и заменить на стриминг. Но со времен появления стримингового апи в .NET мне всегда было интересно, не только то как отдавать стримы, но и как это все получать на клиенте?

https://habr.com/ru/articles/1052622/

#IAsyncEnumerable #streaming #SSE #NDJSON #ASPNET_Core #EF_Core #Kestrel #производительность #net_10

Blogged: "ASP​.NET Core 9 and IAsyncEnumerable - Async Streaming JSON and NDJSON From Blazor WebAssembly"

(https://www.tpeczek.com/2024/09/aspnet-core-9-and-iasyncenumerable.html)

#AspNetCore #Blazor #AsyncStreaming #JSON #NDJSON

I just came across a file format called #ndjson - just a few months after I read about #jsonl - and I thought, seriously? I've been using this format for years, and now all of a sudden it has two different names, within months of each other?

Sometimes they say things come in threes...

I just came across a file format called #ndjson - just a few months after I read about #jsonl - and I thought, seriously? I've been using this format for years, and now all of a sudden it has two different names, within months of each other?

Sometimes they say things come in threes...

zeehaven – a tiny tool to convert data for social media research: https://publicdatalab.org/2023/12/18/zeehaven-social-media-data/

drag and drop ndjson data gathered with @dmi's zeeschuimer tool, and download a csv file. ✨📦✨

#digitalmethods #csv #ndjson #tinytools #osint #newmedia #commodon #mediastudies #data #dataviz

zeehaven – a tiny tool to convert data for social media research: https://publicdatalab.org/2023/12/18/zeehaven-social-media-data/

drag and drop ndjson data gathered with @dmi's zeeschuimer tool, and download a csv file. ✨📦✨

#digitalmethods #csv #ndjson #tinytools #osint #newmedia #commodon #mediastudies #data #dataviz

My family of packages for streaming JSON objects with NDJSON over RESTful APIs in the .NET ecosystem gained a new member - one that brings support for ASP​.NET Core Minimal APIs 🎉

#AspNetCore #MinimalApi #AsyncStreams #NDJSON

https://github.com/tpeczek/Ndjson.AsyncStreams/releases/tag/aspnetcore-v1.0.0

My family of packages for streaming JSON objects with NDJSON over RESTful APIs in the .NET ecosystem gained a new member - one that brings support for ASP​.NET Core Minimal APIs 🎉

#AspNetCore #MinimalApi #AsyncStreams #NDJSON

https://github.com/tpeczek/Ndjson.AsyncStreams/releases/tag/aspnetcore-v1.0.0

My family of packages for streaming JSON objects with NDJSON over RESTful APIs in the .NET ecosystem gained a new member - one that brings support for ASP​.NET Core Minimal APIs 🎉

#AspNetCore #MinimalApi #AsyncStreams #NDJSON

https://github.com/tpeczek/Ndjson.AsyncStreams/releases/tag/aspnetcore-v1.0.0

My family of packages for streaming JSON objects with NDJSON over RESTful APIs in the .NET ecosystem gained a new member - one that brings support for ASP​.NET Core Minimal APIs 🎉

#AspNetCore #MinimalApi #AsyncStreams #NDJSON

https://github.com/tpeczek/Ndjson.AsyncStreams/releases/tag/aspnetcore-v1.0.0

AspNetCore.JsonStreamer 0.2.0 is out now!

JSON Lines streaming serializer on ASP.NET Core, uses standard type of IAsyncEnumerable<T>.

It is simply to use, add AddNewtonsoftJsonStreamer() instead of AddNewtonsoftJson().

https://github.com/kekyo/AspNetCore.JsonStreamer

#dotnet #aspnetcore #jsonlines #ndjson

AspNetCore.JsonStreamer 0.2.0 is out now!

JSON Lines streaming serializer on ASP.NET Core, uses standard type of IAsyncEnumerable<T>.

It is simply to use, add AddNewtonsoftJsonStreamer() instead of AddNewtonsoftJson().

https://github.com/kekyo/AspNetCore.JsonStreamer

#dotnet #aspnetcore #jsonlines #ndjson

I've added requests cancellation capabilities to all (ASP​.NET Core, Blazor, .NET Console) my NDJSON and JSON streaming samples.

#DotNet #AspNetCore #Blazor #NDJSON #AsyncStreams

https://github.com/tpeczek/Demo.Ndjson.AsyncStreams

@bdelacretaz I also came across #NDJSON https://github.com/ndjson/ndjson-spec which seems like the same thing, developed independently

@bdelacretaz I also came across #NDJSON https://github.com/ndjson/ndjson-spec which seems like the same thing, developed independently

I never bothered with optimizing the parsing of #jsonl #ndjson files because in most cases it was an one off task before I put the data into a database or parquet file. But the files got bigger and waiting 20 minutes for the data to load made me reconsider my decision. So tried some different approaches.

Tested with a 1.7 GB file of 300 k Tweets.

jsonlines.reader: 17.2 seconds 100%
orjson: 6.49 seconds 37%
msgspec: 3.06 seconds 17%

I like how orjson cuts the time by two thirds without the need to change anything else. Just use it as a drop in replacement and you are good.

msgspec is twice as fast as orjson or six times as fast as jsonlines if you define the schema of the data that you want. For Tweets that's okay, as I can reuse the schema many times. With data that is used only once, I prefer orjson.

Memory usage was nearly identical across the different solutions. Probably because they all parse the data per line. I restarted the kernel each time to get comparable numbers.

Load time for all 23 million Tweets in the dataset was reduced from 25 to 4 minutes.

This blogposts was useful to me: https://pythonspeed.com/articles/faster-python-json-parsing/ #Python #DataEngineering

I never bothered with optimizing the parsing of #jsonl #ndjson files because in most cases it was an one off task before I put the data into a database or parquet file. But the files got bigger and waiting 20 minutes for the data to load made me reconsider my decision. So tried some different approaches.

Tested with a 1.7 GB file of 300 k Tweets.

jsonlines.reader: 17.2 seconds 100%
orjson: 6.49 seconds 37%
msgspec: 3.06 seconds 17%

I like how orjson cuts the time by two thirds without the need to change anything else. Just use it as a drop in replacement and you are good.

msgspec is twice as fast as orjson or six times as fast as jsonlines if you define the schema of the data that you want. For Tweets that's okay, as I can reuse the schema many times. With data that is used only once, I prefer orjson.

Memory usage was nearly identical across the different solutions. Probably because they all parse the data per line. I restarted the kernel each time to get comparable numbers.

Load time for all 23 million Tweets in the dataset was reduced from 25 to 4 minutes.

This blogposts was useful to me: https://pythonspeed.com/articles/faster-python-json-parsing/ #Python #DataEngineering

[#tools] ndjson-apply: apply a JS function to a stream of newline-delimited JSON
https://github.com/maxlath/ndjson-apply #nodejs #NDJSON