#thingumbrella — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #thingumbrella, aggregated by home.social.
-
Recently, at the library...
"Have you got their new book 'Silk switch system problem'?"
"No. Is that the sequel to 'Equiconnected nanosky oasis'?"
"What's that?"
"Oh, you've never read the 'Magnetic haze cycle' series? That's just too bad. It's full of cool ideas like the ultra-iridescent opcode sentinel, infraconnected division boundary, or even beige frontier mirror..."
"Dude!"
"Yeah, it's real porcelain ultra-peak! Absolutely life changing epoch omnicurve vision!"
"I did like 'Supraviolet ring department'..."
"Well, in that case, I'd recommend 'Macroluminous shimmer facet' instead. It's a good intro to all the abundant assembly craft."In preparation for some name/passphrase generator and LLM-poisoning projects/tools, I updated my online procedural text editor to be able to export generator specs/recipes directly as TypeScript source code for easy integration into your own projects...
The above generator is here (the entire recipe is part of this super long URL):
More project & syntax info here:
https://thi.ng/proctext#Microfiction #ProcGen #ProceduralText #ThingUmbrella #DSL #TypeScript #LLMPoisoning #NoAI
-
Color Therapy. Noisy evolution.
Two snapshots of my piece C-SCAPE, 2022, a multi-organism 1.5D cellular automata simulation. Patterns of symbiosis and annihilation.
Made with https://thi.ng/cellular
#CSCAPE #CellularAutomata #GenerativeArt #AlgorithmicArt #EvolutionaryArt #Abstract #Color #Texture #ThingUmbrella #NoAI
-
Color Therapy. Noisy evolution.
Two snapshots of my piece C-SCAPE, 2022, a multi-organism 1.5D cellular automata simulation. Patterns of symbiosis and annihilation.
Made with https://thi.ng/cellular
#CSCAPE #CellularAutomata #GenerativeArt #AlgorithmicArt #EvolutionaryArt #Abstract #Color #Texture #ThingUmbrella #NoAI
-
Color Therapy. Noisy evolution.
Two snapshots of my piece C-SCAPE, 2022, a multi-organism 1.5D cellular automata simulation. Patterns of symbiosis and annihilation.
Made with https://thi.ng/cellular
#CSCAPE #CellularAutomata #GenerativeArt #AlgorithmicArt #EvolutionaryArt #Abstract #Color #Texture #ThingUmbrella #NoAI
-
Color Therapy. Noisy evolution.
Two snapshots of my piece C-SCAPE, 2022, a multi-organism 1.5D cellular automata simulation. Patterns of symbiosis and annihilation.
Made with https://thi.ng/cellular
#CSCAPE #CellularAutomata #GenerativeArt #AlgorithmicArt #EvolutionaryArt #Abstract #Color #Texture #ThingUmbrella #NoAI
-
Color Therapy. Noisy evolution.
Two snapshots of my piece C-SCAPE, 2022, a multi-organism 1.5D cellular automata simulation. Patterns of symbiosis and annihilation.
Made with https://thi.ng/cellular
#CSCAPE #CellularAutomata #GenerativeArt #AlgorithmicArt #EvolutionaryArt #Abstract #Color #Texture #ThingUmbrella #NoAI
-
#ReleaseDay https://thi.ng/column-store is an in-memory database with customizable column types, extensible query engine, bitfield indexing for query acceleration, JSON serialization with optional RLE compression.
The new version introduces support for arbitrarily nested queries and merging of sub-query results using a choice of AND (intersection/constraint), OR (union/alternative) or NAND/NOR (negated versions) semantics.
https://docs.thi.ng/umbrella/column-store/#nested-queries
Also without query nesting, each individual query term's results can be merged as union now. The previous behavior only allowed for term result intersections, with each subsequent term further narrowing the total result set. This behavior limited the types of compound queries possible and therefore required multiple separate queries and additional user effort to merge results manually. Well, no more! :)
https://docs.thi.ng/umbrella/column-store/#term-union
Another (intentional) side effect: Since queries are defined declaratively, creating complex queries is much easier (and legible) now via composition and re-use of predefined sub-queries. The support for nesting also simplifies the creation of user-defined query DSLs (domain specific languages).
As before, for columns with bitmap indexing enabled, most of the query operators are extremely fast since only bit masks need to be combined and no actual row or column data is being visited. The latter is only necessary for predicate-based
matchXXX()operators...#ThingUmbrella #OpenSource #Database #QueryEngine #TypeScript #BitField
-
#ReleaseThursday Happy to announce that https://thi.ng/units finally has a little CLI wrapper to perform not only simple unit conversions, but also more advanced calculations, using the existing Lisp-like S-expression based DSL and built-in units and constants.
https://docs.thi.ng/umbrella/units/#cli-usage
The CLI also provides a
listcommand to print out the 172 built-in units, their canonical symbol, any aliases and full names. The output can also be filtered via optional regexp pattern. See attached readme screenshots & more advanced linked example:https://mastodon.thi.ng/@toxi/116043223284662742
Hope some of you find this as useful as I do! Any questions, please ask!
#ThingUmbrella #UnitConversion #CLI #Calculator #TypeScript #Lisp #DSL #OpenSource
-
#ReleaseThursday Happy to announce that https://thi.ng/units finally has a little CLI wrapper to perform not only simple unit conversions, but also more advanced calculations, using the existing Lisp-like S-expression based DSL and built-in units and constants.
https://docs.thi.ng/umbrella/units/#cli-usage
The CLI also provides a
listcommand to print out the 172 built-in units, their canonical symbol, any aliases and full names. The output can also be filtered via optional regexp pattern. See attached readme screenshots & more advanced linked example:https://mastodon.thi.ng/@toxi/116043223284662742
Hope some of you find this as useful as I do! Any questions, please ask!
#ThingUmbrella #UnitConversion #CLI #Calculator #TypeScript #Lisp #DSL #OpenSource
-
#ReleaseThursday Happy to announce that https://thi.ng/units finally has a little CLI wrapper to perform not only simple unit conversions, but also more advanced calculations, using the existing Lisp-like S-expression based DSL and built-in units and constants.
https://docs.thi.ng/umbrella/units/#cli-usage
The CLI also provides a
listcommand to print out the 172 built-in units, their canonical symbol, any aliases and full names. The output can also be filtered via optional regexp pattern. See attached readme screenshots & more advanced linked example:https://mastodon.thi.ng/@toxi/116043223284662742
Hope some of you find this as useful as I do! Any questions, please ask!
#ThingUmbrella #UnitConversion #CLI #Calculator #TypeScript #Lisp #DSL #OpenSource
-
#ReleaseThursday Happy to announce that https://thi.ng/units finally has a little CLI wrapper to perform not only simple unit conversions, but also more advanced calculations, using the existing Lisp-like S-expression based DSL and built-in units and constants.
https://docs.thi.ng/umbrella/units/#cli-usage
The CLI also provides a
listcommand to print out the 172 built-in units, their canonical symbol, any aliases and full names. The output can also be filtered via optional regexp pattern. See attached readme screenshots & more advanced linked example:https://mastodon.thi.ng/@toxi/116043223284662742
Hope some of you find this as useful as I do! Any questions, please ask!
#ThingUmbrella #UnitConversion #CLI #Calculator #TypeScript #Lisp #DSL #OpenSource
-
#ReleaseThursday Happy to announce that https://thi.ng/units finally has a little CLI wrapper to perform not only simple unit conversions, but also more advanced calculations, using the existing Lisp-like S-expression based DSL and built-in units and constants.
https://docs.thi.ng/umbrella/units/#cli-usage
The CLI also provides a
listcommand to print out the 172 built-in units, their canonical symbol, any aliases and full names. The output can also be filtered via optional regexp pattern. See attached readme screenshots & more advanced linked example:https://mastodon.thi.ng/@toxi/116043223284662742
Hope some of you find this as useful as I do! Any questions, please ask!
#ThingUmbrella #UnitConversion #CLI #Calculator #TypeScript #Lisp #DSL #OpenSource
-
#ReleaseMonday Yesterday, I took the plunge and updated all 215 https://thi.ng/umbrella packages (plus examples) for TypeScript 6.0, then released new versions just now this morning. After previous preparations and some initial struggles, it thankfully only involved updates to
tsconfig.jsonfiles (accompanied by a lot of head scratching).I'm still confused about the new behavior of TS6 now defaulting to an empty
typesarray[1] incompilerOptions, meaning one has to explicitly state, e.g."types": ["node"]now, seemingly for any dependent package as well. The latter is the confusing part to me!For example, the https://thi.ng/errors package requires NodeJS type definitions for doing some pre-checks, but because of that, any dependent package has to specify the
"types": ["node"]option too now (so it seems...), even if the dependent package itself does not use NodeJS types... If this compiler option isn't given, you'll get errors likeCannot find namespace 'NodeJS'when compiling the downstream package. I find this transitive requirement super weird & nonsense! I also mention it, because it will likely impact your own codebase as well... Please report back, if you could! 🙏 -
#ReleaseMonday Yesterday, I took the plunge and updated all 215 https://thi.ng/umbrella packages (plus examples) for TypeScript 6.0, then released new versions just now this morning. After previous preparations and some initial struggles, it thankfully only involved updates to
tsconfig.jsonfiles (accompanied by a lot of head scratching).I'm still confused about the new behavior of TS6 now defaulting to an empty
typesarray[1] incompilerOptions, meaning one has to explicitly state, e.g."types": ["node"]now, seemingly for any dependent package as well. The latter is the confusing part to me!For example, the https://thi.ng/errors package requires NodeJS type definitions for doing some pre-checks, but because of that, any dependent package has to specify the
"types": ["node"]option too now (so it seems...), even if the dependent package itself does not use NodeJS types... If this compiler option isn't given, you'll get errors likeCannot find namespace 'NodeJS'when compiling the downstream package. I find this transitive requirement super weird & nonsense! I also mention it, because it will likely impact your own codebase as well... Please report back, if you could! 🙏 -
#ReleaseMonday Yesterday, I took the plunge and updated all 215 https://thi.ng/umbrella packages (plus examples) for TypeScript 6.0, then released new versions just now this morning. After previous preparations and some initial struggles, it thankfully only involved updates to
tsconfig.jsonfiles (accompanied by a lot of head scratching).I'm still confused about the new behavior of TS6 now defaulting to an empty
typesarray[1] incompilerOptions, meaning one has to explicitly state, e.g."types": ["node"]now, seemingly for any dependent package as well. The latter is the confusing part to me!For example, the https://thi.ng/errors package requires NodeJS type definitions for doing some pre-checks, but because of that, any dependent package has to specify the
"types": ["node"]option too now (so it seems...), even if the dependent package itself does not use NodeJS types... If this compiler option isn't given, you'll get errors likeCannot find namespace 'NodeJS'when compiling the downstream package. I find this transitive requirement super weird & nonsense! I also mention it, because it will likely impact your own codebase as well... Please report back, if you could! 🙏 -
#ReleaseMonday Yesterday, I took the plunge and updated all 215 https://thi.ng/umbrella packages (plus examples) for TypeScript 6.0, then released new versions just now this morning. After previous preparations and some initial struggles, it thankfully only involved updates to
tsconfig.jsonfiles (accompanied by a lot of head scratching).I'm still confused about the new behavior of TS6 now defaulting to an empty
typesarray[1] incompilerOptions, meaning one has to explicitly state, e.g."types": ["node"]now, seemingly for any dependent package as well. The latter is the confusing part to me!For example, the https://thi.ng/errors package requires NodeJS type definitions for doing some pre-checks, but because of that, any dependent package has to specify the
"types": ["node"]option too now (so it seems...), even if the dependent package itself does not use NodeJS types... If this compiler option isn't given, you'll get errors likeCannot find namespace 'NodeJS'when compiling the downstream package. I find this transitive requirement super weird & nonsense! I also mention it, because it will likely impact your own codebase as well... Please report back, if you could! 🙏 -
#ReleaseMonday Yesterday, I took the plunge and updated all 215 https://thi.ng/umbrella packages (plus examples) for TypeScript 6.0, then released new versions just now this morning. After previous preparations and some initial struggles, it thankfully only involved updates to
tsconfig.jsonfiles (accompanied by a lot of head scratching).I'm still confused about the new behavior of TS6 now defaulting to an empty
typesarray[1] incompilerOptions, meaning one has to explicitly state, e.g."types": ["node"]now, seemingly for any dependent package as well. The latter is the confusing part to me!For example, the https://thi.ng/errors package requires NodeJS type definitions for doing some pre-checks, but because of that, any dependent package has to specify the
"types": ["node"]option too now (so it seems...), even if the dependent package itself does not use NodeJS types... If this compiler option isn't given, you'll get errors likeCannot find namespace 'NodeJS'when compiling the downstream package. I find this transitive requirement super weird & nonsense! I also mention it, because it will likely impact your own codebase as well... Please report back, if you could! 🙏 -
#ReleaseFriday Triggered by a recent feature proposal[1], I went ahead and polished & published a closely related, already work-in-progress (but still private) feature in https://thi.ng/rdom to support something I call "bare lists"[2]. I've started working on this for another project last year, but needed to do more testing (which I think have sufficiently done by now).
These "bare" lists are managed reactive control components which attach items directly to the list's parent DOM element instead of first creating a wrapper/container element for the items and so avoid introducing additional nesting.
There're many use cases where this additional nesting was a real problem with the earlier approach, e.g. in containers with CSS grid or flex layout, tables, or generally situations where we want to have static & reactive list items as true siblings...
The new version of thi.ng/rdom is technically a breaking change (sorry!), but the actual changes required (for you) are tiny and purely limited to the
$list()and$klist()component function calls, which are now accepting a parameter object instead of positional args for the different possible behaviors. Of course, lists with item wrapper elements can still be created too, just as before (but via new args).I've updated & tested all existing examples impacted by this change and also created a new fully commented example project (example #187) to illustrate these "bare" lists in situ (check the DOM inspector to see the shallow structure and how updates are applied):
Demo:
https://demo.thi.ng/umbrella/rdom-bare-lists/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/rdom-bare-lists/src/index.ts[1] github.com/thi-ng/umbrella/discussions/562
[2] My use of "list" here is generic, not limited to<ul>or<ol>...#ThingUmbrella #Reactive #UI #OpenSource #TypeScript #JavaScript #WebDev
-
#ReleaseFriday Triggered by a recent feature proposal[1], I went ahead and polished & published a closely related, already work-in-progress (but still private) feature in https://thi.ng/rdom to support something I call "bare lists"[2]. I've started working on this for another project last year, but needed to do more testing (which I think have sufficiently done by now).
These "bare" lists are managed reactive control components which attach items directly to the list's parent DOM element instead of first creating a wrapper/container element for the items and so avoid introducing additional nesting.
There're many use cases where this additional nesting was a real problem with the earlier approach, e.g. in containers with CSS grid or flex layout, tables, or generally situations where we want to have static & reactive list items as true siblings...
The new version of thi.ng/rdom is technically a breaking change (sorry!), but the actual changes required (for you) are tiny and purely limited to the
$list()and$klist()component function calls, which are now accepting a parameter object instead of positional args for the different possible behaviors. Of course, lists with item wrapper elements can still be created too, just as before (but via new args).I've updated & tested all existing examples impacted by this change and also created a new fully commented example project (example #187) to illustrate these "bare" lists in situ (check the DOM inspector to see the shallow structure and how updates are applied):
Demo:
https://demo.thi.ng/umbrella/rdom-bare-lists/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/rdom-bare-lists/src/index.ts[1] github.com/thi-ng/umbrella/discussions/562
[2] My use of "list" here is generic, not limited to<ul>or<ol>...#ThingUmbrella #Reactive #UI #OpenSource #TypeScript #JavaScript #WebDev
-
#ReleaseFriday Triggered by a recent feature proposal[1], I went ahead and polished & published a closely related, already work-in-progress (but still private) feature in https://thi.ng/rdom to support something I call "bare lists"[2]. I've started working on this for another project last year, but needed to do more testing (which I think have sufficiently done by now).
These "bare" lists are managed reactive control components which attach items directly to the list's parent DOM element instead of first creating a wrapper/container element for the items and so avoid introducing additional nesting.
There're many use cases where this additional nesting was a real problem with the earlier approach, e.g. in containers with CSS grid or flex layout, tables, or generally situations where we want to have static & reactive list items as true siblings...
The new version of thi.ng/rdom is technically a breaking change (sorry!), but the actual changes required (for you) are tiny and purely limited to the
$list()and$klist()component function calls, which are now accepting a parameter object instead of positional args for the different possible behaviors. Of course, lists with item wrapper elements can still be created too, just as before (but via new args).I've updated & tested all existing examples impacted by this change and also created a new fully commented example project (example #187) to illustrate these "bare" lists in situ (check the DOM inspector to see the shallow structure and how updates are applied):
Demo:
https://demo.thi.ng/umbrella/rdom-bare-lists/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/rdom-bare-lists/src/index.ts[1] github.com/thi-ng/umbrella/discussions/562
[2] My use of "list" here is generic, not limited to<ul>or<ol>...#ThingUmbrella #Reactive #UI #OpenSource #TypeScript #JavaScript #WebDev
-
#ReleaseFriday Triggered by a recent feature proposal[1], I went ahead and polished & published a closely related, already work-in-progress (but still private) feature in https://thi.ng/rdom to support something I call "bare lists"[2]. I've started working on this for another project last year, but needed to do more testing (which I think have sufficiently done by now).
These "bare" lists are managed reactive control components which attach items directly to the list's parent DOM element instead of first creating a wrapper/container element for the items and so avoid introducing additional nesting.
There're many use cases where this additional nesting was a real problem with the earlier approach, e.g. in containers with CSS grid or flex layout, tables, or generally situations where we want to have static & reactive list items as true siblings...
The new version of thi.ng/rdom is technically a breaking change (sorry!), but the actual changes required (for you) are tiny and purely limited to the
$list()and$klist()component function calls, which are now accepting a parameter object instead of positional args for the different possible behaviors. Of course, lists with item wrapper elements can still be created too, just as before (but via new args).I've updated & tested all existing examples impacted by this change and also created a new fully commented example project (example #187) to illustrate these "bare" lists in situ (check the DOM inspector to see the shallow structure and how updates are applied):
Demo:
https://demo.thi.ng/umbrella/rdom-bare-lists/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/rdom-bare-lists/src/index.ts[1] github.com/thi-ng/umbrella/discussions/562
[2] My use of "list" here is generic, not limited to<ul>or<ol>...#ThingUmbrella #Reactive #UI #OpenSource #TypeScript #JavaScript #WebDev
-
#ReleaseFriday Triggered by a recent feature proposal[1], I went ahead and polished & published a closely related, already work-in-progress (but still private) feature in https://thi.ng/rdom to support something I call "bare lists"[2]. I've started working on this for another project last year, but needed to do more testing (which I think have sufficiently done by now).
These "bare" lists are managed reactive control components which attach items directly to the list's parent DOM element instead of first creating a wrapper/container element for the items and so avoid introducing additional nesting.
There're many use cases where this additional nesting was a real problem with the earlier approach, e.g. in containers with CSS grid or flex layout, tables, or generally situations where we want to have static & reactive list items as true siblings...
The new version of thi.ng/rdom is technically a breaking change (sorry!), but the actual changes required (for you) are tiny and purely limited to the
$list()and$klist()component function calls, which are now accepting a parameter object instead of positional args for the different possible behaviors. Of course, lists with item wrapper elements can still be created too, just as before (but via new args).I've updated & tested all existing examples impacted by this change and also created a new fully commented example project (example #187) to illustrate these "bare" lists in situ (check the DOM inspector to see the shallow structure and how updates are applied):
Demo:
https://demo.thi.ng/umbrella/rdom-bare-lists/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/rdom-bare-lists/src/index.ts[1] github.com/thi-ng/umbrella/discussions/562
[2] My use of "list" here is generic, not limited to<ul>or<ol>...#ThingUmbrella #Reactive #UI #OpenSource #TypeScript #JavaScript #WebDev
-
@lo Great minds (but not using p5js)... :)
https://demo.thi.ng/umbrella/webgl-shadertoy/
Source:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/webgl-shadertoy/src/index.ts -
@lo Great minds (but not using p5js)... :)
https://demo.thi.ng/umbrella/webgl-shadertoy/
Source:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/webgl-shadertoy/src/index.ts -
@lo Great minds (but not using p5js)... :)
https://demo.thi.ng/umbrella/webgl-shadertoy/
Source:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/webgl-shadertoy/src/index.ts -
@lo Great minds (but not using p5js)... :)
https://demo.thi.ng/umbrella/webgl-shadertoy/
Source:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/webgl-shadertoy/src/index.ts -
@lo Great minds (but not using p5js)... :)
https://demo.thi.ng/umbrella/webgl-shadertoy/
Source:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/webgl-shadertoy/src/index.ts -
#ReleaseSaturday 🚀 — Just pushed the new version of https://thi.ng/hiccup-carbon-icons (now a much larger collection of 2200+ icons, mentioned yesterday[1]) and some other smaller updates/additions to other packages...
This is the last release before switching all packages to the recently released TypeScript 6.0, support for which will likely require some restructuring & refactoring and hopefully will be less painful than it might look so far (I'm also waiting for some dependencies to update their TS type definitions, which are currently breaking, e.g. https://github.com/serialport/node-serialport, used for https://thi.ng/axidraw)
I also added some new async operators for https://thi.ng/transducers-async to simplify some stream processing tasks (e.g. collecting and/or consuming stdout/stderr of a child process by rechunking the stream for line-based processing), for example:
```
import { rechunk } from "@thi.ng/transducers-async";
import { spawn } from "child_process";// launch child process
const child = spawn("ls", ["-l"]);// split child's stdout into single lines
for await(let line of rechunk(/\r?\n/g, child.stdout)) {
console.log("output", line);
}
```[1] https://mastodon.thi.ng/@toxi/116422011357971578
#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript #Transducers #Async #Icons
-
#ReleaseSaturday 🚀 — Just pushed the new version of https://thi.ng/hiccup-carbon-icons (now a much larger collection of 2200+ icons, mentioned yesterday[1]) and some other smaller updates/additions to other packages...
This is the last release before switching all packages to the recently released TypeScript 6.0, support for which will likely require some restructuring & refactoring and hopefully will be less painful than it might look so far (I'm also waiting for some dependencies to update their TS type definitions, which are currently breaking, e.g. https://github.com/serialport/node-serialport, used for https://thi.ng/axidraw)
I also added some new async operators for https://thi.ng/transducers-async to simplify some stream processing tasks (e.g. collecting and/or consuming stdout/stderr of a child process by rechunking the stream for line-based processing), for example:
```
import { rechunk } from "@thi.ng/transducers-async";
import { spawn } from "child_process";// launch child process
const child = spawn("ls", ["-l"]);// split child's stdout into single lines
for await(let line of rechunk(/\r?\n/g, child.stdout)) {
console.log("output", line);
}
```[1] https://mastodon.thi.ng/@toxi/116422011357971578
#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript #Transducers #Async #Icons
-
#ReleaseSaturday 🚀 — Just pushed the new version of https://thi.ng/hiccup-carbon-icons (now a much larger collection of 2200+ icons, mentioned yesterday[1]) and some other smaller updates/additions to other packages...
This is the last release before switching all packages to the recently released TypeScript 6.0, support for which will likely require some restructuring & refactoring and hopefully will be less painful than it might look so far (I'm also waiting for some dependencies to update their TS type definitions, which are currently breaking, e.g. https://github.com/serialport/node-serialport, used for https://thi.ng/axidraw)
I also added some new async operators for https://thi.ng/transducers-async to simplify some stream processing tasks (e.g. collecting and/or consuming stdout/stderr of a child process by rechunking the stream for line-based processing), for example:
```
import { rechunk } from "@thi.ng/transducers-async";
import { spawn } from "child_process";// launch child process
const child = spawn("ls", ["-l"]);// split child's stdout into single lines
for await(let line of rechunk(/\r?\n/g, child.stdout)) {
console.log("output", line);
}
```[1] https://mastodon.thi.ng/@toxi/116422011357971578
#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript #Transducers #Async #Icons
-
#ReleaseSaturday 🚀 — Just pushed the new version of https://thi.ng/hiccup-carbon-icons (now a much larger collection of 2200+ icons, mentioned yesterday[1]) and some other smaller updates/additions to other packages...
This is the last release before switching all packages to the recently released TypeScript 6.0, support for which will likely require some restructuring & refactoring and hopefully will be less painful than it might look so far (I'm also waiting for some dependencies to update their TS type definitions, which are currently breaking, e.g. https://github.com/serialport/node-serialport, used for https://thi.ng/axidraw)
I also added some new async operators for https://thi.ng/transducers-async to simplify some stream processing tasks (e.g. collecting and/or consuming stdout/stderr of a child process by rechunking the stream for line-based processing), for example:
```
import { rechunk } from "@thi.ng/transducers-async";
import { spawn } from "child_process";// launch child process
const child = spawn("ls", ["-l"]);// split child's stdout into single lines
for await(let line of rechunk(/\r?\n/g, child.stdout)) {
console.log("output", line);
}
```[1] https://mastodon.thi.ng/@toxi/116422011357971578
#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript #Transducers #Async #Icons
-
#ReleaseSaturday 🚀 — Just pushed the new version of https://thi.ng/hiccup-carbon-icons (now a much larger collection of 2200+ icons, mentioned yesterday[1]) and some other smaller updates/additions to other packages...
This is the last release before switching all packages to the recently released TypeScript 6.0, support for which will likely require some restructuring & refactoring and hopefully will be less painful than it might look so far (I'm also waiting for some dependencies to update their TS type definitions, which are currently breaking, e.g. https://github.com/serialport/node-serialport, used for https://thi.ng/axidraw)
I also added some new async operators for https://thi.ng/transducers-async to simplify some stream processing tasks (e.g. collecting and/or consuming stdout/stderr of a child process by rechunking the stream for line-based processing), for example:
```
import { rechunk } from "@thi.ng/transducers-async";
import { spawn } from "child_process";// launch child process
const child = spawn("ls", ["-l"]);// split child's stdout into single lines
for await(let line of rechunk(/\r?\n/g, child.stdout)) {
console.log("output", line);
}
```[1] https://mastodon.thi.ng/@toxi/116422011357971578
#ThingUmbrella #OpenSource #Maintenance #TypeScript #JavaScript #Transducers #Async #Icons
-
It's been 4.5 years since I last updated the https://thi.ng/hiccup-carbon-icons collection and synced it with the upstream repo, i.e. IBM's Carbon design system. Spent a few hours today updating the icons to the current version, filtering out a hundred unnecessary ones (e.g. obsolete brand logos, IBM product/service related icons etc.) and updated the converter & code generator[1] to produce more concise outputs, then manually cleaned up the structure for dozens of them (in addition to optimizing/minimizing the SVG sources via the `svgo` CLI).
The new set has exactly 2222 icons in https://thi.ng/hiccup format (SVG expressed as nested JS arrays). These icons can be used in any context where https://thi.ng/hiccup format is supported, i.e. both for static HTML/SVG generation and/or interactive scenarios.
A contact sheet of the full collection (the attached image only shows a tiny selection of this):
https://demo.thi.ng/umbrella/hiccup-carbon-icons/For tree-shaking purposes each icon is defined in its own source file, e.g. the Mastodon logo can be then imported like so:
`import { MASTODON } from "@thi.ng/hiccup-carbon-icons"`
Example icon definition:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/src/logo-mastodon.tsThe new version is still unreleased, but the readme already contains up-to-date information and small usage examples (incl. links to live example projects to see usage in situ).
[1] Converter/codegen tool: https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/tools/convert-icons.ts
-
It's been 4.5 years since I last updated the https://thi.ng/hiccup-carbon-icons collection and synced it with the upstream repo, i.e. IBM's Carbon design system. Spent a few hours today updating the icons to the current version, filtering out a hundred unnecessary ones (e.g. obsolete brand logos, IBM product/service related icons etc.) and updated the converter & code generator[1] to produce more concise outputs, then manually cleaned up the structure for dozens of them (in addition to optimizing/minimizing the SVG sources via the `svgo` CLI).
The new set has exactly 2222 icons in https://thi.ng/hiccup format (SVG expressed as nested JS arrays). These icons can be used in any context where https://thi.ng/hiccup format is supported, i.e. both for static HTML/SVG generation and/or interactive scenarios.
A contact sheet of the full collection (the attached image only shows a tiny selection of this):
https://demo.thi.ng/umbrella/hiccup-carbon-icons/For tree-shaking purposes each icon is defined in its own source file, e.g. the Mastodon logo can be then imported like so:
`import { MASTODON } from "@thi.ng/hiccup-carbon-icons"`
Example icon definition:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/src/logo-mastodon.tsThe new version is still unreleased, but the readme already contains up-to-date information and small usage examples (incl. links to live example projects to see usage in situ).
[1] Converter/codegen tool: https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/tools/convert-icons.ts
-
It's been 4.5 years since I last updated the https://thi.ng/hiccup-carbon-icons collection and synced it with the upstream repo, i.e. IBM's Carbon design system. Spent a few hours today updating the icons to the current version, filtering out a hundred unnecessary ones (e.g. obsolete brand logos, IBM product/service related icons etc.) and updated the converter & code generator[1] to produce more concise outputs, then manually cleaned up the structure for dozens of them (in addition to optimizing/minimizing the SVG sources via the `svgo` CLI).
The new set has exactly 2222 icons in https://thi.ng/hiccup format (SVG expressed as nested JS arrays). These icons can be used in any context where https://thi.ng/hiccup format is supported, i.e. both for static HTML/SVG generation and/or interactive scenarios.
A contact sheet of the full collection (the attached image only shows a tiny selection of this):
https://demo.thi.ng/umbrella/hiccup-carbon-icons/For tree-shaking purposes each icon is defined in its own source file, e.g. the Mastodon logo can be then imported like so:
`import { MASTODON } from "@thi.ng/hiccup-carbon-icons"`
Example icon definition:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/src/logo-mastodon.tsThe new version is still unreleased, but the readme already contains up-to-date information and small usage examples (incl. links to live example projects to see usage in situ).
[1] Converter/codegen tool: https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/tools/convert-icons.ts
-
It's been 4.5 years since I last updated the https://thi.ng/hiccup-carbon-icons collection and synced it with the upstream repo, i.e. IBM's Carbon design system. Spent a few hours today updating the icons to the current version, filtering out a hundred unnecessary ones (e.g. obsolete brand logos, IBM product/service related icons etc.) and updated the converter & code generator[1] to produce more concise outputs, then manually cleaned up the structure for dozens of them (in addition to optimizing/minimizing the SVG sources via the `svgo` CLI).
The new set has exactly 2222 icons in https://thi.ng/hiccup format (SVG expressed as nested JS arrays). These icons can be used in any context where https://thi.ng/hiccup format is supported, i.e. both for static HTML/SVG generation and/or interactive scenarios.
A contact sheet of the full collection (the attached image only shows a tiny selection of this):
https://demo.thi.ng/umbrella/hiccup-carbon-icons/For tree-shaking purposes each icon is defined in its own source file, e.g. the Mastodon logo can be then imported like so:
`import { MASTODON } from "@thi.ng/hiccup-carbon-icons"`
Example icon definition:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/src/logo-mastodon.tsThe new version is still unreleased, but the readme already contains up-to-date information and small usage examples (incl. links to live example projects to see usage in situ).
[1] Converter/codegen tool: https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/tools/convert-icons.ts
-
It's been 4.5 years since I last updated the https://thi.ng/hiccup-carbon-icons collection and synced it with the upstream repo, i.e. IBM's Carbon design system. Spent a few hours today updating the icons to the current version, filtering out a hundred unnecessary ones (e.g. obsolete brand logos, IBM product/service related icons etc.) and updated the converter & code generator[1] to produce more concise outputs, then manually cleaned up the structure for dozens of them (in addition to optimizing/minimizing the SVG sources via the `svgo` CLI).
The new set has exactly 2222 icons in https://thi.ng/hiccup format (SVG expressed as nested JS arrays). These icons can be used in any context where https://thi.ng/hiccup format is supported, i.e. both for static HTML/SVG generation and/or interactive scenarios.
A contact sheet of the full collection (the attached image only shows a tiny selection of this):
https://demo.thi.ng/umbrella/hiccup-carbon-icons/For tree-shaking purposes each icon is defined in its own source file, e.g. the Mastodon logo can be then imported like so:
`import { MASTODON } from "@thi.ng/hiccup-carbon-icons"`
Example icon definition:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/src/logo-mastodon.tsThe new version is still unreleased, but the readme already contains up-to-date information and small usage examples (incl. links to live example projects to see usage in situ).
[1] Converter/codegen tool: https://codeberg.org/thi.ng/umbrella/src/branch/develop/packages/hiccup-carbon-icons/tools/convert-icons.ts
-
Added, updated & simplified the growing collection of darkroom-related calculators and super happy how elegant and concise the code has turned out, making it super easy to add more of them in the future.
I think it's also another great, if minimal, example to illustrate how otherwise completely separate https://thi.ng/umbrella packages can seamlessly compose/combine to enable a reactive dataflow UI, all without the need for any virtual DOMs and/or completely over-the-top frameworks like React & co. It's also doing so via mostly JS-native data structures for declaring the UI (plain objects/arrays/iterables) and various constructs directly managing the reactive value streams, thus providing a lot more finegrained control over UI updates/timing/throttling). Any value changes done by the user only trigger specific, pin-point calculations which then result in equally specific UI updates to show new results. Any user action only ever triggers the minimum amount of work needed to reflect the new state.
Calculators:
https://demo.thi.ng/umbrella/darkroom-calc/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/darkroom-calc/srcThe attached images show the source code of the entire main app (UI root) and one of the calculators...
Ps. Please let me know if you'd like to see more of these posts in the future. I'm tempted to launch season 2 of #HowToThing (see link below for 30 previous mini projects/tutorials) — but since this is very time consuming to produce & document these projects/examples, and because there has been _very little feedback_ to these previous projects/posts, I first need to gauge interest... Thank you! 🫶
https://codeberg.org/thi.ng/umbrella#howtothing
#ThingUmbrella #Darkroom #Calculator #Tool #Reactive #UI #WebDev #TypeScript #JavaScript #OpenSource
-
Added, updated & simplified the growing collection of darkroom-related calculators and super happy how elegant and concise the code has turned out, making it super easy to add more of them in the future.
I think it's also another great, if minimal, example to illustrate how otherwise completely separate https://thi.ng/umbrella packages can seamlessly compose/combine to enable a reactive dataflow UI, all without the need for any virtual DOMs and/or completely over-the-top frameworks like React & co. It's also doing so via mostly JS-native data structures for declaring the UI (plain objects/arrays/iterables) and various constructs directly managing the reactive value streams, thus providing a lot more finegrained control over UI updates/timing/throttling). Any value changes done by the user only trigger specific, pin-point calculations which then result in equally specific UI updates to show new results. Any user action only ever triggers the minimum amount of work needed to reflect the new state.
Calculators:
https://demo.thi.ng/umbrella/darkroom-calc/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/darkroom-calc/srcThe attached images show the source code of the entire main app (UI root) and one of the calculators...
Ps. Please let me know if you'd like to see more of these posts in the future. I'm tempted to launch season 2 of #HowToThing (see link below for 30 previous mini projects/tutorials) — but since this is very time consuming to produce & document these projects/examples, and because there has been _very little feedback_ to these previous projects/posts, I first need to gauge interest... Thank you! 🫶
https://codeberg.org/thi.ng/umbrella#howtothing
#ThingUmbrella #Darkroom #Calculator #Tool #Reactive #UI #WebDev #TypeScript #JavaScript #OpenSource
-
Added, updated & simplified the growing collection of darkroom-related calculators and super happy how elegant and concise the code has turned out, making it super easy to add more of them in the future.
I think it's also another great, if minimal, example to illustrate how otherwise completely separate https://thi.ng/umbrella packages can seamlessly compose/combine to enable a reactive dataflow UI, all without the need for any virtual DOMs and/or completely over-the-top frameworks like React & co. It's also doing so via mostly JS-native data structures for declaring the UI (plain objects/arrays/iterables) and various constructs directly managing the reactive value streams, thus providing a lot more finegrained control over UI updates/timing/throttling). Any value changes done by the user only trigger specific, pin-point calculations which then result in equally specific UI updates to show new results. Any user action only ever triggers the minimum amount of work needed to reflect the new state.
Calculators:
https://demo.thi.ng/umbrella/darkroom-calc/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/darkroom-calc/srcThe attached images show the source code of the entire main app (UI root) and one of the calculators...
Ps. Please let me know if you'd like to see more of these posts in the future. I'm tempted to launch season 2 of #HowToThing (see link below for 30 previous mini projects/tutorials) — but since this is very time consuming to produce & document these projects/examples, and because there has been _very little feedback_ to these previous projects/posts, I first need to gauge interest... Thank you! 🫶
https://codeberg.org/thi.ng/umbrella#howtothing
#ThingUmbrella #Darkroom #Calculator #Tool #Reactive #UI #WebDev #TypeScript #JavaScript #OpenSource
-
Added, updated & simplified the growing collection of darkroom-related calculators and super happy how elegant and concise the code has turned out, making it super easy to add more of them in the future.
I think it's also another great, if minimal, example to illustrate how otherwise completely separate https://thi.ng/umbrella packages can seamlessly compose/combine to enable a reactive dataflow UI, all without the need for any virtual DOMs and/or completely over-the-top frameworks like React & co. It's also doing so via mostly JS-native data structures for declaring the UI (plain objects/arrays/iterables) and various constructs directly managing the reactive value streams, thus providing a lot more finegrained control over UI updates/timing/throttling). Any value changes done by the user only trigger specific, pin-point calculations which then result in equally specific UI updates to show new results. Any user action only ever triggers the minimum amount of work needed to reflect the new state.
Calculators:
https://demo.thi.ng/umbrella/darkroom-calc/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/darkroom-calc/srcThe attached images show the source code of the entire main app (UI root) and one of the calculators...
Ps. Please let me know if you'd like to see more of these posts in the future. I'm tempted to launch season 2 of #HowToThing (see link below for 30 previous mini projects/tutorials) — but since this is very time consuming to produce & document these projects/examples, and because there has been _very little feedback_ to these previous projects/posts, I first need to gauge interest... Thank you! 🫶
https://codeberg.org/thi.ng/umbrella#howtothing
#ThingUmbrella #Darkroom #Calculator #Tool #Reactive #UI #WebDev #TypeScript #JavaScript #OpenSource
-
Added, updated & simplified the growing collection of darkroom-related calculators and super happy how elegant and concise the code has turned out, making it super easy to add more of them in the future.
I think it's also another great, if minimal, example to illustrate how otherwise completely separate https://thi.ng/umbrella packages can seamlessly compose/combine to enable a reactive dataflow UI, all without the need for any virtual DOMs and/or completely over-the-top frameworks like React & co. It's also doing so via mostly JS-native data structures for declaring the UI (plain objects/arrays/iterables) and various constructs directly managing the reactive value streams, thus providing a lot more finegrained control over UI updates/timing/throttling). Any value changes done by the user only trigger specific, pin-point calculations which then result in equally specific UI updates to show new results. Any user action only ever triggers the minimum amount of work needed to reflect the new state.
Calculators:
https://demo.thi.ng/umbrella/darkroom-calc/Source code:
https://codeberg.org/thi.ng/umbrella/src/branch/develop/examples/darkroom-calc/srcThe attached images show the source code of the entire main app (UI root) and one of the calculators...
Ps. Please let me know if you'd like to see more of these posts in the future. I'm tempted to launch season 2 of #HowToThing (see link below for 30 previous mini projects/tutorials) — but since this is very time consuming to produce & document these projects/examples, and because there has been _very little feedback_ to these previous projects/posts, I first need to gauge interest... Thank you! 🫶
https://codeberg.org/thi.ng/umbrella#howtothing
#ThingUmbrella #Darkroom #Calculator #Tool #Reactive #UI #WebDev #TypeScript #JavaScript #OpenSource
-
tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...
Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...
In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.
So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)
Naive approach 🫣:
```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!
Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!
So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...
[1] Permutations = 400 + 400^2 + 400^3
#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization
-
tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...
Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...
In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.
So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)
Naive approach 🫣:
```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!
Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!
So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...
[1] Permutations = 400 + 400^2 + 400^3
#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization
-
tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...
Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...
In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.
So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)
Naive approach 🫣:
```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!
Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!
So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...
[1] Permutations = 400 + 400^2 + 400^3
#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization
-
tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...
Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...
In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.
So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)
Naive approach 🫣:
```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!
Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!
So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...
[1] Permutations = 400 + 400^2 + 400^3
#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization
-
tl;dr Using https://thi.ng/column-store to accelerate tag intersection queries by a factor of 880x...
Working on the static website generator/export plugin for my personal knowledge tool has been one of the main projects this past month. A key part of this setup is tagging, not just simple flat keywords/categories, but actually treating tags as sets. The system doesn't just allow browsing content by a single tag, but also supports adding (or removing) tags to narrow or widen the current topic. E.g. The combination of "3d + geometry + typescript" would select only works which have all of these three tags...
In the local version of my tool there's no limit to the number of tags (and it also supports tag negation), but for the static site generation I have to limit the set size (due to combinatoric explosion) and pre-compute all possible permutations, then create HTML documents for each these individual combinations which actually produce results.
So far I'm having ~400 unique tags in use, meaning if I want to aim for a max set size of 3, there're theoretically ~64,000,000 possibilities to check[1]! For the roughly 3500 content items used for testing, a naive JS approach to filter the result array and only retain items matching the entire current permutation is so extremely slow, that I stopped the process after 3.5 minutes just for the first 250k (aka 0.4%) of the 64 million permutations, i.e. at that rate the full process would have taken ~15 hours, pretty slow for a SSG... :)
Naive approach 🫣:
```
permutation = ["3d", "geometry", "typescript"]
results.filter(item => permutation.every(tag => item.tags.includes(tag)))
```But since I'm using https://thi.ng/column-store as my database, such queries can be optimized by a few magnitudes, since here these intersection queries are applied only to bitfields (explained in the pkg readme). This results in all 64+ million permutations being processed in just 62 seconds (1+ million per second). Quite the difference, i.e. ~880x faster than the above approach!
Also, of these 64 million initial possibilities, there're fewer unique ones (excluding duplicates and ignoring ordering), and currently only ~24,000 are actually producing a result. Still, that's 24,000 index pages to generate & host and it's, of course, far, far too much!
So I will have to also spend more effort curating and severely reducing the tag vocabulary, at least the subset used for the website. On the other hand, I think this system will really help with browsing this large body/archive of work much more meaningfully than the boring single-tag/category approach most websites are offering. And it will do so without any backend (other than file hosting)...
[1] Permutations = 400 + 400^2 + 400^3
#ThingUmbrella #Tagging #Intersection #Query #Bitfield #WebDev #JavaScript #TypeScript #Optimization
-
#ReleaseThursday A historic moment in the https://thi.ng/umbrella microverse!
I just re-published all 214 projects, the first release referencing their new home at @Codeberg with updated links in all packages, examples, readmes and other documentation... 🎉
(Also added a prominent note [hopefully prominent enough!] at the top of the main readme to inform about this project migration...)
Happy Coding! :) The detangling from US big tech continues...
#ThingUmbrella #OpenSource #Codeberg #TypeScript #JavaScript
-
#ReleaseThursday A historic moment in the https://thi.ng/umbrella microverse!
I just re-published all 214 projects, the first release referencing their new home at @Codeberg with updated links in all packages, examples, readmes and other documentation... 🎉
(Also added a prominent note [hopefully prominent enough!] at the top of the main readme to inform about this project migration...)
Happy Coding! :) The detangling from US big tech continues...
#ThingUmbrella #OpenSource #Codeberg #TypeScript #JavaScript
-
#ReleaseThursday A historic moment in the https://thi.ng/umbrella microverse!
I just re-published all 214 projects, the first release referencing their new home at @Codeberg with updated links in all packages, examples, readmes and other documentation... 🎉
(Also added a prominent note [hopefully prominent enough!] at the top of the main readme to inform about this project migration...)
Happy Coding! :) The detangling from US big tech continues...
#ThingUmbrella #OpenSource #Codeberg #TypeScript #JavaScript