#ipld — Public Fediverse posts

I'm happy to announce that today is my first day at the #IPFS Foundation. I'll be working on tooling, #DASL, #IPLD and related tech like #ATProto. With a focus on geo. So I'm doing more #geospatial things again. Stay tuned #FOSS4G.

@agowa338 "self-distract". lol. I can't say for real because i'm just getting into #ipfs and formal local-first, but I've found quite a few videos about #ipfs #ipns #ipld on youtube. published in the last 5 years or so.

You might want to check out https://www.inkandswitch.com You probably heard of them but i wanted to post it for completeness' sake.

@agowa338 "self-distract". lol. I can't say for real because i'm just getting into #ipfs and formal local-first, but I've found quite a few videos about #ipfs #ipns #ipld on youtube. published in the last 5 years or so.

You might want to check out https://www.inkandswitch.com You probably heard of them but i wanted to post it for completeness' sake.

@agowa338 "self-distract". lol. I can't say for real because i'm just getting into #ipfs and formal local-first, but I've found quite a few videos about #ipfs #ipns #ipld on youtube. published in the last 5 years or so.

You might want to check out https://www.inkandswitch.com You probably heard of them but i wanted to post it for completeness' sake.

@agowa338 "self-distract". lol. I can't say for real because i'm just getting into #ipfs and formal local-first, but I've found quite a few videos about #ipfs #ipns #ipld on youtube. published in the last 5 years or so.

You might want to check out https://www.inkandswitch.com You probably heard of them but i wanted to post it for completeness' sake.

@agowa338 "self-distract". lol. I can't say for real because i'm just getting into #ipfs and formal local-first, but I've found quite a few videos about #ipfs #ipns #ipld on youtube. published in the last 5 years or so.

You might want to check out https://www.inkandswitch.com You probably heard of them but i wanted to post it for completeness' sake.

@DimlyLitCorners got some updates from #IPFS leadership on BlueSky:

The crypto algorithm that IPFS uses for public keys and signing mutable names in #IPNS is getting added to #WebCryptography https://groups.google.com/a/chromium.org/g/blink-dev/c/T2kriFdjXsg/m/ZeD_PoLXBwAJ?pli=1

There is a draft spec for CBOR which IPFS uses to store structured data int he #IPLD graph https://datatracker.ietf.org/doc/draft-caballero-cbor-cborc42/

They are discussing it at the #IETF meeting this week in Spain

It seems they are breaking IPFS up into smaller pieces to get it through standards bodies more easily!

@DimlyLitCorners got some updates from #IPFS leadership on BlueSky:

The crypto algorithm that IPFS uses for public keys and signing mutable names in #IPNS is getting added to #WebCryptography https://groups.google.com/a/chromium.org/g/blink-dev/c/T2kriFdjXsg/m/ZeD_PoLXBwAJ?pli=1

There is a draft spec for CBOR which IPFS uses to store structured data int he #IPLD graph https://datatracker.ietf.org/doc/draft-caballero-cbor-cborc42/

They are discussing it at the #IETF meeting this week in Spain

It seems they are breaking IPFS up into smaller pieces to get it through standards bodies more easily!

@DimlyLitCorners got some updates from #IPFS leadership on BlueSky:

The crypto algorithm that IPFS uses for public keys and signing mutable names in #IPNS is getting added to #WebCryptography https://groups.google.com/a/chromium.org/g/blink-dev/c/T2kriFdjXsg/m/ZeD_PoLXBwAJ?pli=1

There is a draft spec for CBOR which IPFS uses to store structured data int he #IPLD graph https://datatracker.ietf.org/doc/draft-caballero-cbor-cborc42/

They are discussing it at the #IETF meeting this week in Spain

It seems they are breaking IPFS up into smaller pieces to get it through standards bodies more easily!

@DimlyLitCorners got some updates from #IPFS leadership on BlueSky:

The crypto algorithm that IPFS uses for public keys and signing mutable names in #IPNS is getting added to #WebCryptography https://groups.google.com/a/chromium.org/g/blink-dev/c/T2kriFdjXsg/m/ZeD_PoLXBwAJ?pli=1

There is a draft spec for CBOR which IPFS uses to store structured data int he #IPLD graph https://datatracker.ietf.org/doc/draft-caballero-cbor-cborc42/

They are discussing it at the #IETF meeting this week in Spain

It seems they are breaking IPFS up into smaller pieces to get it through standards bodies more easily!

@DimlyLitCorners got some updates from #IPFS leadership on BlueSky:

The crypto algorithm that IPFS uses for public keys and signing mutable names in #IPNS is getting added to #WebCryptography https://groups.google.com/a/chromium.org/g/blink-dev/c/T2kriFdjXsg/m/ZeD_PoLXBwAJ?pli=1

There is a draft spec for CBOR which IPFS uses to store structured data int he #IPLD graph https://datatracker.ietf.org/doc/draft-caballero-cbor-cborc42/

They are discussing it at the #IETF meeting this week in Spain

It seems they are breaking IPFS up into smaller pieces to get it through standards bodies more easily!

I'm excited to be at Hashberg event in Berlin. #ipfs #ipld #dweb

Agregore 2.12.0 - Peersky theme compat, Local AI onboarding improvements
=> https://agregore.mauve.moe
A minimal web browser for the distributed web
- Enable people to make and use local first apps using the web
- Be minimal (fewer built-in features, leave more to the OS)
- Be open to anything p2p / decentralized / local-first
- Rely on web extensions for extra functionality
- Work with mesh networks / Bluetooth Low Energy networks
Changelog:
=> https://github.com/AgregoreWeb/agregore-browser/releases/tag/v2.12.0
#IPFS #IPLD #Hypercore #Gemini #SSB #BitTorrent

Agregore 2.8.2 - Font Based Syntax Highlighting
=> https://agregore.mauve.moe
A minimal web browser for the distributed web
- Enable people to make and use local first apps using the web
- Be minimal (fewer built-in features, leave more to the OS)
- Be open to anything p2p / decentralized / local-first
- Rely on web extensions for extra functionality
- Work with mesh networks / Bluetooth Low Energy networks
Changelog:
=> https://github.com/AgregoreWeb/agregore-browser/releases/tag/v2.8.2
#IPFS #IPLD #Hypercore #Gemini #SSB #BitTorrent

Agregore 2.7.0 - Settings Page
=> https://agregore.mauve.moe
A minimal web browser for the distributed web
- Enable people to make and use local first apps using the web
- Be minimal (fewer built-in features, leave more to the OS)
- Be open to anything p2p / decentralized / local-first
- Rely on web extensions for extra functionality
- Work with mesh networks / Bluetooth Low Energy networks
Changelog:
=> https://github.com/AgregoreWeb/agregore-browser/releases/tag/v2.7.0
#IPFS #IPLD #Hypercore #Gemini #SSB #BitTorrent

Agregore 2.7.0 - Settings Page
=> https://agregore.mauve.moe
A minimal web browser for the distributed web
- Enable people to make and use local first apps using the web
- Be minimal (fewer built-in features, leave more to the OS)
- Be open to anything p2p / decentralized / local-first
- Rely on web extensions for extra functionality
- Work with mesh networks / Bluetooth Low Energy networks
Changelog:
=> https://github.com/AgregoreWeb/agregore-browser/releases/tag/v2.7.0
#IPFS #IPLD #Hypercore #Gemini #SSB #BitTorrent

It's been great working with @b_fiive & the @qri_io team on the #WNFS and #UCAN specs.

Encrypted files on @IPFS, with a number of other interesting properties :P
---
RT @qri_io
A presentation & demo of our plan to use our new @golang implementation of @FISSIONcodes's WebNative Filesystem Version 2 in @qri_io:

https://www.youtube.com/watch?v=Hum5vfaQ9e8

#golang #IPFS #filesystem #WNFS #web3 #distributedweb #IPLD #datacatalog #linkeddata #…
https://twitter.com/qri_io/status/1470786956917420037

starting with research data is a constrained problem to get some of the fundamentals of data structures, transport, and interop down so that in the next phase we can build identity, peer federation, and communication on top of that. we want to be able to handle bigg blobs of bits as well as liddel messages, and so making a p2p transport layer for RDF-like things leads naturally into the second phase goal of bridging the fediverse which runs on #JSONLD to p2p.

I still am not committed to actually building for RDF, want to learn from prior art without being weighed down by 20+ years of convoluted technical history, but there is a lot to learn about linking graphs there. I think one of the main things I am uncommitted to is needing globally unique URIs for everything, rather than having some things uniquely identifiable (eg. a dataset, a post) with relative locations for other things. That way you can make containers for encryption - peers can refer to the content hash of the encrypted subgraph as location for querying, zero-knowledge mirroring, etc. as well as opaque references to the graph content itself for peers that can decrypt it, but those inner-locations can't be resolved by peers that can't decrypt.

that is still a very unfinished thought, but it's a big missing piece in related technologies like #IPFS where everything must be addressable by CID. its arguably the need that #IPNS and #IPLD are intended to backfill

ok so re-reading #IPFS paper and there are a few things I think in retrospect are undesirable about the #MerkelDAG spec. it's hard to parse them out as separable ideas because they depend on one another, but the main thing I think is how it conflates the structure of a metadata graph, the content of the graph, and the notion of authorship/identity.

In (basic) IPFS, each node contains some data and some links. the data is some unspecified binary blob, the links are all references to hashes of other nodes, and then the hash of all that identifies the node. There are some abstractions like flattened trees that can represent n-depth links, but that's the gist. I'm refreshing myself, so correct me where I'm wrong.

This makes traversing the graph expensive from a naive (cacheless) state- you have to fetch each node and parse its links serially, and since there isn't a notion of authorship except when used to sign a node, you might have to do the resolution process across a lot of the network instead of being able to say "ah ok this is from this identity so I should ask their neighborhood first"

Since the links are untyped, and because of the need for serial resolution, you can't really "plan" queries and move the query logic to the "edges" (in a networking, rather than graph parlance) of the network - the network resolution logic handles all that.

This structure also makes it so you can't "talk about" a node. A node contains its links. The links are directional, so I could make some statement about a node by pointing to it, but I can't, as a third party make a link under my identity, separate from the author and content of the node, that points from some object to another. That makes the network more like a hard drive than a social space.

Further, since links aren't typed, you have to move that metadata inside the node. This makes you need to re-hash each node more than you need to, and since "keys" for identifying different fields in the node aren't themselves links, you can't have any notion of "schema" where a term can be reused. So there isn't really a facility for being able to do graph queries like "find me this type of data whose field has this value" which restricts a whole huge range of possibilities too long to list here. This also makes knowing what the binary data inside a node is potentially impossible without out of band info, depending on how it's encoded. #IPLD and #Multiformats are intended to solve this, post-hoc.

I'll stop there for now, and save what I think could be a different model for later, but I am thinking along the lines of merging with #LinkedData #Triplets , encoding the notion of authorship into links (so that links can have an "utterance" rather than "fact" ontological status), a notion of container/contained for explicit block formation and metadata separation, and formalizing the notion of orthogonal Merkel DAGs to change the points where the content addressing happens to be able to have "graph subunits" that allow for cycles at a "complete" scope but for the purposes of hashing have no cycles. very much #WIP, still at conceptual stage haven't started writing spec yet.

#LongPost #p2p #WorkingInPublic

Sacrificed my night to the productivity gods. The output is the beginnings of the "ipti" CLI which is a tool for authoring / reading #IPLD databases based on my impl of the Prolly Tree spec.

It's not ready for release yet but this will make it easier to integrate into pipelines with bash or other languages without having to touch golang directly.

Indexer library here: https://github.com/RangerMauve/ipld-prolly-indexer

Note I don't have a standard for replicating over the network so it currently reads/writes CAR files

Soon it becomes apparent that, wait, that same electrode group collects multiple timeseries, so it should probably be its own thing that I can reference in multiple places. So the ability to create references is added, and the ability to create anonymous sub-things is also retained, because not everything is reused.

so electrode is its own thing, but it too has its own anonymous untyped things, and I want to reference those. So now I need to create a reference with subselection - "this electrode within electrode group x." This is often done by creating another anonymous subthing within the referring thing - "a table that contains indices with which I select from the thing I'm referring to." When you un-nest the schema, these kinds of references are actually most of the anonymous classes.

This is a pretty tight analogical map to blank nodes, this tension in #RDF where "everything should be uniquely, globally identifiable" but "actually not literally everything because that would be ridiculous." And blank nodes have caused a ton of work to be needed to eg. canonicalize graphs so you can automatically assign unique names to things.

See also #IPLD which makes some different choices, notably being to treat every subgraph as ordered even though graphs don't have an intrinsic ordering: https://ipld.io/design/tricky-choices/ordering/
(Side note every project should have a "tricky choices" section in the docs. Props to protocol labs for that)

Is there an alternative to #ipfs and #ipld out there?

Essentially I want a way to build a DAG/Merkle Tree, a content-addressable storage, where I can define the node format, can store binary data and can find data from remotes via hashes...

All of that is offered by ipfs 😭 but I don't want to write go or javascript for that, I want to be able to use #rust

I am almost at the point where I think of trying #typescript and use the JS implementation of ipfs... almost.

I am, for geeky reasons, mostly on Bluesky these days. Send me a DM if you’d like an invite.

(Geeky reasons: It uses #IPLD, and I *think* I can build a fully static #ATProto server on #IPFS, so my static blog can use ATProto for comments only in HTML and CSS (no JS!))

#ipld is yet another #Linkeddata format but for content adressable data in decentralised systems. DWN use IPLDs as a schema
https://www.youtube.com/watch?v=totVQXYS1N8

J Chris from #FireproofStorage an #IPLD database https://fireproof.storage

Gifting me some hot sauce. And yes that's obviously a fire 🔥 suit, too!

#IPFSThing

OK I'm starting my #p2p #LinkedData reading list to get started drafting a protocol and I'm checking out #IPLD - lots of really good ideas here, and plenty to learn from. It has a bit of a different focus than what I have planned, but some stuff i like and some stuff I can learn from:

• This typology of complete vs incomplete codecs - I'm learning that one of the major ways I differ in thinking from a lot of prior art is an explicit embrace of heterogeneity, vernacularism, and mess as desirable features of an expressive system rather than designed out by engineer types as an error. I think explicitly allowing for incomplete/imperfect translation between schema is super important for systems of expression, after all it's how language works! So I really liked seeing the "Incompleteness is Valid" section. I also love some of the terminology here, "topowild," "plane-mangling," "underkinded."
• How IPFS gateways work - I have always wondered how this works, and it is a pretty concise access point to seeing why some of the ideas in IPLD are good ones. I like Protocol Labs general approach to bridging across protocols, and being able to access data in #IPFS from HTTP is a really important part of how IPFS gets used (eg. by libgen). I want to read more about how other protocols approach (or don't) interoperability like this.
• CIDs are interesting - - but i ultimately I think they collapse too much information because of how they are intended to be used. It binds the metadata and data to a specific codec, which has attractive qualities, but it becomes clear that they had to do a lot of work around the fact that there is no division between metadata and bytestring/binary data for querying, selecting, etc. Having to expand blocks is not really desirable, and it also is related to one of the bigger problems I see with this approach, which makes data modeling pretty damned complicated having to deal with blocks, models, advanced data layouts, schemas, etc. You can also really see how the blockchain stuff starts to seep into the rest of the ecosystem design starting around here and in adjacent stuff like graphsync
• The tricky-choices section is extremely interesting and i wish more projects had something like it. In particular I liked the discussion on why ordering by default is a good decision in graphs/maps that don't necessarily need order.
• The schemas docs are pretty revealing about the direction of the project, values, design priorities, etc. In particular they are "developer"-oriented interfaces, rather than something intended for any old person out there to be able to structure data with. They share some of what I'm thinking about re: structuring existing data, but the combination of the data schema with the serialization has similar points of difficulty as with CIDs. I want to read more about these and ADLs because i dont' have time rn to do but they seem p subtle and worth spending time with.

I depart from a lot of their design decisions, and it's also clear that this is something that evolved in the process of developing IPFS (they say as much) to fill gaps as they were emerging, rather than a foundational part of the ecosystem. In particular I think the blockchain brain ties them to this notion of immutability, append-only stuff which (imperfectly) trades off with needs for privacy and careful scoping/permissions, valuing verifiability and structuredness above ease of expression, and etc. Regardless, interesting to see a bit of the way they think, particularly since they're a bunch of years ahead of me in dealing with the practicalities of implementation.

I'm gonna try and do this project in public, writing as I go on here rather than limiting to an end piece, so if u want to avoid future posts like this from me in the future u can mute the #Longpost and #WorkingInPublic hashtags which will be sort of wandering like this.

#Longpost #Protocols #WorkingInPublic

@Hyolobrika For my part, the #IPFS / #IPLD based data structure I'm going to be using has it's spec here: https://github.com/ipld/ipld/pull/254

And I'm going to be blabbing about the actual grant we're doing on the fedi or Matrix once we have that together (probably early Feb?).

We're mostly focused on getting #RDF working with the data structure, and ActivityStreams will be a specific type of data we're indexing.

Looks like #ActivityStreams over #IPLD is very likely to be a thing some time this year!

Currently drafting up the grant milestones and figuring out budget.

Likely #rust or #golang to start along with some general #RDF goodies.

My hypothesis is that baking database indexes within the data itself will make it efficient to load people's posts over #p2p without needing to rely as much on indexing servers.

Cool thread from birdsite on how #IPLD fits into the #IPFS stack.

https://twitter.com/ryanshahine/status/1605869247535538177

Wrote up my blog post on #p2p databases on top of #IPLD / #IPFS Prolly Trees. Hoping to switch up how databases are approached with a focus on performance and the ability to combine datasets determenistically.

https://blog.mauve.moe/posts/peer-to-peer-databases

We love this charming mascot that the talented @bruno_monts designed for our friends at @Web3Storage!
---
RT @Web3Storage
Our friends at @FISSIONcodes made http://web3.storage stickers! 😍🐙 Thank you, @bmann and team!! #FissionCodes #UCAN #IPLD
https://twitter.com/Web3Storage/status/1598753314849845248

@mauve I honestly thought the #ipld project was all about that, so I'm surprised to learn it's so far not?

For those that have been seeing me ramble about this every chance I get. We've submitted the initial PR for our #IPLD #ProllyTrees spec.

This is a great building block for building #p2p databases which use sparsely replicated indexes to quickly load just the data that's needed.

Once we get some indexing going on top of these things, waiting for stuff to sync will be something you barely need to think about in P2P social apps! 🎉

https://github.com/ipld/ipld/pull/254

Here's my talk on what #IPLD is at #IPFS Camp 2022!

https://www.youtube.com/watch?v=J_Q6hF_lPiM

I was the third talk I was doing in as many hours so excuse me if I was a bit tired. :P

Still pretty happy with how it turned out.

OH: “No-one should have to directly wield a HAMT as a rite of passage to the world of generalized content addressing.” #ipfs #ipld

For those that have been seeing me ramble about this every chance I get. We've submitted the initial PR for our #IPLD #ProllyTrees spec.

This is a great building block for building #p2p databases which use sparsely replicated indexes to quickly load just the data that's needed.

Once we get some indexing going on top of these things, waiting for stuff to sync will be something you barely need to think about in P2P social apps! 🎉

https://github.com/ipld/ipld/pull/254

For those that have been seeing me ramble about this every chance I get. We've submitted the initial PR for our #IPLD #ProllyTrees spec.

This is a great building block for building #p2p databases which use sparsely replicated indexes to quickly load just the data that's needed.

Once we get some indexing going on top of these things, waiting for stuff to sync will be something you barely need to think about in P2P social apps! 🎉

https://github.com/ipld/ipld/pull/254

For those that have been seeing me ramble about this every chance I get. We've submitted the initial PR for our #IPLD #ProllyTrees spec.

This is a great building block for building #p2p databases which use sparsely replicated indexes to quickly load just the data that's needed.

Once we get some indexing going on top of these things, waiting for stuff to sync will be something you barely need to think about in P2P social apps! 🎉

https://github.com/ipld/ipld/pull/254

starting with research data is a constrained problem to get some of the fundamentals of data structures, transport, and interop down so that in the next phase we can build identity, peer federation, and communication on top of that. we want to be able to handle bigg blobs of bits as well as liddel messages, and so making a p2p transport layer for RDF-like things leads naturally into the second phase goal of bridging the fediverse which runs on #JSONLD to p2p.

I still am not committed to actually building for RDF, want to learn from prior art without being weighed down by 20+ years of convoluted technical history, but there is a lot to learn about linking graphs there. I think one of the main things I am uncommitted to is needing globally unique URIs for everything, rather than having some things uniquely identifiable (eg. a dataset, a post) with relative locations for other things. That way you can make containers for encryption - peers can refer to the content hash of the encrypted subgraph as location for querying, zero-knowledge mirroring, etc. as well as opaque references to the graph content itself for peers that can decrypt it, but those inner-locations can't be resolved by peers that can't decrypt.

that is still a very unfinished thought, but it's a big missing piece in related technologies like #IPFS where everything must be addressable by CID. its arguably the need that #IPNS and #IPLD are intended to backfill

starting with research data is a constrained problem to get some of the fundamentals of data structures, transport, and interop down so that in the next phase we can build identity, peer federation, and communication on top of that. we want to be able to handle bigg blobs of bits as well as liddel messages, and so making a p2p transport layer for RDF-like things leads naturally into the second phase goal of bridging the fediverse which runs on #JSONLD to p2p.

I still am not committed to actually building for RDF, want to learn from prior art without being weighed down by 20+ years of convoluted technical history, but there is a lot to learn about linking graphs there. I think one of the main things I am uncommitted to is needing globally unique URIs for everything, rather than having some things uniquely identifiable (eg. a dataset, a post) with relative locations for other things. That way you can make containers for encryption - peers can refer to the content hash of the encrypted subgraph as location for querying, zero-knowledge mirroring, etc. as well as opaque references to the graph content itself for peers that can decrypt it, but those inner-locations can't be resolved by peers that can't decrypt.

that is still a very unfinished thought, but it's a big missing piece in related technologies like #IPFS where everything must be addressable by CID. its arguably the need that #IPNS and #IPLD are intended to backfill

starting with research data is a constrained problem to get some of the fundamentals of data structures, transport, and interop down so that in the next phase we can build identity, peer federation, and communication on top of that. we want to be able to handle bigg blobs of bits as well as liddel messages, and so making a p2p transport layer for RDF-like things leads naturally into the second phase goal of bridging the fediverse which runs on #JSONLD to p2p.

I still am not committed to actually building for RDF, want to learn from prior art without being weighed down by 20+ years of convoluted technical history, but there is a lot to learn about linking graphs there. I think one of the main things I am uncommitted to is needing globally unique URIs for everything, rather than having some things uniquely identifiable (eg. a dataset, a post) with relative locations for other things. That way you can make containers for encryption - peers can refer to the content hash of the encrypted subgraph as location for querying, zero-knowledge mirroring, etc. as well as opaque references to the graph content itself for peers that can decrypt it, but those inner-locations can't be resolved by peers that can't decrypt.

that is still a very unfinished thought, but it's a big missing piece in related technologies like #IPFS where everything must be addressable by CID. its arguably the need that #IPNS and #IPLD are intended to backfill

starting with research data is a constrained problem to get some of the fundamentals of data structures, transport, and interop down so that in the next phase we can build identity, peer federation, and communication on top of that. we want to be able to handle bigg blobs of bits as well as liddel messages, and so making a p2p transport layer for RDF-like things leads naturally into the second phase goal of bridging the fediverse which runs on #JSONLD to p2p.

I still am not committed to actually building for RDF, want to learn from prior art without being weighed down by 20+ years of convoluted technical history, but there is a lot to learn about linking graphs there. I think one of the main things I am uncommitted to is needing globally unique URIs for everything, rather than having some things uniquely identifiable (eg. a dataset, a post) with relative locations for other things. That way you can make containers for encryption - peers can refer to the content hash of the encrypted subgraph as location for querying, zero-knowledge mirroring, etc. as well as opaque references to the graph content itself for peers that can decrypt it, but those inner-locations can't be resolved by peers that can't decrypt.

that is still a very unfinished thought, but it's a big missing piece in related technologies like #IPFS where everything must be addressable by CID. its arguably the need that #IPNS and #IPLD are intended to backfill