#diyalgorithms — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #diyalgorithms, aggregated by home.social.
-
CW: long, algorithms on the fedi
@hipsterelectron @aud @ireneista @impactology for example, i still haven't had time to get a MWE together even if all the pieces are in place, but it's simply not true that activitypub can't do custom feeds. the protocol explicitly gives a key
streamsas "a list of supplementaryCollectionswhich may be of interest," and it is trivially implemented in mastodon by modifying the list system to allow lists to contain a list of posts, rather than a list of accounts. one could go further by allowing people to make those feeds public and followable, again both implicitly via mastohacking or explicitly with the protocol since they areCollectionslike any other.That would mean that you could have algorithmic feeds computed by a single skiddie with a python script on their laptop that gets run when they feel like it all the way through the same kind of big-world feeds bluesky imagines. your instance is the thing that shoulders all the expensive parts of serving the feed and federation as fedi works now, rather than the atproto model where to have a feed means you need to run a pretty beefy, always-on server. I was working on a json-able DAG system of features and transformations so that if someone had a feed design that they liked, you could just copy it, tweak it, make it your own, and so on - and eventually exposing that via mastodon itself so that you don't even need the additional other computer, your instance handles the whole thing for you.
So algorithmic feeds are things made by people that respect current norms about post access by going through instances with your account as the agent and hosted by the instances we (hopefully) help sustain either with donations or time, rather than products that we are lucky enough to rent.
-
one weird trick to implement #DIYAlgorithms: make the masto
listsAPI endpoint accept an array of Post IDs. boom, then you can implement whatever the fuck algorithm you want, wherever you want, and all the server needs to do is show those posts. wishing a kindly felt "eat my ass" to atprotocol https://github.com/glitch-soc/mastodon/compare/main...sneakers-the-rat:mastodon:feature/postlists -
Alright i am working on #DIYAlgorithms again after a long hiatus for other stuff, and i have finally been coerced into learning Python's asyncio instead of just using
threadsandmultiprocessing. If what ppl need to think the fedi is cool is feed algos, we are gonna have the most transparent, most runnable on a raspi, least server-intensive, least surveillance capitalist algos on the web. -
@mala @algorithms @davidslifka
@mala Hi! Thanks for doing this. Here are a hashtag and some links that may be relevant, in case you haven't seen them:
https://github.com/mitotic/pinafore-mahoot/blob/master/README.md
-
@lili
I agree, but I also think there's a balance to be struck with opinionated design, with masto itself being a decent case study. like inevitably someone will want to make a feed that sorts people they interact with more frequently higher on their feed, which seems very normal to me.so what might other ways of making that healthier look like? one way might be to add some explicit noise, weighting, or damping parameter that prevents that from being the most dominant feature. if that parameter is under their control, they can decrease it until it isn't a harmful feedback loop. to help with that, another idea might be to offer a toggle switch to generate an "inverse feed" across the whole algorithm or a single parameter on it - what is getting filtered out of my feed as a result of how it's designed? that would make it easier to identify things that might be driving them into some unhelpful bubble or etc.
just thoughts, I think the whole question is v far from simple so thinking about what a healthy social internet looks like is never final or complete for me
(forgive me for tagging and making public, this is largely a way of taking notes to self through another imperfect UI lol #DIYAlgorithms )
-
Ok I'm gonna try and do 1 hour on #DIYAlgorithms per programming day until I get a reasonable prototype with some basic Algo examples, that'll be here: https://git.jon-e.net/jonny/diyalgo
prior context here: https://wiki.jon-e.net/Masto_Forks/DIY_Algorithmsthe first things I think are simple but the most requested ones I've seen on here:
- #NRecent : feed with just the most recent n posts from everyone - to balance us adhd hyperposters with ppl more careful with their words.
- #ChronoMatching - to rotate your friends who post at different times from you due to time zone/sleep cycle/etc. into your posting cycle.
- #ExclusiveFeeds (@lina ) - a copy of your home feed without posts that have been included in an exclusive feed (and the exclusive feed itself) for situations like eg. #monsterdon where you don't want to mute something but you don't want it to flood your feed
for now this all just works by POSTing an array of status IDs to a modified lists endpoint in order to be maximally general/interoperable with other Algo implementations. things like NRecent make me think there should be some way to do diffs instead, where you remove one item on the feed and/or insert one in a position too tho. maybe I am preoptimizing.
I am thinking the way to do this will be to have a system of
a) features that can be lazily computed for different object types (status, account, or derived types like lists of accounts, etc.)
b) a tree of procession steps with each node returning a value bounded 0-1 and the root being used as the position on the feed. posts that should not be put on a feed have a special value of NA that terminates processing the rest of the tree.
so that
c) an algo can be defined as an easily serializable eg. JSON/ #JSONLD-like format that specifies features as dependencies and then describes the processing tree with an array of node objects, each pointing to some node type (and where it comes from, to plan ahead for plugins), parameters, and how it's connected in the tree.
d) eventually, new URL routes/API endpoints so you can link to, follow, and re-use other people's (opt-in public) algos in-medium.the goal is to get very low-fi, ie. no big data, inaccessible hardware requiring NLP or neural nets and shit - explainability and transparency by designing simpler qualities that people combine in specific ways. yno. like .... designing an algorithm on purpose. for our own benefit, not some investor.
-
Ok I'm gonna try and do 1 hour on #DIYAlgorithms per programming day until I get a reasonable prototype with some basic Algo examples, that'll be here: https://git.jon-e.net/jonny/diyalgo
prior context here: https://wiki.jon-e.net/Masto_Forks/DIY_Algorithmsthe first things I think are simple but the most requested ones I've seen on here:
- #NRecent : feed with just the most recent n posts from everyone - to balance us adhd hyperposters with ppl more careful with their words.
- #ChronoMatching - to rotate your friends who post at different times from you due to time zone/sleep cycle/etc. into your posting cycle.
- #ExclusiveFeeds (@lina ) - a copy of your home feed without posts that have been included in an exclusive feed (and the exclusive feed itself) for situations like eg. #monsterdon where you don't want to mute something but you don't want it to flood your feed
for now this all just works by POSTing an array of status IDs to a modified lists endpoint in order to be maximally general/interoperable with other Algo implementations. things like NRecent make me think there should be some way to do diffs instead, where you remove one item on the feed and/or insert one in a position too tho. maybe I am preoptimizing.
I am thinking the way to do this will be to have a system of
a) features that can be lazily computed for different object types (status, account, or derived types like lists of accounts, etc.)
b) a tree of procession steps with each node returning a value bounded 0-1 and the root being used as the position on the feed. posts that should not be put on a feed have a special value of NA that terminates processing the rest of the tree.
so that
c) an algo can be defined as an easily serializable eg. JSON/ #JSONLD-like format that specifies features as dependencies and then describes the processing tree with an array of node objects, each pointing to some node type (and where it comes from, to plan ahead for plugins), parameters, and how it's connected in the tree.
d) eventually, new URL routes/API endpoints so you can link to, follow, and re-use other people's (opt-in public) algos in-medium.the goal is to get very low-fi, ie. no big data, inaccessible hardware requiring NLP or neural nets and shit - explainability and transparency by designing simpler qualities that people combine in specific ways. yno. like .... designing an algorithm on purpose. for our own benefit, not some investor.
-
Ok I'm gonna try and do 1 hour on #DIYAlgorithms per programming day until I get a reasonable prototype with some basic Algo examples, that'll be here: https://git.jon-e.net/jonny/diyalgo
prior context here: https://wiki.jon-e.net/Masto_Forks/DIY_Algorithmsthe first things I think are simple but the most requested ones I've seen on here:
- #NRecent : feed with just the most recent n posts from everyone - to balance us adhd hyperposters with ppl more careful with their words.
- #ChronoMatching - to rotate your friends who post at different times from you due to time zone/sleep cycle/etc. into your posting cycle.
- #ExclusiveFeeds (@lina ) - a copy of your home feed without posts that have been included in an exclusive feed (and the exclusive feed itself) for situations like eg. #monsterdon where you don't want to mute something but you don't want it to flood your feed
for now this all just works by POSTing an array of status IDs to a modified lists endpoint in order to be maximally general/interoperable with other Algo implementations. things like NRecent make me think there should be some way to do diffs instead, where you remove one item on the feed and/or insert one in a position too tho. maybe I am preoptimizing.
I am thinking the way to do this will be to have a system of
a) features that can be lazily computed for different object types (status, account, or derived types like lists of accounts, etc.)
b) a tree of procession steps with each node returning a value bounded 0-1 and the root being used as the position on the feed. posts that should not be put on a feed have a special value of NA that terminates processing the rest of the tree.
so that
c) an algo can be defined as an easily serializable eg. JSON/ #JSONLD-like format that specifies features as dependencies and then describes the processing tree with an array of node objects, each pointing to some node type (and where it comes from, to plan ahead for plugins), parameters, and how it's connected in the tree.
d) eventually, new URL routes/API endpoints so you can link to, follow, and re-use other people's (opt-in public) algos in-medium.the goal is to get very low-fi, ie. no big data, inaccessible hardware requiring NLP or neural nets and shit - explainability and transparency by designing simpler qualities that people combine in specific ways. yno. like .... designing an algorithm on purpose. for our own benefit, not some investor.
-
Ok I'm gonna try and do 1 hour on #DIYAlgorithms per programming day until I get a reasonable prototype with some basic Algo examples, that'll be here: https://git.jon-e.net/jonny/diyalgo
prior context here: https://wiki.jon-e.net/Masto_Forks/DIY_Algorithmsthe first things I think are simple but the most requested ones I've seen on here:
- #NRecent : feed with just the most recent n posts from everyone - to balance us adhd hyperposters with ppl more careful with their words.
- #ChronoMatching - to rotate your friends who post at different times from you due to time zone/sleep cycle/etc. into your posting cycle.
- #ExclusiveFeeds (@lina ) - a copy of your home feed without posts that have been included in an exclusive feed (and the exclusive feed itself) for situations like eg. #monsterdon where you don't want to mute something but you don't want it to flood your feed
for now this all just works by POSTing an array of status IDs to a modified lists endpoint in order to be maximally general/interoperable with other Algo implementations. things like NRecent make me think there should be some way to do diffs instead, where you remove one item on the feed and/or insert one in a position too tho. maybe I am preoptimizing.
I am thinking the way to do this will be to have a system of
a) features that can be lazily computed for different object types (status, account, or derived types like lists of accounts, etc.)
b) a tree of procession steps with each node returning a value bounded 0-1 and the root being used as the position on the feed. posts that should not be put on a feed have a special value of NA that terminates processing the rest of the tree.
so that
c) an algo can be defined as an easily serializable eg. JSON/ #JSONLD-like format that specifies features as dependencies and then describes the processing tree with an array of node objects, each pointing to some node type (and where it comes from, to plan ahead for plugins), parameters, and how it's connected in the tree.
d) eventually, new URL routes/API endpoints so you can link to, follow, and re-use other people's (opt-in public) algos in-medium.the goal is to get very low-fi, ie. no big data, inaccessible hardware requiring NLP or neural nets and shit - explainability and transparency by designing simpler qualities that people combine in specific ways. yno. like .... designing an algorithm on purpose. for our own benefit, not some investor.
-
Ok I'm gonna try and do 1 hour on #DIYAlgorithms per programming day until I get a reasonable prototype with some basic Algo examples, that'll be here: https://git.jon-e.net/jonny/diyalgo
prior context here: https://wiki.jon-e.net/Masto_Forks/DIY_Algorithmsthe first things I think are simple but the most requested ones I've seen on here:
- #NRecent : feed with just the most recent n posts from everyone - to balance us adhd hyperposters with ppl more careful with their words.
- #ChronoMatching - to rotate your friends who post at different times from you due to time zone/sleep cycle/etc. into your posting cycle.
- #ExclusiveFeeds (@lina ) - a copy of your home feed without posts that have been included in an exclusive feed (and the exclusive feed itself) for situations like eg. #monsterdon where you don't want to mute something but you don't want it to flood your feed
for now this all just works by POSTing an array of status IDs to a modified lists endpoint in order to be maximally general/interoperable with other Algo implementations. things like NRecent make me think there should be some way to do diffs instead, where you remove one item on the feed and/or insert one in a position too tho. maybe I am preoptimizing.
I am thinking the way to do this will be to have a system of
a) features that can be lazily computed for different object types (status, account, or derived types like lists of accounts, etc.)
b) a tree of procession steps with each node returning a value bounded 0-1 and the root being used as the position on the feed. posts that should not be put on a feed have a special value of NA that terminates processing the rest of the tree.
so that
c) an algo can be defined as an easily serializable eg. JSON/ #JSONLD-like format that specifies features as dependencies and then describes the processing tree with an array of node objects, each pointing to some node type (and where it comes from, to plan ahead for plugins), parameters, and how it's connected in the tree.
d) eventually, new URL routes/API endpoints so you can link to, follow, and re-use other people's (opt-in public) algos in-medium.the goal is to get very low-fi, ie. no big data, inaccessible hardware requiring NLP or neural nets and shit - explainability and transparency by designing simpler qualities that people combine in specific ways. yno. like .... designing an algorithm on purpose. for our own benefit, not some investor.
-
@futurebird
i gotta get back on this. I bet I could put this together by this evening. how about instead of having it be an option for your home feed, it's one of many different feeds you can select between? or would you rather it be an option for every feed?
#DIYAlgorithms -
#DIYAlgorithms on Mastodon:
https://github.com/hodgesmr/mastodon_digest#algorithm-options
https://social.coop/@jonny/109515350051727349
https://social.coop/@jonny/109545449455062668
https://mauforonda.github.io/mastodon_digest/
https://mauforonda.github.io/mastodon_links/
https://social.coop/@[email protected]ne/108220420990358343
Do you have more?
-
it seems like a simple #DIYAlgorithms would be a #TimeWarp algo: the posting time for people you mutually interact with more decays more slowly, or is stickier at the top of your feed.
maybe tunable with two single exponential functions for
- strength of time warping and
- duration of past interactions to compute time warping from
or whatevs -
@LeonDLotter @ryneches @x1l3f @MattHodges @mauforonda @icymi_law @Colarusso @Sho_Ishiguro
there's also a working version that is going to be coming to your very instance shortly:
https://social.coop/@jonny/109545449455062668
#DIYAlgorithms -
ok it's #DIYAlgorithms o'clock
Say I want an algo to surface "posts by my friends who are in different timezones"
at a high level... I could try and estimate a distribution of posting times for my mutuals and then try and promote statuses by the difference... or, I could take my interactions and find ppl who mostly post outside my active times but I still interact with their posts a lot (eg. someone else or they boost around the clock until I am usually awake)... latter seems better to me.
-
CW: long. instance governance and cooperation
@NicoleCRust @admin a couple more things: part of this is about experimenting with different relationships we can have with our communication systems. the normal model in academia is the journal, a communication venue usually run by a small, closed group of board members and editors, usually without much consideration to its economic structure (or so says the preponderance of for-profit APC-driven journals). The question is whether we can operate differently, by taking collective ownership over the means of communication, and practicing what that looks like. So figuring out a way to make that meaningful, and ideally feel good to do by being part of a cooperative community as opposed to feeling like a burden or a regrettable ethical failure as is the case with most contemporary journals is the goal. That, of course necessarily means taking into account the busy schedules of profs and other academic workers, so it's definitely top of mind.
Another part is experimenting with credit for things that usually don't get credit. Sure you can put on a CV that you "helped with neuromatchstodon," but that doesn't really look like anything. So both for professional purposes as well as helping ppl feel proud of the work they've done on infrastructure for the commons, we're building that credit into the communication and information systems themselves.
Eg. on the wiki I have a user page: https://wiki.neuromatch.io/Jonny_Saunders
There is a section "Contributions," which is automatically populated with all of the things I have contributed to on the instance. That's from a property that can be put on any page listing who contributed to something: https://wiki.neuromatch.io/Property:Has_Contributor
usually that is part of a form, so you are both prompted to give yourself credit as well as don't need to understand how the wiki works. We also have a WikiBot on the discord that can take your posts and embed them on the wiki with reference to you as the source, so even very minor contributions to a project can be creditable. The goal is to make the work not be thankless, where it's possible to just point to "look here is what I have contributed," as well as to make the structure of the instance not be some faceless bureaucracy - all information and all rules are socially defined, they were made in a time and a place by people, and those people can be you!We can also build additional things off the wiki or integrate them with it, so eg. @patrickmineault was talking about #DIYAlgorithms earlier today: we can make a framework where we can have Algo "presets" in structured and human readable data on the wiki and swap them around, make them easy to fork and make different versions of, and then also link them to a system of credit and dialogue about them. So that might include the ability to link back to discussions you have about papers, so we can go from "credit for work done on the instance" to "the instance as a medium for a different kind of academic credit en route to replacing the extremely broken system of citations from journal articles to other journal articles being the only thing that matters"
anyway I am just trying to round out the thought a bit less polemically, but this is the subject of https://arxiv.org/abs/2209.07493 which is more of a long-term strategy and technical blueprint but is grounded in making distributed systems of governance and sensemaking
-
@neuralreckoning @patrickmineault @kordinglab yes. this is the idea behind #DIYAlgorithms (am afk but that tag should have my posts at least from this instance kn the matter, otherwise on social.coop) - making an API endpoint where you can POST status IDs into it so they can be read in-medium via the lists interface. so any Algo you want to cook up you could hook up to that. need to see if lists can be made public, if so then we could have any number of different public algo feeds on the instance:)
-
It works!!!!!!!! An API Endpoint to post statuses directly into lists so people can make their own #DIYAlgorithms . Now some cleanup and a bit of UX stuff but i can't believe i actually wrote code in #Ruby i am proud of myself. :)
#MastoDev -
Do we have any #Ruby or #Rails #RubyOnRails people on our instance yet??? would be awesome to have someone to work with because I truly do not know what I'm doing and I want to merge what I've been doing with #DIYAlgorithms to the instance at some point....
-
oh my frick i figured out how to do rails migrations so now it's just a matter of actually adding the statuses to the list feed when posting to the API and then we will have a way to do #DIYAlgorithms using lists.
https://github.com/glitch-soc/mastodon/commit/90b90213f53472bf3d40df5e0d7c65d48b49e38d -
@tychi Yes - I think the shortest path there is to have it handled entirely by the person, with no code running on the server for the algorithm. So at least how i'm thinking about it it would be entirely compatible with lists as implemented in alternate frontends/mobile apps. That I think would be very complementary to how you're thinking :)
-
I need some #FediDev help:
So If we want to really kill birdsite, we need to do things that are impossible on corporate platformsI have been working on one idea, #DIYAlgorithms, by adding an API endpoint to let masto lists be manually controlled lists of statuses (instead of lists of accounts), so an external algo can autopopulate them.
I have no idea what i'm doing tho, and we need to move quick before another centralized platform really takes off.
help plz?
https://github.com/sneakers-the-rat/mastodon/tree/feature/postlists -
Getting near what seems like a first draft of #DIYAlgorithms by allowing you to POST statuses by IDs to a list (instead of just being able to add accounts). Now just need to figure out how to uh get stuff out of redis with a specific set of IDs rather than just by a range that autosorts.
The complete lack of developer documentation has made this... interesting... and i basically am having to write my own. https://wiki.jon-e.net/Masto_Forks/DIY_Algorithms
Well that and I #DontKnowRuby lmao
-
@bo thank you :) so I just set myself up with rubygems as an ide to get started somewhere, and I'm trying to figure out how to modify the Lists feature of masto to let you POST a series of statuses to it for #DIYAlgorithms ( https://wiki.jon-e.net/Masto_Forks/DIY_Algorithms ), but having a hard time stepping back through the models, which seem to be autogenerated by some schema. I found docs for ActiveRecord, but I can't really figure out like what schema tools are in use/how my changes affect build
-
thinking about #DIYAlgorithms again, specifically implementation. one easy lift way of doing that is having "lists" where instead of controlling the accounts or hashtags they contain, you get a POST endpoint where you can inject posts themselves in order. that way rather than needing to implement them serverside it would be possible to have your own skiddy python algo running on a raspi, streaming your feeds and managing your lists