Search
1000 results for “commons_protocol”
-
AWS Machine Learning – Harness the power of MCP servers with Amazon Bedrock Agents
Harness the power of MCP servers with Amazon Bedrock Agents
I’ve been digging into this new AWS blog post about Model Context Protocol (MCP) servers and their integration with Amazon Bedrock Agents, and I have to say, I’m pretty excited about what this means for those of us building with AI.
What the heck is MCP anyway?
MCP (Model Context Protocol) is Anthropic’s open protocol for connecting large language models to basically any data source or tool. Think of it as a standard way for AI models to talk to everything from databases to APIs without needing custom code for every single connection. AWS has now integrated this with Amazon Bedrock Agents.
However, in the past, connecting these agents to diverse enterprise systems has created development bottlenecks, with each integration requiring custom code and ongoing maintenance—a standardization challenge that slows the delivery of contextual AI assistance across an organization’s digital ecosystem. This is a problem that you can solve by using Model Context Protocol (MCP), which provides a standardized way for LLMs to connect to data sources and tools.
I’m always looking for ways to simplify infrastructure. I don’t want to write endless custom integrations that I’ll have to maintain forever. MCP promises to solve exactly this problem, which means I can focus more on what I want to build and less on the plumbing.
An ecosystem in the making
What really caught my attention was this bit:
Today, MCP is providing agents standard access to an expanding list of accessible tools that you can use to accomplish a variety of tasks. In time, MCP can promote better discoverability of agents and tools through marketplaces, enabling agents to share context and have common workspaces for better interaction, and scale agent interoperability across the industry.
This is where things get interesting! It’s not just about making individual connections easier—it’s about creating an entire ecosystem of interoperable tools. Are MCPs like APIs?
The right architecture for the job
The post explains that MCP uses a client-server architecture.
Whether you’re connecting to external systems or internal data stores or tools, you can now use MCP to interface with all of them in the same way. The client-server architecture of MCP enables your agent to access new capabilities as the MCP server updates without requiring any changes to the application code.
Ok, I like the separation of concerns. I can update my data sources without touching my application logic, and vice versa.
Real problems, real solutions
The example AWS gives is all about understanding cloud spending:
Imagine asking questions like “Help me understand my Bedrock spend over the last few weeks” or “What were my EC2 costs last month across regions and instance types?” and getting a human-readable analysis of the data instead of raw numbers on a dashboard. The system interprets your intent and delivers precisely what you need—whether that’s detailed breakdowns, trend analyses, visualizations, or cost-saving recommendations.
I like this. In my work, I am often dealing with complex AWS bills. There’s a whole world of innovation around simplifying IT spend (AWS and beyond).
Simple setup (I hope!)
The blog post claims the setup process is straightforward:
You’re now ready to create an agent that can invoke these MCP servers to provide insights into your AWS spend. You can do this by running the python main.py command.
I think I’m gonna try this walkthrough sometime soon. It looks easy enough, and I have not yet played with Agents enough to fully understand what is going on, so this will be a good way to explore.
Future possibilities
The post lists some additional ideas:
A multi-data source agent that retrieves data from different data sources such as Amazon Bedrock Knowledge Bases, Sqlite, or even your local filesystem.
A developer productivity assistant agent that integrates with Slack and GitHub MCP servers.
A machine learning experiment tracking agent that integrates with the Opik MCP server from Comet ML for managing, visualizing, and tracking machine learning experiments directly within development environments.
It’s going to be a fun year exploring all the potential of MCPs and Amazon Bedrock. Let’s go!
-
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
CW: NGI0 core grant from NLnet for Node9 for Hubzilla and (streams) server deployment; CW: long ( characters), Fediverse meta, non-Mastodon Fediverse meta
In case you haven't read that yet: This month, @node9.org ;-) open culture ~ sustainable media is in for a NGI0 Core grant from @NLnet for automated deployment of Hubzilla nodes and (streams) instances. I do hope this gives especially (streams) the push it deserves, seeing as it barely has any public instances anymore.
Here is the original post:Threadiverse Reproducible Deployment
Reproducible deployment for Threadiverse servers
Fediverse is more than short form microblogging. The ActivityPub protocol connects all kinds of software for various communication needs. Some of those are concentrated on long blogs and threaded discussion forums. A common understanding of conversations in ActivityPub and their secure and safe-from-spam implementation is being developed in several fediverse projects. This project focuses on stable and documented automated deployment for two of them - Hubzilla and Streams, including interoperability tests. This will support threadiverse standardization efforts, and help to bring features like group photoalbums and full channel portability between instances.
Interesting, however, how Hubzilla and (streams) count as parts of the Threadiverse now.
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #NLnet #Node9 #Hubzilla #Streams #(streams) -
For the German OWASP Day in Leipzig on November 13 we're excited to announce the first round of speakers/ talks which the program committee determined yesterday.
* @freddy (builds security for the web as a security engineer and manager for Mozilla Firefox) will present "Modern solutions against Cross-Site Leaks (xs-leaks) and #CSRF"
* Shubham Agarwal will raise his voice against "Double-Edged Crime: How Browser Extension Fingerprinting Might Endanger Users and Extensions Alike"
* Nicolas Schickert, Ole Wagner and Matthias Göhring will tackle most companies problem child "#SAP from an Attacker’s Perspective – Common Vulnerabilities and Pitfalls"
* @bkimminich is celebrating the "OWASP Juice Shop 10th anniversary" . There'll be also a Juice Shop training on the 12th!
* While Dr. Daniel Fett will be talking about "How (Not) to Use OAuth in 2024", Kristina Yasuda will tell you "The Crucial Role of Web Protocols and Standards in Digital Wallet Ecosystems" (EUDI Wallet)
* @TimPhSchaefers will demystify #NIS2 and hopefully #NIS2UmsuCG
* Stephan Pinto Spindler will share his experiences wrt "Network Fingerprinting for Securing User Accounts"
* Thomas Barber will give us a short insights into project #foxhound, a taint tracking project using a patched firefox .
More to announced soon! Expect more excellent topics to be announced during the next days!
-
'We cannot exist if we lose the water': In #Honduras, a community resists a #mine polluting the #RíoGuapinol
"#JuanLópez, representative of the Committee for the Defense of Common and Public Goods, told #RadioProgreso that the activists' next step is advocate for the revoking of Inversiones Los Pinares' mining contracts, the environmental license and the exploitation and exploration permit."
by Soli Salgado
October 18, 2022TOCOA, HONDURAS - "The Río Guapinol that streams through the Bajo Aguán valley in northern Honduras has long been a source of drinking, bathing, cleaning, irrigation and cooking water for the surrounding community in Tocoa.
"In the last few years, the river has also provided an education: how to resist an international mining operation that they say contaminates the river in the name of development; how to confront a seemingly compromised justice system when that #resistance goes awry; and — for the women in particular — how to become leaders of a movement upon the indefinite imprisonment of their husbands and sons.
"Vilma Cruz raised her five children using the river, including a son who wound up spending almost two and a half years in jail for protesting the mining as part of the #Guapinol8, a group whose 2019 arrests and long detentions drew international outrage.
"'We don't feel at peace because our water has been endangered,' she said. 'Now, when I go to the river, I feel my chest swell, like I'm not free.'
A mine emerges, a community responds
"About a decade ago, #InversionesLosPinares, formerly the Honduran #EMCO Mining Company and based in #Tocoa, applied for a concession to build an iron oxide mine in the protected #CarlosEscaleras #NationalPark. Then-President Juan Orlando Hernández authorized the request in 2013, a decision locals said was made without following protocol of consulting residents of the area.
"The #OpenPit #mining project was upstream of the #RíoGuapinol, a channel that stems from the larger #RíoAguan, a river that flows through tropical mountains from the Atlantic on the northern side of the #CentralAmerican country.
"When the Río Guapinol in 2018 started to turn a chocolate brown, locals took that as a cue to act against Inversiones Los Pinares.
"#GuapinolResiste, the local community's movement in response to the mining, said in a 2020 report that even before mining began, the construction of the facilities and roads had polluted both the Guapinol and #SanPedro rivers, affecting 14,000 residents who rely on the water for consumption and domestic purposes.
"'Losing the river would mean buying water, and you can't buy water for everything,' said Leonel George, who serves on the Municipal Committee for the Defense of Common and Public Goods, founded in 2018 in response to the mining project.
"The river, he added, is 'closely linked to life and the existence of everything that surrounds us,' noting that the committee's concern goes beyond the effects that contaminated water would have on human beings to the river's surrounding #forests and #biodiversity."
#JusticeForJuan #GuapinolRiver #WaterDefender #SaveTheForests #DefendTheSacred
#NoMiningWithoutConsent #WaterIsLife #SaveTheRivers #EMCO #InversionesLosPinares
#MunicipalCommitteeForTheDefenseOfCommonAndPublicGoods -
Let’s talk about SD-WAN and MPLS.
I had lunch with a telecom and ISP services aggregator/seller the other day. His opinion is that MPLS is a legacy technology, rapidly being replaced by SD-WAN. That’s a long ways from accurate, and honestly I was kind of stunned. Since that type of thinking is out there, and apparently commonly believed, it’s time to inoculate you from SD-WAN hype with a dose of truth.1) SD-WAN isn’t a protocol. It’s an overlay.
2) MPLS is a transport protocol.
3) SD-WAN can’t exist without transport protocols under it.
4) SD-WAN often uses more than one transport protocol. For example, broadband Internet service from an ISP is less expensive than MPLS. That’s why it’s common – very common – for an SD-WAN overlay to direct high priority traffic over an MPLS link, while directing traffic with lower security requirements or less stringent latency requirements over the cheaper broadband link.Now let’s talk about that “legacy technology” perspective.
--Ethernet was in development for several years, but was standardized by the IEEE as 802.3 in 1983, so let’s call that its birthday and say that Ethernet is now 40 years old. It’s not considered a legacy technology because it's still very much in use.
--MPLS is an IETF standard. It’s first RFC was published in 2001, so it’s now 22 years old. And it’s not a legacy technology, either, because, like Ethernet, it’s still very much in use. In fact, various investment analyses are predicting the growth of MPLS over the next five years.MPLS Growth, Quote 1:
According to Market Statsville Group:
“The global Managed MPLS market size is expected to grow from USD 55.6 million in 2021 to USD 97.9 million by 2030… Carriers have increased their network investment in response to the expansion of cloud-based mobile consumer services.” (Link in comments)MPLS Growth, Quote 2
On it’s website, Cisco says, “Multiprotocol Label Switching (MPLS) enables Enterprises and Service Providers to build next-generation intelligent networks that deliver a wide variety of advanced, value-added services over a single infrastructure… SD-WAN can be seen as a software abstraction of MPLS technology that is applicable to wider scenarios…” (Link in comments)Remember that I said that SD-WAN is an overlay? Cisco agrees. SD-WAN doesn’t replace MPLS. It gives you a convenient way to manage your transport resources – MPLS, broadband Internet, and other.
SUMMARY
SD-WAN gives you the ability to mix-and-match your data transport technologies for the best optimization of cost, QoS, and security.
SD-WAN gives you a convenient dashboard for monitor and control of your WAN resources.
SD-WAN is an overlay. It doesn’t replace any transport layer technology.
Broadband Internet and MPLS are two common transport technologies, each with advantages over the other.
Your SD-WAN solution probably incorporates MPLS, and you just don’t know it. -
@alanz OK, so it turns out that it is probably Onyx who have fuxnored this one:
https://github.com/AntennaPod/AntennaPod/issues/6244#issuecomment-1356846312
I've dropped them a couple of lengthy notes on this and some related UI/UX churn which has become annoying.
@jwz has a few excellent commentaries on the value of GUIs, how much improvement is possible through changing them (little if any), and the importance of consistent and common interfaces, rather than domain-specific specialisation. See especially:
"Unity of Interface" (1998)
Short of being cross-platform (which deserves a document of its own) Mozilla's greatest strength is that it manages to provide a unified interface to a number of useful services and protocols.
It is more interesting to support a concept than to support a particular implementation of it.
https://www.jwz.org/doc/unity-of-interface.html
"Why I use Safari instead of Firefox"
The Firefox UI is a moving target. It is under constant "improvement", which means "change" which means every few months I'm forced to upgrade it and shit has moved around and I need to re-learn how to do a task that I was happily doing before. This does not often happen with Safari. Their UI has been remarkably stable for many, many years. ...
Maybe the Firefox team is right, and you can develop a better UI that way. Well, they haven't yet proved this, because Apple's UI is better.
Look, in the case of all other software, I believe strongly in "release early, release often". Hell, I damned near invented it. But I think history has proven that UI is different than software.
https://www.jwz.org/blog/2012/04/why-i-use-safari-instead-of-firefox/
#Onyx #Boox #UIUX #DontFuckWithTheInterface #jwz #Mozilla #Safari
-
What even is #hydration?
I just published a new video update on #HydroActive exploring exactly what the term means, what problems you encounter when you hydrate with plain #WebComponents and how HydroActive can help with those problems.
I also give an overview of the `defer-hydration` community protocol and how we can use it provide a common hydration interface.
-
No wonder people tend to prefer centralized social media platforms. That's how these products are constantly advertised to us. On the same day, I see two media outlets talking about Bluesky, clearly associated with its CEO, a charismatic young woman.
I think it's good to have another microblogging platform aimed at decentralization, but I wonder why start a new protocol instead of one that has been developed for years. In the world of entrepreneurs, I'm aware it's common for people to reinvent the wheel and have a creation that can be called "their own", but still.
We have many examples of what can happen with super-powerful communication tools whose model is based on indefinite profit. You will have an organization in which it can be more difficult to have accountability and mechanisms to hold the horses if the leadership gets out of control.
What needs to happen for us to understand that we have to have some mechanism to distribute power and be able to reduce the dangers that we are seeing openly before our eyes?
#decentralization #fediverse #power #nonprofit #accountability #x #BlueSky #SocialMedia #farright #energyconservation #Capitalism
-
I cut an initial release (0.1.0-alpha) of the library automerge-repo-swift. A supplemental library to Automerge swift, it adds background networking for sync and storage capabilities. The library extends code I initially created in the Automerge demo app (MeetingNotes), and was common enough to warrant its own library. While I was extracting those pieces, I leaned into the same general pattern that was used in the Javascript library automerge-repo. That library provides largely the same functionality for Automerge in javascript. I borrowed the public API structure, as well as compatibility and implementation details for the Automerge sync protocol. One of my goals while assembling this new library was to build it fully compliant with Swift’s data-race safety. Meaning that it compiles without warnings when I use the Swift compiler’s strict-concurrency mode.
There were some notable challenges in coming up to speed with the concepts of isolation and sendability. In addition to learning the concepts, how to apply them is an open question. Not many Swift developers have embraced strict concurrency and talked about the trade-offs or implications for choices. Because of that, I feel that there’s relatively little available knowledge to understand the trade-offs to make when you protect mutable state. This post shares some of the stumbling blocks I hit, choices I made, and lessons I’ve learned. My hope is that it helps other developers facing a similar challenge.
Framing the problem
The way I try to learn and apply new knowledge to solve these kinds of “new fangled” problems is first working out how to think about the problem. I’ve not come up with a good way to ask other people how to do that. I think when I frame the problem with good first-principles in mind, trade-offs in solutions become easier to understand. Sometimes the answers are even self-obvious.
The foremost principle in strict-concurrency is “protect your mutable state”. The compiler warnings give you feedback about potential hazards and data-races. In Swift, protecting the state uses a concept of an “isolation domain”. My layman’s take on isolation is “How can the compiler verify that only one thread is accessing this bit of data at a time”. There are some places where the compiler infers the state of isolation, and some of them still changing as we progress towards Swift 6. When you’re writing code, the compiler knows what is isolated (and non-isolated) – either by itself or based on what you annotated. When the compiler infers an isolation domain, that detail is not (yet?) easily exposed to developers. It really only shows up when there’s a mismatch in your assumptions vs. what the compiler thinks and it issues a strict-concurrency warning.
Sendability is the second key concept. In my layman’s terms again, something that is sendable is safe to cross over thread boundaries. With Swift 5.10, the compiler has enough knowledge of types to be able to make guarantees about what is safe, and what isn’t.
The first thing I did was lean heavily into making anything and everything
Sendable. In hindsight, that was a bit of a mistake. Not disastrous, but I made a lot more work for myself. Not everything needs to be sendable. Taking advantage of isolation, it is fine – sometimes notably more efficient and easier to reason about – to have and use non-sendable types within an isolation domain. More on that in a bit.My key to framing up the problem was to think in terms of making explicit choices about what data should be in an isolation region along with how I want to pass information from one isolation domain to another. Any types I pass (generally) need to be Sendable, and anything that stays within an isolation domain doesn’t. For this library, I have a lot of mutable state: networking connections, updates from users, and a state machines coordinating it all. All of it needed so a repository can store and synchronize Automerge documents. Automerge documents themselves are Sendable (I had that in place well before starting this work). I made the Automerge documents sendable by wrapping access and updates to anything mutable within a serial dispatch queue. (This was also needed because the core Automerge library – a Rust library accessed through FFI – was not safe for multi-threaded use).
Choosing Isolation
I knew I wanted to make at least one explicit isolation domain, so the first question was “Actor or isolated class?” Honestly, I’m still not sure I understand all the tradeoffs. Without knowing what the effect would be to start off with, I decided to pick “let’s use actors everywhere” and see how it goes. Some of the method calls in the design of the Automerge repository were easily and obviously async, so that seemed like a good first cut. I made the top-level repo an actor, and then I kept making any internal type that had mutable state also be it’s own actor. That included a storage subsystem and a network subsystem, both of which I built to let someone else provide the network or storage provider external to this project. To support external plugins that work with this library, I created protocols for the storage and network provider, as well as one that the network providers use to talk back to the repository.
The downside of that choice was two-fold – first setting things up, then interacting with it from within a SwiftUI app. Because I made every-darn-thing an actor, I hade to await a response, which meant a lot of potential suspension points in my code. That also propagated to imply even setup needed to be done within an async context. Sometimes that’s easy to arrange, but other times it ends up being a complete pain in the butt. More specifically, quite a few of the current Apple-provided frameworks don’t have or provide a clear path to integrate async setup hooks. The server-side Swift world has a lovely “set up and run” mechanism (swift-service-lifecycle) it is adopting, but Apple hasn’t provided a similar concept the frameworks it provides. The one that bites me most frequently is the SwiftUI app and document-based app lifecycle, which are all synchronous.
Initialization Challenges
Making the individual actors – Repo and the two network providers I created – initializable with synchronous calls wasn’t too bad. The stumbling block I hit (that I still don’t have a great solution to) was when I wanted to add and activate the network providers to a repository. To arrange that, I’m currently using a detached Task that I kick off in the SwiftUI App’s initializer:
public let repo = Repo(sharePolicy: .agreeable)public let websocket = WebSocketProvider()public let peerToPeer = PeerToPeerProvider( PeerToPeerProviderConfiguration( passcode: "AutomergeMeetingNotes", reconnectOnError: true, autoconnect: false ))@mainstruct MeetingNotesApp: App { var body: some Scene { DocumentGroup { MeetingNotesDocument() } editor: { file in MeetingNotesDocumentView(document: file.document) } .commands { CommandGroup(replacing: CommandGroupPlacement.toolbar) { } } } init() { Task { await repo.addNetworkAdapter(adapter: websocket) await repo.addNetworkAdapter(adapter: peerToPeer) } }}Swift Async Algorithms
One of the lessons I’ve learned is that if you find yourself stashing a number of actors into an array, and you’re used to interacting with them using functional methods (filter, compactMap, etc), you need to deal with the asynchronous access. The standard library built-in functional methods are all synchronous. Because of that, you can only access non-isolated properties on the actors. For me, that meant working with non-mutable state that I set up during actor initialization.
The second path (and I went there) was to take on a dependency to swift-async-algorithms, and use its async variations of the functional methods. They let you “await” results for anything that needs to cross isolation boundaries. And because it took me an embarrasingly long time to figure it out: If you have an array of actors, the way to get to an AsyncSequence of them is to use the async property on the array after you’ve imported
swift-async-algorithms. For example, something like the following snippet:let arrayOfActors: [YourActorType] = []let filteredResults = arrayOfActors.async.filter(...)Rethinking the isolation choice
That is my first version of this library. I got it functional, then turned around and tore it apart again. In making everything an actor, I was making LOTS of little isolation regions that the code had to hop between. With all the suspension points, that meant a lot of possible re-ordering of what was running. I had to be extrodinarily careful not to assume a copy of some state I’d nabbed earlier was still the same after the await. (I still have to be, but it was a more prominent issue with lots of actors.) All of this boils down to being aware of actor re-entrancy, and when it might invalidate something.
I knew that I wanted at least one isolation region (the repository). I also want to keep mutable state in separate types to preserve an isolation of duties. One particular class highlighted my problems – a wrapper around NWConnection that tracks additional state with it and handles the Automerge sync protocol. It was getting really darned inconvenient with the large number of
awaitsuspension points.I slowly clued in that it would be a lot easier if that were all synchronous – and there was no reason it couldn’t be. In my ideal world, I’d have the type
Repo(my top-level repository) as an non-global actor, and isolate any classes it used to the same isolation zone as that one, non-global, actor. I think that’s a capability that’s coming, or at least I wasn’t sure how to arrange that today with Swift 5.10. Instead I opted to make a single global actor for the library and switch what I previously set up as actors to classes isolated to that global actor.That let me simplify quite a bit, notably when dealing with the state of connections within a network adapter. What surprised me was that when I switched from Actor to isolated class, there were few warnings from the change. The changes were mostly warnings that calls dropped back to synchronous, and no longer needed
await. That was quick to fix up; the change to isolated classes was much faster and easier than I anticipated. After I made the initial changes, I went through the various initializers and associated configuration calls to make more of it explicitly synchronous. The end result was more code that could be set up (initialized) without an async context. And finally, I updated how I handled the networking so that as I needed to track state, I didn’t absolutely have to use the async algorithsm library.A single global actor?
A bit of a side note: I thought about making
Repoa global actor, but I prefer to not demand a singleton style library for it’s usage. That choice made it much easier to host multiple repositories when it came time to run functional tests with a mock In-Memory network, or integration tests with the actual providers. I’m still a slight bit concerned that I might be adding to a long-term potential proliferation of global actors from libraries – but it seems like the best solution at the moment. I’d love it if I could do something that indicated “All these things need a single isolation domain, and you – developer – are responsible for providing one that fits your needs”. I’m not sure that kind of concept is even on the table for future work.Recipes for solving these problems
If you weren’t already aware of it, Matt Massicotte created a GitHub repository called ConcurrencyRecipes. This is a gemstone of knowledge, hints, and possible solutions. I leaned into it again and again while building (and rebuilding) this library. One of the “convert it to async” challenges I encountered was providing an async interface to my own peer-to-peer network protocol. I built the protocol using the Network framework based (partially on Apple’s sample code), which is all synchronous code and callbacks. A high level, I wanted it to act similarly URLSessionWebSocketTask. This gist being a connection has an
async send()and anasync receive()for sending and receiving messages on the connection. With an asyncsendandreceive, you can readily assemble several different patterns of access.To get there, I used a combination of CheckedContinuation (both the throwing and non-throwing variations) to work with what
NWConnectionprovided. I wish that was better documented. How to properly use those APIs is opaque, but that is a digression for another time. I’m particular happy with how my code worked out, including adding a method on the PeerConnection class that used structured concurrency to handle a timeout mechanism.Racing tasks with structured concurrency
One of the harder warnings for me to understand was related to racing concurrent tasks in order to create an async method with a “timeout”. I stashed a pattern for how to do this in my notebook with references to Beyond the basics of structured concurrency from WWDC23.
If the async task returns a value, you can set it up something like this (this is from PeerToPeerConnection.swift):
let msg = try await withThrowingTaskGroup(of: SyncV1Msg.self) { group in group.addTask { // retrieve the next message try await self.receiveSingleMessage() } group.addTask { // Race against the receive call with a continuous timer try await Task.sleep(for: explicitTimeout) throw SyncV1Msg.Errors.Timeout() } guard let msg = try await group.next() else { throw CancellationError() } // cancel all ongoing tasks (the websocket receive request, in this case) group.cancelAll() return msg}There’s a niftier version available in Swift 5.9 (which I didn’t use) for when you don’t care about the return value:
func run() async throws { try await withThrowingDiscardingTaskGroup { group in for cook in staff.keys { group.addTask { try await cook.handleShift() } } group.addTask { // keep the restaurant going until closing time try await Task.sleep(for: shiftDuration) throw TimeToCloseError() } }}With Swift 5.10 compiler, my direct use of this displayed a warning:
warning: passing argument of non-sendable type 'inout ThrowingTaskGroup<SyncV1Msg, any Error>' outside of global actor 'AutomergeRepo'-isolated context may introduce data racesguard let msg = try await group.next() else { ^I didn’t really understand the core of this warning, so I asked on the Swift forums. VNS (on the forums) had run into the same issue and helped explain it:
It’s because
withTaskGroupaccepts a non-Sendableclosure, which means the closure has to be isolated to whatever context it was formed in. If yourtest()function isnonisolated, it means the closure isnonisolated, so callinggroup.waitForAll()doesn’t cross an isolation boundary.The workaround to handle the combination of non-sendable closures and TaskGroup is to make the async method that runs this code
nonisolated. In the context I was using it, the class that contains this method is isolated to a global actor, so it’s inheriting that context. By switching the method to be explicitly non-isolated, the compiler doesn’t complain aboutgroupbeing isolated to that global actor.Sharing information back to SwiftUI
These components have all sorts of interesting internal state, some of which I wanted to export. For example, to provide information from the network providers to make a user interface (in SwiftUI). I want to be able to choose to connect to endpoints, to share what endpoints might be available (from the NWBrowser embedded in the peer to peer network provider), and so forth.
I first tried to lean into AsyncStreams. While they make a great local queue for a single point to point connection, I found they were far less useful to generally make a firehouse of data that SwiftUI knows how to read and react to. While I tried to use all the latest techniques, to handle this part I went to my old friend Combine. Some people are effusing that Combine is dead and dying – but boy it works. And most delightfully, you can have any number of endpoints pick up and subscribe to a shared publisher, which was perfect for my use case. Top that off with SwiftUI having great support to receive streams of data from Combine, and it was an easy choice.
I ended up using Combine publishers to make a a few feeds of data from the PeerToPeerProvider. They share information about what other peers were available, the current state of the listener (that accepts connections) and the browser (that looks for peers), and last a publisher that provides information about active peer to peer connctions. I feel that worked out extremely well. It worked so well that I made an internal publisher (not exposed via the public API) for tests to get events and state updates from within a repository.
Integration Testing
It’s remarkably hard to usefully unit test network providers. Instead of unit testing, I made a separate Swift project for the purposes of running integration tests. It sits in it’s own directory in the git repository and references
automerge-repo-swiftas a local dependency. A side effect is that it let me add in all sorts of wacky dependencies that were handy for the integration testing, but that I really didn’t want exposed and transitive for the main package. I wish that Swift Packages had a means to identify test-only dependencies that didn’t propagate to other packages for situations like this. Ah well, my solution was a separate sub-project.Testing using the Combine publisher worked well. Although it took a little digging to figure out the correct way to set up and use expectations with async XCTests. It feels a bit exhausting to assemble the expectations and fulfillment calls, but its quite possible to get working. If you want to see this in operation, take a look at P2P+explicitConnect.swift. I started to look at potentially using the upcoming swift-testing, but with limited Swift 5.10 support, I decided to hold off for now. If it makes asynchronous testing easier down the road, I may well adopt it quickly after it’s initial release.
The one quirky place that I ran into with that API setup was that
expectation.fulfill()gets cranky with you if you call it more than once. My publisher wasn’t quite so constrained with state updates, so I ended up cobbling a boolean latch variable in asinkwhen I didn’t have a sufficiently constrained closure.The other quirk in integration testing is that while it works beautifully on a local machine, I had a trouble getting it to work in CI (using GitHub Actions). Part of the issue is that the current
swift testdefaults to running all possible tests at once, in parallel. Especially for integration testing of peer to peer networking, that meant a lot of network listeners, and browsers, getting shoved together at once on the local network. I wrote a script to list out the tests and run them one at a time. Even breaking it down like that didn’t consistently get through CI. I also tried higher wait times (120 seconds) on the expectations. When I run them locally, most of those tests take about 5 seconds each.The test that was a real challenge was the cross-platform one. Automerge-repo has a sample sync server (NodeJS, using Automerge through WASM). I created a docker container for it, and my cross-platform integration test pushes and pulls documents to an instance that I can run in Docker. Well… Docker isn’t available for macOS runners, so that’s out for GitHub Actions. I have a script that spins up a local docker instance, and I added a check into the WebSocket network provider test – if it couldn’t find a local instance to work against, it skips the test.
Final Takeaways
Starting with a plan for isolating state made the choices of how and what I used a bit easier, and reaching for global-actor constrained classes made synchronous use of those classes much easier. For me, this mostly played out in better (synchronous) intializers and dealing with collections using functional programming patterns.
I hope there’s some planning/thinking in SwiftUI to update or extend the app structure to accomodate async hooks for things like setup and initialization (FB9221398). That should make it easier for a developer to run an async initializer and verify that it didn’t fail, before continuing into the normal app lifecycle. Likewise, I hope that the Document-based APIs gain an async-context to work with documents to likewise handle asynchronous tasks (FB12243722). Both of these spots are very awkward places for me.
Once you shift to using asynchronous calls, it can have a ripple effect in your code. If you’re looking at converting existing code, start at the “top” and work down. That helped me to make sure there weren’t secondary complications with that choice (such as a a need for an async initializer).
Better yet, step back and take the time to identify where mutable state exists. Group it together as best you can, and review how you’re interacting it, and in what isolation region. In the case of things that need to be available to SwiftUI, you can likely isolate methods appropriately (*cough*
MainActor*cough*). Then make the parts you need to pass between isolation domainsSendable. Recognize that in some cases, it may be fine to do the equivalent of “Here was the state at some recent moment, if you might want to react to that”. There are several places where I pass back a summary snapshot of mutable state to SwiftUI to use in UI elements.And do yourself a favor and keep Matt’s Concurrency Recipes on speed-dial.
Before I finished this post, I listened to episode 43 of the Swift Package Index podcast. It’s a great episode, with Holly Bora, compiler geek and manager of the Swift language team, on as a guest to talk about the Swift 6. A tidbit she shared was that they are creating a Swift 6 migration guide, to be published on the swift.org website. Something to look forward to, in addition to Matt’s collection of recipes!
https://rhonabwy.com/2024/04/29/designing-a-swift-library-with-data-race-safety/
-
'We cannot exist if we lose the water': In #Honduras, a community resists a #mine polluting the #RíoGuapinol
"#JuanLópez, representative of the Committee for the Defense of Common and Public Goods, told #RadioProgreso that the activists' next step is advocate for the revoking of Inversiones Los Pinares' mining contracts, the environmental license and the exploitation and exploration permit."
by Soli Salgado
October 18, 2022TOCOA, HONDURAS - "The Río Guapinol that streams through the Bajo Aguán valley in northern Honduras has long been a source of drinking, bathing, cleaning, irrigation and cooking water for the surrounding community in Tocoa.
"In the last few years, the river has also provided an education: how to resist an international mining operation that they say contaminates the river in the name of development; how to confront a seemingly compromised justice system when that #resistance goes awry; and — for the women in particular — how to become leaders of a movement upon the indefinite imprisonment of their husbands and sons.
"Vilma Cruz raised her five children using the river, including a son who wound up spending almost two and a half years in jail for protesting the mining as part of the #Guapinol8, a group whose 2019 arrests and long detentions drew international outrage.
"'We don't feel at peace because our water has been endangered,' she said. 'Now, when I go to the river, I feel my chest swell, like I'm not free.'
A mine emerges, a community responds
"About a decade ago, #InversionesLosPinares, formerly the Honduran #EMCO Mining Company and based in #Tocoa, applied for a concession to build an iron oxide mine in the protected #CarlosEscaleras #NationalPark. Then-President Juan Orlando Hernández authorized the request in 2013, a decision locals said was made without following protocol of consulting residents of the area.
"The #OpenPit #mining project was upstream of the #RíoGuapinol, a channel that stems from the larger #RíoAguan, a river that flows through tropical mountains from the Atlantic on the northern side of the #CentralAmerican country.
"When the Río Guapinol in 2018 started to turn a chocolate brown, locals took that as a cue to act against Inversiones Los Pinares.
"#GuapinolResiste, the local community's movement in response to the mining, said in a 2020 report that even before mining began, the construction of the facilities and roads had polluted both the Guapinol and #SanPedro rivers, affecting 14,000 residents who rely on the water for consumption and domestic purposes.
"'Losing the river would mean buying water, and you can't buy water for everything,' said Leonel George, who serves on the Municipal Committee for the Defense of Common and Public Goods, founded in 2018 in response to the mining project.
"The river, he added, is 'closely linked to life and the existence of everything that surrounds us,' noting that the committee's concern goes beyond the effects that contaminated water would have on human beings to the river's surrounding #forests and #biodiversity."
#JusticeForJuan #GuapinolRiver #WaterDefender #SaveTheForests #DefendTheSacred
#NoMiningWithoutConsent #WaterIsLife #SaveTheRivers #EMCO #InversionesLosPinares
#MunicipalCommitteeForTheDefenseOfCommonAndPublicGoods -
'We cannot exist if we lose the water': In #Honduras, a community resists a #mine polluting the #RíoGuapinol
"#JuanLópez, representative of the Committee for the Defense of Common and Public Goods, told #RadioProgreso that the activists' next step is advocate for the revoking of Inversiones Los Pinares' mining contracts, the environmental license and the exploitation and exploration permit."
by Soli Salgado
October 18, 2022TOCOA, HONDURAS - "The Río Guapinol that streams through the Bajo Aguán valley in northern Honduras has long been a source of drinking, bathing, cleaning, irrigation and cooking water for the surrounding community in Tocoa.
"In the last few years, the river has also provided an education: how to resist an international mining operation that they say contaminates the river in the name of development; how to confront a seemingly compromised justice system when that #resistance goes awry; and — for the women in particular — how to become leaders of a movement upon the indefinite imprisonment of their husbands and sons.
"Vilma Cruz raised her five children using the river, including a son who wound up spending almost two and a half years in jail for protesting the mining as part of the #Guapinol8, a group whose 2019 arrests and long detentions drew international outrage.
"'We don't feel at peace because our water has been endangered,' she said. 'Now, when I go to the river, I feel my chest swell, like I'm not free.'
A mine emerges, a community responds
"About a decade ago, #InversionesLosPinares, formerly the Honduran #EMCO Mining Company and based in #Tocoa, applied for a concession to build an iron oxide mine in the protected #CarlosEscaleras #NationalPark. Then-President Juan Orlando Hernández authorized the request in 2013, a decision locals said was made without following protocol of consulting residents of the area.
"The #OpenPit #mining project was upstream of the #RíoGuapinol, a channel that stems from the larger #RíoAguan, a river that flows through tropical mountains from the Atlantic on the northern side of the #CentralAmerican country.
"When the Río Guapinol in 2018 started to turn a chocolate brown, locals took that as a cue to act against Inversiones Los Pinares.
"#GuapinolResiste, the local community's movement in response to the mining, said in a 2020 report that even before mining began, the construction of the facilities and roads had polluted both the Guapinol and #SanPedro rivers, affecting 14,000 residents who rely on the water for consumption and domestic purposes.
"'Losing the river would mean buying water, and you can't buy water for everything,' said Leonel George, who serves on the Municipal Committee for the Defense of Common and Public Goods, founded in 2018 in response to the mining project.
"The river, he added, is 'closely linked to life and the existence of everything that surrounds us,' noting that the committee's concern goes beyond the effects that contaminated water would have on human beings to the river's surrounding #forests and #biodiversity."
#JusticeForJuan #GuapinolRiver #WaterDefender #SaveTheForests #DefendTheSacred
#NoMiningWithoutConsent #WaterIsLife #SaveTheRivers #EMCO #InversionesLosPinares
#MunicipalCommitteeForTheDefenseOfCommonAndPublicGoods -
'We cannot exist if we lose the water': In #Honduras, a community resists a #mine polluting the #RíoGuapinol
"#JuanLópez, representative of the Committee for the Defense of Common and Public Goods, told #RadioProgreso that the activists' next step is advocate for the revoking of Inversiones Los Pinares' mining contracts, the environmental license and the exploitation and exploration permit."
by Soli Salgado
October 18, 2022TOCOA, HONDURAS - "The Río Guapinol that streams through the Bajo Aguán valley in northern Honduras has long been a source of drinking, bathing, cleaning, irrigation and cooking water for the surrounding community in Tocoa.
"In the last few years, the river has also provided an education: how to resist an international mining operation that they say contaminates the river in the name of development; how to confront a seemingly compromised justice system when that #resistance goes awry; and — for the women in particular — how to become leaders of a movement upon the indefinite imprisonment of their husbands and sons.
"Vilma Cruz raised her five children using the river, including a son who wound up spending almost two and a half years in jail for protesting the mining as part of the #Guapinol8, a group whose 2019 arrests and long detentions drew international outrage.
"'We don't feel at peace because our water has been endangered,' she said. 'Now, when I go to the river, I feel my chest swell, like I'm not free.'
A mine emerges, a community responds
"About a decade ago, #InversionesLosPinares, formerly the Honduran #EMCO Mining Company and based in #Tocoa, applied for a concession to build an iron oxide mine in the protected #CarlosEscaleras #NationalPark. Then-President Juan Orlando Hernández authorized the request in 2013, a decision locals said was made without following protocol of consulting residents of the area.
"The #OpenPit #mining project was upstream of the #RíoGuapinol, a channel that stems from the larger #RíoAguan, a river that flows through tropical mountains from the Atlantic on the northern side of the #CentralAmerican country.
"When the Río Guapinol in 2018 started to turn a chocolate brown, locals took that as a cue to act against Inversiones Los Pinares.
"#GuapinolResiste, the local community's movement in response to the mining, said in a 2020 report that even before mining began, the construction of the facilities and roads had polluted both the Guapinol and #SanPedro rivers, affecting 14,000 residents who rely on the water for consumption and domestic purposes.
"'Losing the river would mean buying water, and you can't buy water for everything,' said Leonel George, who serves on the Municipal Committee for the Defense of Common and Public Goods, founded in 2018 in response to the mining project.
"The river, he added, is 'closely linked to life and the existence of everything that surrounds us,' noting that the committee's concern goes beyond the effects that contaminated water would have on human beings to the river's surrounding #forests and #biodiversity."
#JusticeForJuan #GuapinolRiver #WaterDefender #SaveTheForests #DefendTheSacred
#NoMiningWithoutConsent #WaterIsLife #SaveTheRivers #EMCO #InversionesLosPinares
#MunicipalCommitteeForTheDefenseOfCommonAndPublicGoods -
'We cannot exist if we lose the water': In #Honduras, a community resists a #mine polluting the #RíoGuapinol
"#JuanLópez, representative of the Committee for the Defense of Common and Public Goods, told #RadioProgreso that the activists' next step is advocate for the revoking of Inversiones Los Pinares' mining contracts, the environmental license and the exploitation and exploration permit."
by Soli Salgado
October 18, 2022TOCOA, HONDURAS - "The Río Guapinol that streams through the Bajo Aguán valley in northern Honduras has long been a source of drinking, bathing, cleaning, irrigation and cooking water for the surrounding community in Tocoa.
"In the last few years, the river has also provided an education: how to resist an international mining operation that they say contaminates the river in the name of development; how to confront a seemingly compromised justice system when that #resistance goes awry; and — for the women in particular — how to become leaders of a movement upon the indefinite imprisonment of their husbands and sons.
"Vilma Cruz raised her five children using the river, including a son who wound up spending almost two and a half years in jail for protesting the mining as part of the #Guapinol8, a group whose 2019 arrests and long detentions drew international outrage.
"'We don't feel at peace because our water has been endangered,' she said. 'Now, when I go to the river, I feel my chest swell, like I'm not free.'
A mine emerges, a community responds
"About a decade ago, #InversionesLosPinares, formerly the Honduran #EMCO Mining Company and based in #Tocoa, applied for a concession to build an iron oxide mine in the protected #CarlosEscaleras #NationalPark. Then-President Juan Orlando Hernández authorized the request in 2013, a decision locals said was made without following protocol of consulting residents of the area.
"The #OpenPit #mining project was upstream of the #RíoGuapinol, a channel that stems from the larger #RíoAguan, a river that flows through tropical mountains from the Atlantic on the northern side of the #CentralAmerican country.
"When the Río Guapinol in 2018 started to turn a chocolate brown, locals took that as a cue to act against Inversiones Los Pinares.
"#GuapinolResiste, the local community's movement in response to the mining, said in a 2020 report that even before mining began, the construction of the facilities and roads had polluted both the Guapinol and #SanPedro rivers, affecting 14,000 residents who rely on the water for consumption and domestic purposes.
"'Losing the river would mean buying water, and you can't buy water for everything,' said Leonel George, who serves on the Municipal Committee for the Defense of Common and Public Goods, founded in 2018 in response to the mining project.
"The river, he added, is 'closely linked to life and the existence of everything that surrounds us,' noting that the committee's concern goes beyond the effects that contaminated water would have on human beings to the river's surrounding #forests and #biodiversity."
#JusticeForJuan #GuapinolRiver #WaterDefender #SaveTheForests #DefendTheSacred
#NoMiningWithoutConsent #WaterIsLife #SaveTheRivers #EMCO #InversionesLosPinares
#MunicipalCommitteeForTheDefenseOfCommonAndPublicGoods -
What even is #hydration?
I just published a new video update on #HydroActive exploring exactly what the term means, what problems you encounter when you hydrate with plain #WebComponents and how HydroActive can help with those problems.
I also give an overview of the `defer-hydration` community protocol and how we can use it provide a common hydration interface.
-
What even is #hydration?
I just published a new video update on #HydroActive exploring exactly what the term means, what problems you encounter when you hydrate with plain #WebComponents and how HydroActive can help with those problems.
I also give an overview of the `defer-hydration` community protocol and how we can use it provide a common hydration interface.
-
What even is #hydration?
I just published a new video update on #HydroActive exploring exactly what the term means, what problems you encounter when you hydrate with plain #WebComponents and how HydroActive can help with those problems.
I also give an overview of the `defer-hydration` community protocol and how we can use it provide a common hydration interface.
-
What even is #hydration?
I just published a new video update on #HydroActive exploring exactly what the term means, what problems you encounter when you hydrate with plain #WebComponents and how HydroActive can help with those problems.
I also give an overview of the `defer-hydration` community protocol and how we can use it provide a common hydration interface.
-
B., the senior officer, claimed that in the current war, “I would invest 20 seconds for each target at this stage, and do dozens of them every day. I had zero added value as a human, apart from being a stamp of approval. It saved a lot of time.”
According to B., a common error occurred “if the [Hamas] target gave [his phone] to his son, his older brother, or just a random man. That person will be bombed in his house with his family. This happened often. These were most of the mistakes caused by Lavender,” B. said.
https://www.972mag.com/lavender-ai-israeli-army-gaza/ @israel @data 🧶
#duty #protocol #conformance #AIRisks #riskApproach #risks #dehumanisation #AIWar #technoCriticism #ethics #efficiency #innovation #Targeting #industrialization #intelligence #AirForce #casualties #DataScience #DataScientist #SIGINT #Unit8200 #patriotism #JewishSupremacy #CrowdSourcing #OSINT #CROSINT #MilInt #military #army #ES2 #IDI #IDF #Unit8200 #Lotem #Habsora #Lavender #dataDon #dataGovernance #Gaza #Hamas #warCrimes #israel
-
Designing a Swift library with data-race safety
I cut an initial release (0.1.0-alpha) of the library automerge-repo-swift. A supplemental library to Automerge swift, it adds background networking for sync and storage capabilities. The library extends code I initially created in the Automerge demo app (MeetingNotes), and was common enough to warrant its own library. While I was extracting those pieces, I leaned into the same general pattern that was used in the Javascript library automerge-repo. That library provides largely the same functionality for Automerge in javascript. I borrowed the public API structure, as well as compatibility and implementation details for the Automerge sync protocol. One of my goals while assembling this new library was to build it fully compliant with Swift’s data-race safety. Meaning that it compiles without warnings when I use the Swift compiler’s strict-concurrency mode.
There were some notable challenges in coming up to speed with the concepts of isolation and sendability. In addition to learning the concepts, how to apply them is an open question. Not many Swift developers have embraced strict concurrency and talked about the trade-offs or implications for choices. Because of that, I feel that there’s relatively little available knowledge to understand the trade-offs to make when you protect mutable state. This post shares some of the stumbling blocks I hit, choices I made, and lessons I’ve learned. My hope is that it helps other developers facing a similar challenge.
Framing the problem
The way I try to learn and apply new knowledge to solve these kinds of “new fangled” problems is first working out how to think about the problem. I’ve not come up with a good way to ask other people how to do that. I think when I frame the problem with good first-principles in mind, trade-offs in solutions become easier to understand. Sometimes the answers are even self-obvious.
The foremost principle in strict-concurrency is “protect your mutable state”. The compiler warnings give you feedback about potential hazards and data-races. In Swift, protecting the state uses a concept of an “isolation domain”. My layman’s take on isolation is “How can the compiler verify that only one thread is accessing this bit of data at a time”. There are some places where the compiler infers the state of isolation, and some of them still changing as we progress towards Swift 6. When you’re writing code, the compiler knows what is isolated (and non-isolated) – either by itself or based on what you annotated. When the compiler infers an isolation domain, that detail is not (yet?) easily exposed to developers. It really only shows up when there’s a mismatch in your assumptions vs. what the compiler thinks and it issues a strict-concurrency warning.
Sendability is the second key concept. In my layman’s terms again, something that is sendable is safe to cross over thread boundaries. With Swift 5.10, the compiler has enough knowledge of types to be able to make guarantees about what is safe, and what isn’t.
The first thing I did was lean heavily into making anything and everything
Sendable. In hindsight, that was a bit of a mistake. Not disastrous, but I made a lot more work for myself. Not everything needs to be sendable. Taking advantage of isolation, it is fine – sometimes notably more efficient and easier to reason about – to have and use non-sendable types within an isolation domain. More on that in a bit.My key to framing up the problem was to think in terms of making explicit choices about what data should be in an isolation region along with how I want to pass information from one isolation domain to another. Any types I pass (generally) need to be Sendable, and anything that stays within an isolation domain doesn’t. For this library, I have a lot of mutable state: networking connections, updates from users, and a state machine coordinating it all. All of it is needed so a repository can store and synchronize Automerge documents. Automerge documents themselves are Sendable (I had that in place well before starting this work). I made the Automerge documents sendable by wrapping access and updates to anything mutable within a serial dispatch queue. (This was also needed because the core Automerge library – a Rust library accessed through FFI – was not safe for multi-threaded use).
Choosing Isolation
I knew I wanted to make at least one explicit isolation domain, so the first question was “Actor or isolated class?” Honestly, I’m still not sure I understand all the tradeoffs. Without knowing what the effect would be to start off with, I decided to pick “let’s use actors everywhere” and see how it goes. Some of the method calls in the design of the Automerge repository were easily and obviously async, so that seemed like a good first cut. I made the top-level repo an actor, and then I kept making any internal type that had mutable state also be it’s own actor. That included a storage subsystem and a network subsystem, both of which I built to let someone else provide the network or storage provider external to this project. To support external plugins that work with this library, I created protocols for the storage and network provider, as well as one that the network providers use to talk back to the repository.
The downside of that choice was two-fold – first setting things up, then interacting with it from within a SwiftUI app. Because I made every-darn-thing an actor, I hade to await a response, which meant a lot of potential suspension points in my code. That also propagated to imply even setup needed to be done within an async context. Sometimes that’s easy to arrange, but other times it ends up being a complete pain in the butt. More specifically, quite a few of the current Apple-provided frameworks don’t have or provide a clear path to integrate async setup hooks. The server-side Swift world has a lovely “set up and run” mechanism (swift-service-lifecycle) it is adopting, but Apple hasn’t provided a similar concept the frameworks it provides. The one that bites me most frequently is the SwiftUI app and document-based app lifecycle, which are all synchronous.
Initialization Challenges
Making the individual actors – Repo and the two network providers I created – initializable with synchronous calls wasn’t too bad. The stumbling block I hit (that I still don’t have a great solution to) was when I wanted to add and activate the network providers to a repository. To arrange that, I’m currently using a detached Task that I kick off in the SwiftUI App’s initializer:
public let repo = Repo(sharePolicy: .agreeable)public let websocket = WebSocketProvider()public let peerToPeer = PeerToPeerProvider( PeerToPeerProviderConfiguration( passcode: "AutomergeMeetingNotes", reconnectOnError: true, autoconnect: false ))@mainstruct MeetingNotesApp: App { var body: some Scene { DocumentGroup { MeetingNotesDocument() } editor: { file in MeetingNotesDocumentView(document: file.document) } .commands { CommandGroup(replacing: CommandGroupPlacement.toolbar) { } } } init() { Task { await repo.addNetworkAdapter(adapter: websocket) await repo.addNetworkAdapter(adapter: peerToPeer) } }}Swift Async Algorithms
One of the lessons I’ve learned is that if you find yourself stashing a number of actors into an array, and you’re used to interacting with them using functional methods (filter, compactMap, etc), you need to deal with the asynchronous access. The standard library built-in functional methods are all synchronous. Because of that, you can only access non-isolated properties on the actors. For me, that meant working with non-mutable state that I set up during actor initialization.
The second path (and I went there) was to take on a dependency to swift-async-algorithms, and use its async variations of the functional methods. They let you “await” results for anything that needs to cross isolation boundaries. And because it took me an embarrasingly long time to figure it out: If you have an array of actors, the way to get to an AsyncSequence of them is to use the async property on the array after you’ve imported
swift-async-algorithms. For example, something like the following snippet:let arrayOfActors: [YourActorType] = []let filteredResults = arrayOfActors.async.filter(...)Rethinking the isolation choice
That is my first version of this library. I got it functional, then turned around and tore it apart again. In making everything an actor, I was making LOTS of little isolation regions that the code had to hop between. With all the suspension points, that meant a lot of possible re-ordering of what was running. I had to be extrodinarily careful not to assume a copy of some state I’d nabbed earlier was still the same after the await. (I still have to be, but it was a more prominent issue with lots of actors.) All of this boils down to being aware of actor re-entrancy, and when it might invalidate something.
I knew that I wanted at least one isolation region (the repository). I also want to keep mutable state in separate types to preserve an isolation of duties. One particular class highlighted my problems – a wrapper around NWConnection that tracks additional state with it and handles the Automerge sync protocol. It was getting really darned inconvenient with the large number of
awaitsuspension points.I slowly clued in that it would be a lot easier if that were all synchronous – and there was no reason it couldn’t be. In my ideal world, I’d have the type
Repo(my top-level repository) as an non-global actor, and isolate any classes it used to the same isolation zone as that one, non-global, actor. I think that’s a capability that’s coming, or at least I wasn’t sure how to arrange that today with Swift 5.10. Instead I opted to make a single global actor for the library and switch what I previously set up as actors to classes isolated to that global actor.That let me simplify quite a bit, notably when dealing with the state of connections within a network adapter. What surprised me was that when I switched from Actor to isolated class, there were few warnings from the change. The changes were mostly warnings that calls dropped back to synchronous, and no longer needed
await. That was quick to fix up; the change to isolated classes was much faster and easier than I anticipated. After I made the initial changes, I went through the various initializers and associated configuration calls to make more of it explicitly synchronous. The end result was more code that could be set up (initialized) without an async context. And finally, I updated how I handled the networking so that as I needed to track state, I didn’t absolutely have to use the async algorithsm library.A single global actor?
A bit of a side note: I thought about making
Repoa global actor, but I prefer to not demand a singleton style library for it’s usage. That choice made it much easier to host multiple repositories when it came time to run functional tests with a mock In-Memory network, or integration tests with the actual providers. I’m still a slight bit concerned that I might be adding to a long-term potential proliferation of global actors from libraries – but it seems like the best solution at the moment. I’d love it if I could do something that indicated “All these things need a single isolation domain, and you – developer – are responsible for providing one that fits your needs”. I’m not sure that kind of concept is even on the table for future work.Recipes for solving these problems
If you weren’t already aware of it, Matt Massicotte created a GitHub repository called ConcurrencyRecipes. This is a gemstone of knowledge, hints, and possible solutions. I leaned into it again and again while building (and rebuilding) this library. One of the “convert it to async” challenges I encountered was providing an async interface to my own peer-to-peer network protocol. I built the protocol using the Network framework based (partially on Apple’s sample code), which is all synchronous code and callbacks. A high level, I wanted it to act similarly URLSessionWebSocketTask. This gist being a connection has an
async send()and anasync receive()for sending and receiving messages on the connection. With an asyncsendandreceive, you can readily assemble several different patterns of access.To get there, I used a combination of CheckedContinuation (both the throwing and non-throwing variations) to work with what
NWConnectionprovided. I wish that was better documented. How to properly use those APIs is opaque, but that is a digression for another time. I’m particular happy with how my code worked out, including adding a method on the PeerConnection class that used structured concurrency to handle a timeout mechanism.Racing tasks with structured concurrency
One of the harder warnings for me to understand was related to racing concurrent tasks in order to create an async method with a “timeout”. I stashed a pattern for how to do this in my notebook with references to Beyond the basics of structured concurrency from WWDC23.
If the async task returns a value, you can set it up something like this (this is from PeerToPeerConnection.swift):
let msg = try await withThrowingTaskGroup(of: SyncV1Msg.self) { group in group.addTask { // retrieve the next message try await self.receiveSingleMessage() } group.addTask { // Race against the receive call with a continuous timer try await Task.sleep(for: explicitTimeout) throw SyncV1Msg.Errors.Timeout() } guard let msg = try await group.next() else { throw CancellationError() } // cancel all ongoing tasks (the websocket receive request, in this case) group.cancelAll() return msg}There’s a niftier version available in Swift 5.9 (which I didn’t use) for when you don’t care about the return value:
func run() async throws { try await withThrowingDiscardingTaskGroup { group in for cook in staff.keys { group.addTask { try await cook.handleShift() } } group.addTask { // keep the restaurant going until closing time try await Task.sleep(for: shiftDuration) throw TimeToCloseError() } }}With Swift 5.10 compiler, my direct use of this displayed a warning:
warning: passing argument of non-sendable type 'inout ThrowingTaskGroup<SyncV1Msg, any Error>' outside of global actor 'AutomergeRepo'-isolated context may introduce data racesguard let msg = try await group.next() else { ^I didn’t really understand the core of this warning, so I asked on the Swift forums. VNS (on the forums) had run into the same issue and helped explain it:
It’s because
withTaskGroupaccepts a non-Sendableclosure, which means the closure has to be isolated to whatever context it was formed in. If yourtest()function isnonisolated, it means the closure isnonisolated, so callinggroup.waitForAll()doesn’t cross an isolation boundary.The workaround to handle the combination of non-sendable closures and TaskGroup is to make the async method that runs this code
nonisolated. In the context I was using it, the class that contains this method is isolated to a global actor, so it’s inheriting that context. By switching the method to be explicitly non-isolated, the compiler doesn’t complain aboutgroupbeing isolated to that global actor.Sharing information back to SwiftUI
These components have all sorts of interesting internal state, some of which I wanted to export. For example, to provide information from the network providers to make a user interface (in SwiftUI). I want to be able to choose to connect to endpoints, to share what endpoints might be available (from the NWBrowser embedded in the peer to peer network provider), and so forth.
I first tried to lean into AsyncStreams. While they make a great local queue for a single point to point connection, I found they were far less useful to generally make a firehouse of data that SwiftUI knows how to read and react to. While I tried to use all the latest techniques, to handle this part I went to my old friend Combine. Some people are effusing that Combine is dead and dying – but boy it works. And most delightfully, you can have any number of endpoints pick up and subscribe to a shared publisher, which was perfect for my use case. Top that off with SwiftUI having great support to receive streams of data from Combine, and it was an easy choice.
I ended up using Combine publishers to make a a few feeds of data from the PeerToPeerProvider. They share information about what other peers were available, the current state of the listener (that accepts connections) and the browser (that looks for peers), and last a publisher that provides information about active peer to peer connctions. I feel that worked out extremely well. It worked so well that I made an internal publisher (not exposed via the public API) for tests to get events and state updates from within a repository.
Integration Testing
It’s remarkably hard to usefully unit test network providers. Instead of unit testing, I made a separate Swift project for the purposes of running integration tests. It sits in it’s own directory in the git repository and references
automerge-repo-swiftas a local dependency. A side effect is that it let me add in all sorts of wacky dependencies that were handy for the integration testing, but that I really didn’t want exposed and transitive for the main package. I wish that Swift Packages had a means to identify test-only dependencies that didn’t propagate to other packages for situations like this. Ah well, my solution was a separate sub-project.Testing using the Combine publisher worked well. Although it took a little digging to figure out the correct way to set up and use expectations with async XCTests. It feels a bit exhausting to assemble the expectations and fulfillment calls, but its quite possible to get working. If you want to see this in operation, take a look at P2P+explicitConnect.swift. I started to look at potentially using the upcoming swift-testing, but with limited Swift 5.10 support, I decided to hold off for now. If it makes asynchronous testing easier down the road, I may well adopt it quickly after it’s initial release.
The one quirky place that I ran into with that API setup was that
expectation.fulfill()gets cranky with you if you call it more than once. My publisher wasn’t quite so constrained with state updates, so I ended up cobbling a boolean latch variable in asinkwhen I didn’t have a sufficiently constrained closure.The other quirk in integration testing is that while it works beautifully on a local machine, I had a trouble getting it to work in CI (using GitHub Actions). Part of the issue is that the current
swift testdefaults to running all possible tests at once, in parallel. Especially for integration testing of peer to peer networking, that meant a lot of network listeners, and browsers, getting shoved together at once on the local network. I wrote a script to list out the tests and run them one at a time. Even breaking it down like that didn’t consistently get through CI. I also tried higher wait times (120 seconds) on the expectations. When I run them locally, most of those tests take about 5 seconds each.The test that was a real challenge was the cross-platform one. Automerge-repo has a sample sync server (NodeJS, using Automerge through WASM). I created a docker container for it, and my cross-platform integration test pushes and pulls documents to an instance that I can run in Docker. Well… Docker isn’t available for macOS runners, so that’s out for GitHub Actions. I have a script that spins up a local docker instance, and I added a check into the WebSocket network provider test – if it couldn’t find a local instance to work against, it skips the test.
Final Takeaways
Starting with a plan for isolating state made the choices of how and what I used a bit easier, and reaching for global-actor constrained classes made synchronous use of those classes much easier. For me, this mostly played out in better (synchronous) intializers and dealing with collections using functional programming patterns.
I hope there’s some planning/thinking in SwiftUI to update or extend the app structure to accomodate async hooks for things like setup and initialization (FB9221398). That should make it easier for a developer to run an async initializer and verify that it didn’t fail, before continuing into the normal app lifecycle. Likewise, I hope that the Document-based APIs gain an async-context to work with documents to likewise handle asynchronous tasks (FB12243722). Both of these spots are very awkward places for me.
Once you shift to using asynchronous calls, it can have a ripple effect in your code. If you’re looking at converting existing code, start at the “top” and work down. That helped me to make sure there weren’t secondary complications with that choice (such as a a need for an async initializer).
Better yet, step back and take the time to identify where mutable state exists. Group it together as best you can, and review how you’re interacting it, and in what isolation region. In the case of things that need to be available to SwiftUI, you can likely isolate methods appropriately (*cough*
MainActor*cough*). Then make the parts you need to pass between isolation domainsSendable. Recognize that in some cases, it may be fine to do the equivalent of “Here was the state at some recent moment, if you might want to react to that”. There are several places where I pass back a summary snapshot of mutable state to SwiftUI to use in UI elements.And do yourself a favor and keep Matt’s Concurrency Recipes on speed-dial.
Before I finished this post, I listened to episode 43 of the Swift Package Index podcast. It’s a great episode, with Holly Bora, compiler geek and manager of the Swift language team, on as a guest to talk about the Swift 6. A tidbit she shared was that they are creating a Swift 6 migration guide, to be published on the swift.org website. Something to look forward to, in addition to Matt’s collection of recipes!
-
I cut an initial release (0.1.0-alpha) of the library automerge-repo-swift. A supplemental library to Automerge swift, it adds background networking for sync and storage capabilities. The library extends code I initially created in the Automerge demo app (MeetingNotes), and was common enough to warrant its own library. While I was extracting those pieces, I leaned into the same general pattern that was used in the Javascript library automerge-repo. That library provides largely the same functionality for Automerge in javascript. I borrowed the public API structure, as well as compatibility and implementation details for the Automerge sync protocol. One of my goals while assembling this new library was to build it fully compliant with Swift’s data-race safety. Meaning that it compiles without warnings when I use the Swift compiler’s strict-concurrency mode.
There were some notable challenges in coming up to speed with the concepts of isolation and sendability. In addition to learning the concepts, how to apply them is an open question. Not many Swift developers have embraced strict concurrency and talked about the trade-offs or implications for choices. Because of that, I feel that there’s relatively little available knowledge to understand the trade-offs to make when you protect mutable state. This post shares some of the stumbling blocks I hit, choices I made, and lessons I’ve learned. My hope is that it helps other developers facing a similar challenge.
Framing the problem
The way I try to learn and apply new knowledge to solve these kinds of “new fangled” problems is first working out how to think about the problem. I’ve not come up with a good way to ask other people how to do that. I think when I frame the problem with good first-principles in mind, trade-offs in solutions become easier to understand. Sometimes the answers are even self-obvious.
The foremost principle in strict-concurrency is “protect your mutable state”. The compiler warnings give you feedback about potential hazards and data-races. In Swift, protecting the state uses a concept of an “isolation domain”. My layman’s take on isolation is “How can the compiler verify that only one thread is accessing this bit of data at a time”. There are some places where the compiler infers the state of isolation, and some of them still changing as we progress towards Swift 6. When you’re writing code, the compiler knows what is isolated (and non-isolated) – either by itself or based on what you annotated. When the compiler infers an isolation domain, that detail is not (yet?) easily exposed to developers. It really only shows up when there’s a mismatch in your assumptions vs. what the compiler thinks and it issues a strict-concurrency warning.
Sendability is the second key concept. In my layman’s terms again, something that is sendable is safe to cross over thread boundaries. With Swift 5.10, the compiler has enough knowledge of types to be able to make guarantees about what is safe, and what isn’t.
The first thing I did was lean heavily into making anything and everything
Sendable. In hindsight, that was a bit of a mistake. Not disastrous, but I made a lot more work for myself. Not everything needs to be sendable. Taking advantage of isolation, it is fine – sometimes notably more efficient and easier to reason about – to have and use non-sendable types within an isolation domain. More on that in a bit.My key to framing up the problem was to think in terms of making explicit choices about what data should be in an isolation region along with how I want to pass information from one isolation domain to another. Any types I pass (generally) need to be Sendable, and anything that stays within an isolation domain doesn’t. For this library, I have a lot of mutable state: networking connections, updates from users, and a state machines coordinating it all. All of it needed so a repository can store and synchronize Automerge documents. Automerge documents themselves are Sendable (I had that in place well before starting this work). I made the Automerge documents sendable by wrapping access and updates to anything mutable within a serial dispatch queue. (This was also needed because the core Automerge library – a Rust library accessed through FFI – was not safe for multi-threaded use).
Choosing Isolation
I knew I wanted to make at least one explicit isolation domain, so the first question was “Actor or isolated class?” Honestly, I’m still not sure I understand all the tradeoffs. Without knowing what the effect would be to start off with, I decided to pick “let’s use actors everywhere” and see how it goes. Some of the method calls in the design of the Automerge repository were easily and obviously async, so that seemed like a good first cut. I made the top-level repo an actor, and then I kept making any internal type that had mutable state also be it’s own actor. That included a storage subsystem and a network subsystem, both of which I built to let someone else provide the network or storage provider external to this project. To support external plugins that work with this library, I created protocols for the storage and network provider, as well as one that the network providers use to talk back to the repository.
The downside of that choice was two-fold – first setting things up, then interacting with it from within a SwiftUI app. Because I made every-darn-thing an actor, I hade to await a response, which meant a lot of potential suspension points in my code. That also propagated to imply even setup needed to be done within an async context. Sometimes that’s easy to arrange, but other times it ends up being a complete pain in the butt. More specifically, quite a few of the current Apple-provided frameworks don’t have or provide a clear path to integrate async setup hooks. The server-side Swift world has a lovely “set up and run” mechanism (swift-service-lifecycle) it is adopting, but Apple hasn’t provided a similar concept the frameworks it provides. The one that bites me most frequently is the SwiftUI app and document-based app lifecycle, which are all synchronous.
Initialization Challenges
Making the individual actors – Repo and the two network providers I created – initializable with synchronous calls wasn’t too bad. The stumbling block I hit (that I still don’t have a great solution to) was when I wanted to add and activate the network providers to a repository. To arrange that, I’m currently using a detached Task that I kick off in the SwiftUI App’s initializer:
public let repo = Repo(sharePolicy: .agreeable)public let websocket = WebSocketProvider()public let peerToPeer = PeerToPeerProvider( PeerToPeerProviderConfiguration( passcode: "AutomergeMeetingNotes", reconnectOnError: true, autoconnect: false ))@mainstruct MeetingNotesApp: App { var body: some Scene { DocumentGroup { MeetingNotesDocument() } editor: { file in MeetingNotesDocumentView(document: file.document) } .commands { CommandGroup(replacing: CommandGroupPlacement.toolbar) { } } } init() { Task { await repo.addNetworkAdapter(adapter: websocket) await repo.addNetworkAdapter(adapter: peerToPeer) } }}Swift Async Algorithms
One of the lessons I’ve learned is that if you find yourself stashing a number of actors into an array, and you’re used to interacting with them using functional methods (filter, compactMap, etc), you need to deal with the asynchronous access. The standard library built-in functional methods are all synchronous. Because of that, you can only access non-isolated properties on the actors. For me, that meant working with non-mutable state that I set up during actor initialization.
The second path (and I went there) was to take on a dependency to swift-async-algorithms, and use its async variations of the functional methods. They let you “await” results for anything that needs to cross isolation boundaries. And because it took me an embarrasingly long time to figure it out: If you have an array of actors, the way to get to an AsyncSequence of them is to use the async property on the array after you’ve imported
swift-async-algorithms. For example, something like the following snippet:let arrayOfActors: [YourActorType] = []let filteredResults = arrayOfActors.async.filter(...)Rethinking the isolation choice
That is my first version of this library. I got it functional, then turned around and tore it apart again. In making everything an actor, I was making LOTS of little isolation regions that the code had to hop between. With all the suspension points, that meant a lot of possible re-ordering of what was running. I had to be extrodinarily careful not to assume a copy of some state I’d nabbed earlier was still the same after the await. (I still have to be, but it was a more prominent issue with lots of actors.) All of this boils down to being aware of actor re-entrancy, and when it might invalidate something.
I knew that I wanted at least one isolation region (the repository). I also want to keep mutable state in separate types to preserve an isolation of duties. One particular class highlighted my problems – a wrapper around NWConnection that tracks additional state with it and handles the Automerge sync protocol. It was getting really darned inconvenient with the large number of
awaitsuspension points.I slowly clued in that it would be a lot easier if that were all synchronous – and there was no reason it couldn’t be. In my ideal world, I’d have the type
Repo(my top-level repository) as an non-global actor, and isolate any classes it used to the same isolation zone as that one, non-global, actor. I think that’s a capability that’s coming, or at least I wasn’t sure how to arrange that today with Swift 5.10. Instead I opted to make a single global actor for the library and switch what I previously set up as actors to classes isolated to that global actor.That let me simplify quite a bit, notably when dealing with the state of connections within a network adapter. What surprised me was that when I switched from Actor to isolated class, there were few warnings from the change. The changes were mostly warnings that calls dropped back to synchronous, and no longer needed
await. That was quick to fix up; the change to isolated classes was much faster and easier than I anticipated. After I made the initial changes, I went through the various initializers and associated configuration calls to make more of it explicitly synchronous. The end result was more code that could be set up (initialized) without an async context. And finally, I updated how I handled the networking so that as I needed to track state, I didn’t absolutely have to use the async algorithsm library.A single global actor?
A bit of a side note: I thought about making
Repoa global actor, but I prefer to not demand a singleton style library for it’s usage. That choice made it much easier to host multiple repositories when it came time to run functional tests with a mock In-Memory network, or integration tests with the actual providers. I’m still a slight bit concerned that I might be adding to a long-term potential proliferation of global actors from libraries – but it seems like the best solution at the moment. I’d love it if I could do something that indicated “All these things need a single isolation domain, and you – developer – are responsible for providing one that fits your needs”. I’m not sure that kind of concept is even on the table for future work.Recipes for solving these problems
If you weren’t already aware of it, Matt Massicotte created a GitHub repository called ConcurrencyRecipes. This is a gemstone of knowledge, hints, and possible solutions. I leaned into it again and again while building (and rebuilding) this library. One of the “convert it to async” challenges I encountered was providing an async interface to my own peer-to-peer network protocol. I built the protocol using the Network framework based (partially on Apple’s sample code), which is all synchronous code and callbacks. A high level, I wanted it to act similarly URLSessionWebSocketTask. This gist being a connection has an
async send()and anasync receive()for sending and receiving messages on the connection. With an asyncsendandreceive, you can readily assemble several different patterns of access.To get there, I used a combination of CheckedContinuation (both the throwing and non-throwing variations) to work with what
NWConnectionprovided. I wish that was better documented. How to properly use those APIs is opaque, but that is a digression for another time. I’m particular happy with how my code worked out, including adding a method on the PeerConnection class that used structured concurrency to handle a timeout mechanism.Racing tasks with structured concurrency
One of the harder warnings for me to understand was related to racing concurrent tasks in order to create an async method with a “timeout”. I stashed a pattern for how to do this in my notebook with references to Beyond the basics of structured concurrency from WWDC23.
If the async task returns a value, you can set it up something like this (this is from PeerToPeerConnection.swift):
let msg = try await withThrowingTaskGroup(of: SyncV1Msg.self) { group in group.addTask { // retrieve the next message try await self.receiveSingleMessage() } group.addTask { // Race against the receive call with a continuous timer try await Task.sleep(for: explicitTimeout) throw SyncV1Msg.Errors.Timeout() } guard let msg = try await group.next() else { throw CancellationError() } // cancel all ongoing tasks (the websocket receive request, in this case) group.cancelAll() return msg}There’s a niftier version available in Swift 5.9 (which I didn’t use) for when you don’t care about the return value:
func run() async throws { try await withThrowingDiscardingTaskGroup { group in for cook in staff.keys { group.addTask { try await cook.handleShift() } } group.addTask { // keep the restaurant going until closing time try await Task.sleep(for: shiftDuration) throw TimeToCloseError() } }}With Swift 5.10 compiler, my direct use of this displayed a warning:
warning: passing argument of non-sendable type 'inout ThrowingTaskGroup<SyncV1Msg, any Error>' outside of global actor 'AutomergeRepo'-isolated context may introduce data racesguard let msg = try await group.next() else { ^I didn’t really understand the core of this warning, so I asked on the Swift forums. VNS (on the forums) had run into the same issue and helped explain it:
It’s because
withTaskGroupaccepts a non-Sendableclosure, which means the closure has to be isolated to whatever context it was formed in. If yourtest()function isnonisolated, it means the closure isnonisolated, so callinggroup.waitForAll()doesn’t cross an isolation boundary.The workaround to handle the combination of non-sendable closures and TaskGroup is to make the async method that runs this code
nonisolated. In the context I was using it, the class that contains this method is isolated to a global actor, so it’s inheriting that context. By switching the method to be explicitly non-isolated, the compiler doesn’t complain aboutgroupbeing isolated to that global actor.Sharing information back to SwiftUI
These components have all sorts of interesting internal state, some of which I wanted to export. For example, to provide information from the network providers to make a user interface (in SwiftUI). I want to be able to choose to connect to endpoints, to share what endpoints might be available (from the NWBrowser embedded in the peer to peer network provider), and so forth.
I first tried to lean into AsyncStreams. While they make a great local queue for a single point to point connection, I found they were far less useful to generally make a firehouse of data that SwiftUI knows how to read and react to. While I tried to use all the latest techniques, to handle this part I went to my old friend Combine. Some people are effusing that Combine is dead and dying – but boy it works. And most delightfully, you can have any number of endpoints pick up and subscribe to a shared publisher, which was perfect for my use case. Top that off with SwiftUI having great support to receive streams of data from Combine, and it was an easy choice.
I ended up using Combine publishers to make a a few feeds of data from the PeerToPeerProvider. They share information about what other peers were available, the current state of the listener (that accepts connections) and the browser (that looks for peers), and last a publisher that provides information about active peer to peer connctions. I feel that worked out extremely well. It worked so well that I made an internal publisher (not exposed via the public API) for tests to get events and state updates from within a repository.
Integration Testing
It’s remarkably hard to usefully unit test network providers. Instead of unit testing, I made a separate Swift project for the purposes of running integration tests. It sits in it’s own directory in the git repository and references
automerge-repo-swiftas a local dependency. A side effect is that it let me add in all sorts of wacky dependencies that were handy for the integration testing, but that I really didn’t want exposed and transitive for the main package. I wish that Swift Packages had a means to identify test-only dependencies that didn’t propagate to other packages for situations like this. Ah well, my solution was a separate sub-project.Testing using the Combine publisher worked well. Although it took a little digging to figure out the correct way to set up and use expectations with async XCTests. It feels a bit exhausting to assemble the expectations and fulfillment calls, but its quite possible to get working. If you want to see this in operation, take a look at P2P+explicitConnect.swift. I started to look at potentially using the upcoming swift-testing, but with limited Swift 5.10 support, I decided to hold off for now. If it makes asynchronous testing easier down the road, I may well adopt it quickly after it’s initial release.
The one quirky place that I ran into with that API setup was that
expectation.fulfill()gets cranky with you if you call it more than once. My publisher wasn’t quite so constrained with state updates, so I ended up cobbling a boolean latch variable in asinkwhen I didn’t have a sufficiently constrained closure.The other quirk in integration testing is that while it works beautifully on a local machine, I had a trouble getting it to work in CI (using GitHub Actions). Part of the issue is that the current
swift testdefaults to running all possible tests at once, in parallel. Especially for integration testing of peer to peer networking, that meant a lot of network listeners, and browsers, getting shoved together at once on the local network. I wrote a script to list out the tests and run them one at a time. Even breaking it down like that didn’t consistently get through CI. I also tried higher wait times (120 seconds) on the expectations. When I run them locally, most of those tests take about 5 seconds each.The test that was a real challenge was the cross-platform one. Automerge-repo has a sample sync server (NodeJS, using Automerge through WASM). I created a docker container for it, and my cross-platform integration test pushes and pulls documents to an instance that I can run in Docker. Well… Docker isn’t available for macOS runners, so that’s out for GitHub Actions. I have a script that spins up a local docker instance, and I added a check into the WebSocket network provider test – if it couldn’t find a local instance to work against, it skips the test.
Final Takeaways
Starting with a plan for isolating state made the choices of how and what I used a bit easier, and reaching for global-actor constrained classes made synchronous use of those classes much easier. For me, this mostly played out in better (synchronous) intializers and dealing with collections using functional programming patterns.
I hope there’s some planning/thinking in SwiftUI to update or extend the app structure to accomodate async hooks for things like setup and initialization (FB9221398). That should make it easier for a developer to run an async initializer and verify that it didn’t fail, before continuing into the normal app lifecycle. Likewise, I hope that the Document-based APIs gain an async-context to work with documents to likewise handle asynchronous tasks (FB12243722). Both of these spots are very awkward places for me.
Once you shift to using asynchronous calls, it can have a ripple effect in your code. If you’re looking at converting existing code, start at the “top” and work down. That helped me to make sure there weren’t secondary complications with that choice (such as a a need for an async initializer).
Better yet, step back and take the time to identify where mutable state exists. Group it together as best you can, and review how you’re interacting it, and in what isolation region. In the case of things that need to be available to SwiftUI, you can likely isolate methods appropriately (*cough*
MainActor*cough*). Then make the parts you need to pass between isolation domainsSendable. Recognize that in some cases, it may be fine to do the equivalent of “Here was the state at some recent moment, if you might want to react to that”. There are several places where I pass back a summary snapshot of mutable state to SwiftUI to use in UI elements.And do yourself a favor and keep Matt’s Concurrency Recipes on speed-dial.
Before I finished this post, I listened to episode 43 of the Swift Package Index podcast. It’s a great episode, with Holly Bora, compiler geek and manager of the Swift language team, on as a guest to talk about the Swift 6. A tidbit she shared was that they are creating a Swift 6 migration guide, to be published on the swift.org website. Something to look forward to, in addition to Matt’s collection of recipes!
https://rhonabwy.com/2024/04/29/designing-a-swift-library-with-data-race-safety/
-
Designing a Swift library with data-race safety
I cut an initial release (0.1.0-alpha) of the library automerge-repo-swift. A supplemental library to Automerge swift, it adds background networking for sync and storage capabilities. The library extends code I initially created in the Automerge demo app (MeetingNotes), and was common enough to warrant its own library. While I was extracting those pieces, I leaned into the same general pattern that was used in the Javascript library automerge-repo. That library provides largely the same functionality for Automerge in javascript. I borrowed the public API structure, as well as compatibility and implementation details for the Automerge sync protocol. One of my goals while assembling this new library was to build it fully compliant with Swift’s data-race safety. Meaning that it compiles without warnings when I use the Swift compiler’s strict-concurrency mode.
There were some notable challenges in coming up to speed with the concepts of isolation and sendability. In addition to learning the concepts, how to apply them is an open question. Not many Swift developers have embraced strict concurrency and talked about the trade-offs or implications for choices. Because of that, I feel that there’s relatively little available knowledge to understand the trade-offs to make when you protect mutable state. This post shares some of the stumbling blocks I hit, choices I made, and lessons I’ve learned. My hope is that it helps other developers facing a similar challenge.
Framing the problem
The way I try to learn and apply new knowledge to solve these kinds of “new fangled” problems is first working out how to think about the problem. I’ve not come up with a good way to ask other people how to do that. I think when I frame the problem with good first-principles in mind, trade-offs in solutions become easier to understand. Sometimes the answers are even self-obvious.
The foremost principle in strict-concurrency is “protect your mutable state”. The compiler warnings give you feedback about potential hazards and data-races. In Swift, protecting the state uses a concept of an “isolation domain”. My layman’s take on isolation is “How can the compiler verify that only one thread is accessing this bit of data at a time”. There are some places where the compiler infers the state of isolation, and some of them still changing as we progress towards Swift 6. When you’re writing code, the compiler knows what is isolated (and non-isolated) – either by itself or based on what you annotated. When the compiler infers an isolation domain, that detail is not (yet?) easily exposed to developers. It really only shows up when there’s a mismatch in your assumptions vs. what the compiler thinks and it issues a strict-concurrency warning.
Sendability is the second key concept. In my layman’s terms again, something that is sendable is safe to cross over thread boundaries. With Swift 5.10, the compiler has enough knowledge of types to be able to make guarantees about what is safe, and what isn’t.
The first thing I did was lean heavily into making anything and everything
Sendable. In hindsight, that was a bit of a mistake. Not disastrous, but I made a lot more work for myself. Not everything needs to be sendable. Taking advantage of isolation, it is fine – sometimes notably more efficient and easier to reason about – to have and use non-sendable types within an isolation domain. More on that in a bit.My key to framing up the problem was to think in terms of making explicit choices about what data should be in an isolation region along with how I want to pass information from one isolation domain to another. Any types I pass (generally) need to be Sendable, and anything that stays within an isolation domain doesn’t. For this library, I have a lot of mutable state: networking connections, updates from users, and a state machine coordinating it all. All of it is needed so a repository can store and synchronize Automerge documents. Automerge documents themselves are Sendable (I had that in place well before starting this work). I made the Automerge documents sendable by wrapping access and updates to anything mutable within a serial dispatch queue. (This was also needed because the core Automerge library – a Rust library accessed through FFI – was not safe for multi-threaded use).
Choosing Isolation
I knew I wanted to make at least one explicit isolation domain, so the first question was “Actor or isolated class?” Honestly, I’m still not sure I understand all the tradeoffs. Without knowing what the effect would be to start off with, I decided to pick “let’s use actors everywhere” and see how it goes. Some of the method calls in the design of the Automerge repository were easily and obviously async, so that seemed like a good first cut. I made the top-level repo an actor, and then I kept making any internal type that had mutable state also be it’s own actor. That included a storage subsystem and a network subsystem, both of which I built to let someone else provide the network or storage provider external to this project. To support external plugins that work with this library, I created protocols for the storage and network provider, as well as one that the network providers use to talk back to the repository.
The downside of that choice was two-fold – first setting things up, then interacting with it from within a SwiftUI app. Because I made every-darn-thing an actor, I hade to await a response, which meant a lot of potential suspension points in my code. That also propagated to imply even setup needed to be done within an async context. Sometimes that’s easy to arrange, but other times it ends up being a complete pain in the butt. More specifically, quite a few of the current Apple-provided frameworks don’t have or provide a clear path to integrate async setup hooks. The server-side Swift world has a lovely “set up and run” mechanism (swift-service-lifecycle) it is adopting, but Apple hasn’t provided a similar concept the frameworks it provides. The one that bites me most frequently is the SwiftUI app and document-based app lifecycle, which are all synchronous.
Initialization Challenges
Making the individual actors – Repo and the two network providers I created – initializable with synchronous calls wasn’t too bad. The stumbling block I hit (that I still don’t have a great solution to) was when I wanted to add and activate the network providers to a repository. To arrange that, I’m currently using a detached Task that I kick off in the SwiftUI App’s initializer:
public let repo = Repo(sharePolicy: .agreeable)public let websocket = WebSocketProvider()public let peerToPeer = PeerToPeerProvider( PeerToPeerProviderConfiguration( passcode: "AutomergeMeetingNotes", reconnectOnError: true, autoconnect: false ))@mainstruct MeetingNotesApp: App { var body: some Scene { DocumentGroup { MeetingNotesDocument() } editor: { file in MeetingNotesDocumentView(document: file.document) } .commands { CommandGroup(replacing: CommandGroupPlacement.toolbar) { } } } init() { Task { await repo.addNetworkAdapter(adapter: websocket) await repo.addNetworkAdapter(adapter: peerToPeer) } }}Swift Async Algorithms
One of the lessons I’ve learned is that if you find yourself stashing a number of actors into an array, and you’re used to interacting with them using functional methods (filter, compactMap, etc), you need to deal with the asynchronous access. The standard library built-in functional methods are all synchronous. Because of that, you can only access non-isolated properties on the actors. For me, that meant working with non-mutable state that I set up during actor initialization.
The second path (and I went there) was to take on a dependency to swift-async-algorithms, and use its async variations of the functional methods. They let you “await” results for anything that needs to cross isolation boundaries. And because it took me an embarrasingly long time to figure it out: If you have an array of actors, the way to get to an AsyncSequence of them is to use the async property on the array after you’ve imported
swift-async-algorithms. For example, something like the following snippet:let arrayOfActors: [YourActorType] = []let filteredResults = arrayOfActors.async.filter(...)Rethinking the isolation choice
That is my first version of this library. I got it functional, then turned around and tore it apart again. In making everything an actor, I was making LOTS of little isolation regions that the code had to hop between. With all the suspension points, that meant a lot of possible re-ordering of what was running. I had to be extrodinarily careful not to assume a copy of some state I’d nabbed earlier was still the same after the await. (I still have to be, but it was a more prominent issue with lots of actors.) All of this boils down to being aware of actor re-entrancy, and when it might invalidate something.
I knew that I wanted at least one isolation region (the repository). I also want to keep mutable state in separate types to preserve an isolation of duties. One particular class highlighted my problems – a wrapper around NWConnection that tracks additional state with it and handles the Automerge sync protocol. It was getting really darned inconvenient with the large number of
awaitsuspension points.I slowly clued in that it would be a lot easier if that were all synchronous – and there was no reason it couldn’t be. In my ideal world, I’d have the type
Repo(my top-level repository) as an non-global actor, and isolate any classes it used to the same isolation zone as that one, non-global, actor. I think that’s a capability that’s coming, or at least I wasn’t sure how to arrange that today with Swift 5.10. Instead I opted to make a single global actor for the library and switch what I previously set up as actors to classes isolated to that global actor.That let me simplify quite a bit, notably when dealing with the state of connections within a network adapter. What surprised me was that when I switched from Actor to isolated class, there were few warnings from the change. The changes were mostly warnings that calls dropped back to synchronous, and no longer needed
await. That was quick to fix up; the change to isolated classes was much faster and easier than I anticipated. After I made the initial changes, I went through the various initializers and associated configuration calls to make more of it explicitly synchronous. The end result was more code that could be set up (initialized) without an async context. And finally, I updated how I handled the networking so that as I needed to track state, I didn’t absolutely have to use the async algorithsm library.A single global actor?
A bit of a side note: I thought about making
Repoa global actor, but I prefer to not demand a singleton style library for it’s usage. That choice made it much easier to host multiple repositories when it came time to run functional tests with a mock In-Memory network, or integration tests with the actual providers. I’m still a slight bit concerned that I might be adding to a long-term potential proliferation of global actors from libraries – but it seems like the best solution at the moment. I’d love it if I could do something that indicated “All these things need a single isolation domain, and you – developer – are responsible for providing one that fits your needs”. I’m not sure that kind of concept is even on the table for future work.Recipes for solving these problems
If you weren’t already aware of it, Matt Massicotte created a GitHub repository called ConcurrencyRecipes. This is a gemstone of knowledge, hints, and possible solutions. I leaned into it again and again while building (and rebuilding) this library. One of the “convert it to async” challenges I encountered was providing an async interface to my own peer-to-peer network protocol. I built the protocol using the Network framework based (partially on Apple’s sample code), which is all synchronous code and callbacks. A high level, I wanted it to act similarly URLSessionWebSocketTask. This gist being a connection has an
async send()and anasync receive()for sending and receiving messages on the connection. With an asyncsendandreceive, you can readily assemble several different patterns of access.To get there, I used a combination of CheckedContinuation (both the throwing and non-throwing variations) to work with what
NWConnectionprovided. I wish that was better documented. How to properly use those APIs is opaque, but that is a digression for another time. I’m particular happy with how my code worked out, including adding a method on the PeerConnection class that used structured concurrency to handle a timeout mechanism.Racing tasks with structured concurrency
One of the harder warnings for me to understand was related to racing concurrent tasks in order to create an async method with a “timeout”. I stashed a pattern for how to do this in my notebook with references to Beyond the basics of structured concurrency from WWDC23.
If the async task returns a value, you can set it up something like this (this is from PeerToPeerConnection.swift):
let msg = try await withThrowingTaskGroup(of: SyncV1Msg.self) { group in group.addTask { // retrieve the next message try await self.receiveSingleMessage() } group.addTask { // Race against the receive call with a continuous timer try await Task.sleep(for: explicitTimeout) throw SyncV1Msg.Errors.Timeout() } guard let msg = try await group.next() else { throw CancellationError() } // cancel all ongoing tasks (the websocket receive request, in this case) group.cancelAll() return msg}There’s a niftier version available in Swift 5.9 (which I didn’t use) for when you don’t care about the return value:
func run() async throws { try await withThrowingDiscardingTaskGroup { group in for cook in staff.keys { group.addTask { try await cook.handleShift() } } group.addTask { // keep the restaurant going until closing time try await Task.sleep(for: shiftDuration) throw TimeToCloseError() } }}With Swift 5.10 compiler, my direct use of this displayed a warning:
warning: passing argument of non-sendable type 'inout ThrowingTaskGroup<SyncV1Msg, any Error>' outside of global actor 'AutomergeRepo'-isolated context may introduce data racesguard let msg = try await group.next() else { ^I didn’t really understand the core of this warning, so I asked on the Swift forums. VNS (on the forums) had run into the same issue and helped explain it:
It’s because
withTaskGroupaccepts a non-Sendableclosure, which means the closure has to be isolated to whatever context it was formed in. If yourtest()function isnonisolated, it means the closure isnonisolated, so callinggroup.waitForAll()doesn’t cross an isolation boundary.The workaround to handle the combination of non-sendable closures and TaskGroup is to make the async method that runs this code
nonisolated. In the context I was using it, the class that contains this method is isolated to a global actor, so it’s inheriting that context. By switching the method to be explicitly non-isolated, the compiler doesn’t complain aboutgroupbeing isolated to that global actor.Sharing information back to SwiftUI
These components have all sorts of interesting internal state, some of which I wanted to export. For example, to provide information from the network providers to make a user interface (in SwiftUI). I want to be able to choose to connect to endpoints, to share what endpoints might be available (from the NWBrowser embedded in the peer to peer network provider), and so forth.
I first tried to lean into AsyncStreams. While they make a great local queue for a single point to point connection, I found they were far less useful to generally make a firehouse of data that SwiftUI knows how to read and react to. While I tried to use all the latest techniques, to handle this part I went to my old friend Combine. Some people are effusing that Combine is dead and dying – but boy it works. And most delightfully, you can have any number of endpoints pick up and subscribe to a shared publisher, which was perfect for my use case. Top that off with SwiftUI having great support to receive streams of data from Combine, and it was an easy choice.
I ended up using Combine publishers to make a a few feeds of data from the PeerToPeerProvider. They share information about what other peers were available, the current state of the listener (that accepts connections) and the browser (that looks for peers), and last a publisher that provides information about active peer to peer connctions. I feel that worked out extremely well. It worked so well that I made an internal publisher (not exposed via the public API) for tests to get events and state updates from within a repository.
Integration Testing
It’s remarkably hard to usefully unit test network providers. Instead of unit testing, I made a separate Swift project for the purposes of running integration tests. It sits in it’s own directory in the git repository and references
automerge-repo-swiftas a local dependency. A side effect is that it let me add in all sorts of wacky dependencies that were handy for the integration testing, but that I really didn’t want exposed and transitive for the main package. I wish that Swift Packages had a means to identify test-only dependencies that didn’t propagate to other packages for situations like this. Ah well, my solution was a separate sub-project.Testing using the Combine publisher worked well. Although it took a little digging to figure out the correct way to set up and use expectations with async XCTests. It feels a bit exhausting to assemble the expectations and fulfillment calls, but its quite possible to get working. If you want to see this in operation, take a look at P2P+explicitConnect.swift. I started to look at potentially using the upcoming swift-testing, but with limited Swift 5.10 support, I decided to hold off for now. If it makes asynchronous testing easier down the road, I may well adopt it quickly after it’s initial release.
The one quirky place that I ran into with that API setup was that
expectation.fulfill()gets cranky with you if you call it more than once. My publisher wasn’t quite so constrained with state updates, so I ended up cobbling a boolean latch variable in asinkwhen I didn’t have a sufficiently constrained closure.The other quirk in integration testing is that while it works beautifully on a local machine, I had a trouble getting it to work in CI (using GitHub Actions). Part of the issue is that the current
swift testdefaults to running all possible tests at once, in parallel. Especially for integration testing of peer to peer networking, that meant a lot of network listeners, and browsers, getting shoved together at once on the local network. I wrote a script to list out the tests and run them one at a time. Even breaking it down like that didn’t consistently get through CI. I also tried higher wait times (120 seconds) on the expectations. When I run them locally, most of those tests take about 5 seconds each.The test that was a real challenge was the cross-platform one. Automerge-repo has a sample sync server (NodeJS, using Automerge through WASM). I created a docker container for it, and my cross-platform integration test pushes and pulls documents to an instance that I can run in Docker. Well… Docker isn’t available for macOS runners, so that’s out for GitHub Actions. I have a script that spins up a local docker instance, and I added a check into the WebSocket network provider test – if it couldn’t find a local instance to work against, it skips the test.
Final Takeaways
Starting with a plan for isolating state made the choices of how and what I used a bit easier, and reaching for global-actor constrained classes made synchronous use of those classes much easier. For me, this mostly played out in better (synchronous) intializers and dealing with collections using functional programming patterns.
I hope there’s some planning/thinking in SwiftUI to update or extend the app structure to accomodate async hooks for things like setup and initialization (FB9221398). That should make it easier for a developer to run an async initializer and verify that it didn’t fail, before continuing into the normal app lifecycle. Likewise, I hope that the Document-based APIs gain an async-context to work with documents to likewise handle asynchronous tasks (FB12243722). Both of these spots are very awkward places for me.
Once you shift to using asynchronous calls, it can have a ripple effect in your code. If you’re looking at converting existing code, start at the “top” and work down. That helped me to make sure there weren’t secondary complications with that choice (such as a a need for an async initializer).
Better yet, step back and take the time to identify where mutable state exists. Group it together as best you can, and review how you’re interacting it, and in what isolation region. In the case of things that need to be available to SwiftUI, you can likely isolate methods appropriately (*cough*
MainActor*cough*). Then make the parts you need to pass between isolation domainsSendable. Recognize that in some cases, it may be fine to do the equivalent of “Here was the state at some recent moment, if you might want to react to that”. There are several places where I pass back a summary snapshot of mutable state to SwiftUI to use in UI elements.And do yourself a favor and keep Matt’s Concurrency Recipes on speed-dial.
Before I finished this post, I listened to episode 43 of the Swift Package Index podcast. It’s a great episode, with Holly Bora, compiler geek and manager of the Swift language team, on as a guest to talk about the Swift 6. A tidbit she shared was that they are creating a Swift 6 migration guide, to be published on the swift.org website. Something to look forward to, in addition to Matt’s collection of recipes!
-
Mavo’s Phantox Pro Wants to Shake Up the Mid-Range Manual Grinder Market
The manual coffee grinder market is a remarkably crowded space right now. There was a time when, if you wanted true precision grind quality, you bought a Comandante and accepted the price tag without flinching. Then 1Zpresso and Timemore arrived, collectively proving that serious burr geometry does not require a serious financial commitment. The mid-range segment they created is now seriously competitive, and a new name is trying to wedge itself right into that conversation.
That name is Mavo. The brand is not exactly a startup; they have been operating out of China since 2012, building a solid domestic presence while selling glass drippers, kettles, and electric grinders. For most of that time, their manual grinders were squarely aimed at the casual consumer market: somewhat capable, not particularly exciting.
The Phantox Pro (Amazon, $129) is a deliberate break from that. It is their first genuine shot at the specialty coffee enthusiast crowd, and it makes a reasonable case for itself. We got a sample unit in to check out for our forthcoming Best Budget Grinders for 2026 Guide, and have been putting it through its paces for a full Snapshot Review. Here are the initial impressions.
The box the Phantox Pro comes in.Build and Feel
Out of the box, the first thing you notice is the weight: a substantial 630 grams of CNC machined aluminium alloy. The fit and finish feel tight from the factory. There is no rattle in the handle assembly, no loose play anywhere. The “champagne” coloured unit we received looks sharp (a black version with slightly more readable markings is also available), and the cutouts and textures in the body give it a purposeful, grippy feel in hand. A silicone ring ships in the box for extra purchase if you need it.
The external grind adjustment dial is the thing you will notice first, and keep noticing. It moves with a dampened, deliberate resistance that is genuinely reminiscent of adjusting a manual f-stop ring on a high end camera lens (I’m talking Leica levels here, folks). There are 120 clicks per full revolution, with each click representing 0.0167mm of burr travel. It is a satisfying mechanism. More practically, it makes documenting and repeating your grind settings refreshingly straightforward. No guessing how many clicks from zero. No fumbling around with an internal adjustment collar while juggling a catch cup full of coffee you spent good money on.
The grind adjustment dial has 120 clicks and feels very nice when in use.Inside the chassis, Mavo uses a triple bearing stabilization system on the central axle to keep the inner burr running concentric under load. It is beefy and designed to work with a power drill if you’re so inclined (and don’t want to hand crank). In fact, the company ships the grinder with a secondary lid designed to work with a cordless drill.
This is the secondary lid the grinder ships with, if you’d prefer to use a cordless drill to electrically churn the burrs.One thing to note here: when you disassemble the grinder for a deep clean, the axle needs to be carefully realigned during reassembly. It is not complicated once you understand what is happening, but the first time you take it apart and find it does not quite want to go back together, do not force anything. Look closely at the axle alignment first.
The Burrs: What They Do Well, and What They Cannot Do
The 45mm seven-sided stainless steel burrs are the big picture story of this grinder. The cutting surface is noticeably larger than the 38mm to 40mm burrs common in this price bracket, and that size advantage pays off in grinding speed. In our initial testing, the Phantox Pro turned out roughly 0.5 grams per second at a standard pour over setting, which is a respectable clip for a hand grinder.
The rotating burr and its cutting pattern.The geometry of these burrs is genuinely unusual. The bottom third of the inner static burr does not use the standard conical cut pattern you would typically expect. Instead, it uses an aggressive hatch design that closely resembles a flat ghost burr, similar in principle to the burr design used in the much pricier Orphan Espresso Apex. Ghost burr geometries are known for producing highly uniform particle sizes while sharply limiting the production of fine coffee dust.
You can see the “ghost burr” pattern on the bottom portion of the static burr inside the grinder housing.That approach is very apparent in the cup. I’ve already pushed roughly 8kg of coffee through our test unit so far, and the output for pour over and full immersion brewing is impressive. The Phantox Pro produces a clean, well-separated flavour profile with good clarity and sweetness. For a Chemex, a press pot, a siphon, or a V60, this grinder punches well above its $140 USD price point. In early head to head testing, it is holding its own against our current filter benchmark, the 1Zpresso K Ultra, which costs considerably more.
The trade-off is straightforward and worth stating plainly: this burr design makes it a poor espresso grinder. Mavo markets the Phantox Pro as a multi-purpose tool capable of everything from Turkish coffee to French press, and the external dial has the mechanical precision to dial very fine. The problem is physics, not mechanics. Espresso extraction relies on a specific volume of fine particles to fill the gaps between larger grounds, building the puck resistance needed to generate nine bars of brew pressure.
Because these burrs are engineered to limit fines production, you simply cannot build that resistance. Shots run fast and extract poorly. No amount of careful dialling changes what the burr geometry is doing. Buyers who primarily pull espresso should look elsewhere, at something like the 1Zpresso J Ultra or the Kingrinder K6.
Where Things Stand
All that volume testing had a purpose: we wanted to know whether the Phantox Pro had earned a spot in our forthcoming Best Budget Grinders for 2026 Guide as a filter-focused recommendation. It has. For a brewer who lives and breathes pour over and immersion methods, $140 is a very reasonable entry point for this level of cup quality and build refinement.
We are wrapping up formal testing protocols now, including particle analysis and head to head comparisons with established competitors. Expect a full breakdown in our forthcoming Best Budget Grinders Guide, and a complete Snapshot Review of the Mavo Phantox Pro later this summer. In the meantime, the grinder is available on Amazon or directly from Mavo.
#brewGrinder #coffeeGrinder #grinder #manualGrinder #Mavo #phantoxPro -
Erzwungene „Grenzpartnerschaft“: EU-Kommission will US-Behörden erlauben, politische Ansichten und „Herkunft“ abzufragen
Laut dem nun vorliegenden Entwurf eines Rahmenabkommens über eine „Grenzpartnerschaft“ mit der Trump-Administration dürfen US-Behörden in EU-Staaten nicht nur Gesichtsbilder, sondern auch Namen, Gesundheitsdaten oder sexuelle Orientierung in Polizeidatenbanken abfragen.
Die Europäische Kommission hat nach Erteilung ihres Verhandlungsmandats im vergangenen Dezember ein Rahmenabkommen mit den USA über eine „Grenzpartnerschaft“ fertig ausgehandelt. Den Entwurf hat die britische Bürgerrechtsorganisation Statewatch veröffentlicht. Demzufolge geht das geplante Abkommen weit über die bislang bekannten US-Forderungen hinaus.
Den Abschluss einer „Enhanced Border Security Partnership“ (EBSP) hatte die US-Regierung bereits 2022 von allen Teilnehmerstaaten des Visa-Waiver-Programms (VWP) verlangt – mit einer Frist bis Ende 2026. Das VWP ermöglicht Staatsangehörigen aus 43 befreundeten Ländern im Rahmen von Kurzaufenthalten bis zu 90 Tagen visafreies Reisen in die USA – und umgekehrt.
Nun knüpft die Regierung in Washington die weitere Teilnahme an dem Programm an den Abschluss der „Grenzpartnerschaft“: Die beteiligten Staaten sollen ihre Polizeidatenbanken für US-Behörden öffnen. Wer sich weigert, verliert den visafreien Status. Das Abkommen soll dem Entwurf zufolge auf dem Prinzip der Gegenseitigkeit beruhen. EU-Mitgliedstaaten sollen also ihrerseits Zugriff auf US-Datenbanken erhalten – sofern sich die US-Regierung nicht dagegen sperrt.
Mehr als Fingerabdrücke und Gesichtsbilder
Im Entwurf für das Rahmenabkommen ist nun auch die Rede davon, die Datenabfrage dazu zu nutzen dass „Personen, die ein echtes Risiko für die öffentliche Sicherheit oder öffentliche Ordnung darstellen“, daran gehindert werden, in den USA „zu verbleiben“. Es geht also auch um Abschiebungen, wie sie derzeit monatlich tausendfach von der brutalen US-Einwanderungsbehörde ICE durchgeführt werden. Ursprünglich hieß es, die „Grenzpartnerschaft“ solle nur bei Einreisen in die USA angewandt werden.
Außerdem galt bislang, dass US-Grenzbehörden nur Zugriff auf Fingerabdrücke und Lichtbilder in Polizeidatenbanken der VWP-Staaten verlangen. Im von der EU-Kommission ausgehandelten Entwurf steht darüber hinaus, dass auch „alphanumerische Daten zur Identifizierung einer Person, wie Vorname, Nachname und Geburtsdatum“ abgefragt werden können.
Kommt es bei einer Anfrage zu einem Treffer, darf die angefragte Behörde – in Deutschland etwa das Bundeskriminalamt – ihrerseits nachfragen, was das Interesse an der Person ausgelöst hat und alle „bei der anfragenden zuständigen Behörde verfügbaren alphanumerischen und kontextuellen Daten zu derselben Person anfordern“.
Weitergabe an Drittstaaten möglich
Unter bestimmten Bedingungen dürfen laut Entwurf auch besonders sensible Kategorien personenbezogener Daten übermittelt werden, darunter Informationen zu „rassischer oder ethnischer Herkunft, politischen Ansichten oder religiösen oder sonstigen Überzeugungen, Gewerkschaftszugehörigkeit“ sowie Angaben zu „Gesundheit oder Sexualleben“.
Der Entwurf erlaubt sogar die Weitergabe empfangener Daten an Behörden in Drittstaaten oder internationale Organisationen – allerdings nur mit vorheriger Zustimmung der übermittelnden Behörde. Welche Drittstaaten konkret gemeint sein könnten, lässt der Entwurf offen. In Betracht kämen neben Interpol auch enge Verbündete der USA, etwa Großbritannien oder andere Staaten des Commonwealth sowie Israel, das eigene Abkommen zum Datentausch mit den USA geschlossen hat.
In Deutschland wären Millionen Datensätze betroffen
Angaben zu den abfrageberechtigten Behörden – auf US-Seite kämen vor allem der Zoll- und Grenzschutz (Customs and Border Protection, CBP) sowie ICE in Frage – enthält der nun veröffentlichte Rahmenentwurf nicht. Das soll jeweils in bilateralen Umsetzungsabkommen geregelt werden, die jeder betroffene Staat separat mit Washington schließen muss.
In Deutschland beträfe dies wohl die INPOL-Datenbank aller Polizeien des Bundes und der Länder, die derzeit Fotos und Fingerabdrücke von 5,4 Millionen Personen enthält – darunter mehr als die Hälfte Asylsuchende. Selbst innerhalb der EU gibt es bislang keinen gegenseitigen Direktzugriff auf derartig umfangreiche Informationssysteme einzelner Mitgliedstaaten – das geplante Abkommen mit den USA ist deshalb besonders intrusiv.
Das Rahmenabkommen regelt auch den Einsatz von Software zur Erstellung von Prognosen aus den abgefragten Datensätzen. Zwar sollen Entscheidungen mit „erheblichen nachteiligen Auswirkungen“ nicht ausschließlich automatisiert erfolgen, sondern stets mit „menschlicher Beteiligung“. Vollautomatische Entscheidungen sind aber erlaubt, wenn dies „nach dem jeweiligen Rechtsrahmen der Vertragsparteien zulässig“ ist. In der EU wäre dies nach der KI-Verordnung ausgeschlossen, in den USA gibt es einen vergleichbaren Rechtsakt nicht.
Eingeschränkte Rechte für Betroffene
Der Entwurf enthält auch Vorgaben zur Protokollierung für „Prüftätigkeiten“ unter anderem von Datenschutzbehörden. Das Abkommen sieht außerdem vor, dass betroffene Personen Auskunft über ihre gespeicherten Daten sowie deren Berichtigung oder Löschung beantragen können.
Diese Rechte stehen jedoch unter Einschränkungsvorbehalt: Verwehrt werden darf der Zugang unter anderem aus Gründen der nationalen Sicherheit, zum Schutz laufender Ermittlungen oder zur Strafverfolgung.
Den Entwurf des Rahmenabkommens werden die EU-Innen- und Justizminister*innen auf einer ihrer kommenden Sitzungen beraten. Nach derzeitigem Stand wird das Parlament daran nicht beteiligt. Die endgültige Entscheidung über den Abschluss wird dann im Rat der Europäischen Union von den 27 Mitgliedstaaten getroffen. Ob dies noch vor der Sommerpause erfolgt, ist unklar. Auch die US-Regierung muss ihre Zustimmung zu dem Entwurf geben. Bis zur Deadline am 31. Dezember 2026 könnte es dann in Kraft treten.
Matthias Monroy, Wissensarbeiter, Aktivist und Mitglied der Redaktion der Zeitschrift Bürgerrechte & Polizei/CILIP. Außerdem Redakteur für Innenpolitik der Zeitung nd.Der Tag. Texte auf Englisch unter digit.site36.net, auf Twitter @matthimon. Dieser Beitrag ist eine Übernahme von netzpolitik, gemäss Lizenz Creative Commons BY-NC-SA 4.0.
-
Erzwungene „Grenzpartnerschaft“: EU-Kommission will US-Behörden erlauben, politische Ansichten und „Herkunft“ abzufragen
Laut dem nun vorliegenden Entwurf eines Rahmenabkommens über eine „Grenzpartnerschaft“ mit der Trump-Administration dürfen US-Behörden in EU-Staaten nicht nur Gesichtsbilder, sondern auch Namen, Gesundheitsdaten oder sexuelle Orientierung in Polizeidatenbanken abfragen.
Die Europäische Kommission hat nach Erteilung ihres Verhandlungsmandats im vergangenen Dezember ein Rahmenabkommen mit den USA über eine „Grenzpartnerschaft“ fertig ausgehandelt. Den Entwurf hat die britische Bürgerrechtsorganisation Statewatch veröffentlicht. Demzufolge geht das geplante Abkommen weit über die bislang bekannten US-Forderungen hinaus.
Den Abschluss einer „Enhanced Border Security Partnership“ (EBSP) hatte die US-Regierung bereits 2022 von allen Teilnehmerstaaten des Visa-Waiver-Programms (VWP) verlangt – mit einer Frist bis Ende 2026. Das VWP ermöglicht Staatsangehörigen aus 43 befreundeten Ländern im Rahmen von Kurzaufenthalten bis zu 90 Tagen visafreies Reisen in die USA – und umgekehrt.
Nun knüpft die Regierung in Washington die weitere Teilnahme an dem Programm an den Abschluss der „Grenzpartnerschaft“: Die beteiligten Staaten sollen ihre Polizeidatenbanken für US-Behörden öffnen. Wer sich weigert, verliert den visafreien Status. Das Abkommen soll dem Entwurf zufolge auf dem Prinzip der Gegenseitigkeit beruhen. EU-Mitgliedstaaten sollen also ihrerseits Zugriff auf US-Datenbanken erhalten – sofern sich die US-Regierung nicht dagegen sperrt.
Mehr als Fingerabdrücke und Gesichtsbilder
Im Entwurf für das Rahmenabkommen ist nun auch die Rede davon, die Datenabfrage dazu zu nutzen dass „Personen, die ein echtes Risiko für die öffentliche Sicherheit oder öffentliche Ordnung darstellen“, daran gehindert werden, in den USA „zu verbleiben“. Es geht also auch um Abschiebungen, wie sie derzeit monatlich tausendfach von der brutalen US-Einwanderungsbehörde ICE durchgeführt werden. Ursprünglich hieß es, die „Grenzpartnerschaft“ solle nur bei Einreisen in die USA angewandt werden.
Außerdem galt bislang, dass US-Grenzbehörden nur Zugriff auf Fingerabdrücke und Lichtbilder in Polizeidatenbanken der VWP-Staaten verlangen. Im von der EU-Kommission ausgehandelten Entwurf steht darüber hinaus, dass auch „alphanumerische Daten zur Identifizierung einer Person, wie Vorname, Nachname und Geburtsdatum“ abgefragt werden können.
Kommt es bei einer Anfrage zu einem Treffer, darf die angefragte Behörde – in Deutschland etwa das Bundeskriminalamt – ihrerseits nachfragen, was das Interesse an der Person ausgelöst hat und alle „bei der anfragenden zuständigen Behörde verfügbaren alphanumerischen und kontextuellen Daten zu derselben Person anfordern“.
Weitergabe an Drittstaaten möglich
Unter bestimmten Bedingungen dürfen laut Entwurf auch besonders sensible Kategorien personenbezogener Daten übermittelt werden, darunter Informationen zu „rassischer oder ethnischer Herkunft, politischen Ansichten oder religiösen oder sonstigen Überzeugungen, Gewerkschaftszugehörigkeit“ sowie Angaben zu „Gesundheit oder Sexualleben“.
Der Entwurf erlaubt sogar die Weitergabe empfangener Daten an Behörden in Drittstaaten oder internationale Organisationen – allerdings nur mit vorheriger Zustimmung der übermittelnden Behörde. Welche Drittstaaten konkret gemeint sein könnten, lässt der Entwurf offen. In Betracht kämen neben Interpol auch enge Verbündete der USA, etwa Großbritannien oder andere Staaten des Commonwealth sowie Israel, das eigene Abkommen zum Datentausch mit den USA geschlossen hat.
In Deutschland wären Millionen Datensätze betroffen
Angaben zu den abfrageberechtigten Behörden – auf US-Seite kämen vor allem der Zoll- und Grenzschutz (Customs and Border Protection, CBP) sowie ICE in Frage – enthält der nun veröffentlichte Rahmenentwurf nicht. Das soll jeweils in bilateralen Umsetzungsabkommen geregelt werden, die jeder betroffene Staat separat mit Washington schließen muss.
In Deutschland beträfe dies wohl die INPOL-Datenbank aller Polizeien des Bundes und der Länder, die derzeit Fotos und Fingerabdrücke von 5,4 Millionen Personen enthält – darunter mehr als die Hälfte Asylsuchende. Selbst innerhalb der EU gibt es bislang keinen gegenseitigen Direktzugriff auf derartig umfangreiche Informationssysteme einzelner Mitgliedstaaten – das geplante Abkommen mit den USA ist deshalb besonders intrusiv.
Das Rahmenabkommen regelt auch den Einsatz von Software zur Erstellung von Prognosen aus den abgefragten Datensätzen. Zwar sollen Entscheidungen mit „erheblichen nachteiligen Auswirkungen“ nicht ausschließlich automatisiert erfolgen, sondern stets mit „menschlicher Beteiligung“. Vollautomatische Entscheidungen sind aber erlaubt, wenn dies „nach dem jeweiligen Rechtsrahmen der Vertragsparteien zulässig“ ist. In der EU wäre dies nach der KI-Verordnung ausgeschlossen, in den USA gibt es einen vergleichbaren Rechtsakt nicht.
Eingeschränkte Rechte für Betroffene
Der Entwurf enthält auch Vorgaben zur Protokollierung für „Prüftätigkeiten“ unter anderem von Datenschutzbehörden. Das Abkommen sieht außerdem vor, dass betroffene Personen Auskunft über ihre gespeicherten Daten sowie deren Berichtigung oder Löschung beantragen können.
Diese Rechte stehen jedoch unter Einschränkungsvorbehalt: Verwehrt werden darf der Zugang unter anderem aus Gründen der nationalen Sicherheit, zum Schutz laufender Ermittlungen oder zur Strafverfolgung.
Den Entwurf des Rahmenabkommens werden die EU-Innen- und Justizminister*innen auf einer ihrer kommenden Sitzungen beraten. Nach derzeitigem Stand wird das Parlament daran nicht beteiligt. Die endgültige Entscheidung über den Abschluss wird dann im Rat der Europäischen Union von den 27 Mitgliedstaaten getroffen. Ob dies noch vor der Sommerpause erfolgt, ist unklar. Auch die US-Regierung muss ihre Zustimmung zu dem Entwurf geben. Bis zur Deadline am 31. Dezember 2026 könnte es dann in Kraft treten.
Matthias Monroy, Wissensarbeiter, Aktivist und Mitglied der Redaktion der Zeitschrift Bürgerrechte & Polizei/CILIP. Außerdem Redakteur für Innenpolitik der Zeitung nd.Der Tag. Texte auf Englisch unter digit.site36.net, auf Twitter @matthimon. Dieser Beitrag ist eine Übernahme von netzpolitik, gemäss Lizenz Creative Commons BY-NC-SA 4.0.
Über Matthias Monroy - netzpolitik:
Unter der Kennung "Gastautor:innen" fassen wir die unterschiedlichsten Beiträge externer Quellen zusammen, die wir dankbar im Beueler-Extradienst (wieder-)veröffentlichen dürfen. Die Autor*innen, Quellen und ggf. Lizenzen sind, soweit bekannt, jeweils im Beitrag vermerkt und/oder verlinkt.
-
Ever since the Invisible Salamanders paper was published, there has been a quiet renaissance within my friends and colleagues in applied cryptography for studying systems that use Authenticated Encryption with Associated Data (AEAD) constructions, understanding what implicit assumptions these systems make about the guarantees of the AEAD mode they chose to build upon, and the consequences of those assumptions being false.
I’ve discussed Invisible Salamanders several times throughout this blog, from my criticisms of AES-GCM and XMPP + OMEMO to my vulnerability disclosures in Threema.
Five years after Invisible Salamanders, it’s become clear to me that many software developers do not fully appreciate the underlying problem discussed in the Invisible Salamanders paper, even when I share trivial proof-of-concept exploits.
Background
Fast AEAD constructions based on polynomial MACs, such as AES-GCM and ChaCha20-Poly1305, were designed to provide confidentiality and integrity for the plaintext data, and optionally integrity for some additional associated data, in systems where both parties already negotiated one shared symmetric key.
The integrity goals of the systems that adopted these AEAD constructions were often accompanied by performance goals–usually to prevent Denial of Service (DoS) attacks in networking protocols. Verification needed to be very fast and consume minimal resources.
In this sense, AEAD constructions were an incredible success. So successful, in fact, that most cryptographers urge application developers to use one of the fast AEAD modes as the default suggestion without looking deeper at the problem being solved. This is a good thing, because most developers will choose something stupid like ECB mode in the absence of guidance from cryptographers, and AEAD modes are much, much safer than any hand-rolled block cipher modes.
The problem is, that one tiny little assumption that both parties (sender, recipient) for a communication have agreed on exactly one symmetric key for use in the protocol.
Fast MACs Are Not Key-Committing
Cryptographers have concluded that AEAD constructions based on polynomial MACs–while great for performance and rejection of malformed packets without creating DoS risks–tend to make the same assumption. This is even true of misuse-resistant modes like AES-GCM-SIV and extended-nonce constructions like XSalsa20-Poly1305.
When discussing this implicit assumption of only one valid key in the systems that use these AEAD modes, we say that the modes are not key-committing. This terminology is based on what happens when this assumption is false.
Consequently, you can take a single, specially crafted ciphertext (with an authentication tag) and decrypt it under multiple different keys. The authentication tags will be valid for all keys, and the plaintext will be different.
Art: SwizzWhat does this look like in practice?
Consider my GCM exploit, which was written to generate puzzle ciphertexts for the DEFCON Furs badge challenge a few years ago. How it works is conceptually simple (although the actual mechanics behind step 4 is a bit technical):
- Generate two keys.
There’s nothing special about these keys, or their relationship to each other, and can be totally random. They just can’t be identical or the exploit is kind of pointless.
- Encrypt some blocks of plaintext with key1.
- Encrypt some more blocks of plaintext with key2.
- Calculate a collision block from the ciphertext in the previous two steps–which is just a bit of polynomial arithmetic in GF(2^128)
- Return the ciphertext (steps 2, 3, 4) and authentication tag calculated over them (which will collide for both keys).
A system that decrypts the output of this exploit under key1 will see some plaintext, followed by some garbage, followed by 1 final block of garbage.
If the same system decrypts under key2, it will see some garbage, followed by some plaintext, followed by 1 final block of garbage.
For many file formats, this garbage isn’t really a problem. Additionally, a bit more precomputation allows you to choose garbage that will be more advantageous to ensuring both outputs are accepted as “valid” by the target system.
For example, choosing two keys and a targeted nonce may allow both the valid plaintext and garbage blocks to begin with a PDF file header.
If you’re familiar with the file polyglot work of Ange Albertini, you can use this to turn the invisible salamanders problem into an artform.
Why is it called Invisible Salamanders?
The proof-of-concept used in the paper involved sending one picture (of a salamander) over an end-to-end encrypted messaging app, but when the recipient flagged it as abusive, the moderator saw a different picture.
https://www.youtube.com/watch?v=3M1jIO-jLHI
Thus, the salamander was invisible to the moderators of the encrypted messaging app.
As for the choice of a “salamander”, I’ve been told by friends familiar with the research that was inspired by the original name of the Signal Protocol being “Axolotl”.
But, like, who cares about these details besides me? It’s a cute and memorable name.
What are the consequences of violating the “one key” assumption?
That depends entirely on what your system does!
In Database Cryptography Fur the Rest of Us, I discussed the use of AEAD modes to prevent confused deputy attacks. This works great, but if you’re building an application that supports multi-tenancy, you suddenly have to care about this issue again.
An earlier design for OPAQUE, a password authenticated key exchange algorithm, was broken by a partitioning oracle attack due to building atop AEAD modes that are not key-committing. This let an attacker recover passwords from Shadowsocks proxy servers with a complexity similar to a binary search algorithm.
These are two very different impacts from the same weakness, which I believe is a significant factor for why the Invisible Salamanders issue isn’t more widely understood.
Sometimes violating the “one key” assumption that went into fast AEAD modes based on Polynomial MACs completely destroys the security of your system.
Other times, it opens the door for a high-complexity but low-impact behavior that simply violates the principle of least astonishment but doesn’t buy the attacker anything useful.
They Just Don’t Get It
The Invisible Salamanders issue is relevant in any system that uses symmetric-key encryption where more than one key can be valid.
This includes, but is not limited to:
- Multi-tenant data warehouses
- Group messaging protocols
- Envelope encryption schemes with multiple wrapping keys
- Bearer tokens (such as JSON Web Tokens) in systems that utilize Key IDs
Systems can mitigate this issue by introducing an explicit key commitment scheme (based on a cryptographic hash rather than a polynomial MAC) or by using a committing cipher mode (such as AES + HMAC, if done carefully).
However, most of the time, this advice falls on deaf ears whenever this concern is brought up by a cryptography engineer who’s more aware of this issue.
“Abuse reporting? We don’t have no stinking abuse reporting!”
The most common misunderstanding is, “We don’t have a report abuse feature, so this issue doesn’t affect us.”
This is because the Invisible Salamanders talk and paper focused on how it could be leveraged to defeat abuse reporting tools and bypass content moderation.
In my experience, many security teams would read the paper and conclude that it only impacts abuse reporting features and not potentially all systems that allow multiple symmetric keys in a given context.
Another Exploit Scenario
Imagine you’re building a Data Loss Prevention product that integrates with corporate file-sharing and collaboration software (e.g. ownCloud) for small and medium businesses.
One day, someone decides to ship an end-to-end encryption feature to the file-sharing software that uses AES-GCM to encrypt files, and then encrypts the keys to each recipient’s public key. This is basically the envelope encryption use-case above.
In order to update your integration to act as another “user”, whose public key must be included in all E2EE transfers, and will block download of ciphertexts it cannot decrypt OR contains sensitive information.
And this works, until an insider threat clever enough to abuse the Invisible Salamanders issue comes along.
In order for said insider threat (e.g., a senior business analyst) to leak sensitive data (e.g., anything that would be useful for illegal insider trading) to another person that shouldn’t have access to it (e.g., a store clerk that’s talking to the press), they just have to do this:
- Encrypt the data they want to exfiltrate using key1.
- Encrypt some innocuous data that won’t trigger your DLP product, using key2.
- Ensure that both messages encrypt to the same ciphertext and authentication tag.
- Give their recipient key1, give everyone else (including your DLP software) key2.
Bam! File leaked, and everyone’s none the wiser, until it’s too late. Let’s actually imagine what happens next:
A random store clerk has leaked sensitive data to the press that only a few analysts had access to.
The only communication between the analyst and the store clerk is a file that was shared to all employees, using the E2EE protocol. No emails or anything else were identified.
Your DLP product didn’t identify any other communications between these two, but somehow the store clerk has the data on their desktop.
A detailed forensics analysis may eventually figure out what happened, but by then, the damage is done and your product’s reputation is irrecoverably damaged.
All because the hypothetical E2EE protocol didn’t include a key-commitment mechanism, and nobody identified this deficit in their designs.
This isn’t to endorse DLP solutions at all, but rather, to highlight one of the many ways that the Invisible Salamander issue can be used creatively by clever attackers.
Art: AJThe Lesson to Learn
If you’re building a network protocol that uses AEAD to encrypt data over an insecure network (e.g., WireGuard), keep up the good work.
If you’re doing anything more involved than that, at the application layer, pause for a moment and consider whether your system will ever need multiple valid symmetric keys at once.
And, if the answer is “yes”, then you should always explicitly add a key-commitment mechanism to your system design.
(Hire a cryptographer if you’re not sure how to proceed.)
In my opinion, hemming and hawing over whether there’s a significant impact to the Invisible Salamanders issue is a worse use of your time than just solving it directly.
Eventually, I expect a new generation of AEAD modes will be standardized that explicitly provide key-commitment.
When these new designs are standardized, widely supported, and sufficiently trusted by experts, feel free to update my advice to “prefer using those modes” instead.
Header art: Harubaki, CMYKat, and Brian Gratwicke
https://soatok.blog/2024/09/10/invisible-salamanders-are-not-what-you-think/
#AEAD #AESGCM #InvisibleSalamanders #randomKeyRobustness #symmetricCryptography
- Generate two keys.