Search
1000 results for “technology_tales”
-
@Kellam⚙️Бур This may come as a surprise, but: Nomadic identity is not an abstract concept or a science-fiction idea for the Fediverse.
It is reality. It exists. Right now. In stable, daily-driver software that's federated with Mastodon. And it has been for over a decade.
I'm literally replying to you here from a nomadic channel that simultaneously exists on two servers.
Nomadic identity was invented by @Mike Macgirvin 🖥️ (formerly American software developer of about half a century who has been living in rural Australia for decades now) in 2011 and first implemented in 2012. Almost four years before Mastodon was first launched.
In 2010, he had invented the Facebook alternative Friendica, originally named Mistpark and based on his own DFRN protocol.
Over the months, he witnessed lots of privately operated public Friendica nodes shut down with or without an announcement and the users on these nodes lose everything. He added the possibility to export and import Friendica accounts. But that would only help if a permanent shutdown was announced. It did not protect you against shutdowns out of the blue.
There was only one solution to this problem. And that was for someone's identity to not be bound to one server, but to exist on multiple servers simultaneously. The whole thing with everything that's attached to it. Name, settings, connections, posts, files in the file storage etc. etc., everything.
So in 2011, Mike designed a whole new protocol named Zot around this brand-new idea of what he called "nomadic identity" back then already.
In 2012, Mike forked Friendica into something called Red, later the Red Matrix, and rebuilt the whole thing from the ground up against Zot. Red was the first nomadic social networking software in the world, almost four years before Mastodon.
In 2015, ten months before Mastodon was first released, the Red Matrix became Hubzilla, the Fediverse's ultimate Swiss army knife.
I am on Hubzilla myself. This channel of mine is constantly being mirrored between its main instance on https://hub.netzgemeinde.eu and its clone on https://hub.hubzilla.de. Anything that happens on the main instance is backed up on the clone. I can also log into the clone and use that, and whatever happens there is backed up on the main instance.
https://hub.netzgemeinde.eu could go down, temporarily, permanently, doesn't matter; I still have my channel, namely the clone. And I can declare the clone my new main instance.
Well, Mike didn't stop at Hubzilla and its original version of the Zot protocol. He wanted to refine it and advance it, but in ways that wouldn't be possible on daily-driver software.
Zot went through several upgrades: Zot6 in 2018 (backported to Hubzilla in 2020, along with OpenWebAuth magic single sign-on). Zot8 in 2020. Zot11 in 2021 which had become incompatible with Zot6 and therefore was renamed to Nomad. Today's Nomad would be Zot12.
Also, in order to advance and test Zot, Mike created a whole bunch of forks and forks of forks. Osada and Zap for Zot6 in 2018, followed by another short-lived Osada in 2019. A third Osada, Mistpark 2020 (a.k.a. Misty) and Redmatrix 2020 in 2020 for Zot8. Roadhouse for Zot11 Nomad in 2021. All Osadas, Zap, Misty, Redmatrix 2020 and Roadhouse were discontinued on New Year's Eve of 2022.
The most recent software based on Nomad is from October, 2021. It can be found in the streams repository. It is officially and intentionally nameless and brandless, it has next to nodeinfo code that could submit statistics, and it is intentionally released into the public domain. The community named it (streams) after the code repository.
I also have two (streams) channels, one of which is cloned so far.
The newest thing, and that's what the Friendica and Hubzilla veteran @Tim Schlotfeldt ⚓?️? referred to, is nomadic identity using nothing but ActivityPub, no longer relying on a special protocol.
This was not Mike Macgirvin's idea. This came from @silverpill, the creator and developer of the microblogging server application Mitra. He wanted to make Mitra nomadic, make it resilient against server shutdown. But he didn't want to port it to Nomad. He wanted to achieve it with nothing but ActivityPub.
So he hit up Mike. The two came to the conclusion: This is actually possible. And they began to work on it. Amongst the results were several FEPs coined by silverpill.
This time, Mike did not create another fork to develop nomadic identity via ActivityPub. He did it all on the nomadic branch of the streams repository while silverpill did his part on a special development branch of Mitra.
In mid-2024, after enough sparring between (streams) instances, between Mitra instances and between (streams) and Mitra, Mike was confident enough that his implementation of support of nomadic identity via ActivityPub was stable enough. He merged the nomadic branch into the dev branch which ended up being merged into the stable release branch in summer.
Now, at this point, (streams) didn't use ActivityPub for nomadic identity. It still used the Nomad protocol for everything first and foremost, including cloning. But it understood nomadic identity via ActivityPub as implemented on experimental Mitra.
However, while it worked under lab conditions, it blew up under real-life conditions. At this point, (streams) had to handle so many different identities that it confused them, and it couldn't federate with anything yet.
In mid-August, while trying to fix the problem, Mike eventually forked the streams repository into Forte. It got a name again, it got a brand identity again, it got its nodeinfo back, it was put under the MIT license again.
But most importantly: Any and all support for Nomad was ripped out, also to get rid of a whole number of IDs, namely those for Nomad-actually-Zot12 and for Hubzilla's Nomad-actually-Zot6. Forte only uses ActivityPub for everything. And so, Forte also had to fully rely on ActivityPub for nomadic identity, cloning and syncing.
For almost seven months, Forte was considered experimental and unstable. For most of the time, the only existing servers were Mike's.
But on March 12th, 2025, Mike Macgirvin released Forte 25.3.12, the first official stable release of Forte. This is what Tim wrote about. Because this actually made it into Fediverse-wide news.
Not because it's nomadic. Nomadic identity has been daily-driven for over a decade now.
But because it uses ActivityPub for nomadic identity. Which means that you can theoretically make any kinds of Fediverse software nomadic now, all without porting it to the Nomad protocol first.
For the future, Mike and silverpill envision a Fediverse in which one can clone between different server applications. A Fediverse in which one can have one and the same identity cloned across multiple servers of Mastodon, Pixelfed, PeerTube, Mitra, Forte, Mobilizon, Lemmy, BookWyrm etc., all with the same name, all with the same content and settings (as far as the software allows; you will certainly not be able to clone your PeerTube videos to Mastodon and Lemmy).
Even if you don't intend to clone, it will make moving instances and even moving from one software to another dramatically easier.
If you're concerned about your privacy, let me tell you this:
Hubzilla's privacy, security and permissions system is unparalleled in the Fediverse. Except for that on (streams) and Forte which is another notch better.
I can define who can see my profile (my default, public profile on Hubzilla where each channel can have multiple profiles).
I can define who can see my stream and my posts when looking at my channel.
I can define who can see my connections (Hubzilla, (streams) and Forte don't distinguish between follower and followed; they aren't Twitter clones).
I can define who can look into my file space (individual permission settings per folder and per file notwithstanding).
I can define who can see my webpages on Hubzilla (if I have any).
I can define who can see my wikis on Hubzilla (no shit, I've got wikis on my Hubzilla channel).
On Hubzilla, I can define individually for any of these whether it's- everyone on the Internet
- everyone with a recognisable Fediverse account
- everyone on Hubzilla (maybe also on (streams); anyone using ActivityPub is definitely excluded here)
- everyone on the same server as myself (AFAIK, only main instances of channels count here, clones don't)
- unapproved (= followers) as well as approved (= mutual) connections
- confirmed connections
- those of my confirmed connections whom I explicitly grant that permission by contact role
- only myself
There's a whole bunch more permissions than these. And they all have seven or eight permission levels (depending on whether the general non-Fediverse public can be given permission).
On (streams) and Forte, I can define whether things are allowed for- everyone on the Internet (where applicable)
- everyone with a recognisable Fediverse account
- all my approved connections
- only me myself plus those whom I explicitly grant that permission in the connection settings
Yes, connection settings. Hubzilla, (streams) and Forte give you various ways of configuring individual connections, much unlike Mastodon. This includes what any individual connection is allowed to do.
Hubzilla uses so-called "contact roles" for that, presets with a whopping 17 permissions to grant or deny for any one individual connection. That is, what the channel generally allows, a contact role can't forbid.
(streams) and Forte still have 15 permissions per contact, but they lack some features which Hubzilla has permissions for. These permissions can be set individually for each connection, or you can define permission roles that cover all 15 permissions to make things easier.
Okay, how about posting in public vs in private? And when I say "private", I mean "private". It's "private messages" on Hubzilla, (streams) and Forte, not "direct messages".
Hubzilla, (streams) and Forte let you post- in public
- only to yourself
- only to your connections ((streams) and Forte only; Hubzilla requires a privacy group with all your connections in it for this)
- to all members of one specific privacy group (Hubzilla)/access list ((streams), Forte); that's like being able to only post to those on one specific list on Mastodon
- to everyone to whom one specific non-default profile is assigned (Hubzilla only)
- to a specific group/forum (I'll get back to that later)
- to a custom one-by-one selection of connections of yours
Now, let's assume I have a privacy group with Alice, Bob and Carol in it. I send a new post to only this privacy group. This means:- Only Alice, Bob and Carol can see the post and the conversation.
- Alice can reply to me, Bob and Carol.
- Bob can reply to me, Alice and Carol.
- Carol can reply to me, Alice and Bob.
- Nobody else can see the post. Not even by searching for it. Not by hashtag either. Not at all.
- Nobody else can see any of the comments.
- Nobody else can comment.
If one of them was on Mastodon, they'd see my post as a DM, by the way, and they could only reply to me. But that's Mastodon's limitation because it understands neither threaded conversations nor permissions.
Or how about reply control? This is something that many Mastodon users have been craving for quite a while now. Hubzilla, (streams) and Forte have them. Right now. And they work. They have since 2012.
Hubzilla optionally lets me disallow comments on either of my posts. Users on Hubzilla, (streams) and Forte won't even be able to comment; they won't have the UI elements to do so. Everyone else is able to comment locally. But that comment will never end up on my channel. It will never officially be added to the conversation. And at least users on Friendica, Hubzilla, (streams) and Forte will never fetch that comment from my channel as part of the conversation, i.e. never at all.
(streams) and Forte can go even further with all available options. They can disallow comments like Hubzilla. But in addition, they can allow only the members of one particular access list to comment, regardless of who can see the post/the conversation. On top of that, comments can be closed at a pre-defined point in the future. And then you even have a channel-wide setting for how long people can comment on your posts.
Oh, and there's even a setting for who is generally permitted to comment on your posts. And you can additionally allow specific connections of yours to comment on your posts.
Lastly, I've already mentioned groups/forums. Like, you know, Web forums or Facebook groups or subreddits or whatever. Like Guppe Groups on a mountain of coke and with moderation and permission control and optionally private.
Hubzilla has them, and it has inherited them from Friendica. (streams) has them. Forte has them. They're basically channels like social networking channels, but with some extra features. This includes that everything that's send to a group/forum as what amounts to a PM is automatically forwarded to all other members.
On Hubzilla, a forum can be gradually made private by denying permission to see certain elements to everyone but its own members (= connections): the profile, the members, what's going on in it. Depending on what you want or do not want people to see.
On (streams) and Forte, you have four types of forums:- public, and members can upload images and other files to the forum channel
- public, but members cannot upload images and other files to the forum channel
- like above, but additionally, posts and comments from new members must be manually approved by the admin(s) until their connections are configured to make them full members
- private, non-members can't see the profile, non-members can't see the connections, non-members can't see what's going on in it, but members can upload images and other files to the forum channel
In addition, on all three, a group/forum channel can choose to hide itself from directories. This is always an extra option that's independent from public/private.
What we have here is the most secure and most private Fediverse software of all.
And, once again, at its core, this is technology from 2012. It pre-dates Mastodon by almost four years.
Finally, if you want to know how Hubzilla and (streams) compare to Mastodon: I have made a number of tables that compare Mastodon, Friendica, Hubzilla and (streams).
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #Mastodon #Mitra #Friendica #Hubzilla #Streams #(streams) #Forte #ActivityPub #Zot #Zot6 #Zot8 #Nomad #NomadicIdentity #Security #FediverseSecurity #Privacy #FediversePrivacy #Permissions -
Samsung releases the changelogs of One UI 8.5 for the Galaxy S25!
One UI 8.5 brings a new ambient design language that is designed to provide you with more modern appearance, as well as new AI features and enhancements to existing features. This version of One UI was found in the Galaxy S26, the Galaxy A57, and the Galaxy A37 before the rollout started.
As One UI 8.5 for the latest flagship devices becomes available in Korea, here is a list of changes that are made (source):
One UI 8.5
Visual design
Fresh new look
One UI seamlessly integrates into your daily routine by combining immersive visuals with meaningful personalization for a more refined and sophisticated design. Transparent blur effects add depth and make content easier to navigate, while floating elements react organically to your workflow for a more focused experience. Through familiar and intuitive data visualization, One UI delivers a design that feels both personal and relatable, helping you focus on what matters.Galaxy AI
Screen calls before answering
See who’s calling so you can decide if you want to talk. You can let a call assistant answer for you and ask the caller who they are why they’re calling.Edit images with text prompts
Image editing has never been easier. Just describe how you want your image to change. You can change the color of someone’s clothes, add something to an empty table, or anything else you can think of.Add items from one image to another
Photo assist makes it easy to combine elements from different images. Take an object from one picture and add it to a different picture. Galaxy AI smooths it out and makes it look natural.Add style to any photo
You can now apply fun styles to any photo with Photo assist, not just pictures of people or pets. In thumbnail view, you can touch and hold to reorder your styles and put your favorites near the front.Continuous image generation
Keep creating without stopping. Photo Assist now lets you generate AI images using different features from the results screen without saving each iteration. When you’re done, you can review all your creations in your history and pick your favorites.Create images with Creative Studio
Creative Studio is now available on the Apps screen for easier access. You can use creative studio to create custom wallpapers, unique stickers, and personalized profile images that you can use throughout on your phone.Now brief on the Lock screen
Get more personalized suggestions on your Lock screen. Now brief will show useful information based on your context.Enhanced AI select
Start AI select instantly by touching and holding the edge handle. Missed something in a video? Use the Rewind button to go back and select exactly what you need.Auto language detection in Interpreter
Keep the conversation flowing. After you choose which languages to translate, Interpreter will detect when each one is being spoken so you don’t need to press the Microphone button each time someone talks.Bixby
Smarter device control
Talk to Bixby in your own words. Bixby is now better at finding the setting or feature you need, even if you don’t use exact commands or feature names. Just say what you need and let Bixby do the rest.Ask anything, anytime
Whether you need a quick answer or detailed information, just ask Bixby for an instant response. There’s no need to spend time on multiple searches or switching between apps.Conversation history
Looking back at past conversations with Bixby is easier than ever. You can now access your conversation history from the side panel in the Bixby app.Camera
Pro-grade document scans
Scanning documents is now faster and more powerful. A scan button will appear automatically whenever you point your camera at a document. You can easily capture multiple pages into a single PDF file, while the new Remove tool automatically cleans up distracting fingers, folded corners, and unwanted moire for a perfect finish.Auto motion photos
When set to Auto, your camera will only record a motion photo if it detects movement in the scene. Otherwise, it’s saved as a still image to save space.Capture both sides of the story
Now you can record yourself and the action in front of you. Just tap the dual recording icon in the Video mode quick controls to start filming with the front and rear cameras at the same time.Log video color previews in real time
Take the guesswork out of filming in Log. You can now apply a cinematic LUT preview while you record, letting you see exactly how your final color-corrected video will look before you even start editing. LUT previews are also available in Gallery and Studio.New portrait filters
Three new filters are available to add vibrant film-like effects to your pictures.Home and Lock screens
Automatic Lock screen layout
Wallpapers with pictures of people or pets now fit perfectly every time. When you choose a photo for your Lock screen, the photo will automatically be adjusted to best fit your clock and widget layout.New downloadable wallpapers
Discover new wallpapers featuring interactive elements. Wallpapers are downloadable so they don’t use up your storage space when not in use.Add weather effects to wallpapers
Bring your wallpaper to life with the current weather conditions. When you choose your wallpaper, you can add weather effects directly from the preview screen.More customizable clock fonts
Personalize your Lock screen clock. You can now adjust the thickness of more font styles to match your preferred look.Weather
Enhanced weather widget
Quickly check upcoming precipitation in the Weather widget on your Home screen. The widget now shows a graph if precipitation is expected in the next few hours.Pollen index
Check how much pollen is in the air to help manage your allergies. You can check pollen levels for trees, grass, and ragweed.Communication
Direct voicemail
Can’t answer right now? Let callers record a voice message directly on your phone that you can listen to later. The message will appear on your screen as it’s being recorded so you can answer at any time.Decline calls with personalized messages
When a call is ringing, new quick decline messages will appear based on your activity. You can tell callers that you can’t answer because you’re driving or exercising even without typing.Clock
Weather alarm backgrounds
Wake up to an alarm that gets you ready for the day’s weather. Your alarm screen can now show the current weather conditions as a background when it rings.Time zone converter
Compare time zones at a glance. The new slider in the Clock app makes it easy to check the time difference between places around the world.Connectivity
Storage Share
Access your files anywhere. Files from your other Samsung phones, tablets, and PCs are available in the My Files app on your phone. You can also access your phone’s files on other Samsung devices, even your TV.Quickly connect to Smart View devices
Connect to your favorite display faster. You can now add a shortcut on your Home screen to instantly mirror your phone’s screen to a TV or other display device.Enhanced Auracast features
It’s easier than ever to listen to and broadcast sound with Auracast. Options for both broadcasting and listening are now located in the Audio broadcast menu in Settings.Voice broadcasts
Broadcast your voice to people around you with Auracast. In addition to media sound, you can now broadcast your voice using your phone’s built-in microphone.Quick Share
Share with Apple devices
Share with even more devices than before. You can now use Quick Share to seamlessly share photos, videos, and other files with iPhones, iPads, Macs, and other devices that support AirDrop.Avoid unwanted sharing requests
You can now set Quick Share to only receive files from other devices signed in to your Samsung account or Google account.Photo sharing suggestions
Share photos with the right people faster. When you share pictures that include friends or family, Quick Share can recognize who’s in them and suggest sharing directly with those people.Samsung Health
Enhanced weekly reports
See a fuller picture of your health each week. Weekly reports now include data from your medication tracker and mindfulness sessions.Upgraded sharing experience
Share your workouts your way. Mix and match your exercise stats with photos from your workout to create the perfect social media post.Start meditations from your watch
Find calm right from your wrist. You can now start favorite or recommended meditations directly on your Galaxy Watch without picking up your phone.Antioxidant measurements from your watch
Check your antioxidant levels anytime. Measure directly from your Galaxy Watch, even if it’s not connected to your phone. Works with Galaxy Watch8 and Galaxy Watch Ultra.Battery and power
Revamped battery info
See your battery use more clearly. The redesigned Battery settings screen makes it easier to check remaining time, charging status, and daily usage over the past week.Improved Power saving
Use Power saving to make your battery last longer without charging. Choose Standard for moderate savings and customizable limits, or choose Maximum to turn off all non-essential features and make your battery last as long as possible.Security and privacy
Privacy alerts
Stay informed about your privacy. You’ll now get alerts when an app’s permissions could put your personal data at risk along with suggestions for what you can do about it.Theft protection
Keep your phone and data safe in case it is lost or stolen. Turn on Failed authentication lock to automatically lock the screen in case there are too many failed attempts to verify your identity using your fingerprints, PIN, pattern, or password. Identity check also protects even more settings than before.Turn off Auto blocker temporarily
If you need to temporarily disable Auto blocker’s security protection, a new option lets you turn it on automatically 30 minutes later so you don’t forget.Check the security status of your devices
Keep all your devices protected. Knox Matrix now shows when any of the supported devices signed in to your Samsung account need a software update for the latest security protections.Accessibility
Easily control Bluetooth hearing aids
Access settings for your Bluetooth hearing aids directly from the Accessibility shortcut. A pop-up will appear that lets you change your hearing program, turn Ambient sound on or off, and more.Control magnification with mouse or keyboard
Keep what you need magnified in view with these new options. You can make the magnified area follow the cursor as you type or move when you change focus using the keyboard. When using a mouse, you can make the magnified area shift as you move the pointer to the edge of the screen.Dwell action and Corner actions
The Auto action after pointer stops feature has been divided into 2 features. Dwell action lets you set custom actions when your mouse stops moving for a certain amount of time. Corner actions let you set a different action for each corner of the screen.Dim strobing in videos
A new setting lets you dim strobing effects in videos for more comfortable viewing.Even more improvements
More customizable quick panel
Arrange your quick settings just the way you like them. You can now add, remove, reorder, and reorganize controls in the quick panel.Customize Calendar countdown widgets
Make your countdown widgets look just the way you like. Use Creative studio to generate a background image, choose an image from Gallery, or go with a solid color.Early alerts for reminders
Get alerts before reminders are due to make sure you don’t forget important tasks. You can choose how far in advance to get an alert for each reminder.Insert tables in Samsung Notes
Organize information in your notes with tables. You can adjust column widths, colors, and border designs while the auto calculation feature helps you stay productive and save time.Redesigned New tab page
The page that appears when you open a new tab in Samsung Browser has been redesigned to help you quickly access the websites and features you need the most. The New tab page now shows the current security status as well as open tabs from Samsung Browser on other devices.Partial screen recording
Include only what you need in your screen recordings. You can now select only the part of the screen that you want to record.Calculator nudges
Save time on calculations. Numbers and formulas copied to your clipboard will be suggested when you open Calculator so you can enter them with a quick tap.Keep window sizes in DeX
DeX now remembers your app window sizes and positions. When you open an app again, it appears just as you left it.To update your Galaxy to One UI 8.5, follow these steps:
- Open Settings, and navigate to System Update
- Tap on Check for Updates
- Press Download and Install
- Wait until the download and the installation is complete, then press Reboot Now
- Wait until the phone reboots successfully
Make sure that you’re connected to a stable and fast Wi-Fi network to avoid high mobile data fees. Never interrupt the update in any way, or problems could occur later. Make sure that you apply all available updates once you update your phone for maximum stability. An update might take 10 to 20 minutes to apply, depending on the device and the type of the update being applied.
If you’re seeing “Your software is up to date,” this means that the update hasn’t reached your device yet. Keep checking for updates, as Samsung rolls it out gradually to all devices from country to country, usually starting from Korea.
#Android #Android16 #news #oneUi #OneUI85 #Samsung #smartphone #Tech #Technology #update -
@Decenta Lyzed
Meanwhile #Friendica
& #Hubzilla have some groups too.
Friendica has groups. Hubzilla has groups called "forums". Both have had groups for longer than Mastodon has even been around.
(streams), a fork of a fork of three forks of a fork (of a fork?) of Hubzilla created and still maintained by Friendica's and Hubzilla's own creator, has groups.
Forte, a fork of (streams) by the same developer again, has groups.
All four are in the Fediverse. All four are federated with Mastodon (Hubzilla optionally and off by default, (streams) optionally and on by default, Friendica and Forte always). By the way, this comment comes from Hubzilla.
For self-hosters: All four are written in PHP, and they require no more than a LAMP stack. But if you don't know them, e.g. if all you know in the Fediverse is Mastodon, I recommend you try them out on a public server before setting up your own one. They're all very different from Mastodon in a lot of ways. Don't just expect Mastodon with groups because that's far from what they are.
Here are several tables that compare the features of Mastodon, Friendica, Hubzilla, (streams) and Forte.How they work
A Friendica group is an account with special settings. Likewise, a Hubzilla forum or a (streams) or Forte group is a channel with special settings.
Speaking in Mastodon terms, what they do is take incoming posts and automatically boost them to all their followers.
An exception exists on Friendica, Hubzilla, (streams) and Forte themselves: If you're there, you must send a DM to the forum/group. Public posts to a group/forum are not forwarded, only DMs are. This ensures that a group/forum doesn't forward any and all posts that happen to mention it.FLOSS
Friendica is open-source (https://github.com/friendica) and under the GNU Affero GPL v3.
Hubzilla is open-source (https://framagit.org/hubzilla/core) and under the MIT license.
(streams) is open-source (https://codeberg.org/streams/streams) and in the public domain.
Forte is open-source (https://codeberg.org/fortified/forte) and under the MIT license.Character limits
The character limit on Friendica and Hubzilla is over 16.7 million.
The character limit on (streams) and Forte is over 24 million.
Nobody will run out of characters anytime soon, no matter from where they post. However, this also means that neither of the four has Mastodon's character-limit-induced culture of brevity.Moderation
Friendica groups can be co-moderated/co-administrated by users on the same server as the group.
Hubzilla forums, (streams) groups and Forte groups can be co-moderated/co-administrated by anyone on Hubzilla, (streams) or Forte.
Two of the public (streams) and Forte group types allow for new content to be moderated: Any new post or comment must be manually approved by the moderators. In both cases, this is mainly for new members. Trustworthy members can be permitted to post or comment immediately.Privacy and security
Friendica groups, Hubzilla forums, (streams) groups and Forte groups can optionally be hidden from directories and made "secret".
Friendica groups can optionally be set to private, i.e. non-members can't see the group profile, the member list or what's going on in the group.
On Hubzilla, (streams) and Forte, the profile, the member list and the stream can be reduced in visibility separately from each other. You can make the group profile public, and at the same time, you can only permit group members to see the member list and/or the stream.
Hubzilla offers eight levels of permission for seeing the forum's main profile, additional profiles that can only be seen by members/certain members, eight levels of permission for seeing the forum's member list and eight levels of permission for seeing the forum stream. One level of permission depends on individual permissions for certain members granted by contact role.
(streams) and Forte offer four group types, one of which is private, four levels of permission for seeing the group's member list and four levels of permission for seeing the group stream. The non-public levels can be overridden by granting individual permissions to certain members.
(streams) and Forte also offer the same four levels of permission plus overrides for searching the group stream.
Note: It may not be possible to join a private group/forum with an account on Mastodon or anything else that isn't one of these four. Public groups/forums can be joined by anyone (unless they're blocked, of course).Resilience
Hubzilla, (streams) and Forte offer nomadic identity, i.e. the forum/group channel can exist simultaneously on multiple servers as live, hot, bidirectional backups of each other. If one server goes down, the forum/group lives on on the other server(s).
Something like this has been announced by Bluesky as a new and revolutionary technology. Bluesky has yet to deliver. Hubzilla has had this technology since 2012.
Downside: Server software that doesn't understand nomadic identity, i.e. everything except Hubzilla, (streams), Forte and at least the development branch of Mitra, sees the instances of a cloned, nomadic channel as multiple individual, independent accounts.
CC: @Fedi.Tips
CC so that everyone else in this thread will read this, even if they're on Mastodon (I wouldn't have to do this if the whole Fediverse supported threaded conversations, and everyone got this automatically anyway, but I don't want to post this several times over because someone on Mastodon hasn't received it due to Mastodon's intentional, by-design limitations): @Georgiana Brummell @Magical Cat @Michael Gisiger :mastodon: @LucileDT @Glowing Cat of the Nuclear Wastelands @AlsoPaisleyCat @Zeroday Podcast (stefan) @mtjm @amy
Also:{re-toot & boost my post if you find it helpful}
Twitter = Retweet
Mastodon = Boost
Twitter = Like
Mastodon = Fave
#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #Fediverse #Friendica #Hubzilla #Streams #(streams) #Forte #Groups #FediGroups #FediverseGroups -
🤖 #Google представила нову функцію #DataTables у своєму #ШІ-сервісі для досліджень #NotebookLM. Інструмент дозволяє збирати та узагальнювати інформацію з кількох джерел у вигляді таблиць, які можна експортувати до #GoogleSheets.
🔗 https://blog.google/technology/google-labs/notebooklm-data-tables/ -
Skill Faire anarchiste
CÉDA, Sunday, May 17 at 10:00 AM EDT
On Sunday, May 17, CÉDA transforms into the sprawling, chaotic Skill Faire—a kind of anarchist science fair where trifold displays are encouraged and “please touch” is the norm. Tables and stations will showcase hands-on skills and tools, inviting people to try things out, ask questions, and leave with practical knowledge. In addition to tabling, there will be rooms reserved for longer, more in-depth skillshares that require time, focus, or specialized equipment. We welcome contributions in a wide range of topics, including:
Street medicine
Autonomous technology
Self-defense
Navigating conflict
Resolving conflict
Song, dance & art
Repairing your stuff
Altered states of consciousness
Construction & deconstruction
“Research”
-
🔍 #LlamaOCR introduces advanced document processing using #Llama32Vision technology
• 📦 Simple #npm package requiring only 5 lines of code to implement advanced #OCR capabilities
• 🔄 Converts complex documents including receipts and tables into structured markdown format
• 🛠️ Built on #TogetherAI API with three model options: free tier, 11B, and 90B parameters for varying performance needs
• 🗺️ Development roadmap includes support for single/multi-page #PDF processing and #JSON output formats
#opensource #artificialintelligence #documentprocessing
Try the demo: https://llamaocr.com
Source: https://github.com/Nutlope/llama-ocr -
I had the great pleasure of being invited to the Open University of the Netherlands and, later in the day, to EdLab, Maastricht University a few weeks ago, giving a slightly different talk in each place based on some of the main themes in my most recent book, How Education Works. Although I adapted my slides a little for each audience, with different titles and a few different slides adjusted to the contexts, I could probably have used either presentation interchangeably. In fact, I could as easily have used the slides from my SITE keynote on which both were quite closely based (which is why I am not sharing them here). As well as most of the same slides, I used some of the same words, many of the same examples, and several of the same anecdotes. For the most part, this was essentially the same presentation given twice. Except, of course, it really, really wasn’t. In fact, the two events could barely have been more different, and what everyone (including me) learned was significantly different in each session.
This is highly self-referential. One of the big points of the book is that it only ever makes sense to consider the entire orchestration, including the roles that learners play in making sense of it all the many components of the assembly, designed for the purpose and otherwise. The slides, structure, and content did provide the theme and a certain amount of hardness, but what we (collectively) did with them led to two very different learning experiences. They shared some components and purposes, just as a car, a truck, and a bicycle share some of the same components and purposes, but the assemblies and orchestrations were quite different, leading to very different outcomes. Some of the variation was planned in advance, including an hour of conversation at the end of each presentation and a structure that encouraged dialogue at various points along the way: these were as much workshops as presentations. However, much of the variance occurred not due to any planning but because of the locations themselves. One of the rooms was a well-appointed conventional lecture theatre, the other an airy space with grouped tables, and with huge windows looking out on a busy and attractive campus. In the lecture theatre I essentially gave a lecture: the interactive parts were very much staged, and I had to devise ways to make them work. In the airy room, I had a conversation and had to devise ways to maintain some structure to the process, that was delightfully disrupted by the occasional passing road train and the very tangible lives of others going on outside, as well as an innately more intimate and conversational atmosphere enabled (not entailed) by the layout. Other parts of the context mattered too: the time of day, the temperature, the different needs and interests of the audience, the fact that one occurred in the midst of planning for a major annual event, and so on. All of this had a big effect on how I and others behaved, and on what and how people learned. From one perspective, in both talks, I was sculpting the available affordances and constraints to achieve my intended ends but, from another equally valid point of view, I was being sculpted by them. The creators and maintainers of the rooms and I were teaching partners, coparticipants in the learning process. Pedagogically, and despite the various things I did to assemble the missing parts in each, they were significantly different learning technologies.
The complexity of distance teaching
Train journeys are great contexts for uninterrupted reflection (trains teach too) so, sitting on the train on my journey back the next day, I began to reflect on what all of this means for my usual teaching practice, and made some notes on which this post is based (notebooks teach, too). I am a distance educator by trade and, as a rule, with exceptions for work-based learning, practicums, co-ops, placements, and a few other limited contexts, distance educators rarely even acknowledge that students occupy a physical space, let alone do we adapt to it. We might sometimes encourage students to use things in their environments as part of a learning activity, but we rarely change our teaching on the fly as a result of the differences between those environments. As I have previously observed, the problem is exacerbated by the illusion that online systems are environments (in the sense of being providers of the context in which we learn) and that we believe we can observe what happens in them. They are not, and we cannot. They are parts of the learners’ own environments, and all we can (ethically) observe are interactions with our designed systems, not the behaviour of the learners within the spaces that they occupy. It is as hard for students to understand our context as it is for us to understand theirs, and that matters too. It makes it trickier to model ways of thinking and approaches to problem solving, for example, if the teacher occupies a different context.
This matters little for some of the harder elements of the teaching process. Information provision, resource design, planning, and at least some forms of assessment and feedback are at least as easy to do at a distance as not. We can certainly do those and make a point of doing them well, thereby providing a little counterbalance. However, facilitation, role modelling, guidance, supporting motivation, fostering networks, monitoring of learning, responsive adaptation, and many other significant teaching roles are more complex to perform because of how little is known about learning activities within an environment. As Peter Goodyear has put it, matter matters. The more that the designated teacher can understand that, the more effective they can be in helping learners to succeed.
Because we are not so able to adapt our teaching to the context, distance learning (more accurately, distance teaching) mostly works because students are the most important teachers, and the pedagogies they add to the raw materials we provide do most of the heavy lifting. Given some shared resources and guided interactions, they are the ones who perform most of the kinds of orchestration and assembly that I added to my two talks in the Netherlands; they are the ones who both adapt and adapt to their spaces for learning. Those better able to do this in the first place tend to do better in the long run, regardless of subject interest or innate ability. This is reflected in the results. In my faculty and on average, more than 95% of our graduate students – who have already proven themselves to be successful learners and so are better able to teach themselves – succeed on any given course, in the sense of reaching the end and achieving a passing grade. 70% of our undergraduates, on the other hand, are the first in their family to take a degree. Many have taken years or even decades out of formal education, and many had poor experiences in school. On average, therefore, they typically have fewer skills in teaching themselves in an academic context (which is a big thing to learn about in and of itself) and we are not able to adapt our teaching to what we cannot perceive, so we are of little assistance either. Without the shared physical context, we can only guess and anticipate when and where they might be learning, and we seldom have the faintest idea how it occurs, save through sparse digital signals that they leave in discussion forums or submitted assignments, or coarse statistics based on web page views. In a few undergraduate core courses within my faculty it is therefore no surprise that the success rates are less than 30%, and (on average) only about half of all our students are successful, with rates that improve dramatically in more senior level courses. The vast majority of those who get to the end pass. Most who don’t succeed drop out. It doesn’t take many core courses with success rates of 30% to eliminate nearly 95% of students by the end of a program.
Teaching with a context
We can better deal with this if we let go of the illusion that we can be in control and, at the same time, find better ways to stay close: to make the learning process including the environment in which it occurs, as visible as possible. It is emphatically not about capturing digital traces and using analytics to reveal patterns. Though such techniques can have a place in helping to build a picture of how learners are responding to our deliberate acts of teaching, they are not even close to a solution for understanding learners in context. Most learning analytics and adaptive systems are McNamara Machines, blind to most of what matters. There’s a huge risk that we start by measuring the easily measurable then wind up not just ignoring but implicitly denying that the things we cannot measure are important. Yes, it might help us to help students who are going to get to the end anyway to get better grades, but it tells us very little about (for instance) how they are learning, what obstacles they face, or how we could help them orchestrate their learning in the contexts in which they live. Could generative AI help with that? I think it might. In conversation, an AI agent could ask leading questions, could recommend things to do with the space, could aggregate and report back on how and where students seem to be learning. Unlike traditional adaptive systems, generative AI can play an active discovery role and make broader connections that have not been scripted. However, this is not and should not be a substitute for an actual teacher: rather, it should mediate between humans, amplifying and feeding back, not guiding or informing.
For the most part, though, I think the trick is to use pedagogical designs that are made to support flexibility, that encourage learners to connect with the spaces live and people they share them with, that support them in understanding the impact of the environments they are in, and, as much as possible, to incorporate conduits that make it likely that participants will share information about their contexts and what they are doing in them, such as through reflective learning diaries, shared videos or audio, or introductory discussions intended to elicit that information. A good trick that I’ve used in the past, for example, is to ask students to send virtual postcards showing where they are and what they have been doing (nowadays a microblog post might serve a similar role). Similarly, it can be useful to start discussions that seek ideas about how to configure time and space for learning, sharing problems and solutions from the students themselves. Modelling behaviours can help: in my own communications, I try to reveal things about where I am and what I have been doing that provide some context and background story, especially when it relates to how I am changing as a result of our shared endeavours. Building social interaction opportunities into every inhabited virtual space would help a lot, making it more likely that students will share more of what they are doing and increasing awareness of both the presence and the non-presence (the difference in context) of others. Learning management systems are almost universally utter rubbish for that, typically relegating interactions to controlled areas of course sites and encouraging instrumental and ephemeral discussions that largely ignore context. We need more, more pervasively, and we need better.
None of this will replicate the rich, shared environments of in-person learning, and that is not the point. This is about acknowledging the differences in online and distance learning and building different orchestrations around them. On the whole, the independence of distance students is an extremely good thing, with great motivational benefits, not to mention convenience, much lower environmental harm, exploitable diversity, and many other valuable features that are hard to reproduce in person. When it works, it works very well. We just need to make it work better for those for whom that is not enough. To do that, we need to understand the whole assembly, not just the pieces we provide.
https://jondron.ca/on-the-importance-of-place/
#analytics #architecture #context #distanceEducation #distanceLearning #environment #learningToLearn #lecture #lms #motivation #onlineEducation #onlineLearning #orchestration #place #technology #visibleLearning
-
Of all the things from my past, the technology that I just cannot believe has completely disappeared is #ColdType pre-press technology – phototypesetting, pasteup, working on light tables to produce mechanicals for anything that had to be printed. These methods are just gone (while letterpress lives on). I made my living at #pasteup for several years. This is from a #journalism institute at #SyracuseUniversity in 1978. #photography #OldPhotographs #Syracuse #OffsetPrinting #FilmPhotography
-
Resilience is community and trust, this resilience grows by connecting the actions of today to the possibilities of tomorrow, even when that future is unknowable. It’s rooted in community, and community thrives on mutual trust. Trust isn’t about keeping a ledger; it’s about giving freely without expectation. Money is not the foundation of resilience. Across the world, billions live resilient lives by supporting each other, because if they don’t, they all go under. From our privileged view, we often forget that resilience is nurtured in these commons.
We need to think about this: The idea of dual power isn’t new. It goes back to revolutionary moments when people realized the need to build alternatives to existing oppressive structures rather than only confronting them head-on. In the current political climate, where the failures of state and capitalist control are glaring, we need to revisit and reframe this idea of “dual power”. This isn’t a utopian dream or a naïve belief that we can merely build around the edges while the world burns. It’s about creating practical, grounded alternatives that directly challenge the existing system by living outside of it and dismantling it from the inside.
The current mess, look around. We are surrounded by a mess of our own making. The relentless march of #neoliberalism has commodified every aspect of our lives, and the #dotcons have taken over our social spaces, transforming genuine human interaction into data points for corporate profit and control. The state, meant to serve the people, is a tool of the greedy and nasty, maintaining control through fear, surveillance, and repression. It doesn’t take much to see that the paths we are currently on are leading to #climatechaos, widespread inequality, social and ecological breakdown.
But here’s the problem: most people still think we have choices within this mess. They talk about reforming the system, fixing capitalism, or making dotcons tech more ethical while continuing to operate on the same lost paths. This is delusion, a comfortable delusion for some, but a delusion nonetheless.
On the #DIY path, dual power is about creating parallel paths that coexist with the current ones but serve entirely different functions. Instead of asking for scraps from the masters’ table, we build our own tables, with food that nourishes everyone. It’s about constructing alternative social, economic, and political structures that are directly in opposition to the current hierarchies and power dynamics.
It’s not just about building alternative structures, though. It’s more important for actively delegitimizing and dismantling the existing power structures of capitalism and the state. This involves #directaction, solidarity, and collective organizing to challenge and change state and capitalist control in all its forms. It’s about a two-fold strategy: building the new while composting the old.
Why dual power matters, for too long, the left and radical movements have been stuck in reactionary paths, fighting battles on terrain chosen by the state and capital. We need to change this by recreating a new path, a space where we shape the traditions and myths that shape us. This is not just some theoretical exercise; it’s already happening in many parts of the world.
We see it in the #fediverse, on #mastodon, #bluesky and #noster networks, in grassroots mutual aid networks springing up during the current crises when the state and corporate structures fail. We see it in community run food cooperatives, decentralized digital spaces, and local assemblies where decisions are made collectively, rather than by a few in power. This is not an abstract idea, it’s lived practice, a shift from fighting against the system to creating something new and more humane.
Building dual power in a digital age, the #openweb and federated networks offer a glimpse of what dual power can look like. Unlike the #dotcons that feed on greed and manipulation, the openweb is rooted in principles that serve the community, #4opens, transparency, open collaboration, and autonomy. But even here, we often fall into the trap of merely copying the structures we’re trying to replace, creating the same mess under a different banner. The next step needs to be truly native to the 4opens path, transparent, open, and accountable, rejecting the commodification that the dotcons have normalized.
But digital spaces alone won’t save us. They are tools, important ones, no doubt, but we need a broader focus. We need to create real-world spaces of resistance and creation. Think community gardens that also serve as meeting points for local decision-making. Think of decentralized energy cooperatives that break free from corporate control. Think of neighbourhood assemblies that replace the hollow, bureaucratic local governments that most people have lost faith in. This is dual power in practice.
The roadblocks, the #Geekproblem and #Fasherista paths, let’s not romanticize this process. We need to acknowledge the challenges within our movements, the #geekproblem and the #fashernista paths that unconsciously block the change we need. The geekproblem is the obsession with technical solutions over social and political ones, while the fashernista path focuses on trendy but superficial activism that serves as more of a social club, careerism, than a serious challenge to power. Both paths have their place, but they should not dominate our paths. We need to keep our focus on the bigger picture.
Moving beyond the noise, to those who say, “Now is not the time,” I ask, “When will it be?” The crisis is here. We are all worshiping the #deathcult, masking 40 years of #neoliberal ideology, pretending we have choices that simply don’t exist. Now is precisely the time to dig in, get our hands dirty, and start composting this mess we’ve been dragged into. The work ahead isn’t easy, and there will be mistakes, missteps, and mess-ups along the way. But that’s okay. Composting is messy work, and so is building a more open and sustainable world.
If you’re waiting for someone to tell you what to do, you’ve already missed the point. Dual power isn’t a blueprint; it’s a living practice. It’s a call to start building the new and composting the old, right now, where you are. Lift your head, look at the mess, and start digging. Together, we can build something better than the scraps we’ve been given. Join us on this humanistic adventure in social technology and direct action. The #openweb, the #commons, and the real-world spaces we build are where the future lies. Let’s make it happen #OMN
#4opens #bluesky #climatechaos #commons #deathcult #directaction #diy #dotcons #Fasherista #fashernista #fediverse #geekproblem #Mastodon #neoliberal #neoliberalism #noster #OMN #openweb
-
The ultimate rebuild of an ancient Yaesu FT-817.
I think it was a couple of years ago now I ordered a QRP Labs QMX transceiver. It quickly, but temporarily, became my favorite radio for portable field operations. I have written before about why I believe the QMX is a mighty fine piece of miniaturized technology but is less suitable for the rigors of being operated in the kind of field operating environment to which I expose my radios. My QMX is the low-band version and I also miss the opportunity to explore the higher bands when propagation conditions permit.
What’s a poor Ham to do?
I could buy another QMX, but order the high band version this time. It would be a very modest investment, but would still require ruggedizing. Another downside is the long, long wait time betwixt ordering and receiving the tiny parcel from Turkey. I could also order a QMX+ which is a fine all HF band radio, but then what to do with the QMX low band? There is another solution.
The Paranoid Android
I recall a quote from the book “The Hitchhikers Guide to the Galaxy” by Douglas Adams in which the perenially depressed robot “Marvin the Paranoid Android” moans: “The first ten million years were the worst.” When I look at the front panel of my ancient Yaesu FT-817 non-ND version it kinda has a Marvin look about it. It has spent almost a quarter of a century waiting patiently in a drawer for the day when it might be called into action again. Many radios have come and gone during that time but – even though I had planned to sell it on many occasions – I still own it and it’s day to see the sunshine again has finally come.
Where are the features?
The non-ND version of the FT-817 is a barebones rig. I needed a CW memory keyer – it doesn’t have one. Activating a POTA park sometimes requires great patience and many, many CQs. My QMX at least has that covered. I also needed an audio filter. It used to be possible to buy a Collins mechanical filter but they are no longer made. My QMX also has that feature covered, but the FT-817 requires an external audio filter.
Failure is not an option
The FT-817 does have a higher level of ruggedness than the QMX. With a few extra precautionary measures it can be protected from the ingress of sand particles during a beach activation, or unexpected spray from waves on the shores of the Great Lakes. The QMX will not tolerate wide variations in DC supply voltage; the FT-817 has that covered. The QMX uses inexpensive but fragile PA transistors (mine have not succumbed to failure – yet). Well, the FT-817 also had fragile PA transistors in its early days and mine did indeed fail during a field deployment. The FT-817’s PA board is a small module that is easily replaced with the new upgraded module – as was mine.
Assembled rebuilt FT-817 portable operations rig. The battered, field protective canvas pouch on the right contains a Talentcell LiFePO4 battery. Right hand side view of the “helper modules” showing the input jack for connecting a cable from the headphone output of the FT-817. The switch allows the K4ICY AF filter to be bypassed for a barn door wide audio bandwidth. Left hand side view of the “helper modules” showing the switch allowing selection of 2-stage or 4-stage audio frequency filtering. To the right of the switch is the AF output jack for connecting headphones. The jack on the K3NG keyer connects to the “Key” jack on the FT-817. On the back of the AF filter module is the power switch controlling the internal
9-volt battery (now replaced by a buck converter) which supplies both modules. Internal view of the keyer module and the filter module. The 9 volt battery has now been replaced with a buck converter that converts the radio’s DC supply from 12.6 volts down to 9 volts to power the helper modules.I get by with a little help from my friends
The feature shortcomings of the FT-817 have been overcome with two “helper modules” assembled inside aluminum Hammond project enclosures. The front enclosure contains a K3NG Arduino nano based CW keyer and a very simple no-thrills set of 3D printed paddles. Well who really needs to spend $300 on a fancy set of paddles for a brief POTA exchange? These paddles get the job done FB. The same cannot be said about the fist that operates them!
The front panel controls are very simple. The paddles protrude through a cutout in the Hammond enclosure.
Beside the paddles is a knob. This knob is used to operate a rotary encoder inside. Clicking the knob operates the switch built into the rotary encoder and triggers the sending of a “CQ CQ POTA de VA3KOT VA3KOT k” stored message in the Arduino keyer.
Rotating the knob adjusts the speed of the CW over a wide range. I have found this to be a very useful feature. I usually send at 20wpm and receive responses that are slower and faster than my sending speed. With this prominent control front-and-center I can quickly adjust my sending speed to suit.
I built the K4ICY audio frequency filter module around a quad op-amp DIL chip. This is a very simple circuit that provides 2 or 4 stages of filtering to narrow the bandwidth of a received signal. Each stage contains identical components whose values are selected according the operator’s desired sidetone frequency. The whole module can be bypassed if required allowing an audio bandwidth wide enough to pass a crosstown bus sideways.
Both modules are rigidly secured to each other using two aluminum rails made from scrap material. I hoard scraps of metal, plastic and other materials – you just never know when you’re gonna need ’em.
The dimensions of the two modules provide an ample flat surface on which to mount the ancient, but revered, transceiver. I purchased some “peel & stick” Gorilla brand “Slipstick” gripper pads and applied four of them to the base of the FT-817. This is a genuinely useful product I recommend to any hambrewer. The radio has been secured to the top of the helper modules with two woodland zip ties made from thin cordage. These simple cord fasteners work just as well as plastic zip ties and can be easily undone for servicing the modules.
I purchased a box load of these Hammond enclosures at an auction many years ago. They have proved very useful. In another build, using the same enclosures configured in an identical manner, I was able to construct two battery modules each containing four 18650 Lithium Ion batteries in 4S1P configuration for powering another one of my ancient QRP transceivers.
This is not the first time I have revived my FT-817, but previous rebuilds were clumsy. It is one thing to put together multiple modules on the shack bench. Clumsy, cluttered, loose modules might work in a picnic-tables-on-the-air type activation. But would it work in a situation where there are no convenient surfaces to mount the equipment; where – at any moment – we might be politely asked to vacate the area by a hungry bear looking for a space to eat his lunch? This new build is a grab-and-go package that works in small, tight spaces – even on top of a rock in the backcountry – and that’s the kind of environment where I like to operate.
Help support HamRadioOutsidetheBox
No “tip-jar”, “buy me a coffee”, Patreon, or Amazon links here. I enjoy my hobby and I enjoy writing about it. If you would like to support this blog please follow/subscribe using the link at the bottom of my home page, or like, comment (links at the bottom of each post), repost or share links to my posts on social media. If you would like to email me directly you will find my email address on my QRZ.com page. Thank you!
The following copyright notice applies to all content on this blog.
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. -
A Story of Radio RSA (7): “A Process of Integration”
Previous blog:
A Story of Radio RSA (6), June 8, 2025Adapting the Message
By the late 1970s, while still an apartheid state, South Africa wasn’t the same as it had been in the 1960s – and it had taken ways neither Hendrik Verwoerd nor Albert Hertzog would have approved of. In 1977, Stellenbosch University opened its doors for a few black students, provided that there were no stages for further education available for them at non-white universities, according to German news magazine “Der Spiegel” at the time. At the same time, some relaxation in racial segregation probably made the rules even more elaborate, like joint inter-racial sports teams were OK, black-and-white staying at the same “international hotels” and having meals there at the same tables was legal, but dancing together or swimming in the same pool was as forbidden as ever. Racial segregation had been formalized and made universal in South Africa in 1953. In 1971, one hotel become multi-racial on an experimental basis, a number of "international hotels" followed, and the end for "whites-only" hotels came in 1986.
A quote from then foreign minister Roelof Botha probably helps to summarize the late-70s status: "I’m prepared to go to war over our right to exist, but I’m not prepared to die for signs in a lift."
Strikes, although illegal according to the apartheid script, had been a way for the black working South African majority to make its power felt to the white rulers, and quotes from ruling Boers, passed down from the 1960s, showed that this power was really feared. The boom years had ended in the 1970s, and at the same time, skilled or half-skilled labor came into high demand. 1973 saw an accelerated emergence of new unions and growing union membership, and although the government did its best to neutralize their attractiveness to the black workforce, it also recognized labor unions, for the first time under apartheid, in 1979. Probably to the chagrin of the government however, the black unions were much more politically aware than expected. And obviously, they were very aware that South Africa was a welfare state for white, but not for black workers.
The ANC thought of itself as a people’s movement and army. That may have been so, but it was probably more radical in its demands, including its call for international economic sanctions, than the general South African population – blacks included. Not only the white minority was divided during its rule, but so was the "black majority", plus Coloreds, Indians, yes, Chinese, too (though not so many after 1910.
I’m not familiar with the programs of the ANC’s Radio Freedom that broadcast from Tanzania, and later from Zambia, but a radio drama on one of the recordings of the ANC’s Radio Freedom that is available on Youtube takes divisions between black South Africans into account – divisions even within families. And while much of the programs may have been rather "foreign" to Radio RSA’s European listeners (and hard to relate to, especially the obligatory machine-gun fire at the opening of each program), there is an element of progressiveness in the radio drama that follows here: the preparedness of a man’s daughters and wife to address Daddy’s erroneous views on the 1984 elections to a tricameral parliament, – right into his face. That was probably in line with the same values that would be expressed in any progressive household in Europe or North America, or in Wembley (on certain days).
The Boer propaganda was elaborate, but it was out of tune with "the trend that normally will triumph because it agrees with the great myths of the time, common to all men".1
So Radio RSA took on some new stories. Rather than praising purported "harmony and peace" between South Africa’s "ethnic groups and races", they now emphasized that "a process of integration" was underway and that the need for skilled labor had "already led to integration at the workplace." The program added that "sanctions would interfere with exactly this process of healthy evolution"2.
Besides content, the station’s technology park had also seen changes. From 1966 to 1979, Chris Greenway counted the installation of seventeen transmitters at Meyerton, seven of which were dedicated exclusively Radio-RSA – the four 250-kW transmitters as had been known since the beginnings in 1966/67, plus three 500-kW transmitters, added "in the late 1970s"3.
The Usual Suspects
Radio RSA continued to discount the ANC as a mere "terrorist organization", alleging that the ANC wouldn’t even be illegal if it wasn’t "responsible for bomb attacks, sabotage, or murder"4.
Which was rich, when you look at the time table of apartheid legislation through the 1950s and 1960s. It didn’t take much to be illegal in South Africa.My hometown wasn’t full of shortwave listeners and DXers. Nobody in my school class shared my radio hobby. In that light alone, Radio RSA couldn’t have been a game changer, even if its message had been whole-heartedly bought by every listener.
But although shortwave wasn’t a universal medium, "Sender und Frequenzen", a German version of the WRTH (but no relation), reportedly had 40,000 users5.Neither TV nor radio made me aware of what was missing in Radio RSA’s presentation of South Africa. As far as I was concerned, the realities presented on South African shortwaves and on German and Dutch VHFs contradicted each other, but they co-existed without demanding judgment.
Two books carried the weight to broaden my horizon.
Can Themba wasn’t well-known even among bibliophiles in my hometown, but an anthology of political essays and prose excerpts had just been published in Hamburg6, almost at the same time I happened on Radio RSA. The collection of articles and short stories included "Crepuscule", published in 1972. It told a few days in the life of a black Sophiatown resident in the 1950s, legally in love with Brandy, and illegally with a white girl ("chocolate on cream").
The story didn’t exactly get me at the time, but it did point out realities to me that weren’t there in Radio RSA’s German programs. There, it was Afrikaans language, Afrikaans poetry, and Afrikaans what-have-you when it wasn’t Our Wildlife Heritage. It helped that the edition of Themba’s story7 that I read was a German translation. That’s how that non-white parallel universe of Radio RSA’s, South Africa’s "crepuscule" world, became visible for me.
Another reality, never mentioned by Radio RSA either, was torture in South Africa. James Michener’s "The Covenant", also in a German translation, informed me. And "Der Spiegel", again from Hamburg, noted that 21 non-white prisoners in South Africa had died, frequently in mysterious ways", within about just twenty months8. Radio RSA did bring up necklacing however – the other guys’ kind of torture and murder.
Themba’s stories also shed light on divisions within racial groups – divisions that were big enough for the Sophiatown first-person narrator to crack jokes about "African nationalists who profess horror at the thought that any self-respecting black man could desire any white woman," and bridges wide enough for a cop to let the narrator emerge from his Johannesburg subway arrival station without passport control because it’s "the one who drank with me Sis Julia’s shebeen on an afternoon off." In short, there was disagreement within the privileged white class and within the oppressed majority respectively, enough to enable a lot of illegal action or nonfeasance.
As far as international radio was concerned, just as there was Radio RSA, always on message with up to 500 kW, there was the ANC’s Radio Freedom, always on message with maybe 50 kW. That was one reason why you wouldn’t usually catch Radio Freedom’s shortwave signals in Europe. Another was that South Africa’s authorities reportedly jammed Radio Freedom’s transmissions. And, of course, the ANC’s target area was always south of Zambia and other "frontline states" that helped Radio Freedom out with airtime on their shortwave stations – the opposite direction of Europe or North America. Radio RSA’s signals always went north, and were carefully targeted at African and Western countries and regions.
Even 50 kWs of ANC radio were too much for Pretoria though. Any listener in South Africa who listened to it could face up to eight years imprisonment, if caught.
How did Radio RSA handle the parallel universe, in that "crepuscule"? At times by demonization, and mostly by omission. There was a series about the Zulus in the late 1980s, but it was about Zulu history, with King Shaka at the center, not about modern, let alone urban, life.
Listener’s Questions
Listeners’ questions didn’t always get answers either: Ake Magnusson, author of an academic booklet about Radio RSA in 1976, wrote that
The foremost programme for listenersis called ‘P.O. Box 4559’. In this so called Mailbag programme it was announced on February 5, 1974: ‘Mr. Ake Magnusson of the Institute of Political Science at Göteborg’s University. We have written to you, giving the details you wanted on Radio RSA.’ Along with a rather stencilled description of a technical nature of the Voice of South Africa, this was the station’s answer to a letter from me concerning four vital questions about the South African shortwave station. Another letter from me has not led to any reactions at all. This example, though it may be a unique case, shows how the letters from listeners are sometimes used in dubious ways.9
To be fair, I’m pretty sure that no shortwave broadcasters at the time answered every question they were asked – nor would they nowadays. But of course, Magnusson’s experience wasn’t unique. When Radio RSA had a news bulletin about a group of French parliamentarians who had concluded their visit to South Africa with a lot of – reportedly – complaisant findings, the newsreader made no mention of the delegation’s party affiliations. In the most unfavorable case, that could have meant a complete National-Rally traveling group, and as there was no internet to answer my question, I turned to P.O. Box 4559’s German edition. Rather than answering my question, the mailbag program simply repeated the original message from its earlier news bulletin:
The deputies declared on a concluding press conference that after their visit, they are of the view that Apartheid had been abolished, and that they also heard in talks with the numerous political groups and population groups that South Africans, with an overwhelming majority, opposed sanctions. Indeed, the parliamentarians noted that there is a problem of rural flight in South Africa,that there are mixed-races residential areas, that the vexed passport legislation that had led to a discriminating cuts into free movement of persons, have been abolished, that the blacks’ standard of living has risen this much, in contrast to rises in the white population’s real incomes, for example, that the question of residence must, logically, beyond the [group areas act] develop into a further process of integration.10
So I sent another letter and repeated my question.
Answer:
We endeavor to give a balanced view of South Africa, and in the cases we remember, when we rendered statements from foreign politicians, their party affiliations were also mentioned as a matter of principle. If that didn’t happen, it was an unintended oversight we would like to apologize for. However, there are German or rather German-speaking politicians who only come here to confirm preconceived views to shine with on TV at home later on. As these views existed before these Gentlemen came to South Africa We don’t consider such statements productive when it is about our mission to raise understanding for the situation of this country.11
I’ll never know who those visiting guys from France were, back in 1986.
Radio Freedom seems to have had some success among younger listeners in South Africa, despite the threat of jailtime, and despite reported South African jamming. One important reason was probably the music they played – music that was frequently banned in South Africa. One of the best-known names among the bands and musicians could be “Dollar Brand”, aka Abdullah Ibrahim, a jazz musician.
Whatever the South African radio landscape looked like, it wasn’t everyone’s stuff, not even among whites. "Lourenco Marques Radio" (LM Radio) from Mozambique wasn’t an opposition broadcaster, but a private station with a lot of the kind of music younger South Africans wanted to listen to, and that wasn’t greatly available on SABC programs. Although medium- and shortwave-based, LM Radio seems to have been a real alternative – until it "lost much of its sparkle", when taken over by the SABC in 1972.
I’m not sure how much Radio RSA really stood out within the South African radio landscape. Above all, its target area, in terms of content and target areas, wasn’t South African. But while the SABC, the domestic service, wasn’t necessarily popular among all segments of their audience, Radio RSA seems to have ruled among international shortwave listeners of the 1960s, 1970s and 1980s. They regularly were among the top-three stations in the German SWL club’s ADDX (Assoziation Deutschsprachiger DXer) popularity polls, often along with the BBC’s and Radio Canada International’s German services, and they were probably liked both for their ways of presentation that came across as rather spontaneous, and for showing appreciation for their listeners. Appreciation manifested itself in effusive praise for listeners’ fidelity, and in unusually refined letter paper, for example, and, according to Peter Orlik during the first years of Radio RSA’s life cycle, the station’s mailbag program P.O. Box 4559 reacted to listener feedback in a respectable manner:
The letters come from all over the world including some of the nations of Black Africa and deal with facts about South Africa as a whole as well as its music and personalities. Little attempt is made to hide the fact that some of these letters are critical of the Republic and its policies. Many of these are rebutted over the air and it is RSA’s hope that "the facts we have given in reply have led to a better understanding of our position: to know is to understand. We believe that most of the misconceptions about South Africa are the result of misunderstanding and misinformation. The task of RSA is clear." (RSA CAlling, No. 1 /1968, p. 1)
It is to the credit of the program that these replies are entertainingly presented with never the slightest hint of ill-feeling.12I don’t know if that remained so until the end. As far as Radio RSA’s German service was concerned, some of their staff sometimes reacted rather accusingly to criticism from the audience. But that probably didn’t hurt in terms of acceptance in the German-speaking target areas. While shortwave doesn’t provide media with market segmentation tools as powerful as the internet, at least the smaller languages – like Dutch, German, and (for a few years) Danish – provided some opportunities to do so after all. In English or French, only the way Radio RSA directed and slewed its antennas would be of some help.
________________Notes
1 Jacques Ellul: Propaganda, New York, 1965, p. 42 2 Radio RSA, German Service, P. O. Box 4559, October 22, 19873 Donald R. Browne: International Radio Broadcasting, New York, 1982, page 2064 Radio RSA, German Service, P.O. Box 4559, October 10, 19865 ADDX-Kurier, Sept 15, 19906 Das Rowohlt aktuell Lesebuch, Reinbek, 1983, 1984, pages 179 to 1897 An audio book of "Crepuscule" can be found here. 8 “Kopf gegen die Wand”, Der Spiegel, December 11, 19779 Ake Magnusson, The Voice of South Africa, Uppsala, 1976, page 5010 Radio RSA, German Service, P. O. Box 4559, October 22, 1987 – same as quoted under FN-2. I’m not sure if the Group Areas Act is what the mailbag program referred to at the time, but I suppose so. In German, Radio RSA said "Gruppenwohnraumvorbehaltsgesetz", and they pointed out that South Africa’s need for skilled labor had already led to "integration at the workplace". Their original answer in German:Die Abgeordneten erklärten auf einer abschließenden Pressekonferenz, dass sie nach ihrem Besuch die Auffassung vertreten, dass die Apartheid abgeschafft worden sei, und sie außerdem im Zuge der Gespräche mit Vertetern der zahlreichen politischen Gruppierungen und Bevölkerungsgruppen vernommen haben, dass man sich in S.A. in der überwältigenden Mehrheit gegen Sanktionen ausgesprochen habe. In der Tat haben die Parlamentarier festgestellt, daß esin S.A. ein Problem der Landflucht gibt, daß es in S.A. gemischtrassige Wohngebiete gibt, daß die leidlichen Paßgesetze, die zu einer diskriminierenden Einschneidung der Freizügigkeit in der Bewegung geführt hatten, abgeschafft worden sind, daß der Lebensstandard der Schwarzen derart gestiegen ist, im Gegensatz zur Steigerung z. B. des Realeinkommens der weissen Bevölkerung, daß sich die Frage des Wohnsitzes über das Gruppenwohnraumvorbehaltsgesetz hinaus in der Zukunft als logische Konsequenz zu einem weiteren Integrationsprozess entwickeln muß. Der Bedarf an Fachkräften durch die südafrikansiche Wirtschaft hat ja letztendlich schon einmal zur Integration am Arbeitsplatz geführt, zu einer gezielten Kampagne um die Schuausbildung der schwarzen Bevölkerung auf den gleichen Stand mit der weissen Bevölkerung zu bringen …
You get the picture.)
11 P.O. Box 4559, Radio RSA German Service, November 5, 1988:
12 Peter Orlik, The South African Broadcasting Corporation, page 186Wir bemühen uns zwar, ein ausgewogenes Bild Südafrikas zu zeichnen und in den uns erinnerlichen Fällen, bei denen wir in unseren Nachrichtebsendungen Aeusserungen ausländischer Politiker wiedergaben, wurde aus Prinzip auch die Parteizugehörigkeit solcher Politiker genannt. Wenn dies nicht geschah, so ist das ein Versehen, das bestimmt nicht beabsichtigt war, und für das wir uns gern entschuldigen wollen. Es gibt jedoch deutsche oder sagen wir besser deutsch-sprechende Politiker, die hier her kommnen, nur um eine vorgefasste Meinung zu bestätigen, mitder sie dann späterim Fernsehen in ihrer Heimat brillieren. Solche Aeusserungen halten wir für wenig produktiv, wenn es um unseren Auftrag geht, Verständnis für die Lage dieses Landes zu wecken. Da die dort geäußerten Meinungen und Ansichten längst bestanden, bevor diese Herren nach Südafrika kamen, sehen wir sie auch kaum als berichtenswerte Neuigkeit. […]
#Africa #foreignRadio #Germany #music #propaganda #RadioRSA #SouthAfrica
-
AI Features in Adobe Photoshop That Actually Changed How I Work: A Designer’s Field Report
This post contains affiliate links. We may earn a commission if you click on them and make a purchase. It’s at no extra cost to you and helps us run this site. Thanks for your support!
Photoshop just became dangerous. Not the old-school dangerous, where you’d accidentally flatten layers at 3 AM. The new kind. The kind where you question whether you’re still designing or just prompting your way through projects.
I spent three weeks testing Adobe’s latest AI toolkit. What started as curiosity turned into something more unsettling: a complete workflow transformation. These aren’t incremental updates. They’re category shifts that redefine what counts as creative labor.
What Makes Adobe’s AI Implementation Different from Generic Tools?
Here’s the framework I developed while testing: Contextual Fidelity versus Prompt Randomness. Most AI image tools operate on the randomness principle. You type words, hope for magic, and regenerate seventeen times. Adobe flipped this model. Their AI features in Adobe Photoshop read existing image data first, then augment rather than replace.
This distinction matters enormously. Generative Fill doesn’t create from nothing. It analyzes surrounding pixels, lighting direction, perspective angles, and color temperature. The AI becomes a collaborator that actually understands your canvas. Traditional generative AI remains blind to context. Adobe’s approach integrates awareness directly into each tool.
The Three-Tier Intelligence Model
I’m proposing a classification system for Photoshop’s AI features based on autonomy levels:
Tier One: Assisted Operations — Tools that require minimal input but significant human decision-making. Remove Tool and Neural Filters fall here. You point, they execute, you validate.
Tier Two: Contextual Generation — Features that create new content while respecting existing parameters. Generative Fill and Generative Expand operate at this level. They produce novelty within constraints.
Tier Three: Semantic Understanding — Advanced capabilities that interpret intent beyond literal commands. Object Selection and the revolutionary new Harmonize feature demonstrate semantic processing. They recognize what things mean, not just what they are.
How Generative Fill Actually Works (And Why Multiple AI Models Matter)
The first time Generative Fill genuinely shocked me: I selected a boring parking lot in a product photo. Typed “cobblestone plaza with cafe tables.” Expected garbage. Got something I’d have spent two hours compositing manually.
But understanding the mechanism reveals why it works. Adobe Firefly is trained on licensed stock imagery. This creates what I call Style Consistency Inheritance. Generated elements match not just your image’s content but its production quality. Stock photo gets stock-quality additions. Illustration gets illustrated elements. The AI doesn’t just add pixels. It matches provenance.
The Partner AI Model Revolution
Here’s where things get genuinely exciting. As of early 2026, Photoshop now offers multiple AI model options within Generative Fill. You’re not locked into Adobe’s Firefly anymore. Google’s Gemini 2.5 Flash Image (nicknamed “Nano Banana”) and Black Forest Labs’ FLUX.1 Kontext Pro now integrates directly into the workflow.
Each model serves different creative purposes:
Gemini 2.5 Flash Image (Nano Banana) excels at stylized elements and imaginative additions. Want surreal, graphic-heavy imagery? This model delivers. It handles text generation inside images remarkably well. The latest Nano Banana Pro variant offers unlimited generations for Creative Cloud subscribers until mid-December.
FLUX.1 Kontext Pro specializes in contextual accuracy and environmental harmony. Need a realistic perspective? Proper lighting integration? This model understands spatial relationships better than alternatives. It generates single variations rather than three, but quality often compensates.
Adobe Firefly models remain the commercially safe choice. Licensed training data means zero copyright concerns. Production-ready results. Up to 2K resolution output. Professional workflows demand this reliability.
The practical workflow integration proves transformative. Generative Fill delivers three variations automatically when using Firefly models. This Constrained Optionality proves more useful than unlimited randomness. Partner models generate single variations but offer a stylistic range Firefly can’t match.
I tested this on client work. Real deadlines, real budgets. Generative Fill replaced background elements in product photography 40% faster than traditional methods. More importantly, it eliminated blank-canvas paralysis. Starting points appeared instantly. Refinement replaced creation as the primary task.
The limitation? Faces still look suspicious. Human features hit an uncanny valley threshold around 60% realism. For anything containing people, expect additional retouching. Adobe acknowledged this gap. Future updates target portrait-specific training data.
Harmonize: The Compositing Breakthrough Nobody Expected
Previously teased as Project Perfect Blend at Adobe MAX 2024, Harmonize launched in beta during the summer of 2025 and became generally available by October. This feature solves the most persistent problem in image compositing: making inserted objects actually belong in their environment.
Traditional compositing required painstaking manual work. Match the lighting direction. Adjust color temperature. Paint shadows manually. Tweak highlights. Hours of labor for a single realistic composite. Harmonize automates this entire process through AI-powered environmental analysis.
How Harmonize Actually Works
The technology reads your background scene’s lighting conditions, color palette, shadow angles, and atmospheric properties. Then it applies corresponding adjustments to your foreground element. Not just color matching—comprehensive environmental harmonization.
I tested Harmonize on real estate photography. Placed furniture into the empty room in the photos. The AI adjusted object shadows to match the window light direction. Colors shifted to match the room temperature. Reflections appeared on glossy surfaces. Results looked photographed, not composited.
The feature generates three variations per use, similar to Generative Fill. Each variation applies a slightly different interpretation of environmental conditions. You choose the most convincing result. Sometimes none work perfectly. Generate again. Eventually, you find the right balance.
Technical implementation: Harmonize consumes five generative credits per generation (standard features use one credit). Available across Photoshop desktop, web, and iOS mobile app through early access. Works only on pixel layers, not adjustment layers or smart objects.
The research behind Harmonize reveals fascinating technical challenges. Adobe’s team experimented with HDR environment mapping but discovered most users work with standard LDR images. They developed specialized diffusion models that extract lighting information from low-dynamic-range backgrounds. This adaptation makes the technology practically usable rather than theoretically impressive.
Where Harmonize Excels and Fails
Harmonize performs brilliantly with clearly defined objects against well-lit backgrounds. Product photography, architectural visualization, marketing composites. The AI understands spatial relationships. It casts appropriate shadows. It adjusts highlights realistically.
Failures occur with complex transparency, overlapping elements, or extreme lighting mismatches. Placing a daylight-shot person into a nighttime scene produces obviously fake results. The AI handles lighting adjustment but can’t relocate light sources. Use judgment. Maintain atmospheric consistency.
The feature doesn’t replace manual compositing for critical projects. It establishes baselines. You still refine. Mask edges. Adjust opacity. Fine-tune color. But starting 80% complete beats starting from zero.
Generative Expand: Solving the Aspect Ratio Problem
Every photographer knows this pain: Perfect composition, wrong dimensions for the platform. Vertical shot needs a horizontal crop. Magazine layout demands a square format. Traditionally, you compromised composition or faked edges with blur and a clone stamp.
Generative Expand eliminates this compromise through Compositional Extrapolation. The tool analyzes scene geometry, then extends canvas edges with contextually appropriate content. Sky continues naturally. Architecture follows perspective lines. Foreground elements expand without distortion.
When Spatial Intelligence Becomes Obvious
I tested Generative Expand on architectural photography. Original image: tight vertical of a building facade. The client needed horizontal orientation for a banner. The AI extended sides by generating accurate brick patterns, window spacing, and atmospheric perspective depth.
The critical insight: it didn’t just repeat patterns. It understood spatial recession. Bricks appeared smaller toward vanishing points. Window reflections showed appropriate sky portions. This demonstrates genuine three-dimensional scene comprehension, not simple pattern replication.
Professional use case? Absolutely viable. I now shoot tighter compositions, knowing expansion handles format variations later. This inverts traditional photography practices. Instead of shooting wide for cropping flexibility, shoot exactly with expansion capacity. The Precision-First Paradigm emerges directly from this capability.
As of early 2026, Generative Expand now supports the new Firefly Fill & Expand model (in beta), delivering higher resolution and cleaner edge detail. Partner models haven’t integrated here yet, but Adobe’s roadmap suggests future expansion.
Generative Upscale: Resolution Enhancement with Partner Models
Generative Upscale launched in beta during mid-2025, addressing one of Photoshop’s most requested features. The tool enlarges images up to 8 megapixels while maintaining detail quality. More significantly, it now integrates Topaz Labs’ Gigapixel AI as a partner model option.
This partnership demonstrates Adobe’s strategic direction. Rather than building every capability in-house, they’re integrating best-in-class external technologies. Topaz has specialized in upscaling for years. Their algorithms outperform generic approaches significantly.
Practical Applications
AI-generated images are frequently output at lower resolutions. Generative Upscale makes them print-ready. Older digital photos lack detail for modern displays. Upscaling recovers sharpness. Social media managers repurpose assets across platforms. Resolution requirements vary. Upscaling accommodates flexibility.
I tested this on archival product photography. Original 1200×800 pixel images needed a 4K output for new marketing materials. Traditional upscaling produced blur and artifacts. Generative Upscale with Topaz integration preserved edge definition. Text remained readable. Product details stayed sharp.
The limitation: extreme upscaling still produces unconvincing results. Doubling resolution works well. Quadrupling shows strain. Realistic expectations matter. This tool enhances, it doesn’t create information that never existed.
Neural Filters: The Uneven Revolution
Neural Filters sound revolutionary. Reality proves more complicated. These AI features in Adobe Photoshop apply machine learning to common editing tasks. Skin smoothing, style transfer, and colorization. Some work brilliantly. Others feel half-baked.
Smart Portrait deserves attention. It manipulates facial features through slider controls. Want wider eyes? Subtle smile? Different head angle? Adjust parameters, watch changes happen. The technology reads facial geometry, then morphs while maintaining photorealism.
Where Neural Filters Stumble
Style Transfer disappoints consistently. Applying artistic styles to photos produces muddy, unconvincing results. The AI can’t distinguish important details from ignorable texture. Faces become abstract when they should remain recognizable. Backgrounds lose necessary definition.
This reveals a fundamental AI limitation I call Semantic Prioritization Failure. Human artists know what matters in an image. They preserve critical elements while stylizing secondary areas. Current AI applies transformations uniformly. Everything gets equal treatment. Results suffer accordingly.
Landscape Mixer shows similar issues. Combining multiple landscape photos theoretically creates new scenes. Practically? Blurry composites that lack coherent lighting or logical geography. The AI merges without understanding environmental logic.
Object Selection and Remove Tool: Speed Improvements That Matter
Selection remains fundamental to image editing. Adobe’s AI-powered Object Selection changed this tedious process into something almost thoughtless. Hover over objects. Click once. Selection appears.
The underlying technology uses Boundary Prediction Networks. The AI doesn’t just detect edges. It predicts where edges should exist based on semantic understanding. A dog obscured by grass? The selection still captures the complete outline. Traditional edge detection would fail here.
Remove Tool Versus Content-Aware Fill
Adobe separated these functions deliberately. Remove Tool handles quick deletions with automatic fill. Content-Aware Fill provides manual control and preview options. Understanding when to use each determines efficiency.
The enhanced Remove Tool launched in August 2025 with improved Firefly Image Model integration. Results show noticeably better quality and accuracy. Tourist removal from landscapes happens cleanly. Power lines disappear convincingly. The AI analyzes the surrounding context more intelligently than previous versions.
Content-Aware Fill becomes necessary for complex removals. Large objects, important compositional elements, and areas requiring precise control. The preview dialogue lets you customize source sampling. Results improve dramatically with manual refinement.
Sky Replacement: Environmental Harmonization Done Right
Sky Replacement sounds gimmicky. Replace boring skies with dramatic alternatives. Seems like Instagram filter territory. Using it seriously changed this perception entirely.
The sophistication lies in Environmental Harmonization. The AI doesn’t just swap skies. It adjusts foreground lighting to match new atmospheric conditions. Sunset sky? Warm tones appear on buildings. Stormy clouds? Cooler color casts throughout the image. The entire scene rebalances automatically.
The Technical Implementation
Adobe’s approach analyzes multiple image layers simultaneously. Horizon detection, subject masking, lighting direction calculation, color temperature assessment. These processes happen instantly but represent complex computational work.
I tested this on real estate photography. Original images showed flat, overcast skies. Replaced with blue sky variations. The AI adjusted building facades to reflect changed lighting conditions. Windows showed appropriate sky reflections. Shadows maintained correct directionality. Professional results in under thirty seconds.
The limitation? Extreme sky changes create obvious discrepancies. A bright midday sky in a scene with long shadows looks wrong. The AI handles lighting adjustment but can’t relocate light sources. Use judgment. Maintain atmospheric consistency.
Sky Replacement launched with Neural Filters in October 2020, but operates independently through Edit > Sky Replacement. It predates the current generative AI wave but demonstrates Adobe’s early commitment to intelligent automated editing.
The Bigger Question: What Happens When AI Does the Boring Parts?
Here’s my forward-looking prediction: Skill Bifurcation Acceleration. As AI handles technical execution, creative direction becomes the differentiating factor. Designers split into two categories—those who use AI as assistants, and those who become AI’s assistants.
The first group maintains creative control. They know what they want. AI speeds execution. These professionals become more productive without sacrificing vision.
The second group outsources decision-making to algorithms. They accept AI suggestions without critical evaluation. They optimize for speed over quality. Their work becomes indistinguishable from anyone else using identical tools.
The New Creative Skillset
Future Photoshop mastery requires what I call Algorithmic Literacy. Understanding how AI features work internally. Knowing their limitations. Recognizing situations where manual methods remain superior.
You need to know when Generative Fill produces better results than manual compositing. When Object Selection fails, manual paths work better. When Neural Filters create unwanted artifacts. This knowledge separates competent AI users from people letting software make decisions.
Additionally, Prompt Engineering becomes crucial. Generative features respond to text descriptions. Precise language produces better results. Vague prompts generate mediocre outputs. The ability to describe desired outcomes clearly determines success.
Understanding model selection adds another layer. Knowing when Gemini produces better stylization than Firefly. When FLUX handles perspective more convincingly. When commercial safety requirements mandate Adobe’s trained models. These decisions require judgment developed through experience.
Real-World Testing: Where Adobe’s AI Actually Saves Time
I tracked time savings across typical projects. E-commerce product editing saw 35% reduction in processing time. Background removal and enhancement happened faster with AI tools. Manual refinement still occurred, but started from better baselines.
Editorial photography showed 25% improvement. Object removal, sky replacement, and compositional expansion handled common requests instantly. Complex retouching still required traditional techniques, but volume work accelerated significantly.
Design mockups gained 40% efficiency. Generative Fill created placeholder content rapidly. Instead of sourcing stock images for concept presentations, AI generated appropriate elements directly. Client presentations happened faster.
This urban billboard Photoshop mockup with generative AI by Pixelbuddha Studio is available for download from Adobe Stock.Harmonize specifically saved approximately two hours per complex composite. Previously, manual color matching, shadow painting, and lighting adjustment now happen automatically. The time redirects toward creative refinement rather than technical correction.
Where AI Doesn’t Help Yet
Detailed illustration work sees minimal benefit. Character design, complex graphic elements, precise vector work. These tasks require human decision-making at every step. AI features in Adobe Photoshop don’t fundamentally accelerate creative processes.
Fine art photography retouching remains largely manual. Subtle color grading, dodging and burning, and selective adjustments. These require artistic judgment that current AI can’t replicate. Tools assist but don’t replace expertise.
Anything requiring brand consistency needs human oversight. AI generates variations but can’t maintain identity guidelines without explicit constraints. Corporate work demands this consistency. Manual verification remains essential.
My Controversial Take: Adobe’s AI Makes Bad Designers Obvious
Unpopular opinion incoming. These tools expose skill gaps ruthlessly. Previously, bad designers hid behind time constraints. “I would have done better work, but deadlines…” AI removes this excuse.
Now you can execute technically proficient images quickly. If results still look amateurish, the problem isn’t tools or time. It’s vision. You can’t blame software for poor compositional choices. You can’t excuse weak color palettes with workflow limitations.
The Democratization Myth
The tech industry loves claiming new tools “democratize creativity.” Anyone can be a designer now. Just use AI. This narrative is fundamentally misleading.
AI democratizes execution, not creativity. Removing technical barriers doesn’t create artistic vision. Someone without compositional understanding produces bad images faster. Tools amplify existing capabilities. They don’t generate taste or judgment.
Professional designers benefit most from these AI features. They already know what good looks like. AI helps them achieve it efficiently. Amateurs generate more content but not better content.
Learning Curve: How Long Before You’re Actually Productive?
Realistic assessment: two weeks of regular use before these tools feel natural. The interfaces seem simple. Click, type, generate. But understanding when and how to use each feature requires experience.
Initial results often disappoint. Generative Fill creates weird artifacts. Neural Filters look obviously filtered. Sky Replacement produces uncanny lighting. This frustration phase lasts about five projects.
The Proficiency Timeline
Week one: Exploration and disappointment. Nothing works as advertised. Results look artificial. You question the hype.
Week two: Pattern recognition begins. You notice which prompts work better. You understand tool limitations. Results improve incrementally.
Week three: Integration starts. AI features become workflow components rather than novelties. You know when to use them versus traditional methods.
Month two: Fluency arrives. Tools feel intuitive. You develop personal techniques. Productivity gains become measurable. Model selection becomes instinctive.
The mistake? Expecting instant mastery. These AI features in Adobe Photoshop require skill development, like any tool. Proficiency demands practice.
What Adobe Should Fix: The Honest Criticism
Generative Fill needs better prompt guidance. The text input box offers zero feedback. You type descriptions blindly, hoping AI interprets correctly. Adobe should implement suggestion systems. Show example prompts. Indicate effective phrasing patterns.
Neural Filters require transparency improvements. What’s actually happening when you apply style transfer? Which aspects can you control? The current black-box approach frustrates professionals who need predictable results.
Performance and Processing Speed
Cloud-based processing creates annoying delays. Generative features send requests to Adobe servers, wait for responses. Fast internet helps, but doesn’t eliminate latency. Local processing options should exist for paying subscribers.
Additionally, batch processing needs implementation. Applying AI features to multiple images requires manual repetition currently. Professional workflows demand automation capabilities. Adobe announced Firefly Creative Production for batch editing, but integration into Photoshop proper remains incomplete.
Preview quality could improve substantially. Low-resolution previews make evaluation difficult. You can’t assess the detail quality until full processing is complete. Better preview rendering would accelerate decision-making.
Partner model integration remains incomplete. Only Generative Fill and Generative Upscale support external models currently. Harmonize, Neural Filters, and Sky Replacement remain Firefly-exclusive. Expanding model choice across all generative features would increase creative flexibility.
The Economics: Is Creative Cloud Worth It for AI Features Alone?
Adobe charges monthly subscriptions. As of February 2026, pricing breaks down as follows:
Photography Plan (1TB): $19.99/month — includes Photoshop, Lightroom, Lightroom Classic, mobile apps, and 1TB cloud storage. This represents the most cost-effective Photoshop access for photographers and most designers.
Single App (Photoshop only): Approximately $22.99/month — provides Photoshop across desktop, web, and mobile, plus 100GB storage.
Creative Cloud Pro: Around $69.99/month for individuals — includes 20+ applications plus Adobe Express Premium, Frame.io, and extensive cloud storage.
Students and Teachers: Currently $24.99/month for the Pro plan — represents a 64% discount from standard pricing.
For professionals billing clients, these costs are easily justified. Time savings generate revenue exceeding subscription expenses. Forty percent efficiency improvement means handling more projects monthly. Increased capacity creates profit.
For hobbyists and students, the calculation differs. AI features provide value but might not justify ongoing expenses for casual use. Alternative software offers similar capabilities at lower prices. Affinity Photo costs $69.99 once. Includes solid AI features without subscriptions.
The Competitive Landscape
Canva integrated AI aggressively. Their generative tools work surprisingly well for basic tasks. Interface simplicity appeals to non-professionals. Monthly cost: around $12.99 for individuals.
Luminar Neo specializes in AI-powered photo editing. Sky replacement, skin retouching, object removal. Subscription model now standard, but pricing remains lower than Adobe.
Adobe maintains advantages in professional workflows. Better color management, extensive plugin ecosystem, and industry-standard file compatibility. Partner model integration creates unique capabilities competitors can’t match. For serious work, these factors outweigh cost considerations.
The generative credits system requires understanding. Standard features (Firefly-powered Generative Fill, Generative Expand, Remove Tool) consume one credit per generation. Premium features (partner AI models, Harmonize at five credits) consume more. Creative Cloud plans include monthly allowances—typically 4,000 credits for premium features.
Future Predictions: Where Adobe’s AI Heads Next
Prediction One: Semantic Style Consistency. Within eighteen months, Adobe will implement style learning from user editing patterns. The AI will observe your color grading choices, compositional preferences, and retouching approaches. It will then suggest adjustments matching your personal style.
Prediction Two: Three-Dimensional Scene Understanding. Next-generation Generative Fill will comprehend spatial relationships better. Perspective-accurate object insertion. Proper occlusion handling. Shadow generation matching light source positions. This requires advanced 3D scene reconstruction capabilities. Early signs appear in FLUX Kontext Pro’s environmental awareness.
Prediction Three: Conversational Editing Interfaces. Late 2025 saw Photoshop integration with ChatGPT, enabling conversational image editing without leaving chat interfaces. This capability will expand. Natural language instructions will replace complex menu navigation. “Make the sky more dramatic” triggers exposure, contrast, and color adjustments automatically.
Prediction Four: Expanded Partner Model Ecosystem. Adobe will integrate specialized models for specific tasks. Medical imaging partners. Architectural visualization specialists. Fashion-specific generators. The model picker becomes a marketplace. Users select tools matching project requirements.
The Augmented Creativity Paradigm
I’m coining a term here: Augmented Creativity Paradigm. This framework describes the emerging relationship between human designers and AI tools. Neither fully automated nor entirely manual. A hybrid state where AI handles bounded tasks while humans maintain strategic control.
This paradigm requires new professional competencies. You must understand AI capabilities and limitations. Furthermore, you must direct tools effectively, and you must evaluate AI outputs critically. Traditional design skills remain essential but insufficient alone.
The designers who thrive will embrace this hybrid model. They will use AI as a tool for efficiency without relinquishing creative control. They will question its outputs rather than accept them at face value, recognizing both its strengths and its limits. Instead of following generic suggestions, they will train the system to reflect their own taste, standards, and creative intent.
Harmonize represents this paradigm perfectly. It automates environmental matching—a technically complex but creatively straightforward task. This frees designers to focus on composition, concept, and narrative. The AI handles photorealistic integration. Humans handle meaning.
Ethical Considerations: The Commercial Safety Advantage
Adobe’s Firefly training exclusively on licensed stock imagery and public domain content creates a genuine competitive advantage. Generated content carries zero copyright liability. Clients accept AI-assisted work without legal concerns.
Partner models introduce complexity. Google’s Gemini and Black Forest Labs’ FLUX are trained on broader datasets. Licensing clarity varies. Professional use requires careful consideration. Adobe maintains that user outputs remain user-owned and aren’t used for AI training, regardless of model choice.
The photography community expresses legitimate concerns about AI replacing human creativity. Stock photography markets face disruption. Junior creative positions evolve. These developments deserve serious discussion rather than dismissal.
My perspective: AI tools amplify rather than replace human creativity when used thoughtfully. They eliminate tedious technical work, accelerate iteration, and democratize execution. But they don’t generate original vision. That remains human domain.
Frequently Asked Questions (FAQ)
How accurate is Generative Fill compared to manual compositing?
Generative Fill achieves roughly 70-80% accuracy for simple background extensions and object additions. Complex composites still require manual work. The AI excels at texture generation and atmospheric consistency but struggles with precise detail matching. Professional results typically need AI generation plus manual refinement. Partner models like FLUX Kontext Pro improve contextual accuracy significantly.
Can AI features in Adobe Photoshop replace traditional retouching skills?
No. AI tools accelerate workflows but don’t eliminate skill requirements. Object removal works automatically for simple cases. Complex retouching demands manual techniques. Color grading, dodging and burning, and detailed masking—these require human judgment that AI can’t replicate currently. Consider AI as efficiency multipliers, not skill replacements. Harmonize automates environmental matching but creative composition decisions remain human.
Do Generative AI features work offline?
Currently, no. Most generative AI features in Adobe Photoshop require internet connectivity. Processing happens on Adobe’s cloud servers. This enables complex computations but creates dependency on network availability. Adobe hasn’t announced local processing options yet. Work requiring offline capability should use traditional tools.
Which AI feature provides the biggest time savings?
Remove Tool delivers the most consistent efficiency gains. Simple object removal that previously took five minutes now completes in seconds. Harmonize ranks second for compositing work, saving approximately two hours per complex project. Generative Expand helps dramatically for photographers needing aspect ratio flexibility. Sky Replacement accelerates real estate and landscape work. Your specific workflow determines which feature saves the most time.
Are there ethical concerns with using AI-generated content commercially?
Adobe’s Firefly AI trains exclusively on licensed stock imagery and public domain content. This addresses copyright concerns other AI tools face. Generated content using Firefly models is commercially safe for most uses. Partner models (Gemini, FLUX) have different training sources—verify licensing terms for specific projects. Client contracts may prohibit AI-generated elements. Check agreements before deploying AI content professionally.
How does Adobe’s AI compare to standalone tools like Midjourney?
Different use cases entirely. Midjourney excels at creating original images from text prompts. Adobe’s AI features augment existing images contextually. Midjourney generates without constraints. Photoshop’s AI respects existing image parameters. For editing workflows, Adobe integrates better. For pure generation, Midjourney offers a more creative range. Most professionals use both for different purposes. Partner model integration now brings some generative flexibility into Photoshop.
Will these AI features make junior designers obsolete?
Unlikely. AI automates technical execution but doesn’t replace design thinking. Junior designers learn by solving problems, not just operating tools. Entry-level positions will shift toward creative direction earlier. Technical proficiency develops faster with AI assistance. Thoughtful employers recognize this creates better-trained professionals, not redundant ones. Design judgment remains fundamentally human. Harmonize automates lighting matching, but can’t decide what should compose the image.
How do generative credits work with partner AI models?
Standard features (Firefly-powered Generative Fill, Remove Tool) consume one credit per generation. Partner AI models like Gemini Nano Banana and FLUX Kontext Pro are premium features consuming variable credits—typically more than standard features. Harmonize consumes five credits per generation. Creative Cloud plans include monthly credit allowances. Photography Plan includes credits for standard features; premium features may require Creative Cloud Pro or additional credit purchases. Check current plan details for specific allocations.
What’s the difference between Harmonize and Color Matching?
Harmonize performs comprehensive environmental integration—adjusting color, lighting, shadows, and visual tone to blend objects realistically into scenes. Color Matching only adjusts the color palette to match reference images. Harmonize goes far beyond color correction. It analyzes light direction, casts appropriate shadows, adjusts highlights, and modifies atmospheric properties. Think of Harmonize as complete compositing automation, while Color Matching handles only color temperature and tones.
Can I use multiple AI models in a single project?
Absolutely. Professional workflows increasingly combine multiple models for different tasks. Use Firefly for commercially safe background generation. Switch to Gemini Nano Banana for stylized graphic elements. Apply FLUX Kontext Pro for perspective-accurate object insertion. Each model serves different creative purposes. Layer these capabilities strategically. The model picker makes switching seamless within the Generative Fill workflow.
Check out WE AND THE COLOR’s AI and Technology categories for more.
Subscribe to our newsletter!
By continuing, you accept the privacy policy -
🖥️ Ah, yes, the riveting tale of how spreadsheets—those glorified tables that haunt our work lives—came to be. #VisiCalc, the "GameBoy game" of its time, apparently revolutionized #computing with its primitive charm and jaw-dropping 16K RAM requirement. 🙄 One can only imagine the thrill of rebuilding a 1979 relic while the rest of us are busy enjoying the luxury of actual modern technology. 😂
https://zserge.com/posts/visicalc/ #spreadsheets #techhistory #nostalgia #HackerNews #ngated -
"Has Britain become an economic colony? | Technology" by Tim Wu
https://www.theguardian.com/books/2025/nov/23/has-britain-become-an-economic-colony "Two and a half centuries ago, the American colonies launched a violent protest against British rule, triggered by parliament’s imposition of a monopoly on the sale of tea and the antics of a vainglorious king. Today, the tables have turned: it is Great Britain that finds itself at the mercy of major US tech firms..." #digitalsovereignty #TechOligarchy -
CW: Classic creations by Arcadia Asylum a.k.a. Lora Lemon/Aley at OpenSimFest 2023; CW: long (post text: 258 characters, first image description: 38,650 characters, second image description: 26,213 characters, third image description: 9,687 characters, full net length: 76,780 characters), eye contact
OpenSimFest 2023 doesn't only offer the latest and greatest in OpenSim creativity. It also shows some classics made by Arcadia Asylum in Second Life and legally preserved in OpenSim. This includes her space themes which are exhibited on three large displays.
The pictures in this post are digital 3-D renderings of exhibition displays at the annual OpenSimFest (official website) which has started on September 15th and will continue until September 30th. They were created using shaders, but without ray-tracing, as they were taken inside a virtual world based on OpenSimulator.
OpenSimulator (official website and wiki), OpenSim in short, is a free and open-source platform for 3-D virtual worlds that uses largely the same technology as the commercial virtual world Second Life. It was launched as early as 2007, and it mostly became a network of federated, interconnected worlds when the Hypergrid was introduced in 2008. It is accessed through client software running on desktop or laptop computers, so-called "viewers". It doesn't require a virtual reality headset, and it actually doesn't support virtual reality headsets.
Just like Second Life's virtual world, worlds based on OpenSim are referred to as "grids" because they are separated into square fields of 256 by 256 metres, so-called "regions". These regions can be empty and inaccessible, or there can be a "simulator" or "sim" running in them. Only these sims count a the actual land area of a grid. It is possible to both look into neighbouring sims and move your avatar across sim borders unless access limitations prevent this.
OpenSimFest takes place on its own grid. It is connected to the Hypergrid, so it isn't necessary to have an avatar on the OpenSimFest grid because you can visit it with an avatar that you have on another grid. It features a schedule of 12 hours each day, starting at 9 a.m. PDT. PDT is the standard time zone both in Second Life and on all OpenSim grids. On most days, from 11 a.m. to 7 p.m., there are live performances or DJ events on one of the four neighbouring event sims. Most of the other sims are much larger. They are so-called "varsims", something specific to OpenSim that doesn't exist in Second Life. They stretch across multiple regions arranged in a square without borders between the regions. Two of them are merchant sims where OpenSim creators offer mostly payware, but also a few freebies. Others are, for example, exhibition sims.
The motto of OpenSimFest 2023 is the Jazz/Blues Era.
These pictures were all taken on Sulfur, one of the two merchant sims. What they show, however, are displays of creations by the famous Second Life creator Arcadia Asylum. She started around 2006, and she was banned from Second Life several times, likely because she refused to charge money for her creations. Instead, she offered them for free, under a free, non-commercial copyleft license and with full permissions. Due to always being banned after a while, she went through several names. Arcadia Asylum was the first, followed by names such as Aley Arai, Lora Lemon and finally, when Second Life had removed the last names feature for new users, Aley Resident or simply Aley.
Her creations were mostly preserved by exporting them from Second Life and importing them into OpenSim. She wasn't opposed to this, she actually eventually gave her own official permission to do so. But she was never in OpenSim herself.
What is shown here are the preserved creations from her Space Pirates and Galactic Trade Union themes which she created from 2008 to 2011 under the guise of Lora Lemon except for a few items which she had to release through her Aley avatar. Most of them are static, unscripted objects. The displays are mostly surrounded by cubes which she had made, too, and which are amongst the items which she released as Aley. These cubes show space pictures on the inside, and they are fully transparent from the outside. Normally, they are the size of a whole region, but they were resized for OpenSimFest to work as display backdrops. The sides show various nebula while the top and the bottom only show a star field.
In the foreground, there is always a small bit of the broad walkways that connect the displays on Sulfur. Their textures give the impression of polished tannish stone tiles cut into two rectangular sizes, one wider, one narrower, but both with the same length. Between the walkways and the space cubes, there is always a very slightly sloped "ramp" with a medium grey texture that shows a stylised moon landscape covered in craters of various sizes.
The direction of view in the first picture is not perpendicular to the edge of the walkways; it is rotated to the left by about 10°.
The space cube in the first picture shows a photograph of otherwise nameless IC 434, as it is designated in the Index Catalogue, on the left-hand side. It is a part of the Orion B molecular cloud, a star-forming region in the constellation of Orion. However, the image seems to be altered. For starters, it is mirrored with the north to the right and the east to the bottom.
IC 434 itself is shown as a rose-coloured nebula extending mostly horizontally with an almost sharp lower edge below its long and thin brightest part, but flaring upwards from there with its brightness increasing in a number of steps. These flares increase in height, the farther to the right they are. In front of it, almost in the middle, there is the dark shape of the Horsehead Nebula, also known by its designations Barnard 33 and LDN 1630 in Lynds' Catalogue of Dark Nebulae.
Below and to the right of the Horsehead Nebula, the star HD 37903 can be seen shining with a faint lavender tint, but not so much the reflection nebula NGC 2023 which it would normally illuminate. What is even stranger is that Alnitak or Zeta Orionis which should be a very bright star to the right of IC 434 in this picture is missing; there is another, smaller purple nebula in its place. The Flame Nebula, also known by its designations NGC 2024 and Sharpless 277, is in its usual place below where Alnitak would be, glowing orange-red while being partially obscured by a darker cloud in the middle.
The texture in the back is a photograph of the Crab Nebula, also known by the designations Messier 1, NGC 1952 in the New General Catalogue, Taurus A and Sharpless 244. It is the remnant of the supernova SN 1054 that was observed on Earth from July 4th, 1054 on, and it is in the constellation of Taurus.
The Crab Nebula generally consists of two components. The remains of the supernova explosion have formed thin, thread-like and chaotic structures that glow from lime green near the upper right limits above the edge of the image to yellow slightly closer to the centre already to orange farther to the lower left to a deep red on the lower left limits. In addition, there is a diffuse, faint aqua blue cloud of glowing gas created by the Crab Pulsar which is what is left of the exploded star. It is smaller than the supernova remnants, and it doesn't reach nearly as far to the lower left.
The texture on the right-hand side is a photograph of otherwise nameless NGC 604, the ninth-largest known nebula. It is a star-forming region that resides in the Triangulum Galaxy which in turn is in the constellation of Triangulum.
The gases that make up the nebula are slowly contracting under their own gravitation to chaotic fibrous structures which might eventually form stars. The brightest parts are in the middle of the nebula, in the upper right corner of the image, where a cluster of about 200 young and massive stars are making the nebula glow by ionising it. Close to these stars, it is shown glowing in a cream or champagne tone. With no gradient in-between, the outer parts are glowing in a much less bright Burgundy red.
On the edge of the walkways towards the ramp, way to the right of the picture, stands a sign-like structure. It is actually a scripted teleporter on which destinations all over OpenSimFest can be chosen, but not outside OpenSimFest. Its basic structure is a flat, rectangular, upright box with a texture that roughly mimicks stainless steel brushed in an elliptical pattern. At the very top, still on the stainless steel texture, there is the name of the device or most of it in vantablack in an unidentified monospace typeface: "MD Teleport System".
Apart from the wide margin, the rest of the front is occupied by various rectangular panels, most of which share the same full width which is about 90% of the width of the whole device. Also, most of these panels are vantablack with yellow writing in the DejaVu Sans Mono monospace typeface on them. The top panel names the destination chosen for teleporting, "Steam Fair by Aley c.2015".
Below it is an image that shows a preview of the destination. The centre of this is a semi-elliptical arch construction that serves as the entrance of a Victorian-age amusement area on a wooden boardwalk. The structure itself is fairly elaborate with a texture that resembles slightly yellowish light wood.
At its front, there is a partially transparent sign that gives the impression of being three-dimensional through highlights and shading, implying light falling in from the left. It shows a light grey silhouette of a boardwalk with four different buildings in various sizes and shapes on it, two of them with one triangular pennant standing off a small flagpole on the roof towards the left, one with two of these. To the right of the left-most building, there is a silhouette of a ferris wheel that bears a strong resemblance to a ship's steering wheel with its eight spokes and no visible cabins.
The boardwalk silhouette is standing on a medium grey horizontal rod. Above it and connected to the rod at its end, there is a semi-elliptical double arch in the same medium grey. This double arch carries the writing "SEAVIEW" in sky blue letters in a very ornate but unidentified serif font which follows the shape of the arch. There is a light grey cartoon crab with big googly eyes and a happy expression with an open mouth and pincers spread wide wearing a white bowtie and a black top hat with a white hatband below the left-hand end of the "SEAVIEW" writing. Likewise, there is a light grey kraken with curled tentacles on the right.
Between the "SEAVIEW" writing and the boardwalk silhouette, there is the horizontal writing "Amusement Park" in a less ornate, Western-style typeface with small capitals, seemingly held in place by two horizontal rods that connect to the inner medium grey arch, one at the bottom and another one at the top only holding the actual capitals. In the same typeface, also in medium grey and with small capitals, but in front of the boardwalk silhouette and only slightly above the horizontal rod that carries it, there is another writing, "Est. 1869".
The floor of the entire structure is textured to resemble longitudinal and transversal wooden beams made of darker wood with diagonal planks made of even darker but still medium brown wood filling the large spaces between them, held in place with one large blue nail at each of their ends.
The point of view is above the bridge that leads to the boardwalk, going slightly uphill. It is lined with fence-like guards on both side, the same as those which surround the whole boardwalk, with a wooden texture that is about as light as that of the wooden "beams" in the ground texture, but more yellowish. These mostly consist of four long planks, the top one being oriented horizontally and mounted against the tops of the support poles, the other three being oriented vertially and mounted against the boardwalk sides of the support poles.
The sides of the bridge are also lined by two strings of small pennants that lead past the on-looker and end on the arch. The one on the left shows eight pennants in, from front to back, green, blue, yellow, blue again, yellow again, red, green again and blue again. The one on the right shows six pennants in, from front to back, green, this one is barely visible, red, green again, blue, yellow and red again.
To the left of the arch, there is a lamp on a lamppost. Both the lamp and the post seem to be made from the same wood as the fence except for the lamp glass which shows a yellow glow. To the right of the arch, there is one dark wooden booth. Behind that booth, there is another different arch with the same wooden texture as the one that is the centrepiece of the preview image.
In extension of the bridge, a wheelbarrow with a market stand with six boxes textured like they contain various unidentified items is standing on the boardwalk. Various other structures are on the boardwalk, mostly in the background, like tents with tarpaulins striped in beige and red or cream and brown as well as a target and another string of pennants.
All objects in this preview image so far were made by Arcadia Asylum.
Above the whole scene, there is a "ceiling" with a transparent texture that refracts the dark purple night sky. Farther in the background, barely identifiable through the "ceiling", two structures can be made out. To the left, there is the semi-transparent, black, box-shaped display booth of the Focus virtual art magazine. The fourth nebula texture of Arcadia Asylum's space cube can be seen through it; it is part of the third Arcadia Asylum space display which will be described along with its image. To the right, there is an enormous static wave with a constantly moving aqua blue texture on it.
Back to the teleporter itself: This image also triggers the actual teleport. If you click it, you are taken to the chosen destination.
The ten vantablack panels below list ten possible destinations. They show these destination labels in yellow writing: "A Whale of a Tale", "Diamond Queen - Ruth 2.4" which refers to the free, open-source mesh body Ruth2 v4, "Fashion Temple", "Kimberley's Fashion Lab", "FOCUS Magazine of Virtual Art", "Galactic Truckstop", "Space Pirates by Aley Arai", "More spaceships by Aley Arai" which is where this teleporter is standing, "Steam Fair by Aley c.2015" which is what is currently selected and "Chez Faire Beach display". Clicking them will select them as the teleport destination. These ten destinations are all on this sim, so they can also be reached by walking which would take a while due to the size of the sim, though.
At the bottom, there are four square vantablack buttons with yellow labels on them. The leftmost one has a question mark in the middle and "(DE)" in the bottom left corner. It gives you a notecard with a manual for the teleporter in German. The rightmost one is similar, only that "(EN)" is written in the bottom left corner. It also works the same, only that it provides an English manual. The buttons in-between are labelled "Pg Up" and "Pg Dn" respectively. With these two buttons, several pages with up to ten teleport destinations each can be flipped through.
On the far edge of the grey ramp, a few metres to the left from where the teleporter is standing, there is a vantablack sign that names and describes the display. In the same yellow DejaVu Sans Mono as the teleporter, it reads, "Name: More spaceships by Aley Arai | Owner: Ada.Radius | Description: Assets from the Arcadia/Aley library owned by New Media Arts, Inc., a 501(c)(3) nonprofit foundation. Complete Opensim collection in Kitely grid." Kitely is the third-largest OpenSim grid and one of the oldest, launched in 2011 after preparations had started as early as 2008. The pipes in this transcription, the vertical lines, mark line breaks in the original.
While the sign is actually hovering above the ground, it is seemingly held by two more of Arcadia's creations, two robots facing the sign from the sides, both of which were released through her Aley avatar.
On the left, it is Rosie the Robot Maid which can also be used as an avatar. She is mostly a darker sky blue. Her head is "cylindrical" with a horizontal, transversal axis. On each side, along the extension of the head axis, sits one horizontal, conical antenna which is about 60% as long as the head is wide and ends in a bright red, double-conical knob. The sockets of the antenna consist of a short cylinder a bit more than 20% the diametre of the head and a slightly longer cone with about 20% the diametre of the head. Two red cylinders are the eyes. On top of each eye, there is one black protrusion in a shape roughly resembling a much-widened obelisk turned slightly outwards, mimicking eyelashes. A red box serves as the mouth with a sky blue box below it serving as the lower lip and another sky blue feature in a harder-to-describe shape above it serving as the upper lip and the tip of the nose.
The upper body is a trapezoid with separate rounded sides that widens more at the front and the back than to the sides. It has three small, cylindrical, bright red buttons at the front. Both arms consist of an upper and a lower arm with separate joints. The shoulder joints are held together and in place with one big bright red slotted pan head screw each. The elbow joints only have one sky blue cylindrical pin each. The hands are two-part clamps with cylindrical pins.
The lower body is shorter, black and can best be described as two trapezoids with rounded sides on top of each other. The top one is rather low and widens downward; the upper body protrudes through the rounded sides. The bottom one is basically of the same shape, but turned upside-down and stretched to about three and a half times the same height. The only foot is sky blue again. It consists of a cylindrical leg, an elliptical, conical upper part, an elliptical, but non-conical lower part and two simple black cylinders as wheels.
Rosie wears multiple white objects shaped like something between half an eight-point star and half a bloom with eight petals. One is standing on top of her head. Two are surrounding each of her wrists. Another two, arranged in an angle of 60 degrees, resemble an apron. Finally, there is a white bow made of two flattened cones on her lower back, right above where her upper and lower body meet.
Due to the low resolution of the meshes, everything that is supposed to be round is actually octogonal.
The robot on the right of the sign is Asteroid Al. He is much more humanoid and uses fewer, but more complex shapes. He also makes more use of complex textures, especially for his body and his head which are mostly covered in detailed metal-like textures. The head is stretched upwards and backwards with large deep cavities for eye sockets and where the ears would be. The mouth is a grille and part of the head texture. The eyes are almost completely baby blue and more reminiscent of two headlights, but they aren't glowing. They're textured onto the ends of what roughly appears like two black cylinders bending into the head towards the back with several golden rings around them.
He is wearing a brick red sleeveless shirt which is textured on rather than a separate object. On the front and the back, there is a picture of a "galactic truck stop" which looks completely different from what Arcadia had built and given the same name and rather resembles that in the Mel Brooks film Spaceballs. From what can be seen in the very-low-resolution image, it seems to have been built on top of an artificially flattened asteroid with the debris from the flattening still around it, and it has three circular landing pads with spaceships on it and structures both above and below landing pad level. The highest point is a purple "GALACTIC TRUCK STOP" sign. The same writing is on the shirt as well, "GALACTIC" above the image, "TRUCK STOP" below it.
Another robot is floating above and behind the image, closer to Asteroid Al than to Rosie: the tiny maintenance bot Widget. Widget is basically a sphere with things attached to it. These are two short clamp arms, three jointless legs with suction-cup-like feet, two eyes similar to Al's, but with silver rings around the cylinders, two tool-like hexagonal attachments on the upper sides, a structure on the back built around an elliptical orb glowing teal and two cylindrical red protrusions, one at the top, one at the bottom. The one at the top serves as the attachment point for a small "satellite dish" that is oriented more forward than upward.
Most of Widget's textures have too low a resolution to be able to identify them. The full-resolution textures might never have made it over from Second Life. But the dominant texture for the attachments is black with diagonal yellow stripes. And the texture of the cylindrical body, probably mostly black with diagonal yellow stripes, too, shows circular blue signs on the left and the right, each surrounded by a silver rim and showing two crossed white tools which may be a pickaxe and a hammer.
Behind the sign and the three robots, slightly to the left again, there is a vortex-like object that is not textured onto the space cube. It looks like the accretion disc of a black hole. The inner half is mostly white, lavender and baby blue whereas the outer half is rose and brick red. It implies a clockwise rotation, but it is static. In order to appear blurrier, it was made of two semi-transparent objects with similar textures. The one up front has a texture that is blurred more, and it has a higher transparency. The object is labelled as a wormhole, but non-functional.
As for spaceships, this display shows tens of them, three of which are parked on the grey ramp in front of the space cube.
The leftmost ship in the foreground, oriented with its bow to the right, is named Retro Rocket Ship. Its fuselage is cylindrical in the middle with slightly elongate spherical ends. It has five fins with a curved, back-swept shape reminiscent of Buck Rogers aesthetics, but with a rectangular cross-section. Two of them are mounted on the sides of the cylindrical part like wings. The other three are mounted on the rear, one on top, the other two rotated slightly downward from the sides. The ship has a short nose that ends in a sphere. Fuselage, nose and fins have textures that suggest that they're riveted together from steel plates with no surface treatment. The ship has no visible landing gear.
The interior of the ship can be seen through 15 portholes which are riveted into the fuselage. The spherical bow contains the cockpit with two padded seats and eight portholes. Behind it, in the fore half of the cylindrical section, there is the crew compartment with a double bunk bed. It has two portholes on each side and more control panels like those in the cockpit below them. The other half of the ship is the engine room with a "rocket engine" that leads to a dark grey rocket nozzle on the back of the ship. It has one porthole and the ship's only door on the port side, two portholes on the starboard site and even more control panels below the portholes. The compartments are separated by double sliding doors which are closed.
While this ship was only inspired by Buck Rogers, the tiny ship in front of its nose, bow oriented roughly towards the front, had its design lifted from actual 1930s Buck Rogers toys. It is one out of only two ships on this display with no visible interior, and it is one out of only two with a glossy surface. Unlike the toys, but following Arcadia's style, it still looks like it's riveted.
Its front contains the cockpit. It is spherical with a strongly semi-elliptical cross-section. On top, there are six roughly rectangular windows in side-by-side pairs of two with light blue panes. Farther back, there are two short, stubby wings, one on each side, with partly elliptical fore edges and straight aft edges. Above each wing, there is one porthole with a light blue pane in it. There is a partly cylindrical, partly conical tube on the nose from which a structure made out of two long, thin, very slight cones protrudes.
Right behind the cockpit, nine elliptical pipe-like objects are arranged in such a way that they appear like 18 small thrust or exhaust nozzles, judging from the soot on their ends and the fuselage aft of them. The soot also covers the door with porthole on each side. The longer aft section of the fuselage is semi-elliptic in shape again, but stretched more than the cockpit section. Four semi-elliptical fins are mounted on it, one on the top, one on each side, one on the bottom. Between these fins, there are four slightly larger and much longer nozzles. Unlike those behind the cockpit, they aren't straight but protruding from the fuselage in a curve. A small normal rocket nozzle sits on the rear end between the fins.
This ship has landing gear. Two wheels are located under the cockpit at the same length as the portholes and the wings. Dark patches in the fuselage texture suggest that they can be retracted; actually, they are static. The third wheel is mounted on the bottom fin slightly aft of the larger nozzles. All three wheels sit in streamlined shrouds which add to its 1930s-style looks, tying in neatly with the motto of OpenSimFest 2023. None of the wheels touch the ground, though.
The ship behind these two is near the ground of the space cube, facing out of the cube, but rotated about 20 degrees clockwise away from standing perpendicularly to the walkways. It is labelled as the Retropolis retro future spaceship. At first glance, it is very similar to the Retro Rocket Ship, but shinier, neater and less grubby. It has a more elaborate nose which ends in a cone. Instead of the two wings, it has three fins around the cylindrical section. These wings are more simple in shape than those on the Retro Rocket Ship. They have multi-part, largely conical insertions vaguely reminiscent of early aircraft jet engines, one sitting on the outer edge. Between the outermost two, the wings are extended towards the back. The stern is surrounded by four diagonally-mounted fins now, similar in shape, but each with only one two-cone construct on the outer edge. The rocket nozzle was replaced with a different, more obviously octogonal and non-conical one, surrounded by four "cylindrical" objects above and below it as well as on its sides which could have been booster nozzles earlier. It's rather obvious that not all original textures have come from Second Life to OpenSim with this ship. Arcadia's space themes are rather problematic in this regard.
The interior is easier to access, also because the single door is open; it opens downwards. The engine compartment has some additional handles and pipes and a larger rocket engine. The other two compartments can be access through the sliding doors, one of each is open now. In fact, one of the cockpit doors is misplaced and clips through the fuselage on the port side. The crew compartment has a small round table, a chair, four cargo shelves, eleven wooden crates all over the port side place and a third bunk. The four shelves even extend into the cockpit where they hold another five crates. Also, the cockpit not only has one handwheel in front of each seat, but it also has a ship's steering wheel in the middle and, for some reason, even a periscope above it.
Another vessel is half-hidden behind the retro future spaceship. The Centurion cargofighter of the Galactic Trade Union has a strong resemblance to post-war jet fighters. It doesn't have a modelled interior; the glass of the cockpit cupola is completely opaque and black. The long nose is slightly curved from the sides, but hexagonal in its cross-section. The two wide wings have winglets on their ends, slightly tilted outward. They also carry what appears to be four hexagonal ray weapons on their weapon mounts.
Most of the rest of the ship is more rounded except for the four square engine nacelles, one mounted into each of the four fins arranged in an X. The nozzles at the ends of the nacelles are round again. Most other details are drawn onto the textures. These include 14 "portholes" similar to those on vintage Buicks, four behind the cockpit and above the wings and three below the wings on each side, as well as several safety markings with black and yellow diagonal stripes, different safety markings with black and red diagonal stripes on the rayguns and a painted-on "shark maw" on the underside below the cockpit.
The "landing gear" consists of three rather elaborate, but unarticulated vertical legs. They suggest or rather require vertical take-off and landing capabilities of which the rest of the Centurion doesn't show any traces.
Right above and farther to the back hovers another, larger, less elegant ship: the Behemoth Freighter of the Galactic Trade Union, facing to the left and slightly to the front. This ship was a cooperation with Jim Carter. It is covered almost entirely in a texture that looks like a wild and chaotic "patchwork" of rectangular metal panels, only that these aren't riveted.
The centre-piece of this ship is its boxy main cargo hold. On each side, two massive cylindrical engines are mounted with intakes at the front which make them partially seem more like turbofan engines. On top of the cargo hold, there is an empty dome which can be accessed via a ladder. The trapezoid stern features the loading ramp which is the only access to the ship's interior. The ramp is lowered almost completely. Above the ramp sits a third engine which is as big as the other two, but of a different type. Invisible from the on-looker's point of view, there are also twelve cube-shaped crates of different kinds in the cargo hold.
Forward from the cargo hold, the fuselage not only narrows, but also rises both at the bottom and at the top towards the elevated cockpit. This section two crew seats, a bed, a touch panel, a microwave and a "Mr. Coffee" coffee maker from Spaceballs. The cockpit has only got one seat which is surrounded by control panels. It is also the only part of the ship with windows. Two rows of ten rectangular windows with rounded corners each are wrapped around the semi-elliptical cockpit section.
The landing gear is lowered again. It consists of four legs, each with four feet with four extending pads each, connected to the upper struts via double shock absorbers. One pair is installed under the rear end of the cargo hold, the other one under the rear end of the "neck" between the cockpit and the cargo hold.
To the right and farther above, there is a ship simply named Saucer that looks the part. It is basically the stereotypical flying saucer. Most of its details are painted onto the texture again. It is oriented in such a way that the open hatch which doubles as a ramp faces the on-looker. The improperly installed double sliding door behind it grants a glimpse into the interior of the ship which, however, lacks any notable details except for the ladder up to the cockpit.
As usually for flying saucers, the cockpit is under a cupola on top and in the middle of the vessel. It shows a great deal of part recycling by Arcadia: Not only are the textures for the control panels standard, but the textures for the 16 windows surrounding the cockpit are the same as on the Behemoth, and the three black padded seats are the same as those in the cockpit of the Retro Rocket. On top of the cupola, there is a structure which is identical to the Retro Rocket's nose except for the texture and the pulsating blue glow around the narrow part below the sphere at the top.
Three partly diagonal, partly vertical struts with round feet make up the extracted landing gear. Also, glowing, translucent purple rings are slowly descending from the bottom of the ship to where the ground would be. Normally, four of them are visible at the same time.
Another much smaller flying saucer is hovering to the right of the Buck Rogers ship. It is simply named the Small Saucer. The very detailed texture on the main fuselage gives it a worn-out apperance. Above it, there is an egg-like cupola with the now-well-known window texture and 32 panes altogether, but the tiny vessel lacks an interior. It emits similar purple rings from the bottom as the Saucer, but they widen on their way down, and only two are visible most of the time.
The light-grey ship seen from behind below the Saucer is the Orion Jump Freighter, another cargo ship. Its rear is facing the on-looker with its rather inconveniently installed cargo hatch open, showing an arch-shaped black and yellow safety marking around its outer edge and granting an easy view into the cargo hold. The surface is mostly covered in identical rectangular texture panels which have rectangular safety markings with diagonal black and yellow stripes and rounded corners in two corners and slightly darker patches with one large circular structure between two smaller ones in the other two corners. The seams are shown as being covered with silver metal stripes.
Its fuselage mostly consists of two half-spheres, connected by a half-cylinder. The bottom halves of all three are actually flattened. The rear half-sphere contains the cargo hold. It also carries four fins with cylindrical engine nacelles at their end, two sloping sharply upward, two sloping slightly downward. The engines have white and orange textures on their rear ends. Between the upper nacelles, there is a rear wing much like on a sports car. The cargo hold contains the usual cube boxes, this time in various sizes. Together with the rear wing, they suggest that the ship has been made in jest.
The front section contains the spacious cockpit with three brown seats with black padding. It is half-surrounded by four rows of ten rectangular windows, re-using the same window texture yet again. In the middle of both the cockpit and the cargo hold, there is one column each carrying what might be a holoprojector. A semi-translucent cube is rotating slowly and clockwise above the one in the cockpit, but showing an animated standard plywood texture because the original texture is lost. Three dark grey ranged weapons are mounted next to each other under the cockpit.
The landing gear consists of three legs of the same kind as under the Behemoth, one under the cockpit, two under the cargo hold, but with different textures.
A rather unusual vessel, created in cooperation with Jim Carter again. is located in the background to the right of the Orion in the image. The Dromedary Super Freighter of the Galactic Trade Union is basically a space-going big rig. It doesn't have a fuselage in the traditional sense. Instead, it has a structure to which six standardised, octogonal cargo containers can be attached, two on each side and two on top. This structure also carries the six landing legs like those on the other two freighters which don't look like they can be retracted here. Three big drive engines are attached to the rear end, the middle one being mounted higher than the other two by a bit more than half its diametre. All four have the same white and orange textures inside their nozzles as also seen on the Orion.
The front, mostly obscured by the Orion and shaped like two joined trapezoids, looks more like the driver's cabin of a lorry. It is mostly red with one wide and two narrow stripes on the sides, the front and the roof. Said front is also adorned by two double headlights and a chrome-plated grille, all of which are part of the texture. There are several hatches and even two stainless-steel fuel tanks textured on the sides.
Access to the cockpit is granted by a door on the port side which leads into a room clad in red artifical leather padding except for the inside of the door and the ceiling which are covered in structured stainless steel. Other than the ladder that leads up to the actual cockpit, it only contains a bed. Likewise, the cockpit is mostly clad in the same padding except for the stainless steel floor. The single seat is the same brown seat as in the Orion's cockpit. The window texture has been re-used, too, but tinted red. On the roof, there is a communications dish aiming forward and slightly upward.
All six container bays are occupied. The front left container is orange. It shows a white label with the black writing "TITAN SHIPPING" and "UNIT 14 ANDROMEDA SPACE DOCK" on it. It is the only one without graffiti. The rear left container is dark red. Any writing it might have is covered in graffiti. The front top container is green with lots of graffiti on it and labelled "GRIDLAG SHIPPING INC." in a white stencil typeface. Grid lag refers to a phenomenon that might happen if a server running an OpenSim grid cannot keep up with its tasks. The rear top container is yellow; any labels on it are covered in graffiti again. The front right container is brown with lots of graffiti and "GREY GOO INTERSIM TRANSPORT" written on it in a yellow military stencil typeface. The rear right container is grey and unlabelled except for the number 3669 and the usual lot of graffiti.
Finally, standing on the grey ramp all the way to the right, the tenth and last spaceship is also the most unusual one. Its shape is mostly defined by a seemingly wild arrangement of ellipsoids, and its glossy surface texture is separated into panels with a large variety of "greebles" on them. Even electric wires appear to be strapped openly across the dome-like shape that makes up the top. The nose object from the Retro Rocket re-appears yet again with the same flashing blue light as on the Saucer, but since the object is much smaller now, the flashing is more easily visible.
The cockpit is almost entirely surrounded by black control panels with mostly white, red and blue buttons on them. It features a single medium grey, dark grey and dark blue seat. It cannot be accessed because the bottom hatch is too small; ironically, the open lid is way too big for the hatch. It can only be seen through the windscreen which is covered in a pattern of ellipses. One ranged weapon is installed on each side. Radiation warning signs on them suggest that they use something radioactive at least internally. The ship is standing on three strangely shaped and actually asymmetrical legs.
To the left, behind the rear end of the Retro Rocket, a bit of natural ground can be seen with the default diffuse green texture with irregular patches of light brown. It ends at a somewhat irregular and steep shoreline. Beyond it and all along the left-hand edge of the image, the ocean is stretching up to the horizon. It is mostly a bluish lilac with slightly less saturated reflections of clouds on the lower parts and a gradient towards purple near the horizon. The sky above it shows a gradient from a faint rose on the horizon to purple near the top left corner.
Close to the horizon, right next to the left-hand edge of the space cube, a single star is glowing.
The scene is illuminated by ambient light and by "sunlight" coming in from a low position opposite the display. There is no actual sun in the scene.
The second picture shows the second display of space-themed Arcadia Asylum creations. It has a direction of view which is straightly eastward and perpendicular to the walkways. These and the grey ramp area between the walkways and the space cube are basically identical to the first image.
The space cube is pushed back by about a metre and a half, uncovering the ground below which has the same mostly green default texture as the ground described in the first picture. It is the same space cube as in the first picture, but rotated counter-clockwise by 90 degrees. The Crab Nebula is on the left-hand side now, and NGC 604 is in the back.
On the right-hand side, there is a photograph of the Lagoon Nebula, also known by the designations Messier 8, Sharpless 25, RCW 146 in the Rodgers, Campbell & Whiteoak Catalogue and Gum 72 in the Gum catalogue of emission nebulae. It is a star-forming region in the constellation of Sagittarius. The nebula itself is greatly shifted to the left on the texture and thereby towards the background from the on-looker's point-of-view.
In comparison with NGC 604, the Lagoon Nebula appears much more as an actual nebula. The gases are more diffuse and spread more homogenously. The brightest spot is also known as the Hourglass Nebula or NGC 6523. On its left-hand edge, hardly separatable from the Hourglass Nebula itself, there is the very young star Herschel 36 which ionises the gases in the nebula and makes them glow, is left of centre in the picture, but far to the right in the nebula. The Hourglass Nebula shows a light teal glow. In turn, it is surrounded by darker clouds through which some more young stars manage shine, including smaller, very dark and dense clouds which are slowly collapsing under their own gravity and may eventually turn into stars themselves. Immediately to the right of the Hourglass Nebula and below it in a dark cloud, there are a few bits of nebula glowing purple.
To the left of the large dark cloud to the left of the bright centre, there is NGC 6530, a particularly striking open cluster of young giant and hypergiant stars which contribute to the brighter glow extending farther to the left than to the right. The cluster extends to the top left of the bright centre behind a very large dark cloud, contributing to the unbalanced spread of the glow some more. Towards the edges, the nebula fades through a faint taupe to Bordeaux red. The faint glow to the left of the star cluster NGC 6523 is designated as an individual nebula, IC 4678; a small part of it is cut off at the texture edge.
Outside the Lagoon Nebula, way to the right and above the bright spot that is the Hourglass Nebula, on the bottom right edge of a Bordeaux red nebula cloud, there is a bright star designated as HD 164385 in the Henry Draper Catalogue. Farther in the same direction are two more, less luminous stars. The brighter one is HD 314895; I couldn't find a designation for the one nearby to its upper left.
In front of NGC 604 on the rear side of the space cube, what appears to be an image of Earth edited into the texture towards its top left corner is an actual sphere with an Earth texture on it. It is oriented so that open international waters east or northeast of French Polynesia in the Pacific Ocean way to the west of South America are facing the on-looker.
On the edge between the walkways, a few metres right of centre, there is a teleporter that is identical to the one in the first picture. I myself have set it to the same teleport target as the one in the first picture so that I wouldn't have to write another description for an image within an image with over 4,000 characters.
On the far edge of the grey ramp, right in the middle, there is a vantablack info sign with yellow writing in the DejaVu sans Mono typeface on it which is similar to the one in the first place. This one reads, "Name: Space pirates by Aley Arai | Owner: Ada.Radius | Description: Astonishing artwork created in Second Life by the avatar known as Arcadia Asylum, Aley Arai, Lora Lemon, Aley and other alts, exported with the artist's permission by many volunteers to OpenSim." "Alts" are short for alternate avatars, avatars which users may have in addition to their primary avatar. In both Second Life and OpenSim, it is possible to have multiple avatars. The pipes in this transcription, the vertical lines, mark line breaks in the original.
To the left of the info sign, there is an info kiosk built by Arcadia Asylum and, like most other displays, published under the guise of Lora Lemon. It has a square footprint, and all four sides are identical. The uppermost part is a symmetrical trapezoid that narrows downward. Its top surface has a science-fiction-style "structured" grey texture which is also used on the sides of the lowermost part with vertical edges. Its sides show a trapeze-shaped image of a dense field of small asteroids in front of a star field, surrounded by frame in very light and medium grey, again, with a "structured", relief-like appearance. On the image, there is a writing in teal letters in a science-fiction-style, unidentified typeface: "The EXCITING History of Asteroid Mining!"
Right below is a semi-translucent cube that serves as an image display. It circles through 16 pictures and changes them every two seconds. All four sides always show the same image. Here it currently shows a picture of what appears to be a small mining vessel in an asteroid field. The latter is shown as about a hundred big and small rocks before a grey "haze" which is tilted from horizontal orientation to the left by about 30 degrees. The ship appears mostly black, so details are hard or impossible to make out. It has a vaguely egg-shaped main fuselage with a single ignited rocket engine nozzle at the back. It also has two probably cylindrical nacelles on the sides with rounded ends and position lights on their rear undersides, red on the left, green on the right. It is located way to the left within the asteroid field and aligned with it, the rear end facing the left, but it is rotated around its own longitudinal axis to the left by almost 90 degrees. It also appears to have made contact with an asteroid slightly smaller than itself on its bow, and the active rocket engine suggests that it is pushing the asteroid.
Between the screen cube and the lowermost box, there is another trapezoid of the same size and shape as the one on the top, but turned around with the small side up. On its top, it has a fairly abstract black and green science-fiction texture which can be seen through the screen cube and which is the same as on the bottom of the upper trapezoid. All four sides show a science-fiction-style control panel which is way too elaborate for an info kiosk with 45 square buttons, three round buttons, nine dial knobs, 34 rocker switches, a 3.5-inch socket and four black-and-mostly-green displays, including at least one radar. All displays are static.
The large display in the centre has a writing on it in the same typeface as on the trapeze-shaped picture on the uppermost part, but it is green and much smaller now. It reads, "80 years ago asteroid minng (sic!) was still a vibrent (sic!) buisness (sic!), With the advent of more advanced atomic fules (sic!) and subsequent improvements in intersteller (sic!) drive engines the need to refule (sic!) from asteroid minning (sic!) pased (sic!) into the glorious annels (sic!) of galactic hstory (sic!)... Blah blah."
The vantablack display info sign is also surrounded by altogether four robots. A worker robot labelled Sculpty Robot (Industrial Model) is standing between the display info sign and the info kiosk, facing the latter. It is similar to Asteroid Al in the first picture, but in a more simple posture, without the sleeveless shirt and with silver rings around the "eye apparatus" instead of golden ones. It seems to be the basic version of these robots. Apparently, it was released by Arcadia under the name Aley Arai.
The same applies to another derivative of this robot which is standing to the right of the display info sign, facing the on-looker. It is called the Bell hop bot (Bellbot). It is identical to the robot looking at the info kiosk except for a red cylindrical bell-hop cap with black and golden trim, a red bell-hop jacket with black and golden trim and many golden buttons, a white button-down shirt underneath and a black bowtie.
Another robot creation released under the Aley Arai guise is still in its original package and lying on its right side underneath the display info sign. It has a much more humanoid and actually female look. Its face is that of a human. She has the same eyes as the two other robots, only that they are elliptical now instead of circular. She has an ellipsoid on her head that stands in for hair; it has an angular cutout for the face all the way back to the ears which are built into the head. She has a more pronounced chest with two round features on the texture that suggest a pair of breasts, a slimmer waist below and a skirt-like attachment around her waist farther below. Also, her arms and legs are somewhat more human-like.
The package box has metal grey science-fiction outer textures all around except for the mostly transparent front where it re-uses the spaceship window texture. The inside is completely textured in what looks like aqua blue padding. There is an opaque, light blue rectangle extending from the left-hand edge of the window about 70% across and from right below the robot's knees to about a third of the way up to its thighs. It has the product name of the robot written on it in black letters in the same science-fiction typeface as used on the asteroid mining info kiosk. The upper line reads, "ROBOMAID-3000", and the lower line reads, "ROBOT DOMESTIC".
The gap between this rectangle and the right-hand edge of the window is closed by another, upright rectangle in the same tone of blue which extends beyond the upper and lower edges of the first rectangle by 75% of the latter's height. It shows the logo of the manufacturer. In the middle, there is a black rectangle with rounded edges surrounded by a thin black line with a light blue robotic hand modelled after a human's left hand in front of a background of zeroes and ones in dark grey digits in an unidentified monospace typeface. "Positronic" is written above the rectangle in what appears to be a bold, italic and condensed Helvetica variant except for the quite stylised, science-fiction-style capital P. Below the rectangle, "ROBOTICS" is written in a different, unidentified sans-serif title typeface. All this writing is in black again.
Below the first rectangle, there are two barcodes. The one to the left appears to follow the Universal Product Code standard as per International Standard ISO/IEC 15420. The code number is 1-33023-81220-0. To my best knowledge, it is either a fantasy code, or it has expired years ago. The barcode to the left is a fantasy code that makes use of what appears to be a 16-colour scheme in three rows with various heights.
At both the top and the bottom of the window, "CAUTION HANDLE WITH CARE" is written in a stencil typeface which can also be seen on one of the containers on the Dromedary Super Freighter in the first picture, but in yellow with thick black outlines. On both sides of each writing, there is a triangular caution sign with rounded corners, a red frame and a black exclamation mark in front of a yellow centre.
The fourth robot in the group, released under the Aley guise again, is another female robot and much more well-known. It is modelled after the robot Maria from Fritz Lang's 1928 silent movie Metropolis. Unlike the other robots, it looks like it was entirely made from brass. It has also got a very detailed body instead of details drawn onto the textures. It is standing to the right of the Bell hop bot on a trapezoid pedestal which has the official logo of the film on the front, surrounded by a golden rectangular frame. The pedestal is partially sunk into the grey ramp below, however. This robot was actually made to be used as an avatar. The avatar variant can be acquired for free elsewhere, and yet another place which I've described several months ago uses it as an automated non-player character.
Above Maria and slightly to the right, a spherical contraption called Confederation Police Drone is hovering, another product by Aley. It is mostly spherical with bright yellow-orange cylindrical extensions at the top and the bottom, similar to the red ones on Widget in the first picture. Again, only a low-quality version of its main body texture seems to have been preserved. It is mostly Prussian blue. Two stripes full of technological "greebles" with dark grey trim along both edges surround the body below the top and above the bottom. One night blue stripe each goes around its vertical axis and its longitudinal axis with black, silver and black again trim along both edges. The "greeble" stripes cut through the blue stripe around the longitudinal axis, but their dark grey trim doesn't.
On both sides, where the blue stripes meet, there is a circular police badge with medium blue as its basic colour. It has a multi-layer frame around its edge which starts with a thin silver grey or bright golden circle, followed inward by a gradient from what appears to be dark grey to medium grey, followed again by a thick golden inner frame which appears to be elevated due to the shading on its edges. The same applies to the other golden elements in the badge. There is a star with four points at the top, at the bottom and on the sides which connect to the thin outer ring of the surrounding frame and cut through the elevated golden ring and four less-than-half-sized points between them. In the middle of the star, the outlines of the number 42 are cut into it. "Confederation" is written across the top of the badge and "Police" across its bottom, both cutting through both the inner frame and the star. "SECURITY DRONE" is written in two lines on the blue stripe to the left and to the right of each badge. All writing uses the previously seen science-fiction typeface again.
It has three horizontally arranged identical weapons at the front. What kinds of weapons they are remains unclear. Otherwise, it doesn't have any farther attachments.
Farther up and fully inside the space cube, there are also four different spaceships. The highest up, right in front of the nebula NGC 604 in the background and the farthest back in the space cube, hovers the Pirate Shark Ship which belongs to the Galactic Trade Union theme. It is pointing past the on-looker to the left. Its fuselage is mostly cylindrical with semi-spherical ends. On the top and slightly below the middles of the sides, three fins are mounted, backswept like those of a shark. Each one of them has an engine installed near its end. The engines have ellipsoid, hollowed out casings with a funnel on each end. The funnels have both the same chaotic blue and teal texture which might originally have been made for water surfaces. Here, however, it is animated. At the front of the engine, it moves inward, at the end, it moves outward. Also, each lateral fin has three dark grey ranged weapons mounted on its front edge.
The name is justified by the front design. From the upper half of the frontal semi-sphere protrudes an ellipsoid that emulates the nose of a shark. On each side of the nose, there is one bulgy, menacing eye with a red iris on a black eyeball mounted on the seam between the cylindrical mid-section and the sphere. Below the nose, on the frontal semi-sphere, there is a shark maw with two rows of pointy teeth on a dark red background which is shaded in order to appear like it's actually built into the ship rather than painted on. And on both sides of the maw, there is a dark grey structure that resembles a cogwheel with no hub which might even imply that the shark maw can be opened. In spite of the many "greebles" on the grey texture that covers almost the whole ship, it doesn't seem to have any hatches or other openings.
Also, it doesn't have an interior. The cockpit is not much more than a cupola on top of the ship, mostly on the nose, with four opaque black panes, one at the front, one at the back, one on each side. Right behind the cupola and in front of the top fin with its engine intake, a small communications dish is rotating clockwise, about once every three seconds, slightly tilted upward.
Both sides of the cylindrical mid-section are labelled. In the middle, there is a white human skill in front of two crossed sabers with black outlines in front of a black playing card spade with a white outline. Next to it, towards the bow, there is a sign made of three black chevrons with cream outlines, one from the top, two slightly tilted from the sides. On the other side, towards the stern, there is a hexagonal badge with black and yellow chevrons and a black outline which has previously appeared on the Orion Jump Freighter in the first image. The chevrons point towards the bow in the usual direction of movement. "PRIVATEER SPACE PIRATES" is written above these three emblems and "RESISTANCE IS FUN!" below them, both in the now usual science-fiction typeface and in black with beige outlines.
The tail consists of a cylindrical cone with a circular cross-section that is mounted against the middle of the rear semi-sphere plus two thin backswept hollow fins with two pointy ends each, one at the top, one at the bottom. Each fin has another one of the same engines installed as the fins on the middle fuselage section. On each side, there are two yellow lightning bolts pointing backwards; the lower one is a mirrored variant of the upper one.
There is no landing gear on the ship.
The ship below the middle that appears to be almost immediately below the Pirate Shark ship is actually a lot closer to the front. It is the Sloth Armored Transport, another ship from the Galactic Trade Union theme.
Its main fuselage section is about three and a half times as long as it is wide or high. It is octogonal from all sides, but differently from ahead or astern than from above or below and from the sides where it has the longest stretches of diagonal shape. It carries two large brackets on each side, consisting of a thick vertical connector plate and two thinner fins protruding from them diagonally which carry the actual turbine-like drive nacelles with the same white and orange rear nozzle textures as the Orion and the Dromedary, one from the top, one from the bottom. The fins also cut through one curved shield-like plate on each side which seems to be part of the brackets, and which seems to imply that the drives are not to be trusted. Most of the outside shows a texture that emulates slightly rusty steel plates with sometimes widening gaps between them.
The rectangular cargo hatch at the end is open, and since the ship is seen from behind, it reveals the interior which mostly shows a different, lighter, embossed texture. The same arrangement of twelve cargo boxes shaped like slightly flattened cubes is standing on each side, but rotated by 180 degrees towards each other. Only the boxes on the port side are visible in the image. Towards the stern, there are four stacks of four boxes each. Farther ahead, there are only two boxes. There are four different boxes, all with the same texture on all sides, arranged in alternating pairs in all directions.
Within the rearmost stack, the box at the top and the second one from the bottom have lids with a picture of a planet apparently made of melting cheese below the centre, slightly cut off at the bottom. A sleek, dark golden flying saucer comes dashing around it from the left, leaving a golden trail behind. Another mostly green flying saucer is in the top right corner, surrounded by a magenta halo and with a beige trail behind it. The planet is surrounded by an arch of golden embossed letters in a serif typeface which spell out, "PLANET CHEESE".
The other two boxes in the stack seem to be held shut with two yellow rectangular brackets riveted around each corner. In their middle, there is a rectangular logo with a medium blue background and a frame with diagonal black and yellow warning stripes around it. Towards the top right corner, there is a somewhat large grey cogwheel with six spokes and black outlines. Another slightly smaller and lighter cogwheel connects to it from the opposite corner. In the top left corner, there is the capital letter "C"; in the bottom right corner, there is the capital letter "G". Below the logo, "CAPITAL GOODS" is written. All writing is yellow and in a stencil typeface.
The boxes ahead of each Planet Cheese box have lime green corner reinforcements. Most of each side shows black and yellow diagonal warning stripes. In the middle, there is a circular logo with a faint golden frame around it and a crossed shovel and pickaxe together with three identical cream-coloured dodecahedra standing in for rock pieces on it. In black letters in the usual science-fiction typeface, "PROCESSED" is written above the logo and "ASTEROID ORE" below it.
The remaining boxes ahead of each Capital Goods box and above and below each processed asteroid ore box appear like they've got internal cooling systems and bolted-on lids. In the middle, there appears to be a window surrounded by black and yellow diagonal warning stripes through which various food items inside the box can be seen; this is of course only part of the texture. On each side of this window, "REFRIGERATED FOOD STORAGE" is written in yellow stencil letters. The writing is always vertical; to the left of the window, it is rotated clockwise, to the right, it is rotated counter-clockwise.
Ahead of the cargo hold, there is a box-shaped "neck" with the cockpit in a structure that can be described as a transversally-mounted barrel. Across the front, two rows of eleven windows using the standard space ship window texture allow for views inside or outside. The cockpit itself has two seats. The control panel ahead of them and below the front window is animated. Underneath the cockpit, there is the ship's only landing leg which is identical to those of the freighters in the first picture. Apparently, the ship uses the lower engines and the curved shields as rear landing gear.
Towards the right, below the Lagoon Nebula as seen in the picture, there is a smaller freighter with a quite angular fuselage whose most striking features are the four big engines protruding from the stern, each again with the white and orange textures. It is called the Tarsus II Tramp Freighter, another Galactic Trade Union ship, the smallest one of the cargo ships and the only one with an elaborate interior, but no access from outside. Its outside texture has lots of hatches, black and yellow safety markings, vents and other "greebles", but it does not have any door or hatch.
Most of the trapezoid bottom of the fuselage is used as the cargo hold which contains six boxes, two Capital Goods and refrigerated food storage boxes each and one Planet Cheese and processed asteroid ore box each. The top part is a bit more complex in shape. At its front, there are two rows of five windows for the cockpit; the usual window textures had to be modified for this use-case. On each side, a ranged weapon is mounted on the angular seam between top and bottom. A static dish is mounted on top of the ship.
The upper level, connected to the cargo hold via a ladder, has to be the most comfortable one on all ships. Apart from lots of control panels and the same brown seats with white seams and black padding seen on other ships before, it has a double bunk bed, and it is also equipped with the column of touch panel, microwave oven and Mr. Coffee also found inside the Behemoth freighter in the first picture. Camming through the ship would reveal a bathroom with a sink, a toilet and a bathtub filled with a slowly rotating bubble bath. Outside the bathroom, the upper has even got a patterned green carpet on the floor.
Last but not least, there is what may be the most unusual vessel of all. Fairly down low and a bit to the left, seen mostly from behind again, the Slee Banana Cruiser 2 is tagged as published by Aley again. Its main fuselage is yellow and actually shaped like a banana, riveted together from steel plates. At the rear end, there is a rocket nozzle which consists of three riveted funnels nested inside one another. The fin shape from the Retro Rocket in the first picture is re-used for one single fin on top of the nozzle, but with a simple brushed metal texture now. The front is decorated with a rusty figurehead shaped like a stylised bald female human.
The wings re-use both the rear fins from the Pirate Shark Ship and, like most of the ship, the riveted steel textures from the Retropolis retro future spaceship in the first picture. Each one holds a hexagonal vertical pod through its leading edge. From each one of these pods, a pair of double-bladed fans like on a gyrocopter protrude upward on a long shaft, and what gives the impression of a machine gun is mounted underneath. The guns seem to also double as landing gear.
The cockpit is inside a yellow sphere which holds onto the banana below with two humanoid arms. The lower half is made of riveted steel on the outside and covered in control panels on the inside. The upper half uses the window texture for a strangely asymmetrical arrangement of 15 windows. Inside, there is a padded red seat behind a hooded yellow instrument panel with a yoke.
To the left of the space cube, a bit of the neighbouring space cube from the first picture can be seen with a small part of its Crab Nebula texture. To the right, there is another bit of land, sea and sky as already described in the first picture. In the top right corner, there is a bit of a cloud. One particular bright star is shining right below the cloud and a smaller one farther below on the edge of the space cube.
Again, the scene is illuminated by ambient light and by "sunlight" coming in from a low position opposite the display. There is no actual sun in the scene.
The third picture shows the third and last display of space-themed Arcadia Asylum creations. Its direction of view has turned southward, but it is perpendicular to the walkways again. These and the grey ramp area between the walkways and the space cube are basically identical to the other two images.
Unlike in the second picture, but like in the first one, the space cube aligns with the ramp. It is placed with the same orientation as in the second picture, but it is being looked at from a different side now, thus showing NGC 604 on the left, the Lagoon Nebula in the back and the mirrored and edited picture of IC 434 with the Horsehead Nebula on the right.
On the edge between the walkways, a few metres right of centre, there is the same teleporter as in the other two pictures again. And once again, I have taken care that it also shows the same teleport destination.
To the left of it, there is a white sign which can also be found between the other two displays, but which cannot be seen in the other two pictures. It is white with black writing on it. It counts down Arcadia Asylum's main avatar names in Second Life and the themes associated with them: "Arcadia Asylum 2006-2007 | Slum City/Urban Blight, Hobos, Street Urchins, Subway System, Modular Sewer System, Greenies, Loli Caverns", a blank line following, "Aley Arai 2008-2011 | Space Pirates, Robots, Space stations, asteroids and wormholes", another blank line following, "Aley 2011 - 2015 | Flotsam Pirate Town, Pirate ships, The Abyss, Nemo's world and submarine, Mer, sunken ruins, Clockworks, Mad Scientist, Aquatics, Steam Fair". Again, the pipes in this transcription, the vertical lines, mark line breaks in the original.
The vantablack display info sign is in its usual place in the middle near the far edge of the grey ramp. This time, it reads, "Galactic Truckstop and Honest Zorg's by Aley | Owner: Ada.Radius | Description: Astonishing artwork created in Second Life by the avatar known as Arcadia Asylum, Aley Arai, Lora Lemon, Aley and other alts, exported with the artist's permission by many volunteers to OpenSim.." Once again, the pipes in this transcription mark line breaks in the original.
The centre-piece of this display is the Galactic Truck Stop in the back and almost in the middle. Its most striking feature is the two-level cupola on top. It uses the same texture for its ten rows of windows as some of the ships in the first two pictures use for their cockpit windows. The top level is a social area with eight dark grey armchairs. The level below has eight sleeping compartments, each with a double bunk bed, a hard-to-define piece of furniture and a scripted sliding door. Still, all these rooms have "glass walls" to the outside all the way to the floor, no curtains and illuminated ceilings.
The structure extends downward in a cylinder with a smaller diametre than the cupola. Directly below the cupola, there are four structures which appear to be fuel storages. They have a label on the side which consists of the Shell logo, a white rectangle below with a red and yellow logo that might be original and "FUEL" written under it in yellow letters with red outlines in a previously unseen science-fiction-style typeface and finally another upside-down Shell logo below that. There is a large porthole divided into 25 panes below each fuel storage.
Between the portholes, there are cylindrical airlocks protruding with two sets of doors. Each airlock leads to a landing pad shaped like a section of a circle. On the edges and towards the airlocks, these pads show striped black and yellow safety marking. The landing areas themselves are marked with white circles from which short white lines protrude into four directions. Inside the white circles, each landing pad has a label with an individual number, always written in red narrow and stencil-like characters with the number at the top, "PAD" below and a horizontal line between the number and the word. The landing pads on this level have even numbers; the ones towards the fronts are numbers 2 on the left and 4 on the right whereas the ones in the back are numbers 8 on the left and 6 on the right.
On the inside, this level contains a diner with a robot similar to Asteroid Al and two tables with four seats each. By default, when trying to enter any of the airlocks by clicking on them, one ends up sitting on one of these chairs.
The bottom level has its own set of four shorter landing pads. These, the corresponding air locks and the portholes between them are offset from those one floor up by 45 degrees. They have odd numbers; number 7 is oriented towards the on-looker. The portholes even have fuel storages above them again which are hanging underneath the upper landing pads. The airlocks appear to be standing wide open because the doors are misaligned; the outer door for airlock number 5 to the left even lacks textures. They are scripted nonetheless.
On the inside, the bottom level holds a big souvenir and duty-free shop with Asteroid Al himself behind the counter as the shopkeep.
All levels inside the station have brownish metal panels with lots of small slot-like holes in them on the ground. They are connected via a lift with red buttons inside which is actually a set of teleporters in disguise.
Above each airlock, the writing "GALACTIC TRUCK STOP" in the same fashion as on Asteroid Al's shirt in the first picture re-appears, but without the picture. In the typical science-fiction typeface from the second picture and the same tone of purple, "Last fule (sic!) n Restrooms for 20,000 Lightyears" is written below. The writing is not connected to anything.
Way to the right and near the front, there is the other part of this display: Honest Zorg's Used Spaceships. It is located on a small, slightly irregular round asteroid with a brown, crater-littered surface and an almost flat top. Interestingly, even the flat top has craters on it, so it came to exist naturally rather than through mining.
Four riveted steel poles are holding up two strings of pennants that surround the dealership. It's a pattern of 16 pennants in five colours: blue, white, red, yellow, green in this order. The 16th pennant to fill the number up after three repetitions is red again. The front left and rear right poles also carry signs painted onto riveted steel plates with the same structure texture as on the Dromedary on the back. The larger part of the sign with a slanted right-hand edge and a pointy upper right corner is painted denim blue with "HONEST ZORG'S" written on it in a heavy stretched Futura typeface and a thick yellow arrow pointing downward on its left-hand side; both are yellow. To the right of the arrow, there is an additional red and rectangular panel with the red writing "USED SPACESHIPS" on it which is made to appear like neon lights.
Within the strings of pennants, there are six small vessels which appear to be personal spacecraft, three on each side, most of which have rather odd shapes. In the back to the left, there is the only ship with transparent windows, another small flying saucer, which doesn't have any cockpit interior, though. Opposite of it, a rocket-style ship stands vertically on eight rear fins. It doesn't even have a cockpit, and with the sleeve around more than half its length, it rather resembles a big turbojet engine with a long pointy nose. The almost white ship to the left in the front has a chubby, mostly ellipsoid fuselage, a "collar" behind its small cockpit and eight tail fins. The oddly shaped one opposite of it in various tones of grey with a cockpit cupola similar to that on the Pirate Shark Ship has even got 16 tail fins. All these ships have in common that they lack visible drive components; this only doesn't matter for the flying saucer. But they all lack landing gear as well. Except for the upright rocket, they are sitting on their bellies.
The office in the back is basically a Retro Rocket Ship as seen in the first picture. However, its wings were cut off with no traces of their whereabouts. The rocket engine was removed; it is standing in front of the ship in two parts with a picnic table in front of the engine. The door is not only open, but dislodged and lying on the ground. The separation wall towards the cockpit including the doors is missing, as is one of the doors to the former crew compartment where the bunk beds have been removed, too. The former engine room is the office with the sales counter now. Honest Zorg turns out to be a grimy worker robot based on the same shape as Asteroid Al again. And the whole ship is slightly tilted forward because it seems to have dug itself into the ground, somewhat like in a crash landing.
There is a sign above the door which is completely obstructed by the dealership sign at the front. Through the porthole ahead of the open door, the bright teal sign on the counter is partially visible. In the same slightly stretched Futura as on the sign on the pole, but in blue, it reads, "Welcome to Honest Zorg's! We sell the finest in used and reconditioned Spacecraft". Below, in red and a bit smaller, there is added, "This is NOT the complaints office!!!" On top of the counter, the dark grey cash register can partially be made out.
This time, the image was taken in such a way that nothing is visible to the sides of the space cube. The scene is illuminated by ambient light and by "sunlight" coming in from the right. Due to the space cube not having visible outside textures, the directed "sunlight" can fall in through it, so, for example, parts of the Galactic Truck Stop or the ships at Honest Zorg's cast shadows. There is no actual sun in the scene.
#OpenSim #OpenSimulator #OpenSimFest #OSFest #OSFest2023 #ArcadiaAsylum #SecondLife #Metaverse #VirtualWorlds #VirtualPhotography #ScienceFiction #Spaceships #Long #LongPost #EyeContact #AltText #ImageDescription #ImageDescriptions -
Why is the Universal Credit website so bad?
I am professionally embarrassed for whoever created the Universal Credit Website (UCW). In this post, I will explore why the website is so bad and what should be done to fix it.
I’m writing this in the hope that someone within the government with the power to do something reads this and implements at least some of the changes that the site is so desperately in need of. (The same reason I analysed the Boots website). There is a summary of changes needed at the end of the article.
Legal disclaimer: I’ve used screenshots of the website. This is legal because (1) fair use and (2) OGL.
A brief intro to Universal Credit
Universal Credit is the UK’s highly criticised unified benefits system for those out of work or currently unable to work. In theory, it was also an income support for those just starting out in self-employment. It was the brainchild of the political right (the Tories). It has an unbelievable number of shortcomings. Many hope that Labour will refine or replace it but I’m not holding my breath on that one.
Claimants of Universal Credit (UC) must make all contact through the Universal Credit Website. Are you old and have never used a computer? Too bad.
You’d think that such a key instrument of the Department of Work and Pensions (DWP) would be fit for purpose and well designed. You’d be wrong to think that.
The DWP are still migrating everyone who claims any sort of assistance to UC. This, I think, is a terrible idea. Not only because UC is a mess but also because the website barely works.
Dangers of the Universal Credit Website journal’s lack of app
The DWP has not (as far as I know) made an app for UC users. So guess what the scammers made. Go on, guess, I can wait.
Yep:
People are being warned about a scam involving a fake Universal Credit app and text messages.
The app is called ‘Universal Credit UK’.
The Department for Work and Pensions is investigating, and is encouraging people not to use the app or respond to any suspicious text messages.
People should use only the DWP Universal Credit website, and if you are unsure how to claim, the Citizens Advice Help To Claim service can offer support on 0800 144 8 444.
Anyone who has given information to the scam should report it to Action Fraud on 0300 123 2040.
Warning over fake Universal Credit app and text messages, CAB Hull and East Riding
I’m going to come back to this topic towards the end where I get technical about how to fix the UCW.
Shortcomings of the Universal Credit Website’s User Interface
There is so much wrong with the UI of the UCW that it is hard to know where to start. It is clear that the site has been created for the benefit of the UC’s management and agents without any consideration for the people it is supposed to serve. This is a common failure by businesses that are all set to fail – they create things for their needs and not the customers’ needs. You’ll see some clear examples of this soon enough.
It was reported that about one-third (32%) of benefit claimants being migrated to Universal Credit failed to make a valid claim.
Statistics released this week by the DWP show that 32% of all claimants sent a universal credit (UC) migration notice up to the end of February 2024 failed to make a successful claim and had their legacy benefits terminated.
In total, a shocking 284,660 individuals did not make a valid claim and had their benefits stopped.
I strongly believe that a large part of the problem is the shockingly bad Universal Credit Website (along with the fact that UC is badly implemented as a whole).
The Universal Credit Website is a terrible CRM
This website that all Universal Credit claimants must use to communicate with the DWP is essentially a CRM (Customer Relationship Manager). CRM software is at the heart of most successful businesses. The UC one is possibly the worst CRM I have ever witnessed.
You could have slapped up an instance of WordPress running a free CRM plugin and gotten something better.
I’m going to show you many of the ways in which the UCW Journal fails at being a simple CRM thing. In short, the design, implementation, and customer interface is a giant steaming turd. I can find, at a push, one nice thing to say about it.
Misleading landing pages
If your session times out and you sign back in, this is what you see.
It looks for all the world like you have to recreate your account. It is only when you more closely examine the page that you find the link to sign in to the UCW. Like, wait a minute Scoobie, didn’t we just sign in to that?
This is either an example of incompetent design or deliberate arse-hole design. Either way, it can only serve to confuse people leading, I have no doubt, to cancelled accounts or sanctions. There is no excuse for this complete failure.
Unclear priority elements
This is what you might see once logged in for real. (I’ve redacted any specific or personal information).
Can you tell just by looking at this what the most important section of the site is? The part you must pay special attention to in order to keep your claim live?
If you said, “Journal” then you are already ahead of the curve. That link top right in a grey box is the thing that matters most. Never mind that grey is UI shorthand for unimportant or disabled.
Not to worry, there are eight boxes lower down. One of which also takes you there. Maybe it is just me but my automatic ad-blindness had me ignoring all of that advert-looking crap until I started writing this article. The UCW “Journal” is a sort of half-arsed CRM (customer relationship manager). As a CRM the Universal Credit Journal fails so hard that I’ve given it its own subheading. A lot of this post will talk about how hard the journal system fails.
The Universal Credit Journal language is all wrong
There is a way to use language such that a task that is needed and a task that has been completed are easy to tell apart. Universal Credit’s designers clearly wanted to have nothing to do with these good design principles.
Someone needs to go back to school and learn that English has tenses (past, present, and future)
Take a look at this and tell me if the two blue links are tasks pending or tasks completed.
If you said tasks completed, you would be right. If you said tasks pending I completely understand. That’s because in English “Report a change” is an order or a thing to do while “Reported a change” is something that happened in the past.
In technical terms, the tenses of the entries are incorrect. I’m a native English speaker and long-time technical user who expected bad UI and even I was unsure. Now, you could blame this on my dyslexia and you may be right to do so. However, if I (a talented geek and native speaker) struggled with the incorrect tenses used here how much more would non-geeks and English learners struggle?
How hard would it be to change “Tell” to “You told” and “Report” to “You reported”? Or, if feeling lazy Prepend “Completed:” to the linked entries? It is only now that I noticed the first one ends with “completed” (the second one does not). If I didn’t see it, who else didn’t?
Send is not the right word here
When I ask for comments on my blog I do not say “send me your comments”, I say “leave me your comments”. That’s because “send” implies transmission not interaction. So, why-oh-why does the reply link say “send reply”? I’m not emailing you, I’m leaving a comment.
Needless priority given to dates
Let’s look at that Journal thing again and tell how important those dates and times are:
The date and author of the entry are what is known as metadata. That is, data about the data. The only key element here is the entry itself. Yet the entries get about half the space as the rest is eaten up by metadata which is treated as equally important.
It looks worse on mobile.
Vital calls to action are unlinked
Let me quickly introduce you to a business term – CTA or Call To Action. A CTA is the thing that you want the person to do in response to what they have read. My CTA is usually, please leave me comments because I love comments.
In this example, the CTA is to carry out a “to-do list” action. Failure to do so means loss of money.
The link, however, is for leaving a reply. Without scrolling back up to see the rest of the UI, can you tell me how to do a “to-do”?
Those of us used to dealing with shite websites could probably find our way to the top and find the other greyed-out link (grey means unimportant remember). Click that and then scroll through any pending items and find the one that means you get paid. I guess the design principle here is: If you want this money, you had better work for it.
On social media, when they want you to fill out your profile, they will show you a CTA with a link to the place where you do the thing. This is because they know that you link vital actions rather than forcing users to go searching for them.
Universal Credit leaves its users searching. No wonder it was reported last year that 900,000 Univeral Credit cases were closed by the DWP
The reading order is potentially confusing
Take a look at this journal entry and see how quickly you can work out if the user needs to go to an appointment or not.
This is where the user is expected to regularly check for important updates.
I’m saying that this UC Journal has a wildly inappropriate reading order. The first thing you see as you scroll down is an appointment has been cancelled. Oh, you might think, did I have an appointment? There was, you learn, an appointment made the day before.
There is, as far as I have been able to learn, no way to connect one journal entry with another.
Universal Credit is not Twitter, FFS!
For no reason that I can ascertain, replies have an arbitrary limit.
If I have several pages of information to share, should I end each section with “1 of 6”, “2/6”, “3/6”, etc.? All that would do is make the textual content hard to follow because the DWP UC “Journal” is not threaded nor is it object-grouped. It’s just a wall of text in reverse chronological order.
The Universal Credit website is buggy as hell
Then there is this shit.
When you click in to “send reply” the whole page flips out. Would you like to see that in a small hand-held viewport?
I hope you like scrolling.
This is not even a new problem. People were complaining about the UI back in 2017
Application online is not user freindly and is quite a messy website. Gov id verification are complicated and have system failures and required me to phone the Gov id verify contractor for technical support. Universal credit website need to be simplified and made user freindly. The goverment id verify contractors will boil yourr blood by asking you to fill in your personell details at least three times, its mad.
Written evidence from Miss Amina Khatun (UCR0073), committees.parliament.uk, October 2017
Yes, that quote is part of the official parliamentary record. Legally speaking, the DWP “knows” there is a problem. I mean, come on, surely they have gotten the message by now, right?
You cannot attach evidentiary documentation
Have a report from a doctor that may be relevant to your UC case worker? Shall I tell you how you can show it to them? Go on, guess.
Did you guess, attach it to a journal entry? Oh, you sweet summer child, no, the journal cannot take attachments. If you want them to see the document, you had better print it off and get a face-to-face in-person interview where they will not make any record of the document at all.
As I have already said, the DWP UCW is an incredibly poor CRM.
Not that I didn’t try (and found more bugs)
I tried to add medical information to the Universal Credit website. I, quite wrongly, guessed that “change of health info” was the right path. It was not. The account now has a pending “to-do” (complete the change of health thing even though there is no change to report).
Bug: It is impossible to cancel an incorrectly started task.
The Universal Website was created for the DWP’s requirements, not your needs
The purpose of the site is to make users jump through the hoops that the DWP (Department of Work and Pensions in case you forgot who those clowns are) demands. As I hope I have shown, they make no effort at all to assist you in this. If you cannot use their CRM, too bad, I guess.
I can think of only two reasons to implement the UCW like this:
- Gross incomitance
- Actual malice
With the DWP those two are hard to tell apart.
The Universal Credit Website is unfit to be the only point of contact; especially for older people
There is so much more I could complain about from an end-user perspective but I hope this brief overview of the UCW’s failings gets my point across.
I’m a technology guy. My profession is making things like this. I struggled to use the system. How the heck would my dad cope? He can barely use his (non-smart) mobile phone and needs help to read his text messages.
In May 2021, Helen Undy, Chief Executive of Money and Mental Health said, “People who need help with Universal Credit are being #SetUpToFail”. Here’s the link if you don’t believe me.
In the last five years, it looks like very little has changed.
I therefore put it to you that the Universal Credit Website is unfit to be the only point of contact; especially for older people and those with limited computer literacy.
The Universal Credit Website HTML and (lack of) web standards
This is the part of the post where I start to get a bit technical as I answer the question I asked at the start – Why is the Universal Credit website so bad?
I’ve already covered the weak and broken UI design approach. In this section, we are going to talk about web standards.
The journal looks the way it does because, for some unknown reason, someone decided to use HTML tables for layout. It is a web standard that tables should be used only for tabular data.
This is semantic, not tabluar data
Whatever junior dev wrote this may have used tables because the data came from a database table and they did not know any better. However, this is not tabular data. Let me give you all the reasons why this is not tabular.
- The journal entry is content and the rest is metadata for context.
- You did not intend for us to sort by date or author
- There is nothing to take a sum, average, or other calculation from
- Some of these have subforms (send reply) that break tabulation
- It would look nicer if the metadata were stacked on smaller screen sizes
- The table imparts no additional data to screenreaders
- CSS grid is faster, harder to break, and easier to make responsive
Tables for layout are dumb
This is what the experts say about using tables for layout:
It was common in the early days of the web to use tables as a layout device. Before the advent of modern standards-based browsers, this was the easiest way to make sure that page elements were arranged properly on the screen.
This design pattern is now considered very bad. It is bad for the user experience, bad for SEO, and bad for developers who have to maintain pages.
You should not use table-based layout under any circumstances.
HTML Tables: Find Out When To Use Them (And When To Avoid), Adam Wood, html.com
Here’s another that explains why HTML table layout is bad for end users:
1) Tables shouldn’t be used for page layouts because they are:
- Slow to render as the browser needs to download most – if not all – of the table to render it properly
- They require more HTML than non-table layouts which means slower loading and rendering, as well as an increased bandwidth usage
- They can be a nightmare to maintain as they can quickly get complex
- They can break text copying
- They negatively affect screen readers and may make your content inaccessible to some users
- They are not as flexible as using proper semantic markup
- They were never intended to be used for page layouts
- Making tables into a responsive layout is very difficult to control
2) Use a table for tabular data. That’s what tables are for.
Let’s break down a few of these:
Tables negatively affect screen readers and may make your content inaccessible to some users
A good chunk of the users of the Universal Credit website will have long-term disabilities. Those who rely on screen readers are likely to face a nasty mess as they try to use the website.
This is likely in breach of the UK’s Disability Discrimination Act 1995 and the Equality Act 2010. I am not a lawyer but activist groups might want to look into this.
Tables are not as flexible as using proper semantic markup
This is why the site looks god-awful on mobile. Using any one of the many flexible standards could make the journal resize in a useful and pleasing way on smaller screens. As I have said, the date and author are metadata and have no business taking the prominence they have been given. As the journal entry is the salient thing, the journal is NOT TABULAR DATA.
The nineteen-nineties called – they want their web design back.
Tables are a nightmare to maintain as they can quickly get complex
This is why the reply thing is broken. It has the wrong number of table divisions which breaks readability and looks bloody stupid on a government website.
Tables are slow and use excessive data
This is a website for people who have limited spending money. Why then, choose a markup that burns data faster than needed, loads slowly, and is frustrating to use with disability support software?
Tables are for data only – not journal entries with sub-forms
I put it to you and the DWP that using tables in the UCW at all was a failure in design, planning, and implementation. Replace the tables with nice semantic div tags, some CSS flex-box, auto margins, and responsive design. The metadata should stack under or over the entry on smaller screens.
This is beginner-level flexible and accessible design stuff. Not to mention the DWP is, in my opinion, failing to uphold the laws about accessibility made by the institution (the government) that they work as part of.
I’m not going to teach the DWP how to do this properly. They should know that if you pay peanuts you get monkeys.
The weird thing is the DWP do it right elsewhere
On the UCW home page (even though it defaults to to-do) the supplementary navigation items are responsive and stack appropriately on smaller screens.
If you can do it on the home page, why not on the Journal too?
The Universal Credit CRM is missing a few things
I’ve said quite clearly that the UCW is a terrible CRM example but it can be fixed. We just need to add a few features.
No tables, darling
I covered this enough, I think.
Threading or object grouping
The journal is currently a list of unconnected text things. You can do better than this.
As it stands UC journal users must scroll back down the page and take a guess from context alone what the entry refers to. How much easier would this be if things were grouped with or threaded under related content?
When you reply to something in the UC Journal, your reply is added to the top without any indication of what you replied to. The same is true of the replies from the UC person responding to you. This renders the entire journal almost impossible to use to keep track of a conversation thread.
Modern email clients solved the threaded discussion issue a long time ago. Replies are grouped with the email they are replying to. Thus the conversation thread can be followed. We do this in forums, and in comments, and on social media. Topic threading is not even remotely new.
That’s not even the only way to do things. There is also object grouping. An object can be a task, a record, a question, a ticket, or whatever. All content generated is assigned to an object. For example, if I open a ticket with my ISP about an issue I am having, we can both see what the issue topic is and the history of the conversation along with any actions taken.
On a blog, the post is the object and the comments, replies, and mentions are listed under it. Thus we know what the response was responding to. Forums do that too.
Look how easy this conversation is to follow (no tables used):
I cannot grep this logfile (where’s the flippin’ search option?)
The Universal Credit Journal system reads a lot like a poorly implemented log file. That’s how we store debug information when doing things like testing (which should have caught the deformed page on reply bug before it went live).
Grep is a command line utility for searching text-based log files. I’m using it as a nerdy way to signal to my fellow developers that the UC Journal needs a bloody search box.
Attachments as standard
A good CRM has attachments as standard. Even log entries are attachments. How this works is every entry is a small text field. It only needs to be large enough to summarise what the entry is about. If there is a lot to say, this is stored as a text attachment with metadata indicating that it is a native note (or whatever). That metadata should have some sort of document type indicator so the CRM knows if it is an image, a PDF, or note, or whatever.
Email solved this issue years and years ago with Multipurpose Internet Mail Extensions or MIME types.
HTML solved this with content-type headers like this:
Content-Type: text/html; charset=UTF-8Content-Type: multipart/form-data; boundary=something
Other departments (especially those giving out grants) do this already. Maybe ask another department to lend you their developer for an afternoon and copy their homework.
Even if you only allow uploads upon request, using an attachment feature would allow agents to add helpful and informative media to a journal. Attachments would also allow long entries to take up as much space as a short one until clicked.
Make questions their own content type
User questions should not be an unspecialised text entry. They should be of type “question” with metadata showing who has handled the question or the agent responsible for the question. The reply should be of type linked-entry so the user can find the question and the answer together in their journal.
This would speed up handling questions as stock answer libraries can be grown so standard answers to common questions can be given in just a few clicks.
You could separate technical support questions from UC advisor-type questions. It is likely you could separate out more question types with enough use case data.
You might even want to make the question a knowledge base search that the user must check to see if said article solves their question before they ask it.
This is pretty much a standard thing in even basic CRM systems. UC needs this too.
Auto-link tasks
It should be impossible for the system to add a vital task (do this or lose money) without making the task a link to the place where said tusk must be undertaken.
Autolink your CTA’s please DWP. You do it for appointments (even if you do wrap a link around a pre-tag for no good reason. That’s not what pre-tags are for but that’s another story.)
Stop misusing HTML tags
Actually, no. This is important. Some tags have very specific meanings.
The
pre-tagsuggests that the text is preformatted. Usually, this is for code and other times when whitespace (space, tab, etc) must be preserved in the rendering. Screenreaders will struggle to deal with your weird HTML tag abuse. Which may (NAL) be in breach of disability laws.The
pre-tagincreases what is already a broken and inconsistent way that HTML handles white space characters (tab, new line, space, etc.). The pre-tag semantically insists that the white space is all significant even though a “bug” in the pre-tag implementation means empty lines at the start and end are collapsed or ignored. Either way, the DWP should not have client-critical architecture that depends on a mistake that could be fixed at some future point.Here is a long article explaining just how crazy HTML whitespace gets. DWP’s dev’s please read it and understand it. If that is too hard, here is an expert reading the article and explaining it. The video is almost an hour long.
https://www.youtube.com/watch?v=qF7iXBk1s5o
There’s no need for abusing tags in this way. It is asking for bugs and possible accessibility problems. After all, every space (meaningful or not) has just been declared meaningful in a way that can only be fully understood by also reading the site’s CSS. Use a div-tag and a monospaced font if you must but leave
pre-tagfor white space preserving data.Why I think the DWP uses pre
After some poking around, I think I have worked out why the dev used the
pre-tag. It’s a terrible reason and shows they don’t actually know what they are doing.Block-level elements (which they want for styling reasons) break the table by force-resize (a lesser-known Jedi power) by making the link box element expand to take up as much space as it wants. Rather than CSS styling it to wrap, they depend on the
text-wrap-mode: wrapas thepre-tag‘s default behaviour rather than setting that value in the CSS for the block. They would not have had this problem had they not used tables for layout.If this is the case, the developer who was hired to make this website lacked anything approaching modern web design understanding and possesses barely functional development skills. I was able to identify all of these faults without even seeing the code. If I get my hands on the code, I expect unreadable spaghetti programming.
Show/Hide metadata
When reading log files (which is what the journal resembles) it is often helpful to use a tool to hide verbose information. In this case, date, time, and author should be elements the user can minify or hide. This would aid reading comprehension.
Show/Hide completed
Cancelled appointments, stuff the user or agent marks as complete, and old “you did the thing” journal entries should be something clients (users) can opt to hide. That makes getting to the critical and actionable entries so much easier.
Inbox Zero is a thing. Embrace it.
Critical actions should not be hidden in a log file
Critical activities that have yet to be completed should be more prominent. There are a number of good ways to do this.
Style to highlight: Add a “critical” class to the containing element of the journal entry that has the critical instruction. Use CSS to style it with a colour indicating urgency. Traditionally, this is red.
Dynamic banner notices: Use some JavaScript to parse the journal for classes marked critical and add a banner to the top of the page to strongly notify the user that a critical action is needed. It is only critical if not doing it costs money or risks sanctions. Everything else is at best urgent.
Critical messages set to app global: Put the critical information on all pages until undertaken.
Land the user on the page: Make the first page the user sees when logging in, the page where the critical action must be done.
Marked-up appointments to work with calendar apps
I’m not asking that you allow people to add Google Calander or other task management systems as guest apps but could you at least mark up appointments with h-event or hCalendar microformats?
Given the aim of UC is to support people, making things like appointments and to-do lists something they can export to their productivity software would make the UCW much more helpful.
If that’s too much hard work, an “add to Google Calendar” button would be something. If you are doing that anyway, why not go the whole way and make export appointment a fully working feature?
Show the next appointment prominently
Talking of appointments, like the critical actions, the client’s next appointment should be displayed prominently or at least have the option to do so. Top of the page or a toaster notification when they log in.
Push notifications?
Users should have the option to enable push notifications. These are little pop-ups that appear to tell you there’s a new message to read. Most phone apps do this and some news websites do too.
While I strongly suggest push notifications MUST be opt-in, they could prove highly useful for users of the UCW. You could further expand this to remind about appointments and urgent critical tasks that are running out of time.
You want people to use your atrocious website regularly, right? So why not prompt them to come back when something new needs their attention?
Why is this not a basic app?
The DWP need to have there (soon to be improved I hope) UCW as an app. At the very least as a dedicated one-site browser which can be knocked up in a few hours.
A dedicated UC app could provide an additional layer of protection (not only from scammers) by making the app a two-factor auth code generator. This is slightly more secure than texting a code as a text can be intercepted and is not encrypted. This, of course, depends on the devs knowing what they are doing – something I doubt from the
tableandpre tagabuse on display.An app would ensure users get push notifications on their smartphones.
An official app would make scams harder to pull off.
The problem at the moment is the incorrect use of whitespace affecting markup. Apps for Apple and Android are quite likely to handle whitespace differently. You’re already borking things with pre-tags and tables. Thankfully, a few hours of reading up on CSS and adaptive design should fix those errors.
Don’t gamify just yet
I was going to suggest you have a points-based system to indicate completeness and progress. We call this gamification and it can help encourage users to be a little more active.
However, you cannot even roll out a simple CRM so I do not trust you to gamify it.
A wild guess (contains mild politics)
I have no evidence (thus this is a wild guess) but I would not be surprised if this site was not commissioned under a “cash for mates” program during the Tory administration. I would not be shocked to learn that someone got a lot of tax-payer money in their pocket and then hired an intern to knock out the system In an afternoon.
This would explain the way the UCW does not follow standard government website standards as well as why exactly I found so many instances of mistakes I would expect from a complete beginner.
Summary of fixes needed on the Universal Credit website
In this section, I use industry-standard language and so: Bug indicates something that is wrong, broken, or in error and so are critical items. Enhancement indicates something that can be enhanced and so are urgent items. Feature Request is a thing the site is not doing but probably should and should be addressed at the next development meeting (assuming you have those). I’ve also added action points for the managers to do at start of business of the next working day.
- Bug: Deformed page on reply
- Bug: The Journal HTML may breach the law – reevaluate concerning accessibility standards
- Bug: The journal is not responsive but should be (for smaller screen sizes)
- Bug: It is impossible to cancel an incorrectly started task.
- Bug: Using
pre-tagsfor word wrap is an error use CSS - Bug: Switch tables to responsive divs – should fix most bugs
- Bug: Some key tasks do not auto-link. Consistently auto-link required tasks
- [new] Bug: Markdown weirdness
- Enhancement: Use semantically meaningful tags
- Enhancement: use syntactically appropriate language
- Enhancement: Make critical messages more prominent
- Enhancement: Add toggle hidden for metadata
- Enhancement: Add toggle hidden for expired, cancelled, or old entries
- Enhancement: Add microformats to make data more useful
- Feature Request: Search feature
- Feature Request: Threading, ticketing, or task-object grouping
- Feature Request: An attachment system
- Feature Request: Add a question entry type with a linked answer
- Feature Request: Enable clients to add appointments to personal calendar apps
- Feature Request: Optional push notifications
- Feature Request: Make a dedicated app
- Action point: Hire a domain-expert systems analyst who will confirm everything on this list.
- Action point: Read the analyst’s report and action it ASAP.
Alternatively, dear DWP managers, put together a budget to hire me and a small team I will select – we will fix it for you; budget for ongoing development as the UCW could be so much more than it is. You got this expert analysis from me for free which is already a massive saving. Make the most of it.
Over to you, dear readers
I think I have covered every fault with, and improvement needed for, the DWP’s Universal Credit Website. Is there anything I have missed? Are there more bugs to report? Would you like an enhancement I’ve not mentioned or hate one I have suggested?
I want to know your thoughts. Don’t send me a comment, leave me one. Or a reply via federation, or a WebMention via whatever content publisher you use. I really like comments so please leave me some.
Your replies will not be shoved inside a table and no
pre-tagswill be harmed. -
At @joyofcoding!
Great talk by Hannes Mühleisen of #DuckDB about tables being a fundamental technology to civilization and not dismissing databases, SQL & ACID just because some implementation are getting old in the tooth.
DuckDB sounds awesome and I know @bert_hubert is a big fan.
-
🎉 New Kitten¹ Release: A little housekeeping 🧹
Today’s release only concerns production servers:
• Kitten no longer counts all *hits* in its stats. You can still see which of your *pages* are most popular, etc., and see stats for missing URLs, etc., as before from either the web interface or the interactive shell, but not every hit is logged. Instead, you can see the latest 25 served routes in Kitten’s Settings (at /🐱/settings/state/requests/ via the web on your server).
- Kitten production servers now carry out an automatic daily maintenance restart at some time between 3AM and 5AM local server time. (“Have you tried turning it off and on again?” as a Service™) This is to allow JSDB² tables a chance to compact themselves (especially important for high traffic/high mutation tables like sessions, so they don’t balloon up to take up all available memory on small VPS instances).
I don’t think anyone but us (Small Technology Foundation³) is running Kitten in production at the moment but, still. If you are playing with Kitten and experimenting with it in production, your servers will update to this latest version in a few hours.
Full details: https://codeberg.org/kitten/app/src/branch/main/CHANGELOG.md#2025-07-29
:kitten: 💕
¹ https://kitten.small-web.org
² https://codeberg.org/small-tech/jsdb
³ https://small-tech.org -
"If you attempt to take a screenshot of Signal Desktop when screen security is enabled, nothing will appear. This limitation can be frustrating, but it might look familiar to you if you’ve ever had the audacity to try and take a screenshot of a movie or TV show on Windows. According to Microsoft’s official developer documentation, setting the correct Digital Rights Management (DRM) flag on the application window will ensure that “content won’t show up in Recall or any other screenshot application.” So that’s exactly what Signal Desktop is now doing on Windows 11 by default.
A stylized close-up crop of a movie screenplay that says "INT. COPILOT+ PC MANUFACTURING FACILITY - NIGHT - METALLIC SHELVES in endless rows stretch into the darkness. Two figures crouch in the shadows. ALICE: DRM technology has been consistently used against us. BOB: It won't be the first time we've turned the tables. ALICE: My life has always felt like a movie."
Apps like Signal have essentially no control over what content Recall is able to capture, and implementing “DRM” that works for you (not against you) is the best choice that we had. It’s like a scene in a movie where the villain has switched sides, and you can’t screenshot this one by default either."
https://signal.org/blog/signal-doesnt-recall/
#CyberSecurity #Privacy #DataProtection #Microsoft #Windows #WindowsRecall #Signal #Messaging
-
Earlier this year, Cendyne wrote a blog post covering the use of HKDF, building partially upon my own blog post about HKDF and the KDF security definition, but moreso inspired by a cryptographic issue they identified in another company’s product (dubbed AnonCo).
At the bottom they teased:
Database cryptography is hard. The above sketch is not complete and does not address several threats! This article is quite long, so I will not be sharing the fixes.
Cendyne
If you read Cendyne’s post, you may have nodded along with that remark and not appreciate the degree to which our naga friend was putting it mildly. So I thought I’d share some of my knowledge about real-world database cryptography in an accessible and fun format in the hopes that it might serve as an introduction to the specialization.
Note: I’m also not going to fix Cendyne’s sketch of AnonCo’s software here–partly because I don’t want to get in the habit of assigning homework or required reading, but mostly because it’s kind of obvious once you’ve learned the basics.
I’m including art of my fursona in this post… as is tradition for furry blogs.If you don’t like furries, please feel free to leave this blog and read about this topic elsewhere.
Thanks to CMYKat for the awesome stickers.
Contents
- Database Cryptography?
- Cryptography for Relational Databases
- The Perils of Built-in Encryption Functions
- Application-Layer Relational Database Cryptography
- Confused Deputies
- Canonicalization Attacks
- Multi-Tenancy
- Cryptography for NoSQL Databases
- NoSQL is Built Different
- Record Authentication
- Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
- Searchable Encryption
- Order-{Preserving, Revealing} Encryption
- Deterministic Encryption
- Homomorphic Encryption
- Searchable Symmetric Encryption (SSE)
- You Can Have Little a HMAC, As a Treat
- Intermission
- Case Study: MongoDB Client-Side Encryption
- MongoCrypt: The Good
- How is Queryable Encryption Implemented?
- MongoCrypt: The Bad
- MongoCrypt: The Ugly
- MongoCrypt: The Good
- Wrapping Up
Database Cryptography?
The premise of database cryptography is deceptively simple: You have a database, of some sort, and you want to store sensitive data in said database.
The consequences of this simple premise are anything but simple. Let me explain.
Art: ScruffKerfluffThe sensitive data you want to store may need to remain confidential, or you may need to provide some sort of integrity guarantees throughout your entire system, or sometimes both. Sometimes all of your data is sensitive, sometimes only some of it is. Sometimes the confidentiality requirements of your data extends to where within a dataset the record you want actually lives. Sometimes that’s true of some data, but not others, so your cryptography has to be flexible to support multiple types of workloads.
Other times, you just want your disks encrypted at rest so if they grow legs and walk out of the data center, the data cannot be comprehended by an attacker. And you can’t be bothered to work on this problem any deeper. This is usually what compliance requirements cover. Boxes get checked, executives feel safer about their operation, and the whole time nobody has really analyzed the risks they’re facing.
But we’re not settling for mere compliance on this blog. Furries have standards, after all.
So the first thing you need to do before diving into database cryptography is threat modelling. The first step in any good threat model is taking inventory; especially of assumptions, requirements, and desired outcomes. A few good starter questions:
- What database software is being used? Is it up to date?
- What data is being stored in which database software?
- How are databases oriented in the network of the overall system?
- Is your database properly firewalled from the public Internet?
- How does data flow throughout the network, and when do these data flows intersect with the database?
- Which applications talk to the database? What languages are they written in? Which APIs do they use?
- How will cryptography secrets be managed?
- Is there one key for everyone, one key per tenant, etc.?
- How are keys rotated?
- Do you use envelope encryption with an HSM, or vend the raw materials to your end devices?
The first two questions are paramount for deciding how to write software for database cryptography, before you even get to thinking about the cryptography itself.
(This is not a comprehensive set of questions to ask, either. A formal threat model is much deeper in the weeds.)
The kind of cryptography protocol you need for, say, storing encrypted CSV files an S3 bucket is vastly different from relational (SQL) databases, which in turn will be significantly different from schema-free (NoSQL) databases.
Furthermore, when you get to the point that you can start to think about the cryptography, you’ll often need to tackle confidentiality and integrity separately.
If that’s unclear, think of a scenario like, “I need to encrypt PII, but I also need to digitally sign the lab results so I know it wasn’t tampered with at rest.”
My point is, right off the bat, we’ve got a three-dimensional matrix of complexity to contend with:
- On one axis, we have the type of database.
- Flat-file
- Relational
- Schema-free
- On another, we have the basic confidentiality requirements of the data.
- Field encryption
- Row encryption
- Column encryption
- Unstructured record encryption
- Encrypting entire collections of records
- Finally, we have the integrity requirements of the data.
- Field authentication
- Row/column authentication
- Unstructured record authentication
- Collection authentication (based on e.g. Sparse Merkle Trees)
And then you have a fourth dimension that often falls out of operational requirements for databases: Searchability.
Why store data in a database if you have no way to index or search the data for fast retrieval?
Credit: HarubakiIf you’re starting to feel overwhelmed, you’re not alone. A lot of developers drastically underestimate the difficulty of the undertaking, until they run head-first into the complexity.
Some just phone it in with
AES_Encrypt()calls in their MySQL queries. (Too bad ECB mode doesn’t provide semantic security!)Which brings us to the meat of this blog post: The actual cryptography part.
Cryptography is the art of transforming information security problems into key management problems.
Former coworker
Note: In the interest of time, I’m skipping over flat files and focusing instead on actual database technologies.
Cryptography for Relational Databases
Encrypting data in an SQL database seems simple enough, even if you’ve managed to shake off the complexity I teased from the introduction.
You’ve got data, you’ve got a column on a table. Just encrypt the data and shove it in a cell on that column and call it a day, right?
But, alas, this is a trap. There are so many gotchas that I can’t weave a coherent, easy-to-follow narrative between them all.
So let’s start with a simple question: where and how are you performing your encryption?
The Perils of Built-in Encryption Functions
MySQL provides functions called AES_Encrypt and AES_Decrypt, which many developers have unfortunately decided to rely on in the past.
It’s unfortunate because these functions implement ECB mode. To illustrate why ECB mode is bad, I encrypted one of my art commissions with AES in ECB mode:
Art by Riley, encrypted with AES-ECBThe problems with ECB mode aren’t exactly “you can see the image through it,” because ECB-encrypting a compressed image won’t have redundancy (and thus can make you feel safer than you are).
ECB art is a good visual for the actual issue you should care about, however: A lack of semantic security.
A cryptosystem is considered semantically secure if observing the ciphertext doesn’t reveal information about the plaintext (except, perhaps, the length; which all cryptosystems leak to some extent). More information here.
ECB art isn’t to be confused with ECB poetry, which looks like this:
Oh little one, you’re growing up
You’ll soon be writing C
You’ll treat your ints as pointers
You’ll nest the ternary
You’ll cut and paste from github
And try cryptography
But even in your darkest hour
Do not use ECBCBC’s BEASTly when padding’s abused
And CTR’s fine til a nonce is reused
Some say it’s a CRIME to compress then encrypt
Or store keys in the browser (or use javascript)
Diffie Hellman will collapse if hackers choose your g
And RSA is full of traps when e is set to 3
Whiten! Blind! In constant time! Don’t write an RNG!
But failing all, and listen well: Do not use ECBThey’ll say “It’s like a one-time-pad!
The data’s short, it’s not so bad
the keys are long–they’re iron clad
I have a PhD!”
And then you’re front page Hacker News
Your passwords cracked–Adobe Blues.
Don’t leave your penguins showing through,
Do not use ECB— Ben Nagy, PoC||GTFO 0x04:13
Most people reading this probably know better than to use ECB mode already, and don’t need any of these reminders, but there is still a lot of code that inadvertently uses ECB mode to encrypt data in the database.
Also,
Credit: CMYKattSHOW processlist;leaks your encryption keys. Oops.Application-layer Relational Database Cryptography
Whether burned by ECB or just cautious about not giving your secrets to the system that stores all the ciphertext protected by said secret, a common next step for developers is to simply encrypt in their server-side application code.
And, yes, that’s part of the answer. But how you encrypt is important.
Credit: Harubaki“I’ll encrypt with CBC mode.”
If you don’t authenticate your ciphertext, you’ll be sorry. Maybe try again?“Okay, fine, I’ll use an authenticated mode like GCM.”
Did you remember to make the table and column name part of your AAD? What about the primary key of the record?“What on Earth are you talking about, Soatok?”
Welcome to the first footgun of database cryptography!Confused Deputies
Encrypting your sensitive data is necessary, but not sufficient. You need to also bind your ciphertexts to the specific context in which they are stored.
To understand why, let’s take a step back: What specific threat does encrypting your database records protect against?
We’ve already established that “your disks walk out of the datacenter” is a “full disk encryption” problem, so if you’re using application-layer cryptography to encrypt data in a relational database, your threat model probably involves unauthorized access to the database server.
What, then, stops an attacker from copying ciphertexts around?
Credit: CMYKattLet’s say I have a legitimate user account with an ID 12345, and I want to read your street address, but it’s encrypted in the database. But because I’m a clever hacker, I have unfettered access to your relational database server.
All I would need to do is simply…
UPDATE table SET addr_encrypted = 'your-ciphertext' WHERE id = 12345…and then access the application through my legitimate access. Bam, data leaked. As an attacker, I can probably even copy fields from other columns and it will just decrypt. Even if you’re using an authenticated mode.
We call this a confused deputy attack, because the deputy (the component of the system that has been delegated some authority or privilege) has become confused by the attacker, and thus undermined an intended security goal.
The fix is to use the AAD parameter from the authenticated mode to bind the data to a given context. (AAD = Additional Authenticated Data.)
- $addr = aes_gcm_encrypt($addr, $key);+ $addr = aes_gcm_encrypt($addr, $key, canonicalize([+ $tableName,+ $columnName,+ $primaryKey+ ]);
Now if I start cutting and pasting ciphertexts around, I get a decryption failure instead of silently decrypting plaintext.
This may sound like a specific vulnerability, but it’s more of a failure to understand an important general lesson with database cryptography:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
Canonicalization Attacks
In the previous section, I introduced a pseudocode called
canonicalize(). This isn’t a pasto from some reference code; it’s an important design detail that I will elaborate on now.First, consider you didn’t do anything to canonicalize your data, and you just joined strings together and called it a day…
function dumbCanonicalize( string $tableName, string $columnName, string|int $primaryKey): string { return $tableName . '_' . $columnName . '#' . $primaryKey;}Consider these two inputs to this function:
dumbCanonicalize('customers', 'last_order_uuid', 123);dumbCanonicalize('customers_last_order', 'uuid', 123);
In this case, your AAD would be the same, and therefore, your deputy can still be confused (albeit in a narrower use case).
In Cendyne’s article, AnonCo did something more subtle: The canonicalization bug created a collision on the inputs to HKDF, which resulted in an unintentional key reuse.
Up until this point, their mistake isn’t relevant to us, because we haven’t even explored key management at all. But the same design flaw can re-emerge in multiple locations, with drastically different consequence.
Multi-Tenancy
Once you’ve implemented a mitigation against Confused Deputies, you may think your job is done. And it very well could be.
Often times, however, software developers are tasked with building support for Bring Your Own Key (BYOK).
This is often spawned from a specific compliance requirement (such as cryptographic shredding; i.e. if you erase the key, you can no longer recover the plaintext, so it may as well be deleted).
Other times, this is driven by a need to cut costs: Storing different users’ data in the same database server, but encrypting it such that they can only encrypt their own records.
Two things can happen when you introduce multi-tenancy into your database cryptography designs:
- Invisible Salamanders becomes a risk, due to multiple keys being possible for any given encrypted record.
- Failure to address the risk of Invisible Salamanders can undermine your protection against Confused Deputies, thereby returning you to a state before you properly used the AAD.
So now you have to revisit your designs and ensure you’re using a key-committing authenticated mode, rather than just a regular authenticated mode.
Isn’t cryptography fun?
“What Are Invisible Salamanders?”
This refers to a fun property of AEAD modes based on Polynomical MACs. Basically, if you:
- Encrypt one message under a specific key and nonce.
- Encrypt another message under a separate key and nonce.
…Then you can get the same exact ciphertext and authentication tag. Performing this attack requires you to control the keys for both encryption operations.
This was first demonstrated in an attack against encrypted messaging applications, where a picture of a salamander was hidden from the abuse reporting feature because another attached file had the same authentication tag and ciphertext, and you could trick the system if you disclosed the second key instead of the first. Thus, the salamander is invisible to attackers.
Art: CMYKatWe’re not quite done with relational databases yet, but we should talk about NoSQL databases for a bit. The final topic in scope applies equally to both, after all.
Cryptography for NoSQL Databases
Most of the topics from relational databases also apply to NoSQL databases, so I shall refrain from duplicating them here. This article is already sufficiently long to read, after all, and I dislike redundancy.
NoSQL is Built Different
The main thing that NoSQL databases offer in the service of making cryptographers lose sleep at night is the schema-free nature of NoSQL designs.
What this means is that, if you’re using a client-side encryption library for a NoSQL database, the previous concerns about confused deputy attacks are amplified by the malleability of the document structure.
Additionally, the previously discussed cryptographic attacks against the encryption mode may be less expensive for an attacker to pull off.
Consider the following record structure, which stores a bunch of data stored with AES in CBC mode:
{ "encrypted-data-key": "<blob>", "name": "<ciphertext>", "address": [ "<ciphertext>", "<ciphertext>" ], "social-security": "<ciphertext>", "zip-code": "<ciphertext>"}If this record is decrypted with code that looks something like this:
$decrypted = [];// ... snip ...foreach ($record['address'] as $i => $addrLine) { try { $decrypted['address'][$i] = $this->decrypt($addrLine); } catch (Throwable $ex) { // You'd never deliberately do this, but it's for illustration $this->doSomethingAnOracleCanObserve($i); // This is more believable, of course: $this->logDecryptionError($ex, $addrLine); $decrypted['address'][$i] = ''; }}Then you can keep appending rows to the
Art: Harubaki"address"field to reduce the number of writes needed to exploit a padding oracle attack against any of the<ciphertext>fields.This isn’t to say that NoSQL is less secure than SQL, from the context of client-side encryption. However, the powerful feature sets that NoSQL users are accustomed to may also give attackers a more versatile toolkit to work with.
Record Authentication
A pedant may point out that record authentication applies to both SQL and NoSQL. However, I mostly only observe this feature in NoSQL databases and document storage systems in the wild, so I’m shoving it in here.
Encrypting fields is nice and all, but sometimes what you want to know is that your unencrypted data hasn’t been tampered with as it flows through your system.
The trivial way this is done is by using a digital signature algorithm over the whole record, and then appending the signature to the end. When you go to verify the record, all of the information you need is right there.
This works well enough for most use cases, and everyone can pack up and go home. Nothing more to see here.
Except…
When you’re working with NoSQL databases, you often want systems to be able to write to additional fields, and since you’re working with schema-free blobs of data rather than a normalized set of relatable tables, the most sensible thing to do is to is to append this data to the same record.
Except, oops! You can’t do that if you’re shoving a digital signature over the record. So now you need to specify which fields are to be included in the signature.
And you need to think about how to model that in a way that doesn’t prohibit schema upgrades nor allow attackers to perform downgrade attacks. (See below.)
I don’t have any specific real-world examples here that I can point to of this problem being solved well.Art: CMYKat
Furthermore, as with preventing confused deputy and/or canonicalization attacks above, you must also include the fully qualified path of each field in the data that gets signed.
As I said with encryption before, but also true here:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
This requirement holds true whether you’re using symmetric-key authentication (i.e. HMAC) or asymmetric-key digital signatures (e.g. EdDSA).
Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
Art: HarubakiOkay, how do you solve this problem so that you can perform updates and upgrades to your schema but without enabling attackers to downgrade the security? Here’s one possible design.
Let’s say you have two metadata fields on each record:
- A compressed binary string representing which fields should be authenticated. This field is, itself, not authenticated. Let’s call this
meta-auth. - A compressed binary string representing which of the authenticated fields should also be encrypted. This field is also authenticated. This is at most the same length as the first metadata field. Let’s call this
meta-enc.
Furthermore, you will specify a canonical field ordering for both how data is fed into the signature algorithm as well as the field mappings in
meta-authandmeta-enc.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-enc */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false /* example.superfluous.rewards-member */ ]), "signature": /* -- snip -- */}When you go to append data to an existing record, you’ll need to update
meta-authto include the mapping of fields based on this canonical ordering to ensure only the intended fields get validated.When you update your code to add an additional field that is intended to be signed, you can roll that out for new records and the record will continue to be self-describing:
- New records will have the additional field flagged as authenticated in
meta-auth(andmeta-encwill grow) - Old records will not, but your code will still sign them successfully
- To prevent downgrade attacks, simply include a schema version ID as an additional plaintext field that gets authenticated. An attacker who tries to downgrade will need to be able to produce a valid signature too.
You might think
meta-authgives an attacker some advantage, but this only includes which fields are included in the security boundary of the signature or MAC, which allows unauthenticated data to be appended for whatever operational purpose without having to update signatures or expose signing keys to a wider part of the network.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true, /* meta-enc */ true /* meta-version */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-version */ ]), "meta-version": 0x01000000, "signature": /* -- snip -- */}If an attacker tries to use the
meta-authfield to mess with a record, the best they can hope for is an Invalid Signature exception (assuming the signature algorithm is secure to begin with).Even if they keep all of the fields the same, but play around with the structure of the record (e.g. changing the XPath or equivalent), so long as the path is authenticated with each field, breaking this is computationally infeasible.
Searchable Encryption
If you’ve managed to make it through the previous sections, congratulations, you now know enough to build a secure but completely useless database.
Art: CMYKatOkay, put away the pitchforks; I will explain.
Part of the reason why we store data in a database, rather than a flat file, is because we want to do more than just read and write. Sometimes computer scientists want to compute. Almost always, you want to be able to query your database for a subset of records based on your specific business logic needs.
And so, a database which doesn’t do anything more than store ciphertext and maybe signatures is pretty useless to most people. You’d have better luck selling Monkey JPEGs to furries than convincing most businesses to part with their precious database-driven report generators.
Art: SophieSo whenever one of your users wants to actually use their data, rather than just store it, they’re forced to decide between two mutually exclusive options:
- Encrypting the data, to protect it from unauthorized disclosure, but render it useless
- Doing anything useful with the data, but leaving it unencrypted in the database
This is especially annoying for business types that are all in on the Zero Trust buzzword.
Fortunately, the cryptographers are at it again, and boy howdy do they have a lot of solutions for this problem.
Order-{Preserving, Revealing} Encryption
On the fun side of things, you have things like Order-Preserving and Order-Revealing Encryption, which Matthew Green wrote about at length.
[D]atabase encryption has been a controversial subject in our field. I wish I could say that there’s been an actual debate, but it’s more that different researchers have fallen into different camps, and nobody has really had the data to make their position in a compelling way. There have actually been some very personal arguments made about it.
Attack of the week: searchable encryption and the ever-expanding leakage function
The problem with these designs is that they have a significant enough leakage that it no longer provides semantic security.
From Grubbs, et al. (GLMP, 2019.)
Colors inverted to fit my blog’s theme better.To put it in other words: These designs are only marginally better than ECB mode, and probably deserve their own poems too.
Order revealing
Reveals much more than order
Softcore ECBOrder preserving
Semantic security?
Only in your dreamsHaiku for your consideration
Deterministic Encryption
Here’s a simpler, but also terrible, idea for searchable encryption: Simply give up on semantic security entirely.
If you recall the
AES_{De,En}crypt()functions built into MySQL I mentioned at the start of this article, those are the most common form of deterministic encryption I’ve seen in use.SELECT * FROM foo WHERE bar = AES_Encrypt('query', 'key');However, there are slightly less bad variants. If you use AES-GCM-SIV with a static nonce, your ciphertexts are fully deterministic, and you can encrypt a small number of distinct records safely before you’re no longer secure.
From Page 14 of the linked paper. Full view.That’s certainly better than nothing, but you also can’t mitigate confused deputy attacks. But we can do better than this.
Homomorphic Encryption
In a safer plane of academia, you’ll find homomorphic encryption, which researchers recently demonstrated with serving Wikipedia pages in a reasonable amount of time.
Homomorphic encryption allows computations over the ciphertext, which will be reflected in the plaintext, without ever revealing the key to the entity performing the computation.
If this sounds vaguely similar to the conditions that enable chosen-ciphertext attacks, you probably have a good intuition for how it works: RSA is homomorphic to multiplication, AES-CTR is homomorphic to XOR. Fully homomorphic encryption uses lattices, which enables multiple operations but carries a relatively enormous performance cost.
Art: HarubakiHomomorphic encryption sometimes intersects with machine learning, because the notion of training an encrypted model by feeding it encrypted data, then decrypting it after-the-fact is desirable for certain business verticals. Your data scientists never see your data, and you have some plausible deniability about the final ML model this work produces. This is like a Siren song for Venture Capitalist-backed medical technology companies. Tech journalists love writing about it.
However, a less-explored use case is the ability to encrypt your programs but still get the correct behavior and outputs. Although this sounds like a DRM technology, it’s actually something that individuals could one day use to prevent their ISPs or cloud providers from knowing what software is being executed on the customer’s leased hardware. The potential for a privacy win here is certainly worth pondering, even if you’re a tried and true Pirate Party member.
Just say “NO” to the copyright cartels.Art: CMYKat
Searchable Symmetric Encryption (SSE)
Forget about working at the level of fields and rows or individual records. What if we, instead, worked over collections of documents, where each document is viewed as a set of keywords from a keyword space?
Art: CMYKatThat’s the basic premise of SSE: Encrypting collections of documents rather than individual records.
The actual implementation details differ greatly between designs. They also differ greatly in their leakage profiles and susceptibility to side-channel attacks.
Some schemes use a so-called trapdoor permutation, such as RSA, as one of their building blocks.
Some schemes only allow for searching a static set of records, while others can accommodate new data over time (with the trade-off between more leakage or worse performance).
If you’re curious, you can learn more about SSE here, and see some open source SEE implementations online here.
You’re probably wondering, “If SSE is this well-studied and there are open source implementations available, why isn’t it more widely used?”
Your guess is as good as mine, but I can think of a few reasons:
- The protocols can be a little complicated to implement, and aren’t shipped by default in cryptography libraries (i.e. OpenSSL’s libcrypto or libsodium).
- Every known security risk in SSE is the product of a trade-offs, rather than there being a single winner for all use cases that developers can feel comfortable picking.
- Insufficient marketing and developer advocacy.
SSE schemes are mostly of interest to academics, although Seny Kamara (Brown Univeristy professior and one of the luminaries of searchable encryption) did try to develop an app called Pixek which used SSE to encrypt photos.
Maybe there’s room for a cryptography competition on searchable encryption schemes in the future.
You Can Have Little a HMAC, As a Treat
Finally, I can’t talk about searchable encryption without discussing a technique that’s older than dirt by Internet standards, that has been independently reinvented by countless software developers tasked with encrypting database records.
The oldest version I’ve been able to track down dates to 2006 by Raul Garcia at Microsoft, but I’m not confident that it didn’t exist before.
The idea I’m alluding to goes like this:
- Encrypt your data, securely, using symmetric cryptography.
(Hopefully your encryption addresses the considerations outlined in the relevant sections above.) - Separately, calculate an HMAC over the unencrypted data with a separate key used exclusively for indexing.
When you need to query your data, you can just recalculate the HMAC of your challenge and fetch the records that match it. Easy, right?
Even if you rotate your keys for encryption, you keep your indexing keys static across your entire data set. This lets you have durable indexes for encrypted data, which gives you the ability to do literal lookups for the performance hit of a hash function.
Additionally, everyone has HMAC in their toolkit, so you don’t have to move around implementations of complex cryptographic building blocks. You can live off the land. What’s not to love?
Hooray!However, if you stopped here, we regret to inform you that your data is no longer indistinguishable from random, which probably undermines the security proof for your encryption scheme.
How annoying!Of course, you don’t have to stop with the addition of plain HMAC to your database encryption software.
Take a page from Troy Hunt: Truncate the output to provide k-anonymity rather than a direct literal look-up.
“K-What Now?”
Imagine you have a full HMAC-SHA256 of the plaintext next to every ciphertext record with a static key, for searchability.
Each HMAC output corresponds 1:1 with a unique plaintext.
Because you’re using HMAC with a secret key, an attacker can’t just build a rainbow table like they would when attempting password cracking, but it still leaks duplicate plaintexts.
For example, an HMAC-SHA256 output might look like this:
Art: CMYKat\04a74e4c0158e34a566785d1a5e1167c4e3455c42aea173104e48ca810a8b1aeIf you were to slice off most of those bytes (e.g. leaving only the last 3, which in the previous example yields
a8b1ae), then with sufficient records, multiple plaintexts will now map to the same truncated HMAC tag.Which means if you’re only revealing a truncated HMAC tag to the database server (both when storing records or retrieving them), you can now expect false positives due to collisions in your truncated HMAC tag.
These false positives give your data a discrete set of anonymity (called k-anonymity), which means an attacker with access to your database cannot:
- Distinguish between two encrypted records with the same short HMAC tag.
- Reverse engineer the short HMAC tag into a single possible plaintext value, even if they can supply candidate queries and study the tags sent to the database.
As with SSE above, this short HMAC technique exposes a trade-off to users.
- Too much k-anonymity (i.e. too many false positives), and you will have to decrypt-then-discard multiple mismatching records. This can make queries slow.
- Not enough k-anonymity (i.e. insufficient false positives), and you’re no better off than a full HMAC.
Even more troublesome, the right amount to truncate is expressed in bits (not bytes), and calculating this value depends on the number of unique plaintext values you anticipate in your dataset. (Fortunately, it grows logarithmically, so you’ll rarely if ever have to tune this.)
If you’d like to play with this idea, here’s a quick and dirty demo script.
Intermission
If you started reading this post with any doubts about Cendyne’s statement that “Database cryptography is hard”, by making it to this point, they’ve probably been long since put to rest.
Art: HarubakiConversely, anyone that specializes in this topic is probably waiting for me to say anything novel or interesting; their patience wearing thin as I continue to rehash a surface-level introduction of their field without really diving deep into anything.
Thus, if you’ve read this far, I’d like to demonstrate the application of what I’ve covered thus far into a real-world case study into an database cryptography product.
Case Study: MongoDB Client-Side Encryption
MongoDB is an open source schema-free NoSQL database. Last year, MongoDB made waves when they announced Queryable Encryption in their upcoming client-side encryption release.
Taken from the press release, but adapted for dark themes.A statement at the bottom of their press release indicates that this isn’t clown-shoes:
Queryable Encryption was designed by MongoDB’s Advanced Cryptography Research Group, headed by Seny Kamara and Tarik Moataz, who are pioneers in the field of encrypted search. The Group conducts cutting-edge peer-reviewed research in cryptography and works with MongoDB engineering teams to transfer and deploy the latest innovations in cryptography and privacy to the MongoDB data platform.
If you recall, I mentioned Seny Kamara in the SSE section of this post. They certainly aren’t wrong about Kamara and Moataz being pioneers in this field.
So with that in mind, let’s explore the implementation in libmongocrypt and see how it stands up to scrutiny.
MongoCrypt: The Good
MongoDB’s encryption library takes key management seriously: They provide a KMS integration for cloud users by default (supporting both AWS and Azure).
MongoDB uses Encrypt-then-MAC with AES-CBC and HMAC-SHA256, which is congruent to what Signal does for message encryption.
How Is Queryable Encryption Implemented?
From the current source code, we can see that MongoCrypt generates several different types of tokens, using HMAC (calculation defined here).
According to their press release:
The feature supports equality searches, with additional query types such as range, prefix, suffix, and substring planned for future releases.
Which means that most of the juicy details probably aren’t public yet.
These HMAC-derived tokens are stored wholesale in the data structure, but most are encrypted before storage using AES-CTR.
There are more layers of encryption (using AEAD), server-side token processing, and more AES-CTR-encrypted edge tokens. All of this is finally serialized (implementation) as one blob for storage.
Since only the equality operation is currently supported (which is the same feature you’d get from HMAC), it’s difficult to speculate what the full feature set looks like.
However, since Kamara and Moataz are leading its development, it’s likely that this feature set will be excellent.
MongoCrypt: The Bad
Every call to
do_encrypt()includes at most the Key ID (but typicallyNULL) as the AAD. This means that the concerns over Confused Deputies (and NoSQL specifically) are relevant to MongoDB.However, even if they did support authenticating the fully qualified path to a field in the AAD for their encryption, their AEAD construction is vulnerable to the kind of canonicalization attack I wrote about previously.
First, observe this code which assembles the multi-part inputs into HMAC.
/* Construct the input to the HMAC */uint32_t num_intermediates = 0;_mongocrypt_buffer_t intermediates[3];// -- snip --if (!_mongocrypt_buffer_concat ( &to_hmac, intermediates, num_intermediates)) { CLIENT_ERR ("failed to allocate buffer"); goto done;}if (hmac == HMAC_SHA_512_256) { uint8_t storage[64]; _mongocrypt_buffer_t tag = {.data = storage, .len = sizeof (storage)}; if (!_crypto_hmac_sha_512 (crypto, Km, &to_hmac, &tag, status)) { goto done; } // Truncate sha512 to first 256 bits. memcpy (out->data, tag.data, MONGOCRYPT_HMAC_LEN);} else { BSON_ASSERT (hmac == HMAC_SHA_256); if (!_mongocrypt_hmac_sha_256 (crypto, Km, &to_hmac, out, status)) { goto done; }}The implementation of
_mongocrypt_buffer_concat()can be found here.If either the implementation of that function, or the code I snipped from my excerpt, had contained code that prefixed every segment of the AAD with the length of the segment (represented as a
uint64_tto make overflow infeasible), then their AEAD mode would not be vulnerable to canonicalization issues.Using TupleHash would also have prevented this issue.
Silver lining for MongoDB developers: Because the AAD is either a key ID or NULL, this isn’t exploitable in practice.
The first cryptographic flaw sort of cancels the second out.
If the libmongocrypt developers ever want to mitigate Confused Deputy attacks, they’ll need to address this canonicalization issue too.
MongoCrypt: The Ugly
MongoCrypt supports deterministic encryption.
If you specify deterministic encryption for a field, your application passes a deterministic initialization vector to AEAD.
We already discussed why this is bad above.
Wrapping Up
This was not a comprehensive treatment of the field of database cryptography. There are many areas of this field that I did not cover, nor do I feel qualified to discuss.
However, I hope anyone who takes the time to read this finds themselves more familiar with the subject.
Additionally, I hope any developers who think “encrypting data in a database is [easy, trivial] (select appropriate)” will find this broad introduction a humbling experience.
Art: CMYKathttps://soatok.blog/2023/03/01/database-cryptography-fur-the-rest-of-us/
#appliedCryptography #blockCipherModes #cryptography #databaseCryptography #databases #encryptedSearch #HMAC #MongoCrypt #MongoDB #QueryableEncryption #realWorldCryptography #security #SecurityGuidance #SQL #SSE #symmetricCryptography #symmetricSearchableEncryption
-
Earlier this year, Cendyne wrote a blog post covering the use of HKDF, building partially upon my own blog post about HKDF and the KDF security definition, but moreso inspired by a cryptographic issue they identified in another company’s product (dubbed AnonCo).
At the bottom they teased:
Database cryptography is hard. The above sketch is not complete and does not address several threats! This article is quite long, so I will not be sharing the fixes.
Cendyne
If you read Cendyne’s post, you may have nodded along with that remark and not appreciate the degree to which our naga friend was putting it mildly. So I thought I’d share some of my knowledge about real-world database cryptography in an accessible and fun format in the hopes that it might serve as an introduction to the specialization.
Note: I’m also not going to fix Cendyne’s sketch of AnonCo’s software here–partly because I don’t want to get in the habit of assigning homework or required reading, but mostly because it’s kind of obvious once you’ve learned the basics.
I’m including art of my fursona in this post… as is tradition for furry blogs.If you don’t like furries, please feel free to leave this blog and read about this topic elsewhere.
Thanks to CMYKat for the awesome stickers.
Contents
- Database Cryptography?
- Cryptography for Relational Databases
- The Perils of Built-in Encryption Functions
- Application-Layer Relational Database Cryptography
- Confused Deputies
- Canonicalization Attacks
- Multi-Tenancy
- Cryptography for NoSQL Databases
- NoSQL is Built Different
- Record Authentication
- Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
- Searchable Encryption
- Order-{Preserving, Revealing} Encryption
- Deterministic Encryption
- Homomorphic Encryption
- Searchable Symmetric Encryption (SSE)
- You Can Have Little a HMAC, As a Treat
- Intermission
- Case Study: MongoDB Client-Side Encryption
- MongoCrypt: The Good
- How is Queryable Encryption Implemented?
- MongoCrypt: The Bad
- MongoCrypt: The Ugly
- MongoCrypt: The Good
- Wrapping Up
Database Cryptography?
The premise of database cryptography is deceptively simple: You have a database, of some sort, and you want to store sensitive data in said database.
The consequences of this simple premise are anything but simple. Let me explain.
Art: ScruffKerfluffThe sensitive data you want to store may need to remain confidential, or you may need to provide some sort of integrity guarantees throughout your entire system, or sometimes both. Sometimes all of your data is sensitive, sometimes only some of it is. Sometimes the confidentiality requirements of your data extends to where within a dataset the record you want actually lives. Sometimes that’s true of some data, but not others, so your cryptography has to be flexible to support multiple types of workloads.
Other times, you just want your disks encrypted at rest so if they grow legs and walk out of the data center, the data cannot be comprehended by an attacker. And you can’t be bothered to work on this problem any deeper. This is usually what compliance requirements cover. Boxes get checked, executives feel safer about their operation, and the whole time nobody has really analyzed the risks they’re facing.
But we’re not settling for mere compliance on this blog. Furries have standards, after all.
So the first thing you need to do before diving into database cryptography is threat modelling. The first step in any good threat model is taking inventory; especially of assumptions, requirements, and desired outcomes. A few good starter questions:
- What database software is being used? Is it up to date?
- What data is being stored in which database software?
- How are databases oriented in the network of the overall system?
- Is your database properly firewalled from the public Internet?
- How does data flow throughout the network, and when do these data flows intersect with the database?
- Which applications talk to the database? What languages are they written in? Which APIs do they use?
- How will cryptography secrets be managed?
- Is there one key for everyone, one key per tenant, etc.?
- How are keys rotated?
- Do you use envelope encryption with an HSM, or vend the raw materials to your end devices?
The first two questions are paramount for deciding how to write software for database cryptography, before you even get to thinking about the cryptography itself.
(This is not a comprehensive set of questions to ask, either. A formal threat model is much deeper in the weeds.)
The kind of cryptography protocol you need for, say, storing encrypted CSV files an S3 bucket is vastly different from relational (SQL) databases, which in turn will be significantly different from schema-free (NoSQL) databases.
Furthermore, when you get to the point that you can start to think about the cryptography, you’ll often need to tackle confidentiality and integrity separately.
If that’s unclear, think of a scenario like, “I need to encrypt PII, but I also need to digitally sign the lab results so I know it wasn’t tampered with at rest.”
My point is, right off the bat, we’ve got a three-dimensional matrix of complexity to contend with:
- On one axis, we have the type of database.
- Flat-file
- Relational
- Schema-free
- On another, we have the basic confidentiality requirements of the data.
- Field encryption
- Row encryption
- Column encryption
- Unstructured record encryption
- Encrypting entire collections of records
- Finally, we have the integrity requirements of the data.
- Field authentication
- Row/column authentication
- Unstructured record authentication
- Collection authentication (based on e.g. Sparse Merkle Trees)
And then you have a fourth dimension that often falls out of operational requirements for databases: Searchability.
Why store data in a database if you have no way to index or search the data for fast retrieval?
Credit: HarubakiIf you’re starting to feel overwhelmed, you’re not alone. A lot of developers drastically underestimate the difficulty of the undertaking, until they run head-first into the complexity.
Some just phone it in with
AES_Encrypt()calls in their MySQL queries. (Too bad ECB mode doesn’t provide semantic security!)Which brings us to the meat of this blog post: The actual cryptography part.
Cryptography is the art of transforming information security problems into key management problems.
Former coworker
Note: In the interest of time, I’m skipping over flat files and focusing instead on actual database technologies.
Cryptography for Relational Databases
Encrypting data in an SQL database seems simple enough, even if you’ve managed to shake off the complexity I teased from the introduction.
You’ve got data, you’ve got a column on a table. Just encrypt the data and shove it in a cell on that column and call it a day, right?
But, alas, this is a trap. There are so many gotchas that I can’t weave a coherent, easy-to-follow narrative between them all.
So let’s start with a simple question: where and how are you performing your encryption?
The Perils of Built-in Encryption Functions
MySQL provides functions called AES_Encrypt and AES_Decrypt, which many developers have unfortunately decided to rely on in the past.
It’s unfortunate because these functions implement ECB mode. To illustrate why ECB mode is bad, I encrypted one of my art commissions with AES in ECB mode:
Art by Riley, encrypted with AES-ECBThe problems with ECB mode aren’t exactly “you can see the image through it,” because ECB-encrypting a compressed image won’t have redundancy (and thus can make you feel safer than you are).
ECB art is a good visual for the actual issue you should care about, however: A lack of semantic security.
A cryptosystem is considered semantically secure if observing the ciphertext doesn’t reveal information about the plaintext (except, perhaps, the length; which all cryptosystems leak to some extent). More information here.
ECB art isn’t to be confused with ECB poetry, which looks like this:
Oh little one, you’re growing up
You’ll soon be writing C
You’ll treat your ints as pointers
You’ll nest the ternary
You’ll cut and paste from github
And try cryptography
But even in your darkest hour
Do not use ECBCBC’s BEASTly when padding’s abused
And CTR’s fine til a nonce is reused
Some say it’s a CRIME to compress then encrypt
Or store keys in the browser (or use javascript)
Diffie Hellman will collapse if hackers choose your g
And RSA is full of traps when e is set to 3
Whiten! Blind! In constant time! Don’t write an RNG!
But failing all, and listen well: Do not use ECBThey’ll say “It’s like a one-time-pad!
The data’s short, it’s not so bad
the keys are long–they’re iron clad
I have a PhD!”
And then you’re front page Hacker News
Your passwords cracked–Adobe Blues.
Don’t leave your penguins showing through,
Do not use ECB— Ben Nagy, PoC||GTFO 0x04:13
Most people reading this probably know better than to use ECB mode already, and don’t need any of these reminders, but there is still a lot of code that inadvertently uses ECB mode to encrypt data in the database.
Also,
Credit: CMYKattSHOW processlist;leaks your encryption keys. Oops.Application-layer Relational Database Cryptography
Whether burned by ECB or just cautious about not giving your secrets to the system that stores all the ciphertext protected by said secret, a common next step for developers is to simply encrypt in their server-side application code.
And, yes, that’s part of the answer. But how you encrypt is important.
Credit: Harubaki“I’ll encrypt with CBC mode.”
If you don’t authenticate your ciphertext, you’ll be sorry. Maybe try again?“Okay, fine, I’ll use an authenticated mode like GCM.”
Did you remember to make the table and column name part of your AAD? What about the primary key of the record?“What on Earth are you talking about, Soatok?”
Welcome to the first footgun of database cryptography!Confused Deputies
Encrypting your sensitive data is necessary, but not sufficient. You need to also bind your ciphertexts to the specific context in which they are stored.
To understand why, let’s take a step back: What specific threat does encrypting your database records protect against?
We’ve already established that “your disks walk out of the datacenter” is a “full disk encryption” problem, so if you’re using application-layer cryptography to encrypt data in a relational database, your threat model probably involves unauthorized access to the database server.
What, then, stops an attacker from copying ciphertexts around?
Credit: CMYKattLet’s say I have a legitimate user account with an ID 12345, and I want to read your street address, but it’s encrypted in the database. But because I’m a clever hacker, I have unfettered access to your relational database server.
All I would need to do is simply…
UPDATE table SET addr_encrypted = 'your-ciphertext' WHERE id = 12345…and then access the application through my legitimate access. Bam, data leaked. As an attacker, I can probably even copy fields from other columns and it will just decrypt. Even if you’re using an authenticated mode.
We call this a confused deputy attack, because the deputy (the component of the system that has been delegated some authority or privilege) has become confused by the attacker, and thus undermined an intended security goal.
The fix is to use the AAD parameter from the authenticated mode to bind the data to a given context. (AAD = Additional Authenticated Data.)
- $addr = aes_gcm_encrypt($addr, $key);+ $addr = aes_gcm_encrypt($addr, $key, canonicalize([+ $tableName,+ $columnName,+ $primaryKey+ ]);
Now if I start cutting and pasting ciphertexts around, I get a decryption failure instead of silently decrypting plaintext.
This may sound like a specific vulnerability, but it’s more of a failure to understand an important general lesson with database cryptography:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
Canonicalization Attacks
In the previous section, I introduced a pseudocode called
canonicalize(). This isn’t a pasto from some reference code; it’s an important design detail that I will elaborate on now.First, consider you didn’t do anything to canonicalize your data, and you just joined strings together and called it a day…
function dumbCanonicalize( string $tableName, string $columnName, string|int $primaryKey): string { return $tableName . '_' . $columnName . '#' . $primaryKey;}Consider these two inputs to this function:
dumbCanonicalize('customers', 'last_order_uuid', 123);dumbCanonicalize('customers_last_order', 'uuid', 123);
In this case, your AAD would be the same, and therefore, your deputy can still be confused (albeit in a narrower use case).
In Cendyne’s article, AnonCo did something more subtle: The canonicalization bug created a collision on the inputs to HKDF, which resulted in an unintentional key reuse.
Up until this point, their mistake isn’t relevant to us, because we haven’t even explored key management at all. But the same design flaw can re-emerge in multiple locations, with drastically different consequence.
Multi-Tenancy
Once you’ve implemented a mitigation against Confused Deputies, you may think your job is done. And it very well could be.
Often times, however, software developers are tasked with building support for Bring Your Own Key (BYOK).
This is often spawned from a specific compliance requirement (such as cryptographic shredding; i.e. if you erase the key, you can no longer recover the plaintext, so it may as well be deleted).
Other times, this is driven by a need to cut costs: Storing different users’ data in the same database server, but encrypting it such that they can only encrypt their own records.
Two things can happen when you introduce multi-tenancy into your database cryptography designs:
- Invisible Salamanders becomes a risk, due to multiple keys being possible for any given encrypted record.
- Failure to address the risk of Invisible Salamanders can undermine your protection against Confused Deputies, thereby returning you to a state before you properly used the AAD.
So now you have to revisit your designs and ensure you’re using a key-committing authenticated mode, rather than just a regular authenticated mode.
Isn’t cryptography fun?
“What Are Invisible Salamanders?”
This refers to a fun property of AEAD modes based on Polynomical MACs. Basically, if you:
- Encrypt one message under a specific key and nonce.
- Encrypt another message under a separate key and nonce.
…Then you can get the same exact ciphertext and authentication tag. Performing this attack requires you to control the keys for both encryption operations.
This was first demonstrated in an attack against encrypted messaging applications, where a picture of a salamander was hidden from the abuse reporting feature because another attached file had the same authentication tag and ciphertext, and you could trick the system if you disclosed the second key instead of the first. Thus, the salamander is invisible to attackers.
Art: CMYKatWe’re not quite done with relational databases yet, but we should talk about NoSQL databases for a bit. The final topic in scope applies equally to both, after all.
Cryptography for NoSQL Databases
Most of the topics from relational databases also apply to NoSQL databases, so I shall refrain from duplicating them here. This article is already sufficiently long to read, after all, and I dislike redundancy.
NoSQL is Built Different
The main thing that NoSQL databases offer in the service of making cryptographers lose sleep at night is the schema-free nature of NoSQL designs.
What this means is that, if you’re using a client-side encryption library for a NoSQL database, the previous concerns about confused deputy attacks are amplified by the malleability of the document structure.
Additionally, the previously discussed cryptographic attacks against the encryption mode may be less expensive for an attacker to pull off.
Consider the following record structure, which stores a bunch of data stored with AES in CBC mode:
{ "encrypted-data-key": "<blob>", "name": "<ciphertext>", "address": [ "<ciphertext>", "<ciphertext>" ], "social-security": "<ciphertext>", "zip-code": "<ciphertext>"}If this record is decrypted with code that looks something like this:
$decrypted = [];// ... snip ...foreach ($record['address'] as $i => $addrLine) { try { $decrypted['address'][$i] = $this->decrypt($addrLine); } catch (Throwable $ex) { // You'd never deliberately do this, but it's for illustration $this->doSomethingAnOracleCanObserve($i); // This is more believable, of course: $this->logDecryptionError($ex, $addrLine); $decrypted['address'][$i] = ''; }}Then you can keep appending rows to the
Art: Harubaki"address"field to reduce the number of writes needed to exploit a padding oracle attack against any of the<ciphertext>fields.This isn’t to say that NoSQL is less secure than SQL, from the context of client-side encryption. However, the powerful feature sets that NoSQL users are accustomed to may also give attackers a more versatile toolkit to work with.
Record Authentication
A pedant may point out that record authentication applies to both SQL and NoSQL. However, I mostly only observe this feature in NoSQL databases and document storage systems in the wild, so I’m shoving it in here.
Encrypting fields is nice and all, but sometimes what you want to know is that your unencrypted data hasn’t been tampered with as it flows through your system.
The trivial way this is done is by using a digital signature algorithm over the whole record, and then appending the signature to the end. When you go to verify the record, all of the information you need is right there.
This works well enough for most use cases, and everyone can pack up and go home. Nothing more to see here.
Except…
When you’re working with NoSQL databases, you often want systems to be able to write to additional fields, and since you’re working with schema-free blobs of data rather than a normalized set of relatable tables, the most sensible thing to do is to is to append this data to the same record.
Except, oops! You can’t do that if you’re shoving a digital signature over the record. So now you need to specify which fields are to be included in the signature.
And you need to think about how to model that in a way that doesn’t prohibit schema upgrades nor allow attackers to perform downgrade attacks. (See below.)
I don’t have any specific real-world examples here that I can point to of this problem being solved well.Art: CMYKat
Furthermore, as with preventing confused deputy and/or canonicalization attacks above, you must also include the fully qualified path of each field in the data that gets signed.
As I said with encryption before, but also true here:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
This requirement holds true whether you’re using symmetric-key authentication (i.e. HMAC) or asymmetric-key digital signatures (e.g. EdDSA).
Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
Art: HarubakiOkay, how do you solve this problem so that you can perform updates and upgrades to your schema but without enabling attackers to downgrade the security? Here’s one possible design.
Let’s say you have two metadata fields on each record:
- A compressed binary string representing which fields should be authenticated. This field is, itself, not authenticated. Let’s call this
meta-auth. - A compressed binary string representing which of the authenticated fields should also be encrypted. This field is also authenticated. This is at most the same length as the first metadata field. Let’s call this
meta-enc.
Furthermore, you will specify a canonical field ordering for both how data is fed into the signature algorithm as well as the field mappings in
meta-authandmeta-enc.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-enc */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false /* example.superfluous.rewards-member */ ]), "signature": /* -- snip -- */}When you go to append data to an existing record, you’ll need to update
meta-authto include the mapping of fields based on this canonical ordering to ensure only the intended fields get validated.When you update your code to add an additional field that is intended to be signed, you can roll that out for new records and the record will continue to be self-describing:
- New records will have the additional field flagged as authenticated in
meta-auth(andmeta-encwill grow) - Old records will not, but your code will still sign them successfully
- To prevent downgrade attacks, simply include a schema version ID as an additional plaintext field that gets authenticated. An attacker who tries to downgrade will need to be able to produce a valid signature too.
You might think
meta-authgives an attacker some advantage, but this only includes which fields are included in the security boundary of the signature or MAC, which allows unauthenticated data to be appended for whatever operational purpose without having to update signatures or expose signing keys to a wider part of the network.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true, /* meta-enc */ true /* meta-version */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-version */ ]), "meta-version": 0x01000000, "signature": /* -- snip -- */}If an attacker tries to use the
meta-authfield to mess with a record, the best they can hope for is an Invalid Signature exception (assuming the signature algorithm is secure to begin with).Even if they keep all of the fields the same, but play around with the structure of the record (e.g. changing the XPath or equivalent), so long as the path is authenticated with each field, breaking this is computationally infeasible.
Searchable Encryption
If you’ve managed to make it through the previous sections, congratulations, you now know enough to build a secure but completely useless database.
Art: CMYKatOkay, put away the pitchforks; I will explain.
Part of the reason why we store data in a database, rather than a flat file, is because we want to do more than just read and write. Sometimes computer scientists want to compute. Almost always, you want to be able to query your database for a subset of records based on your specific business logic needs.
And so, a database which doesn’t do anything more than store ciphertext and maybe signatures is pretty useless to most people. You’d have better luck selling Monkey JPEGs to furries than convincing most businesses to part with their precious database-driven report generators.
Art: SophieSo whenever one of your users wants to actually use their data, rather than just store it, they’re forced to decide between two mutually exclusive options:
- Encrypting the data, to protect it from unauthorized disclosure, but render it useless
- Doing anything useful with the data, but leaving it unencrypted in the database
This is especially annoying for business types that are all in on the Zero Trust buzzword.
Fortunately, the cryptographers are at it again, and boy howdy do they have a lot of solutions for this problem.
Order-{Preserving, Revealing} Encryption
On the fun side of things, you have things like Order-Preserving and Order-Revealing Encryption, which Matthew Green wrote about at length.
[D]atabase encryption has been a controversial subject in our field. I wish I could say that there’s been an actual debate, but it’s more that different researchers have fallen into different camps, and nobody has really had the data to make their position in a compelling way. There have actually been some very personal arguments made about it.
Attack of the week: searchable encryption and the ever-expanding leakage function
The problem with these designs is that they have a significant enough leakage that it no longer provides semantic security.
From Grubbs, et al. (GLMP, 2019.)
Colors inverted to fit my blog’s theme better.To put it in other words: These designs are only marginally better than ECB mode, and probably deserve their own poems too.
Order revealing
Reveals much more than order
Softcore ECBOrder preserving
Semantic security?
Only in your dreamsHaiku for your consideration
Deterministic Encryption
Here’s a simpler, but also terrible, idea for searchable encryption: Simply give up on semantic security entirely.
If you recall the
AES_{De,En}crypt()functions built into MySQL I mentioned at the start of this article, those are the most common form of deterministic encryption I’ve seen in use.SELECT * FROM foo WHERE bar = AES_Encrypt('query', 'key');However, there are slightly less bad variants. If you use AES-GCM-SIV with a static nonce, your ciphertexts are fully deterministic, and you can encrypt a small number of distinct records safely before you’re no longer secure.
From Page 14 of the linked paper. Full view.That’s certainly better than nothing, but you also can’t mitigate confused deputy attacks. But we can do better than this.
Homomorphic Encryption
In a safer plane of academia, you’ll find homomorphic encryption, which researchers recently demonstrated with serving Wikipedia pages in a reasonable amount of time.
Homomorphic encryption allows computations over the ciphertext, which will be reflected in the plaintext, without ever revealing the key to the entity performing the computation.
If this sounds vaguely similar to the conditions that enable chosen-ciphertext attacks, you probably have a good intuition for how it works: RSA is homomorphic to multiplication, AES-CTR is homomorphic to XOR. Fully homomorphic encryption uses lattices, which enables multiple operations but carries a relatively enormous performance cost.
Art: HarubakiHomomorphic encryption sometimes intersects with machine learning, because the notion of training an encrypted model by feeding it encrypted data, then decrypting it after-the-fact is desirable for certain business verticals. Your data scientists never see your data, and you have some plausible deniability about the final ML model this work produces. This is like a Siren song for Venture Capitalist-backed medical technology companies. Tech journalists love writing about it.
However, a less-explored use case is the ability to encrypt your programs but still get the correct behavior and outputs. Although this sounds like a DRM technology, it’s actually something that individuals could one day use to prevent their ISPs or cloud providers from knowing what software is being executed on the customer’s leased hardware. The potential for a privacy win here is certainly worth pondering, even if you’re a tried and true Pirate Party member.
Just say “NO” to the copyright cartels.Art: CMYKat
Searchable Symmetric Encryption (SSE)
Forget about working at the level of fields and rows or individual records. What if we, instead, worked over collections of documents, where each document is viewed as a set of keywords from a keyword space?
Art: CMYKatThat’s the basic premise of SSE: Encrypting collections of documents rather than individual records.
The actual implementation details differ greatly between designs. They also differ greatly in their leakage profiles and susceptibility to side-channel attacks.
Some schemes use a so-called trapdoor permutation, such as RSA, as one of their building blocks.
Some schemes only allow for searching a static set of records, while others can accommodate new data over time (with the trade-off between more leakage or worse performance).
If you’re curious, you can learn more about SSE here, and see some open source SEE implementations online here.
You’re probably wondering, “If SSE is this well-studied and there are open source implementations available, why isn’t it more widely used?”
Your guess is as good as mine, but I can think of a few reasons:
- The protocols can be a little complicated to implement, and aren’t shipped by default in cryptography libraries (i.e. OpenSSL’s libcrypto or libsodium).
- Every known security risk in SSE is the product of a trade-offs, rather than there being a single winner for all use cases that developers can feel comfortable picking.
- Insufficient marketing and developer advocacy.
SSE schemes are mostly of interest to academics, although Seny Kamara (Brown Univeristy professior and one of the luminaries of searchable encryption) did try to develop an app called Pixek which used SSE to encrypt photos.
Maybe there’s room for a cryptography competition on searchable encryption schemes in the future.
You Can Have Little a HMAC, As a Treat
Finally, I can’t talk about searchable encryption without discussing a technique that’s older than dirt by Internet standards, that has been independently reinvented by countless software developers tasked with encrypting database records.
The oldest version I’ve been able to track down dates to 2006 by Raul Garcia at Microsoft, but I’m not confident that it didn’t exist before.
The idea I’m alluding to goes like this:
- Encrypt your data, securely, using symmetric cryptography.
(Hopefully your encryption addresses the considerations outlined in the relevant sections above.) - Separately, calculate an HMAC over the unencrypted data with a separate key used exclusively for indexing.
When you need to query your data, you can just recalculate the HMAC of your challenge and fetch the records that match it. Easy, right?
Even if you rotate your keys for encryption, you keep your indexing keys static across your entire data set. This lets you have durable indexes for encrypted data, which gives you the ability to do literal lookups for the performance hit of a hash function.
Additionally, everyone has HMAC in their toolkit, so you don’t have to move around implementations of complex cryptographic building blocks. You can live off the land. What’s not to love?
Hooray!However, if you stopped here, we regret to inform you that your data is no longer indistinguishable from random, which probably undermines the security proof for your encryption scheme.
How annoying!Of course, you don’t have to stop with the addition of plain HMAC to your database encryption software.
Take a page from Troy Hunt: Truncate the output to provide k-anonymity rather than a direct literal look-up.
“K-What Now?”
Imagine you have a full HMAC-SHA256 of the plaintext next to every ciphertext record with a static key, for searchability.
Each HMAC output corresponds 1:1 with a unique plaintext.
Because you’re using HMAC with a secret key, an attacker can’t just build a rainbow table like they would when attempting password cracking, but it still leaks duplicate plaintexts.
For example, an HMAC-SHA256 output might look like this:
Art: CMYKat\04a74e4c0158e34a566785d1a5e1167c4e3455c42aea173104e48ca810a8b1aeIf you were to slice off most of those bytes (e.g. leaving only the last 3, which in the previous example yields
a8b1ae), then with sufficient records, multiple plaintexts will now map to the same truncated HMAC tag.Which means if you’re only revealing a truncated HMAC tag to the database server (both when storing records or retrieving them), you can now expect false positives due to collisions in your truncated HMAC tag.
These false positives give your data a discrete set of anonymity (called k-anonymity), which means an attacker with access to your database cannot:
- Distinguish between two encrypted records with the same short HMAC tag.
- Reverse engineer the short HMAC tag into a single possible plaintext value, even if they can supply candidate queries and study the tags sent to the database.
As with SSE above, this short HMAC technique exposes a trade-off to users.
- Too much k-anonymity (i.e. too many false positives), and you will have to decrypt-then-discard multiple mismatching records. This can make queries slow.
- Not enough k-anonymity (i.e. insufficient false positives), and you’re no better off than a full HMAC.
Even more troublesome, the right amount to truncate is expressed in bits (not bytes), and calculating this value depends on the number of unique plaintext values you anticipate in your dataset. (Fortunately, it grows logarithmically, so you’ll rarely if ever have to tune this.)
If you’d like to play with this idea, here’s a quick and dirty demo script.
Intermission
If you started reading this post with any doubts about Cendyne’s statement that “Database cryptography is hard”, by making it to this point, they’ve probably been long since put to rest.
Art: HarubakiConversely, anyone that specializes in this topic is probably waiting for me to say anything novel or interesting; their patience wearing thin as I continue to rehash a surface-level introduction of their field without really diving deep into anything.
Thus, if you’ve read this far, I’d like to demonstrate the application of what I’ve covered thus far into a real-world case study into an database cryptography product.
Case Study: MongoDB Client-Side Encryption
MongoDB is an open source schema-free NoSQL database. Last year, MongoDB made waves when they announced Queryable Encryption in their upcoming client-side encryption release.
Taken from the press release, but adapted for dark themes.A statement at the bottom of their press release indicates that this isn’t clown-shoes:
Queryable Encryption was designed by MongoDB’s Advanced Cryptography Research Group, headed by Seny Kamara and Tarik Moataz, who are pioneers in the field of encrypted search. The Group conducts cutting-edge peer-reviewed research in cryptography and works with MongoDB engineering teams to transfer and deploy the latest innovations in cryptography and privacy to the MongoDB data platform.
If you recall, I mentioned Seny Kamara in the SSE section of this post. They certainly aren’t wrong about Kamara and Moataz being pioneers in this field.
So with that in mind, let’s explore the implementation in libmongocrypt and see how it stands up to scrutiny.
MongoCrypt: The Good
MongoDB’s encryption library takes key management seriously: They provide a KMS integration for cloud users by default (supporting both AWS and Azure).
MongoDB uses Encrypt-then-MAC with AES-CBC and HMAC-SHA256, which is congruent to what Signal does for message encryption.
How Is Queryable Encryption Implemented?
From the current source code, we can see that MongoCrypt generates several different types of tokens, using HMAC (calculation defined here).
According to their press release:
The feature supports equality searches, with additional query types such as range, prefix, suffix, and substring planned for future releases.
Which means that most of the juicy details probably aren’t public yet.
These HMAC-derived tokens are stored wholesale in the data structure, but most are encrypted before storage using AES-CTR.
There are more layers of encryption (using AEAD), server-side token processing, and more AES-CTR-encrypted edge tokens. All of this is finally serialized (implementation) as one blob for storage.
Since only the equality operation is currently supported (which is the same feature you’d get from HMAC), it’s difficult to speculate what the full feature set looks like.
However, since Kamara and Moataz are leading its development, it’s likely that this feature set will be excellent.
MongoCrypt: The Bad
Every call to
do_encrypt()includes at most the Key ID (but typicallyNULL) as the AAD. This means that the concerns over Confused Deputies (and NoSQL specifically) are relevant to MongoDB.However, even if they did support authenticating the fully qualified path to a field in the AAD for their encryption, their AEAD construction is vulnerable to the kind of canonicalization attack I wrote about previously.
First, observe this code which assembles the multi-part inputs into HMAC.
/* Construct the input to the HMAC */uint32_t num_intermediates = 0;_mongocrypt_buffer_t intermediates[3];// -- snip --if (!_mongocrypt_buffer_concat ( &to_hmac, intermediates, num_intermediates)) { CLIENT_ERR ("failed to allocate buffer"); goto done;}if (hmac == HMAC_SHA_512_256) { uint8_t storage[64]; _mongocrypt_buffer_t tag = {.data = storage, .len = sizeof (storage)}; if (!_crypto_hmac_sha_512 (crypto, Km, &to_hmac, &tag, status)) { goto done; } // Truncate sha512 to first 256 bits. memcpy (out->data, tag.data, MONGOCRYPT_HMAC_LEN);} else { BSON_ASSERT (hmac == HMAC_SHA_256); if (!_mongocrypt_hmac_sha_256 (crypto, Km, &to_hmac, out, status)) { goto done; }}The implementation of
_mongocrypt_buffer_concat()can be found here.If either the implementation of that function, or the code I snipped from my excerpt, had contained code that prefixed every segment of the AAD with the length of the segment (represented as a
uint64_tto make overflow infeasible), then their AEAD mode would not be vulnerable to canonicalization issues.Using TupleHash would also have prevented this issue.
Silver lining for MongoDB developers: Because the AAD is either a key ID or NULL, this isn’t exploitable in practice.
The first cryptographic flaw sort of cancels the second out.
If the libmongocrypt developers ever want to mitigate Confused Deputy attacks, they’ll need to address this canonicalization issue too.
MongoCrypt: The Ugly
MongoCrypt supports deterministic encryption.
If you specify deterministic encryption for a field, your application passes a deterministic initialization vector to AEAD.
We already discussed why this is bad above.
Wrapping Up
This was not a comprehensive treatment of the field of database cryptography. There are many areas of this field that I did not cover, nor do I feel qualified to discuss.
However, I hope anyone who takes the time to read this finds themselves more familiar with the subject.
Additionally, I hope any developers who think “encrypting data in a database is [easy, trivial] (select appropriate)” will find this broad introduction a humbling experience.
Art: CMYKathttps://soatok.blog/2023/03/01/database-cryptography-fur-the-rest-of-us/
#appliedCryptography #blockCipherModes #cryptography #databaseCryptography #databases #encryptedSearch #HMAC #MongoCrypt #MongoDB #QueryableEncryption #realWorldCryptography #security #SecurityGuidance #SQL #SSE #symmetricCryptography #symmetricSearchableEncryption
-
Earlier this year, Cendyne wrote a blog post covering the use of HKDF, building partially upon my own blog post about HKDF and the KDF security definition, but moreso inspired by a cryptographic issue they identified in another company’s product (dubbed AnonCo).
At the bottom they teased:
Database cryptography is hard. The above sketch is not complete and does not address several threats! This article is quite long, so I will not be sharing the fixes.
Cendyne
If you read Cendyne’s post, you may have nodded along with that remark and not appreciate the degree to which our naga friend was putting it mildly. So I thought I’d share some of my knowledge about real-world database cryptography in an accessible and fun format in the hopes that it might serve as an introduction to the specialization.
Note: I’m also not going to fix Cendyne’s sketch of AnonCo’s software here–partly because I don’t want to get in the habit of assigning homework or required reading, but mostly because it’s kind of obvious once you’ve learned the basics.
I’m including art of my fursona in this post… as is tradition for furry blogs.If you don’t like furries, please feel free to leave this blog and read about this topic elsewhere.
Thanks to CMYKat for the awesome stickers.
Contents
- Database Cryptography?
- Cryptography for Relational Databases
- The Perils of Built-in Encryption Functions
- Application-Layer Relational Database Cryptography
- Confused Deputies
- Canonicalization Attacks
- Multi-Tenancy
- Cryptography for NoSQL Databases
- NoSQL is Built Different
- Record Authentication
- Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
- Searchable Encryption
- Order-{Preserving, Revealing} Encryption
- Deterministic Encryption
- Homomorphic Encryption
- Searchable Symmetric Encryption (SSE)
- You Can Have Little a HMAC, As a Treat
- Intermission
- Case Study: MongoDB Client-Side Encryption
- MongoCrypt: The Good
- How is Queryable Encryption Implemented?
- MongoCrypt: The Bad
- MongoCrypt: The Ugly
- MongoCrypt: The Good
- Wrapping Up
Database Cryptography?
The premise of database cryptography is deceptively simple: You have a database, of some sort, and you want to store sensitive data in said database.
The consequences of this simple premise are anything but simple. Let me explain.
Art: ScruffKerfluffThe sensitive data you want to store may need to remain confidential, or you may need to provide some sort of integrity guarantees throughout your entire system, or sometimes both. Sometimes all of your data is sensitive, sometimes only some of it is. Sometimes the confidentiality requirements of your data extends to where within a dataset the record you want actually lives. Sometimes that’s true of some data, but not others, so your cryptography has to be flexible to support multiple types of workloads.
Other times, you just want your disks encrypted at rest so if they grow legs and walk out of the data center, the data cannot be comprehended by an attacker. And you can’t be bothered to work on this problem any deeper. This is usually what compliance requirements cover. Boxes get checked, executives feel safer about their operation, and the whole time nobody has really analyzed the risks they’re facing.
But we’re not settling for mere compliance on this blog. Furries have standards, after all.
So the first thing you need to do before diving into database cryptography is threat modelling. The first step in any good threat model is taking inventory; especially of assumptions, requirements, and desired outcomes. A few good starter questions:
- What database software is being used? Is it up to date?
- What data is being stored in which database software?
- How are databases oriented in the network of the overall system?
- Is your database properly firewalled from the public Internet?
- How does data flow throughout the network, and when do these data flows intersect with the database?
- Which applications talk to the database? What languages are they written in? Which APIs do they use?
- How will cryptography secrets be managed?
- Is there one key for everyone, one key per tenant, etc.?
- How are keys rotated?
- Do you use envelope encryption with an HSM, or vend the raw materials to your end devices?
The first two questions are paramount for deciding how to write software for database cryptography, before you even get to thinking about the cryptography itself.
(This is not a comprehensive set of questions to ask, either. A formal threat model is much deeper in the weeds.)
The kind of cryptography protocol you need for, say, storing encrypted CSV files an S3 bucket is vastly different from relational (SQL) databases, which in turn will be significantly different from schema-free (NoSQL) databases.
Furthermore, when you get to the point that you can start to think about the cryptography, you’ll often need to tackle confidentiality and integrity separately.
If that’s unclear, think of a scenario like, “I need to encrypt PII, but I also need to digitally sign the lab results so I know it wasn’t tampered with at rest.”
My point is, right off the bat, we’ve got a three-dimensional matrix of complexity to contend with:
- On one axis, we have the type of database.
- Flat-file
- Relational
- Schema-free
- On another, we have the basic confidentiality requirements of the data.
- Field encryption
- Row encryption
- Column encryption
- Unstructured record encryption
- Encrypting entire collections of records
- Finally, we have the integrity requirements of the data.
- Field authentication
- Row/column authentication
- Unstructured record authentication
- Collection authentication (based on e.g. Sparse Merkle Trees)
And then you have a fourth dimension that often falls out of operational requirements for databases: Searchability.
Why store data in a database if you have no way to index or search the data for fast retrieval?
Credit: HarubakiIf you’re starting to feel overwhelmed, you’re not alone. A lot of developers drastically underestimate the difficulty of the undertaking, until they run head-first into the complexity.
Some just phone it in with
AES_Encrypt()calls in their MySQL queries. (Too bad ECB mode doesn’t provide semantic security!)Which brings us to the meat of this blog post: The actual cryptography part.
Cryptography is the art of transforming information security problems into key management problems.
Former coworker
Note: In the interest of time, I’m skipping over flat files and focusing instead on actual database technologies.
Cryptography for Relational Databases
Encrypting data in an SQL database seems simple enough, even if you’ve managed to shake off the complexity I teased from the introduction.
You’ve got data, you’ve got a column on a table. Just encrypt the data and shove it in a cell on that column and call it a day, right?
But, alas, this is a trap. There are so many gotchas that I can’t weave a coherent, easy-to-follow narrative between them all.
So let’s start with a simple question: where and how are you performing your encryption?
The Perils of Built-in Encryption Functions
MySQL provides functions called AES_Encrypt and AES_Decrypt, which many developers have unfortunately decided to rely on in the past.
It’s unfortunate because these functions implement ECB mode. To illustrate why ECB mode is bad, I encrypted one of my art commissions with AES in ECB mode:
Art by Riley, encrypted with AES-ECBThe problems with ECB mode aren’t exactly “you can see the image through it,” because ECB-encrypting a compressed image won’t have redundancy (and thus can make you feel safer than you are).
ECB art is a good visual for the actual issue you should care about, however: A lack of semantic security.
A cryptosystem is considered semantically secure if observing the ciphertext doesn’t reveal information about the plaintext (except, perhaps, the length; which all cryptosystems leak to some extent). More information here.
ECB art isn’t to be confused with ECB poetry, which looks like this:
Oh little one, you’re growing up
You’ll soon be writing C
You’ll treat your ints as pointers
You’ll nest the ternary
You’ll cut and paste from github
And try cryptography
But even in your darkest hour
Do not use ECBCBC’s BEASTly when padding’s abused
And CTR’s fine til a nonce is reused
Some say it’s a CRIME to compress then encrypt
Or store keys in the browser (or use javascript)
Diffie Hellman will collapse if hackers choose your g
And RSA is full of traps when e is set to 3
Whiten! Blind! In constant time! Don’t write an RNG!
But failing all, and listen well: Do not use ECBThey’ll say “It’s like a one-time-pad!
The data’s short, it’s not so bad
the keys are long–they’re iron clad
I have a PhD!”
And then you’re front page Hacker News
Your passwords cracked–Adobe Blues.
Don’t leave your penguins showing through,
Do not use ECB— Ben Nagy, PoC||GTFO 0x04:13
Most people reading this probably know better than to use ECB mode already, and don’t need any of these reminders, but there is still a lot of code that inadvertently uses ECB mode to encrypt data in the database.
Also,
Credit: CMYKattSHOW processlist;leaks your encryption keys. Oops.Application-layer Relational Database Cryptography
Whether burned by ECB or just cautious about not giving your secrets to the system that stores all the ciphertext protected by said secret, a common next step for developers is to simply encrypt in their server-side application code.
And, yes, that’s part of the answer. But how you encrypt is important.
Credit: Harubaki“I’ll encrypt with CBC mode.”
If you don’t authenticate your ciphertext, you’ll be sorry. Maybe try again?“Okay, fine, I’ll use an authenticated mode like GCM.”
Did you remember to make the table and column name part of your AAD? What about the primary key of the record?“What on Earth are you talking about, Soatok?”
Welcome to the first footgun of database cryptography!Confused Deputies
Encrypting your sensitive data is necessary, but not sufficient. You need to also bind your ciphertexts to the specific context in which they are stored.
To understand why, let’s take a step back: What specific threat does encrypting your database records protect against?
We’ve already established that “your disks walk out of the datacenter” is a “full disk encryption” problem, so if you’re using application-layer cryptography to encrypt data in a relational database, your threat model probably involves unauthorized access to the database server.
What, then, stops an attacker from copying ciphertexts around?
Credit: CMYKattLet’s say I have a legitimate user account with an ID 12345, and I want to read your street address, but it’s encrypted in the database. But because I’m a clever hacker, I have unfettered access to your relational database server.
All I would need to do is simply…
UPDATE table SET addr_encrypted = 'your-ciphertext' WHERE id = 12345…and then access the application through my legitimate access. Bam, data leaked. As an attacker, I can probably even copy fields from other columns and it will just decrypt. Even if you’re using an authenticated mode.
We call this a confused deputy attack, because the deputy (the component of the system that has been delegated some authority or privilege) has become confused by the attacker, and thus undermined an intended security goal.
The fix is to use the AAD parameter from the authenticated mode to bind the data to a given context. (AAD = Additional Authenticated Data.)
- $addr = aes_gcm_encrypt($addr, $key);+ $addr = aes_gcm_encrypt($addr, $key, canonicalize([+ $tableName,+ $columnName,+ $primaryKey+ ]);
Now if I start cutting and pasting ciphertexts around, I get a decryption failure instead of silently decrypting plaintext.
This may sound like a specific vulnerability, but it’s more of a failure to understand an important general lesson with database cryptography:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
Canonicalization Attacks
In the previous section, I introduced a pseudocode called
canonicalize(). This isn’t a pasto from some reference code; it’s an important design detail that I will elaborate on now.First, consider you didn’t do anything to canonicalize your data, and you just joined strings together and called it a day…
function dumbCanonicalize( string $tableName, string $columnName, string|int $primaryKey): string { return $tableName . '_' . $columnName . '#' . $primaryKey;}Consider these two inputs to this function:
dumbCanonicalize('customers', 'last_order_uuid', 123);dumbCanonicalize('customers_last_order', 'uuid', 123);
In this case, your AAD would be the same, and therefore, your deputy can still be confused (albeit in a narrower use case).
In Cendyne’s article, AnonCo did something more subtle: The canonicalization bug created a collision on the inputs to HKDF, which resulted in an unintentional key reuse.
Up until this point, their mistake isn’t relevant to us, because we haven’t even explored key management at all. But the same design flaw can re-emerge in multiple locations, with drastically different consequence.
Multi-Tenancy
Once you’ve implemented a mitigation against Confused Deputies, you may think your job is done. And it very well could be.
Often times, however, software developers are tasked with building support for Bring Your Own Key (BYOK).
This is often spawned from a specific compliance requirement (such as cryptographic shredding; i.e. if you erase the key, you can no longer recover the plaintext, so it may as well be deleted).
Other times, this is driven by a need to cut costs: Storing different users’ data in the same database server, but encrypting it such that they can only encrypt their own records.
Two things can happen when you introduce multi-tenancy into your database cryptography designs:
- Invisible Salamanders becomes a risk, due to multiple keys being possible for any given encrypted record.
- Failure to address the risk of Invisible Salamanders can undermine your protection against Confused Deputies, thereby returning you to a state before you properly used the AAD.
So now you have to revisit your designs and ensure you’re using a key-committing authenticated mode, rather than just a regular authenticated mode.
Isn’t cryptography fun?
“What Are Invisible Salamanders?”
This refers to a fun property of AEAD modes based on Polynomical MACs. Basically, if you:
- Encrypt one message under a specific key and nonce.
- Encrypt another message under a separate key and nonce.
…Then you can get the same exact ciphertext and authentication tag. Performing this attack requires you to control the keys for both encryption operations.
This was first demonstrated in an attack against encrypted messaging applications, where a picture of a salamander was hidden from the abuse reporting feature because another attached file had the same authentication tag and ciphertext, and you could trick the system if you disclosed the second key instead of the first. Thus, the salamander is invisible to attackers.
Art: CMYKatWe’re not quite done with relational databases yet, but we should talk about NoSQL databases for a bit. The final topic in scope applies equally to both, after all.
Cryptography for NoSQL Databases
Most of the topics from relational databases also apply to NoSQL databases, so I shall refrain from duplicating them here. This article is already sufficiently long to read, after all, and I dislike redundancy.
NoSQL is Built Different
The main thing that NoSQL databases offer in the service of making cryptographers lose sleep at night is the schema-free nature of NoSQL designs.
What this means is that, if you’re using a client-side encryption library for a NoSQL database, the previous concerns about confused deputy attacks are amplified by the malleability of the document structure.
Additionally, the previously discussed cryptographic attacks against the encryption mode may be less expensive for an attacker to pull off.
Consider the following record structure, which stores a bunch of data stored with AES in CBC mode:
{ "encrypted-data-key": "<blob>", "name": "<ciphertext>", "address": [ "<ciphertext>", "<ciphertext>" ], "social-security": "<ciphertext>", "zip-code": "<ciphertext>"}If this record is decrypted with code that looks something like this:
$decrypted = [];// ... snip ...foreach ($record['address'] as $i => $addrLine) { try { $decrypted['address'][$i] = $this->decrypt($addrLine); } catch (Throwable $ex) { // You'd never deliberately do this, but it's for illustration $this->doSomethingAnOracleCanObserve($i); // This is more believable, of course: $this->logDecryptionError($ex, $addrLine); $decrypted['address'][$i] = ''; }}Then you can keep appending rows to the
Art: Harubaki"address"field to reduce the number of writes needed to exploit a padding oracle attack against any of the<ciphertext>fields.This isn’t to say that NoSQL is less secure than SQL, from the context of client-side encryption. However, the powerful feature sets that NoSQL users are accustomed to may also give attackers a more versatile toolkit to work with.
Record Authentication
A pedant may point out that record authentication applies to both SQL and NoSQL. However, I mostly only observe this feature in NoSQL databases and document storage systems in the wild, so I’m shoving it in here.
Encrypting fields is nice and all, but sometimes what you want to know is that your unencrypted data hasn’t been tampered with as it flows through your system.
The trivial way this is done is by using a digital signature algorithm over the whole record, and then appending the signature to the end. When you go to verify the record, all of the information you need is right there.
This works well enough for most use cases, and everyone can pack up and go home. Nothing more to see here.
Except…
When you’re working with NoSQL databases, you often want systems to be able to write to additional fields, and since you’re working with schema-free blobs of data rather than a normalized set of relatable tables, the most sensible thing to do is to is to append this data to the same record.
Except, oops! You can’t do that if you’re shoving a digital signature over the record. So now you need to specify which fields are to be included in the signature.
And you need to think about how to model that in a way that doesn’t prohibit schema upgrades nor allow attackers to perform downgrade attacks. (See below.)
I don’t have any specific real-world examples here that I can point to of this problem being solved well.Art: CMYKat
Furthermore, as with preventing confused deputy and/or canonicalization attacks above, you must also include the fully qualified path of each field in the data that gets signed.
As I said with encryption before, but also true here:
Where your data lives is part of its identity, and MUST be authenticated.
Soatok’s Rule of Database Cryptography
This requirement holds true whether you’re using symmetric-key authentication (i.e. HMAC) or asymmetric-key digital signatures (e.g. EdDSA).
Bonus: A Maximally Schema-Free, Upgradeable Authentication Design
Art: HarubakiOkay, how do you solve this problem so that you can perform updates and upgrades to your schema but without enabling attackers to downgrade the security? Here’s one possible design.
Let’s say you have two metadata fields on each record:
- A compressed binary string representing which fields should be authenticated. This field is, itself, not authenticated. Let’s call this
meta-auth. - A compressed binary string representing which of the authenticated fields should also be encrypted. This field is also authenticated. This is at most the same length as the first metadata field. Let’s call this
meta-enc.
Furthermore, you will specify a canonical field ordering for both how data is fed into the signature algorithm as well as the field mappings in
meta-authandmeta-enc.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-enc */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false /* example.superfluous.rewards-member */ ]), "signature": /* -- snip -- */}When you go to append data to an existing record, you’ll need to update
meta-authto include the mapping of fields based on this canonical ordering to ensure only the intended fields get validated.When you update your code to add an additional field that is intended to be signed, you can roll that out for new records and the record will continue to be self-describing:
- New records will have the additional field flagged as authenticated in
meta-auth(andmeta-encwill grow) - Old records will not, but your code will still sign them successfully
- To prevent downgrade attacks, simply include a schema version ID as an additional plaintext field that gets authenticated. An attacker who tries to downgrade will need to be able to produce a valid signature too.
You might think
meta-authgives an attacker some advantage, but this only includes which fields are included in the security boundary of the signature or MAC, which allows unauthenticated data to be appended for whatever operational purpose without having to update signatures or expose signing keys to a wider part of the network.{ "example": { "credit-card": { "number": /* encrypted */, "expiration": /* encrypted */, "ccv": /* encrypted */ }, "superfluous": { "rewards-member": null } }, "meta-auth": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true, /* meta-enc */ true /* meta-version */ ]), "meta-enc": compress_bools([ true, /* example.credit-card.number */ true, /* example.credit-card.expiration */ true, /* example.credit-card.ccv */ false, /* example.superfluous.rewards-member */ true /* meta-version */ ]), "meta-version": 0x01000000, "signature": /* -- snip -- */}If an attacker tries to use the
meta-authfield to mess with a record, the best they can hope for is an Invalid Signature exception (assuming the signature algorithm is secure to begin with).Even if they keep all of the fields the same, but play around with the structure of the record (e.g. changing the XPath or equivalent), so long as the path is authenticated with each field, breaking this is computationally infeasible.
Searchable Encryption
If you’ve managed to make it through the previous sections, congratulations, you now know enough to build a secure but completely useless database.
Art: CMYKatOkay, put away the pitchforks; I will explain.
Part of the reason why we store data in a database, rather than a flat file, is because we want to do more than just read and write. Sometimes computer scientists want to compute. Almost always, you want to be able to query your database for a subset of records based on your specific business logic needs.
And so, a database which doesn’t do anything more than store ciphertext and maybe signatures is pretty useless to most people. You’d have better luck selling Monkey JPEGs to furries than convincing most businesses to part with their precious database-driven report generators.
Art: SophieSo whenever one of your users wants to actually use their data, rather than just store it, they’re forced to decide between two mutually exclusive options:
- Encrypting the data, to protect it from unauthorized disclosure, but render it useless
- Doing anything useful with the data, but leaving it unencrypted in the database
This is especially annoying for business types that are all in on the Zero Trust buzzword.
Fortunately, the cryptographers are at it again, and boy howdy do they have a lot of solutions for this problem.
Order-{Preserving, Revealing} Encryption
On the fun side of things, you have things like Order-Preserving and Order-Revealing Encryption, which Matthew Green wrote about at length.
[D]atabase encryption has been a controversial subject in our field. I wish I could say that there’s been an actual debate, but it’s more that different researchers have fallen into different camps, and nobody has really had the data to make their position in a compelling way. There have actually been some very personal arguments made about it.
Attack of the week: searchable encryption and the ever-expanding leakage function
The problem with these designs is that they have a significant enough leakage that it no longer provides semantic security.
From Grubbs, et al. (GLMP, 2019.)
Colors inverted to fit my blog’s theme better.To put it in other words: These designs are only marginally better than ECB mode, and probably deserve their own poems too.
Order revealing
Reveals much more than order
Softcore ECBOrder preserving
Semantic security?
Only in your dreamsHaiku for your consideration
Deterministic Encryption
Here’s a simpler, but also terrible, idea for searchable encryption: Simply give up on semantic security entirely.
If you recall the
AES_{De,En}crypt()functions built into MySQL I mentioned at the start of this article, those are the most common form of deterministic encryption I’ve seen in use.SELECT * FROM foo WHERE bar = AES_Encrypt('query', 'key');However, there are slightly less bad variants. If you use AES-GCM-SIV with a static nonce, your ciphertexts are fully deterministic, and you can encrypt a small number of distinct records safely before you’re no longer secure.
From Page 14 of the linked paper. Full view.That’s certainly better than nothing, but you also can’t mitigate confused deputy attacks. But we can do better than this.
Homomorphic Encryption
In a safer plane of academia, you’ll find homomorphic encryption, which researchers recently demonstrated with serving Wikipedia pages in a reasonable amount of time.
Homomorphic encryption allows computations over the ciphertext, which will be reflected in the plaintext, without ever revealing the key to the entity performing the computation.
If this sounds vaguely similar to the conditions that enable chosen-ciphertext attacks, you probably have a good intuition for how it works: RSA is homomorphic to multiplication, AES-CTR is homomorphic to XOR. Fully homomorphic encryption uses lattices, which enables multiple operations but carries a relatively enormous performance cost.
Art: HarubakiHomomorphic encryption sometimes intersects with machine learning, because the notion of training an encrypted model by feeding it encrypted data, then decrypting it after-the-fact is desirable for certain business verticals. Your data scientists never see your data, and you have some plausible deniability about the final ML model this work produces. This is like a Siren song for Venture Capitalist-backed medical technology companies. Tech journalists love writing about it.
However, a less-explored use case is the ability to encrypt your programs but still get the correct behavior and outputs. Although this sounds like a DRM technology, it’s actually something that individuals could one day use to prevent their ISPs or cloud providers from knowing what software is being executed on the customer’s leased hardware. The potential for a privacy win here is certainly worth pondering, even if you’re a tried and true Pirate Party member.
Just say “NO” to the copyright cartels.Art: CMYKat
Searchable Symmetric Encryption (SSE)
Forget about working at the level of fields and rows or individual records. What if we, instead, worked over collections of documents, where each document is viewed as a set of keywords from a keyword space?
Art: CMYKatThat’s the basic premise of SSE: Encrypting collections of documents rather than individual records.
The actual implementation details differ greatly between designs. They also differ greatly in their leakage profiles and susceptibility to side-channel attacks.
Some schemes use a so-called trapdoor permutation, such as RSA, as one of their building blocks.
Some schemes only allow for searching a static set of records, while others can accommodate new data over time (with the trade-off between more leakage or worse performance).
If you’re curious, you can learn more about SSE here, and see some open source SEE implementations online here.
You’re probably wondering, “If SSE is this well-studied and there are open source implementations available, why isn’t it more widely used?”
Your guess is as good as mine, but I can think of a few reasons:
- The protocols can be a little complicated to implement, and aren’t shipped by default in cryptography libraries (i.e. OpenSSL’s libcrypto or libsodium).
- Every known security risk in SSE is the product of a trade-offs, rather than there being a single winner for all use cases that developers can feel comfortable picking.
- Insufficient marketing and developer advocacy.
SSE schemes are mostly of interest to academics, although Seny Kamara (Brown Univeristy professior and one of the luminaries of searchable encryption) did try to develop an app called Pixek which used SSE to encrypt photos.
Maybe there’s room for a cryptography competition on searchable encryption schemes in the future.
You Can Have Little a HMAC, As a Treat
Finally, I can’t talk about searchable encryption without discussing a technique that’s older than dirt by Internet standards, that has been independently reinvented by countless software developers tasked with encrypting database records.
The oldest version I’ve been able to track down dates to 2006 by Raul Garcia at Microsoft, but I’m not confident that it didn’t exist before.
The idea I’m alluding to goes like this:
- Encrypt your data, securely, using symmetric cryptography.
(Hopefully your encryption addresses the considerations outlined in the relevant sections above.) - Separately, calculate an HMAC over the unencrypted data with a separate key used exclusively for indexing.
When you need to query your data, you can just recalculate the HMAC of your challenge and fetch the records that match it. Easy, right?
Even if you rotate your keys for encryption, you keep your indexing keys static across your entire data set. This lets you have durable indexes for encrypted data, which gives you the ability to do literal lookups for the performance hit of a hash function.
Additionally, everyone has HMAC in their toolkit, so you don’t have to move around implementations of complex cryptographic building blocks. You can live off the land. What’s not to love?
Hooray!However, if you stopped here, we regret to inform you that your data is no longer indistinguishable from random, which probably undermines the security proof for your encryption scheme.
How annoying!Of course, you don’t have to stop with the addition of plain HMAC to your database encryption software.
Take a page from Troy Hunt: Truncate the output to provide k-anonymity rather than a direct literal look-up.
“K-What Now?”
Imagine you have a full HMAC-SHA256 of the plaintext next to every ciphertext record with a static key, for searchability.
Each HMAC output corresponds 1:1 with a unique plaintext.
Because you’re using HMAC with a secret key, an attacker can’t just build a rainbow table like they would when attempting password cracking, but it still leaks duplicate plaintexts.
For example, an HMAC-SHA256 output might look like this:
Art: CMYKat\04a74e4c0158e34a566785d1a5e1167c4e3455c42aea173104e48ca810a8b1aeIf you were to slice off most of those bytes (e.g. leaving only the last 3, which in the previous example yields
a8b1ae), then with sufficient records, multiple plaintexts will now map to the same truncated HMAC tag.Which means if you’re only revealing a truncated HMAC tag to the database server (both when storing records or retrieving them), you can now expect false positives due to collisions in your truncated HMAC tag.
These false positives give your data a discrete set of anonymity (called k-anonymity), which means an attacker with access to your database cannot:
- Distinguish between two encrypted records with the same short HMAC tag.
- Reverse engineer the short HMAC tag into a single possible plaintext value, even if they can supply candidate queries and study the tags sent to the database.
As with SSE above, this short HMAC technique exposes a trade-off to users.
- Too much k-anonymity (i.e. too many false positives), and you will have to decrypt-then-discard multiple mismatching records. This can make queries slow.
- Not enough k-anonymity (i.e. insufficient false positives), and you’re no better off than a full HMAC.
Even more troublesome, the right amount to truncate is expressed in bits (not bytes), and calculating this value depends on the number of unique plaintext values you anticipate in your dataset. (Fortunately, it grows logarithmically, so you’ll rarely if ever have to tune this.)
If you’d like to play with this idea, here’s a quick and dirty demo script.
Intermission
If you started reading this post with any doubts about Cendyne’s statement that “Database cryptography is hard”, by making it to this point, they’ve probably been long since put to rest.
Art: HarubakiConversely, anyone that specializes in this topic is probably waiting for me to say anything novel or interesting; their patience wearing thin as I continue to rehash a surface-level introduction of their field without really diving deep into anything.
Thus, if you’ve read this far, I’d like to demonstrate the application of what I’ve covered thus far into a real-world case study into an database cryptography product.
Case Study: MongoDB Client-Side Encryption
MongoDB is an open source schema-free NoSQL database. Last year, MongoDB made waves when they announced Queryable Encryption in their upcoming client-side encryption release.
Taken from the press release, but adapted for dark themes.A statement at the bottom of their press release indicates that this isn’t clown-shoes:
Queryable Encryption was designed by MongoDB’s Advanced Cryptography Research Group, headed by Seny Kamara and Tarik Moataz, who are pioneers in the field of encrypted search. The Group conducts cutting-edge peer-reviewed research in cryptography and works with MongoDB engineering teams to transfer and deploy the latest innovations in cryptography and privacy to the MongoDB data platform.
If you recall, I mentioned Seny Kamara in the SSE section of this post. They certainly aren’t wrong about Kamara and Moataz being pioneers in this field.
So with that in mind, let’s explore the implementation in libmongocrypt and see how it stands up to scrutiny.
MongoCrypt: The Good
MongoDB’s encryption library takes key management seriously: They provide a KMS integration for cloud users by default (supporting both AWS and Azure).
MongoDB uses Encrypt-then-MAC with AES-CBC and HMAC-SHA256, which is congruent to what Signal does for message encryption.
How Is Queryable Encryption Implemented?
From the current source code, we can see that MongoCrypt generates several different types of tokens, using HMAC (calculation defined here).
According to their press release:
The feature supports equality searches, with additional query types such as range, prefix, suffix, and substring planned for future releases.
Which means that most of the juicy details probably aren’t public yet.
These HMAC-derived tokens are stored wholesale in the data structure, but most are encrypted before storage using AES-CTR.
There are more layers of encryption (using AEAD), server-side token processing, and more AES-CTR-encrypted edge tokens. All of this is finally serialized (implementation) as one blob for storage.
Since only the equality operation is currently supported (which is the same feature you’d get from HMAC), it’s difficult to speculate what the full feature set looks like.
However, since Kamara and Moataz are leading its development, it’s likely that this feature set will be excellent.
MongoCrypt: The Bad
Every call to
do_encrypt()includes at most the Key ID (but typicallyNULL) as the AAD. This means that the concerns over Confused Deputies (and NoSQL specifically) are relevant to MongoDB.However, even if they did support authenticating the fully qualified path to a field in the AAD for their encryption, their AEAD construction is vulnerable to the kind of canonicalization attack I wrote about previously.
First, observe this code which assembles the multi-part inputs into HMAC.
/* Construct the input to the HMAC */uint32_t num_intermediates = 0;_mongocrypt_buffer_t intermediates[3];// -- snip --if (!_mongocrypt_buffer_concat ( &to_hmac, intermediates, num_intermediates)) { CLIENT_ERR ("failed to allocate buffer"); goto done;}if (hmac == HMAC_SHA_512_256) { uint8_t storage[64]; _mongocrypt_buffer_t tag = {.data = storage, .len = sizeof (storage)}; if (!_crypto_hmac_sha_512 (crypto, Km, &to_hmac, &tag, status)) { goto done; } // Truncate sha512 to first 256 bits. memcpy (out->data, tag.data, MONGOCRYPT_HMAC_LEN);} else { BSON_ASSERT (hmac == HMAC_SHA_256); if (!_mongocrypt_hmac_sha_256 (crypto, Km, &to_hmac, out, status)) { goto done; }}The implementation of
_mongocrypt_buffer_concat()can be found here.If either the implementation of that function, or the code I snipped from my excerpt, had contained code that prefixed every segment of the AAD with the length of the segment (represented as a
uint64_tto make overflow infeasible), then their AEAD mode would not be vulnerable to canonicalization issues.Using TupleHash would also have prevented this issue.
Silver lining for MongoDB developers: Because the AAD is either a key ID or NULL, this isn’t exploitable in practice.
The first cryptographic flaw sort of cancels the second out.
If the libmongocrypt developers ever want to mitigate Confused Deputy attacks, they’ll need to address this canonicalization issue too.
MongoCrypt: The Ugly
MongoCrypt supports deterministic encryption.
If you specify deterministic encryption for a field, your application passes a deterministic initialization vector to AEAD.
We already discussed why this is bad above.
Wrapping Up
This was not a comprehensive treatment of the field of database cryptography. There are many areas of this field that I did not cover, nor do I feel qualified to discuss.
However, I hope anyone who takes the time to read this finds themselves more familiar with the subject.
Additionally, I hope any developers who think “encrypting data in a database is [easy, trivial] (select appropriate)” will find this broad introduction a humbling experience.
Art: CMYKathttps://soatok.blog/2023/03/01/database-cryptography-fur-the-rest-of-us/
#appliedCryptography #blockCipherModes #cryptography #databaseCryptography #databases #encryptedSearch #HMAC #MongoCrypt #MongoDB #QueryableEncryption #realWorldCryptography #security #SecurityGuidance #SQL #SSE #symmetricCryptography #symmetricSearchableEncryption
-
Google’s AirTable rival, Tables, graduates from beta test to become a Google Cloud product - Last fall, Google’s in-house incubator Area 120 introduced a new work-tracking too... - http://feedproxy.google.com/~r/Techcrunch/~3/JCzROdZ7KaE/ #inventorymanagement #cloudapplications #googlesheets #technology #computing #airtable #appsheet #google #tables #apps #crm #tc
-
Key Transparency and the Right to be Forgotten
This post is the first in a new series covering some of the reasoning behind decisions made in my project to build end-to-end encryption for direct messages on the Fediverse.
(Collectively, Fedi-E2EE.)
Although the reasons for specific design decisions should be immediately obvious from reading the relevant specification (and if not, I consider that a bug in the specification), I believe writing about it less formally will improve the clarity behind the specific design decisions taken.
In the inaugural post for this series, I’d like to focus on how the Fedi-E2EE Public Key Directory specification aims to provide Key Transparency and an Authority-free PKI for the Fediverse without making GDPR compliance logically impossible.
CMYKat‘s art, edited by me.Background
Key Transparency
For a clearer background, I recommend reading my blog post announcing the focused effort on a Public Key Directory, and then my update from August 2024.
If you’re in a hurry, I’ll be brief:
The goal of Key Transparency is to ensure everyone in a network sees the same view of who has which public key.
How it accomplishes this is a little complicated: It involves Merkle trees, digital signatures, and a higher-level protocol of distinct actions that affect the state machine.
If you’re thinking “blockchain”, you’re in the right ballpark, but we aren’t propping up a cryptocurrency. Instead, we’re using a centralized publisher model (per Public Key Directory instance) with decentralized verification.
Add a bit of cross-signing and replication, and you can stitch together a robust network of Public Key Directories that can be queried to obtain the currently-trusted list of public keys (or other auxiliary data) for a given Fediverse user. This can then be used to build application-layer protocols (i.e., end-to-end encryption with an identity key more robust than “trust on first use” due to the built-in audit trail to Merkle trees).
I’m handwaving a lot of details here. The Architecture and Specification documents are both worth a read if you’re curious to learn more.
HarubakiRight To Be Forgotten
I am not a lawyer, nor do I play one on TV. This is not legal advice. Other standard disclaimers go here.
Okay, now that we’ve got that out of the way, Article 17 of the GDPR establishes a “Right to erasure” for Personal Data.
What this actually means in practice has not been consistently decided by the courts yet. However, a publicly readable, immutable ledger that maps public keys (which may be considered Personal Data) with Actor IDs (which includes usernames, which are definitely Personal Data) goes against the grain when it comes to GDPR.
It remains an open question of there is public interest in this data persisting in a read-only ledger ad infinitum, which could override the right to be forgotten. If there is, that’s for the courts to decide, not furry tech bloggers.
I know it can be tempting, especially as an American with no presence in the European Union, to shrug and say, “That seems like a them problem.” However, if other folks want to be able to use my designs within the EU, I would be remiss to at least consider this potential pitfall and try to mitigate it in my designs.
So that’s exactly what I did.
AJAlmost Contradictory
At first glance, the privacy goals of both Key Transparency and the GDPR’s Right To Erasure are at odds.
- One creates an immutable, append-only history.
- The other establishes a right for EU citizens’ history to be selectively censored, which means history has to be mutable.
However, they’re not totally impossible to reconcile.
An untested legal theory circulating around large American tech companies is that “crypto shredding” is legally equivalent to erasure.
Crypto shredding is the act of storing encrypted data, and then when given a legal takedown request from an EU citizen, deleting the key instead of the data.
AJThis works from a purely technical perspective: If the data is encrypted, and you don’t know the key, to you it’s indistinguishable from someone who encrypted the same number of NUL bytes.
In fact, many security proofs for encryption schemes are satisfied by reaching this conclusion, so this isn’t a crazy notion.
Is Crypto Shredding Plausible?
In 2019, the European Parliamentary Research Service published a lengthy report titled Blockchain and the General Data Protection Regulation which states the following:
Before any examination of whether blockchain technology is capable of complying with Article 17 GDPR; it must be underscored that the precise meaning of the term ‘erasure’ remains unclear.
Article 17 GDPR does not define erasure, and the Regulation’s recitals are equally mum on how this term should be understood. It might be assumed that a common-sense understanding of this terminology ought to be embraced. According to the Oxford English Dictionary, erasure means ‘the removal or writing, recorded material, or data’ or ‘the removal of all traces of something: obliteration’.494
From this perspective, erasure could be taken to equal destruction. It has, however, already been stressed that the destruction of data on blockchains, particularly these of a public and permissionless nature, is far from straightforward.
There are, however, indications that the obligation inherent to Article 17 GDPR does not have to be interpreted as requiring the outright destruction of data. In Google Spain, the delisting of information from research results was considered to amount to erasure. It is important to note, however, that in this case, this is all that was requested of Google by the claimant, who did not have control over the original data source (an online newspaper publication). Had the claimant wished to obtain the outright destruction of the relevant data it would have had to address the newspaper, not Google. This may be taken as an indication that what the GDPR requires is that the obligation resting on data controllers is to do all they can to secure a result as close as possible to the destruction of their data within the limits of [their] own factual possibilities.
Dr Michèle Finck, Blockchain and the General Data Protection Regulation, pp. 75-76
From this, we can kind of intuit that the courts aren’t pedantic: The cited Google Spain case was satisfied by merely delisting the content, not the erasure of the newspaper’s archives.
The report goes on to say:
As awareness regarding the tricky reconciliation between Article 17 GDPR and distributed ledgers grows, a number of technical alternatives to the outright destruction of data have been considered by various actors. An often-mentioned solution is that of the destruction of the private key, which would have the effect of making data encrypted with a public key inaccessible. This is indeed the solution that has been put forward by the French data protection authority CNIL in its guidance on blockchains and the GDPR. The CNIL has suggested that erasure could be obtained where the keyed hash function’s secret key is deleted together with information from other systems where it was stored for processing.
Dr Michèle Finck, Blockchain and the General Data Protection Regulation, pp. 76-77
That said, I cannot locate a specific court decision that affirms that crypto erasure is legally sufficient for complying with data erasure requests (nor any that affirm that it’s necessary).
I don’t have a crystal ball that can read the future on what government compliance will decide, nor am I an expert in legal matters.
Given the absence of a clear legal framework, I do think it’s totally reasonable to consider crypto-shredding equivalent to data erasure. Most experts would probably agree with this. But it’s also possible that the courts could rule totally stupidly on this one day.
Therefore, I must caution anyone that follows a similar path: Do not claim GDPR compliance just because you implement crypto-shredding in a distributed ledger. All you can realistically promise is that you’re not going out of your way to make compliance logically impossible. All we have to go by are untested legal hypotheses, and very little clarity (even if the technologists are near-unanimous on the topic!).
Towards A Solution
With all that in mind, let’s start with “crypto shredding” as the answer to the GDPR + transparency log conundrum.
This is only the start of our complications.
CMYKatProtocol Risks Introduced by Crypto Shredding
Before the introduction of crypto shredding, the job of the Public Key Directory was simple:
- Receive a protocol message.
- Validate the protocol message.
- Commit the protocol message to a transparency log (in this case, Sigsum).
- Retrieve the protocol message whenever someone requests it to independently verify its inclusion.
- Miscellaneous other protocol things (cross-directory checkpoint commitment, replication, etc.).
Point being: there was very little that the directory could do to be dishonest. If they lied about the contents of a record, it would invalidate the inclusion proofs of every successive record in the ledger.
In order to make a given record crypto-shreddable without breaking the inclusion proofs for every record that follows, we need to commit to the ciphertext, not the plaintext. (And then, when a takedown request comes in, wipe the key.)
Now, things are quite more interesting.
Do you…
- …Distribute the encryption key alongside the ciphertext and let independent third parties decrypt it on demand?
…OR…
- Decrypt the ciphertext and serve plaintext through the public API, keeping the encryption key private so that it may be shredded later?
The first option seems simple, but runs into governance issues: How do you claim the data was crypto-shredded if countless individuals have a copy of the encryption key, and can therefore recover the plaintext from the ciphertext?
I don’t think that would stand up in court.
CMYKatClearly, your best option is the second one.
Okay, so how does an end user know that the ciphertext that was committed to the transparency ledger decrypts to the specific plaintext value served by the Public Key Directory? How do users know it’s not lying?
Quick aside: This question is also relevant if you went with the first option and used a non-committing AEAD mode for the actual encryption scheme.
In that scenario, a hostile nation state adversary could pressure a Public Key Directory to selectively give one decryption key to targeted users, and another to the rest of the Internet, in order to perform a targeted attack against citizens they’d rather didn’t have civil rights.
My entire goal with introducing key transparency to my end-to-end encryption proposal is to prevent these sorts of attacks, not enable them.
There are a lot of avenues we could explore here, but it’s always worth outlining the specific assumptions and security goals of any design before you start perusing the literature.
AJAssumptions
This is just a list of things we assume are true, and do not need to prove for the sake of our discussion here today. The first two are legal assumptions; the remainder are cryptographic.
Ask your lawyer if you want advice about the first two assumptions. Ask your cryptographer if you suspect any of the remaining assumptions are false.
- Crypto-shredding is a legally valid way to provide data erasure (as discussed above).
- EU courts will consider public keys to be Personal Data.
- The SHA-2 family of hash functions is secure (ignoring length-extension attacks, which won’t matter for how we’re using them).
- HMAC is a secure way to build a MAC algorithm out of a secure hash function.
- HKDF is a secure KDF if used correctly.
- AES is a secure 128-bit block cipher.
- Counter Mode (CTR) is a secure way to turn a block cipher into a stream cipher.
- AES-CTR + HMAC-SHA2 can be turned into a secure AEAD mode, if done carefully.
- Ed25519 is a digital signature algorithm that provides strong security against existent forgery under a chosen-message attack (SUF-CMA).
- Argon2id is a secure, memory-hard password KDF, when used with reasonable parameters. (You’ll see why in a moment.)
- Sigsum is a secure mechanism for building a transparency log.
This list isn’t exhaustive or formal, but should be sufficient for our purposes.
Security Goals
- The protocol messages stored in the Public Key Directory are accompanied by a Merkle tree proof of inclusion. This makes it append-only with an immutable history.
- The Public Key Directory cannot behave dishonestly about the decrypted plaintext for a given ciphertext without clients detecting the deception.
- Whatever strategy we use to solve this should be resistant to economic precomputation and brute-force attacks.
Can We Use Zero-Knowledge Proofs?
At first, this seems like an ideal situation for a succinct, non-interactive zero-knowledge proof.
After all, you’ve got some secret data that you hold, and you want to prove that a calculation is correct without revealing the data to the end user. This seems like the ideal setup for Schnorr’s identification protocol.
CMYKatUnfortunately, the second assumption (public keys being considered Personal Data by courts, even though they’re derived from random secret keys) makes implementing a Zero-Knowledge Proof here very challenging.
First, if you look at Ed25519 carefully, you’ll realize that it’s just a digital signature algorithm built atop a Schnorr proof, which requires some sort of public key (even an ephemeral one) to be managed.
Worse, if you try to derive this value solely from public inputs (rather than creating a key management catch-22), the secret scalar your system derives at will have been calculated from the user’s Personal Data–which only strengthens a court’s argument that the public key is therefore personally identifiable.
CMKatThere may be a more exotic zero-knowledge proof scheme that might be appropriate for our needs, but I’m generally wary of fancy new cryptography.
Here are two rules I live by in this context:
- If I can’t get the algorithms out of the crypto module for whatever programming language I find myself working with, it may as well not even exist.
- Corollary: If libsodium bindings are available, that counts as “the crypto module” too.
- If a developer needs to reach for a generic Big Integer library (e.g., GMP) for any reason in the course of implementing a protocol, I do not trust their implementation.
Unfortunately, a lot of zero-knowledge proof designs fail one or both of these rules in practice.
(Sorry not sorry, homomorphic encryption enthusiasts! The real world hasn’t caught up to your ideas yet.)
What About Verifiable Random Functions (VRFs)?
It may be tempting to use VRFs (i.e., RFC 9381), but this runs into the same problem as zero-knowledge proofs: we’re assuming that an EU court would deem public keys Personal Data.
But even if that assumption turns out false, the lifecycle of a protocol message looks like this:
- User wants to perform an action (e.g.,
AddKey). - Their client software creates a plaintext protocol message.
- Their client software generates a random 256-bit key for each potentially-sensitive attribute, so it can be shredded later.
- Their client software encrypts each attribute of the protocol message.
- The ciphertext and keys are sent to the Public Key Directory.
- For each attribute, the Public Key Directory decrypts the ciphertext with the key, verifies the contents, and then stores both. The ciphertext is used to generate a commitment on Sigsum (signed by the Public Key Directory’s keypair).
- The Public Key Directory serves plaintext to requestors, but does not disclose the key.
- In the future, the end user can demand a legal takedown, which just wipes the key.
Let’s assume I wanted to build a VRF out of Ed25519 (similar to what Signal does with VXEdDSA). Now I have a key management problem, which is pretty much what this project was meant to address in the first place.
VRFs are really cool, and more projects should use them, but I don’t think they will help me.
CMYKatSoatok’s Proposed Solution
If you want to fully understand the nitty-gritty implementation details, I encourage you to read the current draft specification, plus the section describing the encryption algorithm, and finally the plaintext commitment algorithm.
Now that we’ve established all that, I can begin to describe my approach to solving this problem.
First, we will encrypt each attribute of a protocol message, as follows:
- For subkey derivation, we use HKDF-HMAC-SHA512.
- For encrypting the actual plaintext, we use AES-256-CTR.
- For message authentication, we use HMAC-SHA512.
- Additional associated data (AAD) is accepted and handled securely; i.e., we don’t use YOLO as a hash construction.
This prevents an Invisible Salamander attack from being possible.
This encryption is performed client-side, by each user, and the symmetric key for each attribute is shared with the Public Key Directory when publishing protocol messages.
If they later issue a legal request for erasure, they can be sure that the key used to encrypt the data they previously published isn’t secretly the same key used by every other user’s records.
They always know this because they selected the key, not the server. Furthermore, everyone can verify that the hash published to the Merkle tree matches a locally generated hash of the ciphertext they just emitted.
This provides a mechanism to keep everyone honest. If anything goes wrong, it will be detected.
Next, to prevent the server from being dishonest, we include a plaintext commitment hash, which is included as part of the AAD (alongside the attribute name).
(Implementing crypto-shredding is straightforward: simply wipe the encryption keys for the attributes of the records in scope for the request.)
If you’ve read this far, you’re probably wondering, “What exactly do you mean by plaintext commitment?”
Art by Scruff.Plaintext Commitments
The security of a plaintext commitment is attained by the Argon2id password hashing function.
By using the Argon2id KDF, you can make an effective trapdoor that is easy to calculate if you know the plaintext, but economically infeasible to brute-force attack if you do not.
However, you need to do a little more work to make it safe.
HarubakiThe details here matter a lot, so this section is unavoidably going to be a little dense.
Pass the Salt?
Argon2id expects both a password and a salt.
If you eschew the salt (i.e., zero it out), you open the door to precomputation attacks (see also: rainbow tables) that would greatly weaken the security of this plaintext commitment scheme.
You need a salt.
If you generate the salt randomly, this commitment property isn’t guaranteed by the algorithm. It would be difficult, but probably not impossible, to find two salts (, ) such that .
Deriving the salt from public inputs eliminates this flexibility.
By itself, this reintroduces the risk of making salts totally deterministic, which reintroduces the risk of precomputation attacks (which motivated the salt in the first place).
If you include the plaintext in this calculation, it could also create a crib that gives attackers a shortcut for bypassing the cost of password hashing.
Furthermore, any two encryptions operations that act over the same plaintext would, without any additional design considerations, produce an identical value for the plaintext commitment.
CMYKatPublic Inputs for Salt Derivation
The initial proposal included the plaintext value for Argon2 salt derivation, and published the salt and Argon2 output next to each other.
Hacker News comex pointed out a flaw with this technique, so I’ve since revised how salts are selected to make them independent of the plaintext.
The public inputs for the Argon2 salt are now:
- The version identifier prefix for the ciphertext blob.
- The 256-bit random value used as a KDF salt (also stored in the ciphertext blob).
- A recent Merkle tree root.
- The attribute name (prefixed by its length).
These values are all hashed together with SHA-512, and then truncated to 128 bits (the length required by libsodium for Argon2 salts).
This salt is not stored, but can deterministically be calculated from public information.
Crisis Averted?
This sure sounds like we’ve arrived at a solution, but let’s also consider another situation before we declare our job done.
High-traffic Public Key Directories may have multiple users push a protocol message with the same recent Merkle root.
This may happen if two or more users query the directory to obtain the latest Merkle root before either of them publish their updates.
Later, if both of these users issue a legal takedown, someone might observe that the
recent-merkle-rootis the same for two messages, but their commitments differ.Is this enough leakage to distinguish plaintext records?
In my earlier design, we needed to truncate the salt and rely on understanding the birthday bound to reason about its security. This is no longer the case, since each salt is randomized by the same random value used in key derivation.
Choosing Other Parameters
As mentioned a second ago, we set the output length of the Argon2id KDF to 32 bytes (256 bits). We expect the security of this KDF to exceed , which to most users might as well be infinity.
With apologies to Filippo.The other Argon2id parameters are a bit hand-wavey. Although the general recommendation for Argon2id is to use as much memory as possible, this code will inevitably run in some low-memory environments, so asking for several gigabytes isn’t reasonable.
For the first draft, I settled on 16 MiB of memory, 3 iterations, and a parallelism degree of 1 (for widespread platform support).
Plaintext Commitment Algorithm
With all that figured out, our plaintext commitment algorithm looks something like this:
- Calculate the SHA512 hash of:
- A domain separation constant
- The header prefix (stored in the ciphertext)
- The randomness used for key-splitting in encryption (stored in the ciphertext)
- Recent Merkle Root
- Attribute Name Length (64-bit unsigned integer)
- Attribute Name
- Truncate this hash to the rightmost 16 bytes (128 bits). This is the salt.
- Calculate Argon2id over the following inputs concatenated in this order, with an output length of 32 bytes (256 bits), using the salt from step 2:
- Recent Merle Root Length (64-bit unsigned integer)
- Recent Merkle Root
- Attribute Name Length (64-bit unsigned integer)
- Attribute Name
- Plaintext Length (64-bit unsigned integer)
- Plaintext
The output (step 3) is included as the AAD in the attribute encryption step, so the authentication tag is calculated over both the randomness and the commitment.
To verify a commitment (which is extractable from the ciphertext), simply recalculate the commitment you expect (using the recent Merkle root specified by the record), and compare the two in constant-time.
If they match, then you know the plaintext you’re seeing is the correct value for the ciphertext value that was committed to the Merkle tree.
If the encryption key is shredded in the future, an attacker without knowledge of the plaintext will have an enormous uphill battle recovering it from the KDF output (and the salt will prove to be somewhat useless as a crib).
AJCaveats and Limitations
Although this design does satisfy the specific criteria we’ve established, an attacker that already knows the correct plaintext can confirm that a specific record matches it via the plaintext commitment.
This cannot be avoided: If we are to publish a commitment of the plaintext, someone with the plaintext can always confirm the commitment after the fact.
CMYKatWhether this matters at all to the courts is a question for which I cannot offer any insight.
Remember, we don’t even know if any of this is actually necessary, or if “moderation and platform safety” is a sufficient reason to sidestep the right to erasure.
If the courts ever clarify this adequately, we can simply publish the mapping of Actor IDs to public keys and auxiliary data without any crypto-shredding at all.
Trying to attack it from the other direction (download a crypto-shredded record and try to recover the plaintext without knowing it ahead of time) is attack angle we’re interested in.
Herd Immunity for the Forgotten
Another interesting implication that might not be obvious: The more Fediverse servers and users publish to a single Public Key Directory, the greater the anonymity pool available to each of them.
Consider the case where a user has erased their previous Fediverse account and used the GDPR to also crypto-shred the Public Key Directory entries containing their old Actor ID.
To guess the correct plaintext, you must not only brute-force guessing possible usernames, but also permute your guesses across all of the instances in scope.
The more instances there are, the higher the cost of the attack.
CMYKatRecap
I tasked myself with designing a Key Transparency solution that doesn’t make complying with Article 17 of the GDPR nigh-impossible. To that end, crypto-shredding seemed like the only viable way forward.
A serialized record containing ciphertext for each sensitive attribute would be committed to the Merkle tree. The directory would store the key locally and serve plaintext until a legal takedown was requested by the user who owns the data. Afterwards, the stored ciphertext committed to the Merkle tree is indistinguishable from random for any party that doesn’t already know the plaintext value.
I didn’t want to allow Public Key Directories to lie about the plaintext for a given ciphertext, given that they know the key and the requestor doesn’t.
After considering zero-knowledge proofs and finding them to not be a perfect fit, I settled on designing a plaintext commitment scheme based on the Argon2id password KDF. The KDF salts can be calculated from public inputs.
Altogether, this meets the requirements of enabling crypto-shredding while keeping the Public Key Directory honest. All known attacks for this design are prohibitively expensive for any terrestrial threat actors.
As an added bonus, I didn’t introduce anything fancy. You can build all of this with the cryptography available to your favorite programming language today.
CMYKatClosing Thoughts
If you’ve made it this far without being horribly confused, you’ve successfully followed my thought process for developing message attribute shreddability in my Public Key Directory specification.
This is just one component of the overall design proposal, but one that I thought my readers would enjoy exploring in greater detail than the specification needed to capture.
(This post was updated on 2024-11-22 to replace the incorrect term “PII” with “personal data”. Apologies for the confusion!)
#Argon2 #crypto #cryptography #E2EE #encryption #FederatedPKI #fediverse #passwordHashing #symmetricCryptography
-
Magazine Review: Galaxy Science Fiction, ed. H. L. Gold (November 1950) (Brown, Asimov, Boucher, Leiber, Knight, Simak)
Preliminary Note: I plan on reading all 116 issues of the influential, and iconic, SF magazine Galaxy under H. L. Gold’s editorship (October 1950-October 1961) in chronological order. How long this project will take or how seriously/systematically I will take it remain complete unknowns.
See my inaugural post in this series for my reasoning behind selecting Galaxy under H. L. Gold.
Previously: the October 1950 issue.
Up Next: the December 1950 issue.
Let’s get to the stories. We have the first Galaxy masterpiece!
- Don Sibley’s cover for Galaxy Science Fiction, ed. H. L. Gold (November 1950)
You can read the entire issue here.
Fredric Brown’s “Honeymoon in Hell” (1950), 3/5 (Average): The year is 1962. The Cold War heats up. The race for a permanent presence on the Moon takes center stage. Each side “had landed a few men” and claimed it as their own (4). Each side races to construct a space station in orbit to facilitate the construction of a permanent base on the moon. But there’s another worrying world-wide trend–a massive gender imbalance in new births! Not enough boys! Riots. Cults. What’s the plan?
Capt. Raymond F. Carmody, retired from the space service (at age 27) after a successful flight to the Moon, steps into the ring. Resisting an administrative role in the service, he’d chosen a new career: cybernetics, “the science of electronic calculating machines” (9). In his new position, he had access to a powerful computer called Junior, built in 1958, tasked with issues of national security. Alone with the machine, he feeds Junior the data. Junior doesn’t have an answer. But Junior does offer a rare extrapolation that Carmody will be married on the morrow.
And a meeting with the President reveals the nature of the plan to birth a male child on the moon to avoid whatever on Earth is causing the problem! He’ll be legally married before they head to the moon and divorced if the pairing doesn’t work out. The catch? His wife will be Russian and their honeymoon will be Hell Crater.1 The “lucky” woman? Anna Borisovna is also a pilot of “experimental rockets on short-range flights” (16). Alcohol included as “icebreaker” for a “happy honeymoon” (19). The twelve day stay will be “plenty of time to get off before the Lunar night” (18) (Brown certainly intends the pun). And then the story morphs, abruptly, into a first contact story. Or does it?
This is an odd story. At its core it’s about a man and a woman (and mortal enemies) who go to the moon to have sex. But it’s the 50s. They need to be married! And all the references to the act are double entendres. As the ridiculousness fades, Brown settles on a rather enlightened position considering the Cold War terror of the moment–détente with the Soviets, politics and all, remains possible (under some circumstances). The story implies that Carmody falls head-over-heels for Anna due to the similarities of their careers and status as intellectual equals despite their divergent politics. Don Sibley’s issue cover shows her abilities under stressful circumstances. Carmody’s even willing to head to the Soviet Union to be with her! Love trumps all message aside, I am not convinced by the reading experience. Brown relays the strange events that transpire on Mars, and almost all of Carmody and Anna’s interactions, after they occur. It weakens the effect.
Somewhat recommended.
Isaac Asimov’s “Misbegotten Missionary” (variant title: “Green Patches”) (1950), 3/5 (Average): “Misbegotten Missionary” begins from the perspective of an alien entity that slipped onboard a human ship after its barrier faltered for a moment. The alien utterly believes that it is a superior “unified organism” (34) over the “life fragments” that populate the ship (34). Fanatical in its mindset, the shape-shifting alien wants to convert the entire vessel to its ways–without their consent. Slowly the nature of its own world, the purpose of the human vessel, and the fate of a past voyage become clear.
While not a miserable entry in his canon, I am starting to dread the Asimov stories in Galaxy and struggle to write coherently about them. And there’s a serialized novel on the horizon that I haven’t read yet and thus cannot skip– The Stars, Like Dust (1951). While far superior to “Darwinian Pool Room” (1950), “Misbegotten Missionary” defeats its initial success with a laborious exposition of what happened before. I appreciated the Asimov’s attempt to convey alienness of the entity’s perspective. Maybe if you’re interested in the evolution of Asimov’s attempts to write about entire planets as alien consciousness this is worth tracking down.
I reviewed this in 2021 and completely forgot. I was even more cruel in the earlier review!
Anthony Boucher’s “Transfer Point” (1950), 3/5 (Average): Three survivors retreat beneath the Earth’s surface after two apocalyptic events–the release of a new element (agnoton) and an attack by mysterious “yellow bands” (are they light-like? It’s not entirely clear. It’s pulpy on purpose). The scientist Kirth-Labbery constructed the self-sufficient retreat due to his allergies (!). His daughter Lavra spends her time eating fruit grown in the hydroponics bay. And Vyrko, a self-described intellectual poet, observes and writes about the end of the world, pines after his lost love, and reads historical pulp science fiction –including Damon Knight’s “Not with a Bang” (1950) and Robert A. Heinlein’s “By His Bootstraps” (1941). He notices that only one author seems to predict correctly what will happen. And also strange narrative parallels with himself…
I’m a sucker for metafictional science fiction that contains references and quotations from other authors both real and invented. Boucher’s “Transfer Point” serves as a recursive commentary on the nature of genre and its favorite tropes (last man and woman as Adam and Eve, time travel, etc.). Behind the tale’s ultra-pulpy exterior and sappy silliness, Boucher jabs (gently and with a smile) at science fiction’s Campbellian delusion of future prediction. Despite its moments, Boucher can’t approach the heights of Richard Matheson’s “Patterns of Survival” (1955), a far more complex commentary on the power of science fiction.
Somewhat recommended.
Fritz Leiber’s “Coming Attraction” (1950), 5/5 (Masterpiece): I reviewed this story in 2013. I’ve decided to reread it and modify my earlier review.
In Arthur Schlesinger, Jr.’s influential The Vital Center: The Politics of Freedom (1949), a blueprint of the “new liberal self-image,” he describes the post-WWII period as an “age of anxiety” in which “Western man” is “tense, uncertain, adrift.”2 Channeling this sentiment, branded as an “American brand of misery” (83), Leiber imagines an America transformed after a limited nuclear war with the Soviet Union.
The physical landscape mirrors the psychological scars of New York’s inhabitants. “H-Bomb scars” tunnel faces (78). The Empire State Building thrusts out of “Inferno like a mangled finger” (77). In a disturbed attempt to maintain control, a new “puritanical morality” (80) replete with “anti-sex songs” (78) and required masks to cover female faces takes hold. A sinister media landscape manifests the corruption within. Billboards promote “hysterical slogans” in which “the very letters of the advertiser’s alphabet have begin to crawl with sex” (78). New TV gadgets facilitate touch and pseudo-connection (80). Perverse new forms of TV entertainment, in particular male wrestlers pitted against masked women, transfix all audiences.
Wysten Turner, the British narrator, gets caught up in the disturbing changes that have swept the US. He rescues a masked woman from a car driven by youths replete with hooks designed to snag the dresses of passing women. She embodies loneliness and despair. And he wants to help. Soon he finds himself unable to identify the new erotic and violent rituals of control and release. The games layer on themselves. Our narrator, also manipulated, flees in shame when the bizarre tableau’s true nature is unmasked.
Leiber doles out fascinating and punchy commentary on the anxieties of the modern world. A disturbed, erotic, creepy, and hyper-violent exploration of that reflexive Cold War tendency to equate the inability to control and triumph abroad as caused by internal crisis within society as a whole. A brilliant satire of late 40s/early 50s American Cold War culture.
Highly recommended.
Damon Knight’s “To Serve Man” (1950), 3.5/5 (Good): I reviewed this story in 2023. I decided not to reread it. I’ve reproduced the review below.
The Kanamit, pig-like humanoid aliens, arrive on Earth with a promise to assist humanity that appears to have zero caveats. Their similarity to a human food animal creates a disquieting horror: “when a think with the countenance of a fiend comes from the stars and offers a gift, you are disinclined to accept” (91). The Kanama proclaim that they want “to bring you the peace and plenty which we ourselves enjoy, and which we have in the past brought to other races throughout the galaxy” (92). They introduce fantastic power sources, anti-nuclear explosion shields, and technology to exponentially enhance agricultural productivity. Soon there are no “more standing armies, no more shortages, and no unemployment” (98). But no one can decode their language. And when someone finally figures it out, it will be too late.
I don’t completely understand why “To Serve Man” is one of Knight’s best-known short fictions. It won the 2001 Retro Hugo Award for Best Short Story. I would have voted for Fritz Leiber’s “Coming Attraction” (1950) from the list of nominees! That said, “To Serve Man” is an effective twist-ending story that plays with our expectations but doesn’t have the reflective or incisive impact of Knight’s best — for example “The Enemy” (1958), “You’re Another” (1955), or even “Time Enough” (1960) in Far Out (1961). I’m probably in the minority in this view.
Somewhat recommended.
Clifford D. Simak’s Time Quarry (variant title: Time and Again) (1950). Serialized over three issues. I will post an individual review after I complete the serialization.
Notes
- Brown adheres to the theory that the Moon is covered with deep dust. He claims that Hell Crater is a bit more solid than other points. Arthur C. Clarke’s A Fall of Moondust (1961) is another example. ↩︎
- See Ch. 1 of K. A. Courdileone’s Manhood and American Political Culture in the Cold War (2005) for a discussion of Schlesinger. ↩︎
For book reviews consult the INDEX
For cover art posts consult the INDEX
For TV and film reviews consult the INDEX
#1950s #avantGarde #bookReviews #books #CliffordDSimak #DamonKnight #FredricBrown #fritzLeiber #HLGold #IsaacAsimov #sciFi #scienceFiction #ShortStories -
Entlang des Tages entwickeln die Gespräche ihr Eigenleben und eine andere als die angedachte Richtung. Schnittstellenprobleme, Reibungsverluste, die Herausforderungen von deutlichen Größenunterschieden, Defensivkommunikation, ausgebremst durch Dienst nach Vorschrift. Auf dem Bürodach gegenüber sitzen zwei Tauben und der Blick kehrt immer wieder dorthin zurück, das Gefühl unverborgener Beobachtung verstärkt sich. Zwei Etagen weiter unten räumt der Student seinen Schreibtisch auf, legt Buch auf Buch, nebenan schwärzt der Drucker seitenweise Papier, Informationen stecken in Warteschlangen und der Horizont zieht sich weißgraue, dichte Wolken über die waldbedeckten Hügel. Sonne kurz vor dem Zenit, Stadt, Beton, Dunst. #outerworld #office_hours #concrete_city #stories_of_technology_and_backlogs
-
I make daily use of a note-taking app to record what I learn when problem-solving, prepare for YouTube videos, and much more. I save everything in Markdown format so that it is future-proof and stays compatible with whatever I'll be using in 20 years time.
I had originally got started with Evernote but exported everything to QOwnNotes later on. After that I started using Obsidian (powerful, free but not open source), and I'm thinking of making a change again back to something open source. Although I've also tried Joplin and Standard Notes, I never fully migrated to them. The beauty of an open standard like Markdown is that you can switch apps and just continue using all your notes from 10+ years ago.
The only issues you may experience are that some "features" like say generated tables of contents, Kanban, to-do's, highlighting, etc are not standardised in Markdown, so you can lose these when changing apps. But generally, headings, bold, italics, indenting, links, images, tables, etc are all fine. So, if you stick to one editor then go wild with the extra plugins, but if you want to retain compatibility across editors then you may want to think about what plugins you make use of.
Although not listed in the linked article, I see that open source and cross-platform (including mobile) Logseq has vastly improved, and I may want to give that a spin. Its feature is "everything is a referenceable block" with a block being a paragraph of text, and it calls notes pages. Only thing is it seems to mark the start of every block with a dot (and this shows in other editors) so hopefully I can disable that being inserted into the saved text. Its philosophy also centres around a daily journal where you just write your notes and can flag to-dos or link/search for anything. It also has whiteboards and graph views of linked notes (just like Obsidian).
Very interestingly, I noted that both Logseq and also Obsidian, have various Chat-GPT plugins to help generate content or even to rewrite your rough drafts. So yes, AI has already invaded open-source text editors!
See 11 Best Note-Taking Apps for Linux Desktop
#technology #notes #opensource #Logsec #LinuxA note-taking app allows you to record notes on the go, whether you are in class or studying, reading somewhere, at work, or in a meeting.
-
Perplexity releases AI web browser Comet for free – TechHQ
Perplexity releases AI web browser Comet for free – TechHQ
October 20, 2025
- AI browser Comet from Perplexity rolls out for free.
- iOS version still in the works, despite rogue App Store appearance.
- Company CEO declares bogus app “fake and spam”.
The new AI-powered browser from Perplexity, Comet, is currently winning headlines and gaining popularity – but in the mobile space, only on Android devices.
Aravind Srinivas, the CEO of AI startup Perplexity has issued a public warning via social media that the iOS version of the app, which was available on the App Store for iOS users, is a hoax application and should not be downloaded.
As of the time of writing, the listing for the bogus app has been removed, although links from search engines still exist.
Srinivas called the app “fake and spam”, stating the app is not affiliated with Perplexity, and that the company has yet to release the browser for iOS. “You will directly hear from us when Comet iOS is ready for pre-registering and downloading.”
The Comet AI browser has received a positive reception on the Google Play Store in the geographies where it’s available. The company has also released the desktop Comet browser for free in some areas – previously it was only an option to paying subscribers of Perplexity’s AI services. Comet now offers AI-powered features to users at zero cost (depending on their location, release cadences may vary), including recommendations, shopping, and smart search. Features include AI-powered summaries of searches, a contextual recommendation engine, and various integrated tools for content discovery.
The company has been careful to position Comet as a direct Chrome rival; a tall order given the latter’s ubiquitous position in the browser league tables. Chrome currently dominates web browser choice, with approximately 65% of the market across mobile and desktop. Main rival, Apple’s Safari, holds around 15% of the total
Continue/Read Original Article Here: Perplexity releases AI web browser Comet for free – TechHQ
#2025 #AI #AIBrowser #artificialIntelligence #Comet #CometBrowser #Education #Free #Health #History #Libraries #Library #ModernBrowsers #Perplexity #PerplexityAI #Science #TechHQ #Technology #UnitedStates
-
Destroying Autocracy – July 24, 2025
Welcome to this week’s “Destroying Autocracy”.
It’s your source for curated news affecting democracy in the cyber arena with a focus on protecting it. That necessitates an opinionated Butlerian jihad against big tech as well as evangelizing for open-source and the Fediverse. Since big media’s journalism wing is flailing and failing in its core duty to democracy, this is also a collection of alternative reporting on the eternal battle between autocracy and democracy. We also cover the cybersecurity world. You can’t be free without safety and privacy.
DA comes out on Thursday and is updated through the end of day on Friday. Then we start over. So take your time in perusing it and check back in over the weekend.
FYI, my opinions will be in bold. And will often involve cursing. Because humans. Especially tech bros. And fascists. Fuck ’em.
Featured Item
TechDirt writes:
Over the last year or so I’ve seen a disturbing tendency in tech/startup/VC worlds to buy into the neoreactionary view that for startups to be successful they need to get on board the Trump train.
Yes, there are the big name folks who everyone knows about and who didn’t really surprise anyone—Peter Thiel, Marc Andreessen, David Sacks, Elon Musk (pre-fallout)—but the more troubling trend has been watching younger entrepreneurs and VCs listen to their podcasts, read their posts and books, and slowly nod along to the idea that democracy is holding back innovation.
Fascism For First Time Founders
We start and end with good news to make the middle bearable.
The response to Russia’s War Crimes, Techno Feudalism, and other douchebaggery
BleepingComputer reports:
Ukraine arrests suspected admin of XSS Russian hacking forum
Radio Free Europe reports:
Drone Attacks Even The Odds For Ukrainian Frontline Units
BitDefender reports:
Europol targets Kremlin-backed cybercrime gang NoName057(16)
Bruce Lawson reports:
CMA designates Google and Apple, proposes measures
TechCentral reports:
Italy takes Meta, X and LinkedIn to court over unpaid tax
404 Media reports:
Hacker Plants Computer ‘Wiping’ Commands in Amazon’s AI Coding Agent
Archivists Recreate Pre-Trump CDC Website, Are Hosting It in Europe
The Register reports:
Radio geeks reveal how to access crucial hurricane data after US Department of Defense cut it off
The Register reports:
AI data-suckers would have to ask permission first under new bill
Laptop farmer behind $17M North Korean IT worker scam locked up for 8.5 years
TechPolicy reports:
The Case for Europe’s Backing of Digital Civil Society Groups
Open_Future shares:
Licensing, Levies, and the Limits of Copyright
Open Forum Europe announces:
DarkReading reports:
Stop AI Bot Traffic: Protecting Your Organization’s Website
Speaking of your websites, LocalGhost has:
This page is under construction: a love letter to the personal website
Hamish Campbell has:
The Open Media Network: More Than Just a Tech Project
This is what your site could be a part of.
Neutral
TechPolicy reports:
Brazil Has a Bridge to Defending the Internet
The Financial Times:
UK government seeks way out of clash with US over Apple encryption
When you have three sets of c^nts involved, it’s hard to know who to route for.
TechPolicy opines:
Enforcement of EU’s Tech Laws Should Not Be Traded Away
And they are right.
The Evil Empire (AKA Autocracy) Strikes Back
So-called newspaper, The Wall Street Journal reports:
White House Prepares Executive Order Targeting ‘Woke AI’
MIT Technology Review reports:
America’s AI watchdog is losing its bite
The Electronic Frontier Foundation reports:
Axon’s Draft One is Designed to Defy Transparency
EuroNews reports:
UK online legislation could threaten Wikipedia volunteer safety, group to argue in court
Pariah States
The Register reports:
UK uncovers novel Microsoft snooping malware, blames and sanctions GRU cyberspies
Silicon Valley engineer admits theft of US missile tech secrets
Four new Android spyware samples linked to Iran’s intel agency
TechCrunch reports:
A surveillance vendor was caught exploiting a new SS7 attack to track people’s phone locations
Hackers exploiting SharePoint zero-day seen targeting government agencies
BleepingComputer reports:
Microsoft links Sharepoint ToolShell attacks to Chinese hackers
Big Media
Today in Tabs reports:
Billionaires Destroyed American News Media On Purpose
Mother Jones reports:
Colbert’s Cancellation Is a Dark Warning
Akademie shares:
Investigating AI datasets: A journalist’s guide
Big Tech
Where’s Your ‘Ed shares:
The Hater’s Guide To The AI Bubble
🙂
The Next Web reports:
ChatGPT advises women to ask for lower salaries, study finds
404 Media reports:
A Startup is Selling Data Hacked from Peoples’ Computers to Debt Collectors
Spotify Publishes AI-Generated Songs From Dead Artists Without Permission
Google’s AI Is Destroying Search, the Internet, and Your Brain
Grindr Won’t Let Users Say ‘No Zionists’
EuroNews reports:
Meta ran ads that fundraised for Israeli Defence Forces, analysis shows
Meta won’t sign EU’s AI Code, but who will?
The Electronic Frontier Foundation reports:
Amazon Ring Cashes in on Techno-Authoritarianism and Mass Surveillance
Ars Technica reports:
Researcher threatens X with lawsuit after falsely linking him to French probe
xAI workers balked over training request to help “give Grok a face,” docs show
TechCrunch reports:
Microsoft says it will no longer use engineers in China for Department of Defense work
For privacy and security, think twice before granting AI access to your personal data
Terror
The Register reports:
IRL Com recruits teens for real-life stabbings, shootings, FBI warns
Cybersecurity/Privacy
TechCrunch reports:
Serial spyware founder Scott Zuckerman wants the FTC to unban him from the surveillance industry
BleepingComputer reports:
CISA and FBI warn of escalating Interlock ransomware attacks
DarkReading reports:
Translating Cyber-Risk for the Boardroom
Fediverse
Connected Places has:
Fediverse Report – #126 July 22, 2025
Bonfire is:
Exploring a Bonfire Geosocial Extension
MarkWrites reflects on:
Mastodon announces:
If you are on the Fediverse please donate to your instance’s maintainers. Especially if they ask nicely.
Aphyr opines:
The Future of Forums is Lies, I Guess
Fediverse favorite, Elena Rossini shares:
The Future is Federated: Year 2
ActivityPub for WordPress has an update:
We Distribute has details:
WordPress-ActivityPub v 7.1.0 Introduces Following Capabilities
Randall Black show us:
How to Install and Set Up Castopod for Your Podcast
TechCrunch reports:
Threads adds improved content performance metrics for creators
Slightly Federated Social Media
The Register reports:
Selling your digital soul to use Bluesky’s DMs isn’t just a bad idea, it’s the law
CTAs (aka show us some free love)
- That’s it for this week. Please share this edition of Destroying Autocracy.
- Follow me on the Fediverse. Or this site via the button in the footer. Or via RSS.
Keep fighting!
Ringleader, Battalion
Reuben Walker
Follow me on the Fediverse#126 #ActivityPub #AI #Autocracy #BigJournalism #BigTech #Bluesky #Castopod #Democracy #Fascism #Fediverse #Mastodon #StopChina #StopIsrael #StopRedAmerica #StopRussia #SupportUkraine #TechnoAnarchism #TechnoFeudalism #Threads #WordPress
-
Japandi Style: A Harmonious Fusion of Japanese and Scandinavian Interior Design
A timely trend rooted in history.
The Japandi style has risen from niche fusion to mainstream movement. During the global shift toward wellness and sustainability, more people are drawn to interiors that feel calm yet intentional. This unique blend of Japanese and Scandinavian design, sometimes described as East‑meets‑West minimalism, marries the wabi‑sabi philosophy of rustic simplicity with the warmth of hygge. As a result, the Japandi style prioritizes natural materials, muted colors, and uncluttered spaces while remaining comfortable and inviting. According to interior designers, its roots stretch back to the mid‑19th century when Scandinavian travelers first visited a newly opened Japan. It is not just an aesthetic; it reflects a lifestyle that values quality, craftsmanship, and mindful living.
What is the Japandi style?
Definition and core principles
The Japandi style fuses Japanese minimalism with Scandinavian simplicity. It balances clean lines, functional pieces, and a tranquil atmosphere. In practice, this hybrid aesthetic celebrates simplicity, craftsmanship, and a deep connection to nature. Both cultural traditions appreciate neutral color palettes, natural materials, and uncluttered rooms. Interior designer Nareg Taimoorian notes that the Japandi style blends “clean lines, natural materials, and functionality” with the tranquility and craftsmanship of Japanese design [homesandgardens.com]. Consequently, rooms feel calm and balanced rather than stark or sterile.
Origins and philosophy
While the term Japandi style gained popularity only in recent years, the design dialogue began around 150 years ago. After Japan reopened to global trade in the 1850s, Scandinavian designers visited and were captivated by Japanese art and craftsmanship. This cultural exchange revealed surprising commonalities: both traditions valued simplicity, handcrafted objects, and harmonious living. In her book Japandi Living, author Laila Rietbergen explains that the wabi‑sabi philosophy embraces rustic simplicity and the beauty of imperfections, while Scandinavian hygge emphasizes comfort and warmth. This shared appreciation for natural materials and understated elegance allowed the two aesthetics to merge seamlessly. Today, design historians credit a Danish naval lieutenant’s 1860s trip to Japan for sparking the fusion [vogue.com]. The resulting Japandi style invites you to celebrate imperfections, invest in enduring quality, and find beauty in everyday objects.
Why the Japandi style resonates today
A response to modern life
Amid urban clutter and constant digital stimulation, many seek spaces that nurture mental health. The Japandi style answers this need with its serene atmosphere and focus on well‑being. Vogue reports that the style’s search popularity has reached record levels [vogue.com], and Google Trends confirms rising interest since 2020. Designers explain that people crave interiors that will stand the test of time and offer a sense of calm. As technology permeates daily life, the directness and simplicity of Japandi style solutions feel refreshing. The approach resonates because it provides a mindful alternative to fast décor trends and encourages deeper connections to nature and craft.
Enduring relevance for 2025 and beyond
Design forecasts suggest that the Japandi style will remain influential in 2025. A Decorilla trend report notes that Scandinavian design “is here to stay” and that Japandi, combining minimalism and natural materials, continues to be a relevant trend for the year [decorilla.com]. Homes & Gardens echoes this sentiment, stating that Japandi style has become more than a trend; it is a lifestyle promoting wellbeing and sustainability [homesandgardens.com]. Unlike short‑lived fashions, its emphasis on timeless craftsmanship and quality materials makes it resilient. Therefore, investing in this aesthetic means choosing pieces that will continue to feel fresh and meaningful.
Key elements of Japandi design
Neutral color palettes
The Japandi style relies on a calm palette to create a serene backdrop. Typical hues include warm whites, soft beige, taupe, charcoal gray, and muted greens. Homes & Gardens advises using neutral schemes with subtle pops of color to introduce Scandinavian influences [homesandgardens.com]. Earthy tones and natural wood finishes reflect the beauty of the outdoors. These gentle shades prevent overstimulation, allowing textures and materials to take center stage.
Emphasis on natural materials
Authentic materials are fundamental to the Japandi style. Japanese interiors favor organic materials such as wood, stone, paper, and cotton, while Scandinavian design emphasizes light woods like birch and white oak. Combining darker Japanese woods with lighter Scandinavian timbers creates contrast and depth. The use of bamboo, linen, rattan, and stone connects indoor spaces to nature and highlights craftsmanship. Selecting eco‑friendly materials also aligns with contemporary values of sustainability.
Functional and minimalist furniture
Furniture in the Japandi style is low‑profile and multifunctional. It blends clean lines with gentle curves to offer both comfort and refined form. Pieces are handcrafted, emphasizing durability and simple beauty. Handmade ceramics, carved wooden bowls, and woven baskets introduce character without clutter. Every item serves a purpose; designers recommend choosing fewer but better‑quality pieces, reflecting the style’s philosophy of intentional living. By avoiding excess ornamentation, rooms feel open and purposeful.
Clutter‑free spaces and intentional decor
Minimalism lies at the heart of the Japandi style. Both Japanese and Scandinavian traditions prize uncluttered rooms and open layouts [architecturaldigest.com]. Decluttering is often the first step toward achieving this look. Homes & Gardens suggests that everything introduced into a Japandi scheme must have a purpose and that adequate storage is essential [homesandgardens.com]. Rather than filling shelves with decorations, you place a handful of meaningful objects such as a teapot, bonsai, or sculptural vase. The result is a calm space that encourages mindfulness.
Connection to nature and indoor–outdoor harmony
Bringing the outdoors inside is integral to the Japandi style. Both cultures share a love for nature and seek to incorporate it into their homes. Plants like ferns, snake plants, or bonsai add life without overwhelming the serene atmosphere. Large windows, shoji screens, and open floor plans maximise natural light, while organic shapes and textures reference rivers, mountains, and forests. Hovia explains that this hybrid aesthetic provides mindfulness through a calming décor sympathetic to local landscapes [hovia.com]. By connecting indoor and outdoor environments, the Japandi design fosters well‑being.
How to incorporate the Japandi style in your home
Start by simplifying and decluttering
Creating a Japandi-style interior begins with removing excess. Designers recommend decluttering to reveal open spaces and free movement. Use the 20/80 rule: display twenty percent of your possessions and hide the rest behind closed doors [homesandgardens.com]. This exercise encourages you to ask which objects truly matter in your daily life. After clearing the space, assess how the room feels; the calm of an uncluttered environment is fundamental to the Japandi style.
Use natural materials and layered textures
Next, choose materials that reflect nature. Wood, bamboo, seagrass, rattan, and stone are staples. Scandinavian light wood brings an airy feel, while darker Japanese woods add depth. Layering textures such as linen curtains, woven rugs, and knit throws enhances warmth without clutter. Soft lighting—think paper lanterns, diffused pendants or linen shades—creates a gentle ambience [arterahome.com]. When selecting lighting, consider how each fixture contributes to the overall mood.
Balance contrast and muted tones
Although the Japandi interior design favors muted colors, it also embraces thoughtful contrast. Layer pale woods with deeper hues like charcoal or rust. Textured rugs, statement cushions, and curated wall art provide opportunities to introduce darker tones without overwhelming the space. The goal is harmony rather than uniformity. Contrast adds dimension, while neutral backdrops maintain a calming foundation.
Select streamlined furniture and handcrafted decor
Opt for furniture that is simple yet comfortable. Low seating, sleek tables, and storage benches crafted from natural materials reflect the Japandi style. Multi‑functional pieces, such as stacking tables or woven baskets used for storage, maximise functionality. Handcrafted ceramics, woven textiles, and carved wood objects bring soul and personal meaning to the space. These items celebrate imperfection and reflect the wabi‑sabi concept.
Invite nature indoors
Integrate living plants, branches, river stones, or dried grasses to strengthen the connection to nature. Bonsai trees, snake plants, or ferns provide greenery without crowding. Organic sculptures and driftwood art blur the boundary between interior and exterior spaces. Using natural elements encourages mindfulness and fosters a peaceful atmosphere. Remember that each addition should enhance the sense of balance central to the Japandi style.
Japandi versus similar design styles
Japandi vs Scandinavian minimalism
While both aesthetics prioritize simplicity and function, the Japandi style introduces a richer palette and handcrafted details. Scandinavian interiors often feature bright white bases and cozy accessories like knitted throws and candles. In contrast, Japandi incorporates darker hues, low‑profile furniture, and Zen‑inspired objects such as shoji screens. It also embraces imperfection through wabi‑sabi pottery and aged finishes. Therefore, Japandi feels more grounded and organic than pure Scandinavian minimalism.
Japandi vs organic modern
Organic modern combines minimalism with mid‑century and boho influences, using earthy tones and sculptural furniture. The Japandi style shares an appreciation for natural materials but focuses on tranquility and balance. Organic modern spaces may incorporate bold contrasts and varied textures for a layered effect, whereas Japandi remains restrained, prioritizing calm and intentionality. The differences underscore how subtle choices in color, proportion, and decor shift a room’s mood.
Japandi vs minimalism
Minimalism aims to eliminate distractions, often resulting in stark, monochromatic spaces. Although the Japandi style embraces minimalism, it warms the palette and adds tactile elements to avoid coldness. Using warm neutrals, crafted objects, and natural materials invites comfort while maintaining the clarity of a minimalist approach. In short, Japandi interior refines minimalism with a human touch.
Beyond decor: Japandi as lifestyle and mindset
Mindful consumption and sustainability
Choosing the Japandi style extends beyond selecting furniture; it reflects a commitment to mindful living. Emphasis on quality over quantity encourages investing in pieces that last [arterahome.com]. Using eco‑friendly materials like bamboo, wood, and linen aligns with sustainable values. This mindset discourages impulsive buying and promotes responsible consumption. It also honors artisans and traditional techniques, supporting cultural heritage. Therefore, the Japandi style becomes a statement of environmental and social awareness.
Emotional well‑being and mental health
Calm spaces influence mental health. The Japandi style encourages you to slow down, appreciate your surroundings, and prioritise daily rituals. The neutral palette and organic textures reduce sensory overload, while natural light and plants uplift moods. Homes & Gardens notes that the style transcends decoration to become a way of life that promotes wellbeing [homesandgardens.com]. Creating a sanctuary at home fosters mindfulness and helps counteract the stress of modern life. As you design with intention, you practice self‑care and find beauty in simplicity.
Cultural appreciation and respectful fusion
At its best, the Japandi style honors both Japanese and Scandinavian traditions rather than appropriating them. Understanding the origins—Scandinavian designers’ early encounters with Japanese art and the shared values of wabi‑sabi and hygge—ensures respectful adoption. Recognising that this aesthetic is a Western invention with deep roots [vogue.com] reminds designers to credit both cultures. When selecting décor, choose authentic pieces from artisans or reputable brands that support traditional craft. Such consideration deepens the connection between design and cultural heritage.
Japandi Style Living RoomQuestions to consider when adopting Japandi
Do you value quality over quantity?
Adopting the Japandi style means investing in well‑made pieces and avoiding disposable décor. Reflect on whether you are ready to curate rather than accumulate.How do you connect with nature at home?
The Japandi style invites greenery, natural light, and organic materials into your space. Consider how plants, textures, and natural colors can improve your daily environment.Which objects truly matter?
Decluttering is essential. Assess which possessions carry meaning and which simply take up space. Let go of items that no longer serve your purpose.Are you prepared to embrace imperfections?
Wabi‑sabi teaches that beauty exists in flaws and aging. Handmade ceramics, weathered wood, and uneven textures embody this idea. Allow yourself to appreciate irregularity.Will you honor cultural roots?
Understanding the historical exchange between Japan and Scandinavia deepens the practice. Seek knowledge about both traditions to ensure a respectful approach.Inspiration and next steps
The Japandi style offers more than a set of décor guidelines; it presents a pathway to intentional living. By blending minimalism with warmth, it creates spaces where one can breathe, think, and feel grounded. If you are searching for design ideas, start with a single room. Apply neutral colors, remove clutter, and add a handmade ceramic bowl or a potted tree. Observe how the mood shifts. Over time, extend the Japandi style throughout your home, adapting it to your habits and cultural context. With each choice, you cultivate calm and connection—qualities that define this enduring aesthetic.
#decor #decoration #home #interiorDeign #Japandi #JapandiDesign #JapandiStyle #japanese #JapaneseInteriorDesign #scandinavianDesign #ScandinavianInteriorDesign