#enigmatick — Public Fediverse posts

Anatomy of an E2EE ActivityPub Note in Enigmatick

Over the break I spent some time revamping my structs and enums for handling #E2EE #ActivityPub exchanges. This is what I have at the moment for the initial Create from the client (this is subject to change).

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://purl.archive.org/socialweb/mls"
  ],
  "type": "Create",
  "actor": "https://enigmatick.social/user/jdt",
  "to": [
    "https://enigmatick.social/user/mls-tester"
  ],
  "cc": [],
  "object": {
    "mediaType": "message/mls",
    "encoding": "base64",
    "published": "2025-12-29T21:11:17Z",
    "type": "Note",
    "to": [
      "https://enigmatick.social/user/mls-tester"
    ],
    "cc": [],
    "attributedTo": "https://enigmatick.social/user/jdt",
    "content": "AAEAAhDIkTdahK5U3s1SQBbA2fIvAAAAAAAAAAEBABzHJj3+WZDE3Hw6H5W9YNhE3bhDSyExL4tnn0E5QOvPUeOgYdo/2n+rGUI1M8wupeK2bt919jAs8yTj2awvBAsprxAInHm8qWgq2FER2xjcpAx9f42ncAL8Xehfy3v6vonGuAPkYteAzdFCC8mGWxg3slpI+VoPrSWThSKqjMNxttZSfvbnhEFYw0jUFmJwxOecDS8n3UVjuLDt14V0Fw9YMv+XXUgKj9bWaCPi0qhuatcgxCqXwXzpevJBhY0trbPZgjB8LlhC/zcx5NNLI9WJtCF9zq3RPkG6ByPdH6IWTH5drOJyvefILElybP4eGMSoGWZATmGUQm/k3iKG6tRyzMM/fHxpIz34",
    "tag": [
      {
        "type": "Mention",
        "name": "[at][email protected]",
        "href": "https://enigmatick.social/user/mls-tester"
      }
    ],
    "attachment": []
  },
  "published": "2025-12-29T21:11:17Z",
  "instrument": [
    {
      "type": "VaultItem",
      "content": "oAjN0c2+q5gCtrFlauGU1kbKGizm5a5pH+iY5cObdihVBjjsSiSbkeu6XQ7/RR4lcn0OIqS9O+MVJ0QyympRxt7JI9qzCGlkCFcAyp4oe8nhh/YG7WHoLWL272vPqlq8A/Yf0YFprhIBlU48sJF6A1iRlU4fEBaRwnwV4gniSCQr55jpDe++M/HD3bSFJNcvpmf5BH9G4Z3VBaB8HMnDECGzFrB8zFycAR0w1SNYbGJuXFsopfMb4EfPkZoHT28="
    },
    {
      "type": "Welcome",
      "mediaType": "message/mls",
      "encoding": "base64",
      "content": "AAFAmCDd32lw8FF8QC6uFjAHr3mgvxBPjuGV1Pssxx+h5CNhcCAv4/tP+dWUbCq6L4A+aMr+L1uP3Xkusb4ea2H80EgFOUBUqHDCdGzjAhZdLCrjE0kM/aou36ZSZc0ahftKmE07chTiMUo8FgIj12deS4A0l3sXAR8x724HeigtyObI1GK7fxMGXU5oWemHGhS1unxK6XVyiuqvQriKP+VJPZa6JzN3Yb4r4l9B5jUfNUc5gX3XvzkzcO57Exj8SgpTTygo2zBTOPQ+6steYycUA70ed3LPKEfSCllIWeWAV+J+EFh7ywm90XFYzaPrXVudIx3orRzVMFnIldLax3oHUW5uXIrGQ06JWfgdOQ6DByb2iy/Dw4zJlbZP1oJptBTgO19IqUQaz58wrWDBSiYIT3h1jOXE3GvJgeeQIiW78noO5MF3xI3WRXB4C0cttyKm7Od2CBOk1USo+TDLCevnmExhlKlVTobTMMI4xZ1vVCdYpnb0k9rO7coVKPueBqVg2mE20GS/IoRG0v8+h0xUKIgsbYua41krjXoB5hctX4kH3lRO+qERkoXEcLVxRJuyeZEGsk4XrJ18AigU1a8gB6cZZ5LyvcV756yelu6F6MRQKba6zawDLOn4Takm3Lr3urfWo5Br7DBZcigMVjzGn8MhLSIEMhv6Ehmqajj9Vl6sTnpi90tVYWUtKO1buPrk3AiXrsbnEoNubqCs1sVmA93/Z0CcE8GQa1dO7NywofSV4/ax/tmBCZEoU6/uL7ygZHt8ZvSyd6wniPCR6sfd4ys4O8C7qW1KKFtEBbb1lMre494ZeXI5v0sG7MwbwFtI3d2uRHtWZ8v9khGr1ds4AFz3QnvoMSUtsR2kvq+C6H45kUs2ae957+txpKt5MDrcmK+CTTcX/FhlAtib4mlbs+Aj4WJFLOkXuB+MlHOMBlMz3ndJ5I3GDbo+VQvdJC2mCUOBWraihOV92mQQHi2gLXAA1NSsZIjXgnyVSYsrUvNJgTvw+AOVmJ7q5tXENPlZoOKUjSB3fEmKlywf90EQLUjDpphnYtvdFz8Oh8Vbgze+QAQp2VVYImI+1voBErK1EMKqEDFz1DC1CWUzqhl3k2pMgx00JhP3ZZpbX7YtjRmLjJM="
    }
  ]
}

The mention of mls-tester has an [at] to avoid triggering my automatic webfinger resolution.

I've adopted the mediaType and encoding attributes from the draft ActivityPub specification. And I've standardized the Welcome message to incorporate those.

The Create includes the Welcome and a VaultItem as instruments rather than being sent as separate activities. The Welcome can include any number of recipients, so one within the activity should be sufficient.

The VaultItem is specific to the #Enigmatick client and isn't intended to be federated. That's how I'm handling ongoing access to the decrypted data; the VaultItem is a symmetrically encrypted representation of the original content that can be safely stored on the server for repeated use by the client. I'll need to eventually update the @context with that extension.

Anatomy of an E2EE ActivityPub Note in Enigmatick

Over the break I spent some time revamping my structs and enums for handling #E2EE #ActivityPub exchanges. This is what I have at the moment for the initial Create from the client (this is subject to change).

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://purl.archive.org/socialweb/mls"
  ],
  "type": "Create",
  "actor": "https://enigmatick.social/user/jdt",
  "to": [
    "https://enigmatick.social/user/mls-tester"
  ],
  "cc": [],
  "object": {
    "mediaType": "message/mls",
    "encoding": "base64",
    "published": "2025-12-29T21:11:17Z",
    "type": "Note",
    "to": [
      "https://enigmatick.social/user/mls-tester"
    ],
    "cc": [],
    "attributedTo": "https://enigmatick.social/user/jdt",
    "content": "AAEAAhDIkTdahK5U3s1SQBbA2fIvAAAAAAAAAAEBABzHJj3+WZDE3Hw6H5W9YNhE3bhDSyExL4tnn0E5QOvPUeOgYdo/2n+rGUI1M8wupeK2bt919jAs8yTj2awvBAsprxAInHm8qWgq2FER2xjcpAx9f42ncAL8Xehfy3v6vonGuAPkYteAzdFCC8mGWxg3slpI+VoPrSWThSKqjMNxttZSfvbnhEFYw0jUFmJwxOecDS8n3UVjuLDt14V0Fw9YMv+XXUgKj9bWaCPi0qhuatcgxCqXwXzpevJBhY0trbPZgjB8LlhC/zcx5NNLI9WJtCF9zq3RPkG6ByPdH6IWTH5drOJyvefILElybP4eGMSoGWZATmGUQm/k3iKG6tRyzMM/fHxpIz34",
    "tag": [
      {
        "type": "Mention",
        "name": "[at][email protected]",
        "href": "https://enigmatick.social/user/mls-tester"
      }
    ],
    "attachment": []
  },
  "published": "2025-12-29T21:11:17Z",
  "instrument": [
    {
      "type": "VaultItem",
      "content": "oAjN0c2+q5gCtrFlauGU1kbKGizm5a5pH+iY5cObdihVBjjsSiSbkeu6XQ7/RR4lcn0OIqS9O+MVJ0QyympRxt7JI9qzCGlkCFcAyp4oe8nhh/YG7WHoLWL272vPqlq8A/Yf0YFprhIBlU48sJF6A1iRlU4fEBaRwnwV4gniSCQr55jpDe++M/HD3bSFJNcvpmf5BH9G4Z3VBaB8HMnDECGzFrB8zFycAR0w1SNYbGJuXFsopfMb4EfPkZoHT28="
    },
    {
      "type": "Welcome",
      "mediaType": "message/mls",
      "encoding": "base64",
      "content": "AAFAmCDd32lw8FF8QC6uFjAHr3mgvxBPjuGV1Pssxx+h5CNhcCAv4/tP+dWUbCq6L4A+aMr+L1uP3Xkusb4ea2H80EgFOUBUqHDCdGzjAhZdLCrjE0kM/aou36ZSZc0ahftKmE07chTiMUo8FgIj12deS4A0l3sXAR8x724HeigtyObI1GK7fxMGXU5oWemHGhS1unxK6XVyiuqvQriKP+VJPZa6JzN3Yb4r4l9B5jUfNUc5gX3XvzkzcO57Exj8SgpTTygo2zBTOPQ+6steYycUA70ed3LPKEfSCllIWeWAV+J+EFh7ywm90XFYzaPrXVudIx3orRzVMFnIldLax3oHUW5uXIrGQ06JWfgdOQ6DByb2iy/Dw4zJlbZP1oJptBTgO19IqUQaz58wrWDBSiYIT3h1jOXE3GvJgeeQIiW78noO5MF3xI3WRXB4C0cttyKm7Od2CBOk1USo+TDLCevnmExhlKlVTobTMMI4xZ1vVCdYpnb0k9rO7coVKPueBqVg2mE20GS/IoRG0v8+h0xUKIgsbYua41krjXoB5hctX4kH3lRO+qERkoXEcLVxRJuyeZEGsk4XrJ18AigU1a8gB6cZZ5LyvcV756yelu6F6MRQKba6zawDLOn4Takm3Lr3urfWo5Br7DBZcigMVjzGn8MhLSIEMhv6Ehmqajj9Vl6sTnpi90tVYWUtKO1buPrk3AiXrsbnEoNubqCs1sVmA93/Z0CcE8GQa1dO7NywofSV4/ax/tmBCZEoU6/uL7ygZHt8ZvSyd6wniPCR6sfd4ys4O8C7qW1KKFtEBbb1lMre494ZeXI5v0sG7MwbwFtI3d2uRHtWZ8v9khGr1ds4AFz3QnvoMSUtsR2kvq+C6H45kUs2ae957+txpKt5MDrcmK+CTTcX/FhlAtib4mlbs+Aj4WJFLOkXuB+MlHOMBlMz3ndJ5I3GDbo+VQvdJC2mCUOBWraihOV92mQQHi2gLXAA1NSsZIjXgnyVSYsrUvNJgTvw+AOVmJ7q5tXENPlZoOKUjSB3fEmKlywf90EQLUjDpphnYtvdFz8Oh8Vbgze+QAQp2VVYImI+1voBErK1EMKqEDFz1DC1CWUzqhl3k2pMgx00JhP3ZZpbX7YtjRmLjJM="
    }
  ]
}

The mention of mls-tester has an [at] to avoid triggering my automatic webfinger resolution.

I've adopted the mediaType and encoding attributes from the draft ActivityPub specification. And I've standardized the Welcome message to incorporate those.

The Create includes the Welcome and a VaultItem as instruments rather than being sent as separate activities. The Welcome can include any number of recipients, so one within the activity should be sufficient.

The VaultItem is specific to the #Enigmatick client and isn't intended to be federated. That's how I'm handling ongoing access to the decrypted data; the VaultItem is a symmetrically encrypted representation of the original content that can be safely stored on the server for repeated use by the client. I'll need to eventually update the @context with that extension.

Adjusting #Enigmatick's previously built functions to align with the #ActivityPub #E2EE draft.

KeyPackage collection on the Actor object

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt | jq '.keyPackages'

Response

"https://enigmatick.social/user/jdt/keys"

Retrieve the KeyPackage Collection

These are populated by the client (the browser using the wasm module loaded in the Svelte SPA). I had a different mechanism in place for managing this (using the instrument field on Activities) but am updating it to use the Add flow for the Collection as specified by the draft.

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt/keys | jq

Response

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Collection",
  "id": "https://enigmatick.social/user/jdt/keys",
  "totalItems": 39,
  "next": "https://enigmatick.social/user/jdt/keys?mkp=true"
}

Follow the next link to retrieve a KeyPackage

This will mark the KeyPackage as distributed and would typically link it to the Actor that signed the request (that's temporarily disabled for testing).

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt/keys?mkp=true | jq

Response

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://purl.archive.org/socialweb/mls"
  ],
  "type": "Collection",
  "totalItems": 1,
  "items": [
    {
      "type": "KeyPackage",
      "id": "https://enigmatick.social/key-package/cfde7b54-8d08-41e9-8eb5-d997a12e9347",
      "attributedTo": "https://enigmatick.social/user/jdt",
      "to": [
        "https://www.w3.org/ns/activitystreams#Public"
      ],
      "mediaType": "message/mls",
      "encoding": "base64",
      "content": "AAEAASCOWe7uxA8tPeI01nDP8ka2irEhgfkK2qM81/o+tEKPEiCz2h2Vv6VVuGSx/C0WINQCuazZm4rQAymxd1JdHGVvaSDurWzIE3z5eXUFz5coFGv9db7DxZDx0arb/gyRuidCAgABImh0dHBzOi8vZW5pZ21hdGljay5zb2NpYWwvdXNlci9qZHQCAAEIAAEAAgADAE0KAAEAAgADAAQABQwAAQACAAMABAAFAAcCAAEBAAAAAGeDNwsAAAAAZ/IDGwBAQJUffd/8/Efgt0ITBeMWfJYcb7S8E5U7yyUINiIUfqmsGwcvTyam6C+9F+k64NBEs0rXjGMjPA6AgiZ2oR4/1wwAQECliseeQfQb0zgq755ZiiujRqLkEWt8WOTJSsBXHK6hey6bNs4wet782NL8k92h1BNuPZDzxHaHXdXe1wgBO6QJ"
    }
  ]
}

Adjusting #Enigmatick's previously built functions to align with the #ActivityPub #E2EE draft.

KeyPackage collection on the Actor object

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt | jq '.keyPackages'

Response

"https://enigmatick.social/user/jdt/keys"

Retrieve the KeyPackage Collection

These are populated by the client (the browser using the wasm module loaded in the Svelte SPA). I had a different mechanism in place for managing this (using the instrument field on Activities) but am updating it to use the Add flow for the Collection as specified by the draft.

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt/keys | jq

Response

{
  "@context": "https://www.w3.org/ns/activitystreams",
  "type": "Collection",
  "id": "https://enigmatick.social/user/jdt/keys",
  "totalItems": 39,
  "next": "https://enigmatick.social/user/jdt/keys?mkp=true"
}

Follow the next link to retrieve a KeyPackage

This will mark the KeyPackage as distributed and would typically link it to the Actor that signed the request (that's temporarily disabled for testing).

Request

> curl -H "Accept: application/activity+json" https://enigmatick.social/user/jdt/keys?mkp=true | jq

Response

{
  "@context": [
    "https://www.w3.org/ns/activitystreams",
    "https://purl.archive.org/socialweb/mls"
  ],
  "type": "Collection",
  "totalItems": 1,
  "items": [
    {
      "type": "KeyPackage",
      "id": "https://enigmatick.social/key-package/cfde7b54-8d08-41e9-8eb5-d997a12e9347",
      "attributedTo": "https://enigmatick.social/user/jdt",
      "to": [
        "https://www.w3.org/ns/activitystreams#Public"
      ],
      "mediaType": "message/mls",
      "encoding": "base64",
      "content": "AAEAASCOWe7uxA8tPeI01nDP8ka2irEhgfkK2qM81/o+tEKPEiCz2h2Vv6VVuGSx/C0WINQCuazZm4rQAymxd1JdHGVvaSDurWzIE3z5eXUFz5coFGv9db7DxZDx0arb/gyRuidCAgABImh0dHBzOi8vZW5pZ21hdGljay5zb2NpYWwvdXNlci9qZHQCAAEIAAEAAgADAE0KAAEAAgADAAQABQwAAQACAAMABAAFAAcCAAEBAAAAAGeDNwsAAAAAZ/IDGwBAQJUffd/8/Efgt0ITBeMWfJYcb7S8E5U7yyUINiIUfqmsGwcvTyam6C+9F+k64NBEs0rXjGMjPA6AgiZ2oR4/1wwAQECliseeQfQb0zgq755ZiiujRqLkEWt8WOTJSsBXHK6hey6bNs4wet782NL8k92h1BNuPZDzxHaHXdXe1wgBO6QJ"
    }
  ]
}

detecting a couple of new platforms #Enigmatick #Stegodon

detecting a couple of new platforms #Enigmatick #Stegodon

FEP-8fcf implemented last night in #Enigmatick.

[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: pool acquire took 436.968µs
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] Request[target: post /inbox, host: enigmatick.social, signature: keyId="https://[redacted]main-key",algorithm="rsa-sha256",headers="(request-target) host date digest content-type collection-synchronization",signature="G...s=, content-type: application/activity+json, content-length: 427, user-agent: http.rb/5.1.1 (Mastodon/4.2.10; +[redacted])]
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: verify_axum took 5.040791ms
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: update_instance took 26.262904ms
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: total time 32.021645ms
[2025-12-18T16:24:56Z DEBUG enigmatick::blocklist] checking [redacted] against BlockList
[2025-12-18T16:24:56Z DEBUG enigmatick::blocklist] checking actor [redacted] against BlockList
[2025-12-18T16:24:56Z DEBUG enigmatick::server::routes::inbox] Processing Collection-Synchronization header: collectionId=[redacted]/followers, digest=195650b06cb44278929fd1c2320831f17a2899312dc482de7fc41301778d210a
[2025-12-18T16:24:56Z DEBUG enigmatick::server::routes::inbox::announce] Announce[by: [redacted], obj: ref:[redacted], at: 2025-12-18T16:24:55Z, id: [redacted]/activity]
[2025-12-18T16:24:56Z DEBUG enigmatick::server::follower_sync] Follower sync digest matches for remote actor [redacted], no reconciliation needed

This also fixes some signature verification issues I hadn't noticed were occurring due to the collection-synchronization header not being included in my verification code. I see now how to make that more flexible to accommodate any headers and will be improving that further.

FEP-8fcf implemented last night in #Enigmatick.

[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: pool acquire took 436.968µs
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] Request[target: post /inbox, host: enigmatick.social, signature: keyId="https://[redacted]main-key",algorithm="rsa-sha256",headers="(request-target) host date digest content-type collection-synchronization",signature="G...s=, content-type: application/activity+json, content-length: 427, user-agent: http.rb/5.1.1 (Mastodon/4.2.10; +[redacted])]
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: verify_axum took 5.040791ms
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: update_instance took 26.262904ms
[2025-12-18T16:24:56Z DEBUG enigmatick::server::extractors] AxumSigned: total time 32.021645ms
[2025-12-18T16:24:56Z DEBUG enigmatick::blocklist] checking [redacted] against BlockList
[2025-12-18T16:24:56Z DEBUG enigmatick::blocklist] checking actor [redacted] against BlockList
[2025-12-18T16:24:56Z DEBUG enigmatick::server::routes::inbox] Processing Collection-Synchronization header: collectionId=[redacted]/followers, digest=195650b06cb44278929fd1c2320831f17a2899312dc482de7fc41301778d210a
[2025-12-18T16:24:56Z DEBUG enigmatick::server::routes::inbox::announce] Announce[by: [redacted], obj: ref:[redacted], at: 2025-12-18T16:24:55Z, id: [redacted]/activity]
[2025-12-18T16:24:56Z DEBUG enigmatick::server::follower_sync] Follower sync digest matches for remote actor [redacted], no reconciliation needed

This also fixes some signature verification issues I hadn't noticed were occurring due to the collection-synchronization header not being included in my verification code. I see now how to make that more flexible to accommodate any headers and will be improving that further.

@AdamStuartSmith I had that problem with vanilla Mastodon for a bit when I first installed it. Any posts that got any sort of activity from anyone with a significant following would overwhelm my server for several minutes.

I solved that by setting up a #Varnish cache in front of my server. I don't use Mastodon anymore, but I still use the cache in front of my bespoke software (#Enigmatick). It works really well to take the load off of the core server for those fan-out posts. Varnish handles the load with aplomb.

@AdamStuartSmith I had that problem with vanilla Mastodon for a bit when I first installed it. Any posts that got any sort of activity from anyone with a significant following would overwhelm my server for several minutes.

I solved that by setting up a #Varnish cache in front of my server. I don't use Mastodon anymore, but I still use the cache in front of my bespoke software (#Enigmatick). It works really well to take the load off of the core server for those fan-out posts. Varnish handles the load with aplomb.

Enigmatick ActivityPub C2S

https://enigmatick.social/objects?uuid=9d4f1be4-ca56-4176-87a7-49fd72ddc98a

Enigmatick ActivityPub C2S

https://enigmatick.social/objects?uuid=9d4f1be4-ca56-4176-87a7-49fd72ddc98a

@reiver …and it's not just the sheer amount of storage; prudent expiration and cleanup (e.g., #Enigmatick clears cache data each night that's greater than 30 days old) helps to manage that. It's the application load of pulling up the same records tens of thousands of times within a few seconds when new content is created (i.e., the fan-out of initial delivery and the resulting requests back for content).

I use Varnish and that helps a lot with the latter case. But a #FediCDN could be built on Varnish (like Fastly is) and provide decentralization in that service.

@reiver …and it's not just the sheer amount of storage; prudent expiration and cleanup (e.g., #Enigmatick clears cache data each night that's greater than 30 days old) helps to manage that. It's the application load of pulling up the same records tens of thousands of times within a few seconds when new content is created (i.e., the fan-out of initial delivery and the resulting requests back for content).

I use Varnish and that helps a lot with the latter case. But a #FediCDN could be built on Varnish (like Fastly is) and provide decentralization in that service.

I finally got around to setting up #nodeinfo properly on #Enigmatick as an effort to be a more compliant #ActivityPub participant. Coincidentally (and unexpectedly), my connected instances have jumped substantially (>15% in 24 hours):

enigmatick=> SELECT                                                                                                          CASE                                                                                                                       WHEN created_at > NOW() - INTERVAL '1 day' THEN 'last 24h'                                                               WHEN created_at > NOW() - INTERVAL '2 days' THEN '24-48h ago'                                                            WHEN created_at > NOW() - INTERVAL '3 days' THEN '48-72h ago'                                                          END as period,                                                                                                           COUNT(*)                                                                                                               FROM instances                                                                                                           WHERE created_at > NOW() - INTERVAL '3 days'                                                                             GROUP BY period                                                                                                          ORDER BY period;
   period   | count
------------+-------
 24-48h ago |     1
 48-72h ago |     3
 last 24h   |   164
(3 rows)

Maybe it's not directly related - I also fixed some direct object and tags link references to provide proper ActivityPub representations.

I finally got around to setting up #nodeinfo properly on #Enigmatick as an effort to be a more compliant #ActivityPub participant. Coincidentally (and unexpectedly), my connected instances have jumped substantially (>15% in 24 hours):

enigmatick=> SELECT                                                                                                          CASE                                                                                                                       WHEN created_at > NOW() - INTERVAL '1 day' THEN 'last 24h'                                                               WHEN created_at > NOW() - INTERVAL '2 days' THEN '24-48h ago'                                                            WHEN created_at > NOW() - INTERVAL '3 days' THEN '48-72h ago'                                                          END as period,                                                                                                           COUNT(*)                                                                                                               FROM instances                                                                                                           WHERE created_at > NOW() - INTERVAL '3 days'                                                                             GROUP BY period                                                                                                          ORDER BY period;
   period   | count
------------+-------
 24-48h ago |     1
 48-72h ago |     3
 last 24h   |   164
(3 rows)

Maybe it's not directly related - I also fixed some direct object and tags link references to provide proper ActivityPub representations.

I added the ability to manage property values on users' profiles to #Enigmatick. As I'm using this more as my full-time #Fediverse platform, I'm running into things that I've sort of blown off in the past as silly or unnecessary but finding useful now.

In this case, I implemented it using Add and Remove messages posted by the client (a single-page application in Svelte mediated by a wasm module) to the user's outbox on the server. The Actor is the target and the PropertyValue is the Object.

It's not canonically how those #ActivityPub messages should be used (e.g., managing a Collection - the attachment field on an Actor is not really a Collection). But I think logically it makes sense. And it's client communication only; those activities are not federated. When a profile is updated via the Add or Remove, a separate Update is sent out to known servers to update the profile.

Also, I may have gone a little overboard with my own properties.

I added the ability to manage property values on users' profiles to #Enigmatick. As I'm using this more as my full-time #Fediverse platform, I'm running into things that I've sort of blown off in the past as silly or unnecessary but finding useful now.

In this case, I implemented it using Add and Remove messages posted by the client (a single-page application in Svelte mediated by a wasm module) to the user's outbox on the server. The Actor is the target and the PropertyValue is the Object.

It's not canonically how those #ActivityPub messages should be used (e.g., managing a Collection - the attachment field on an Actor is not really a Collection). But I think logically it makes sense. And it's client communication only; those activities are not federated. When a profile is updated via the Add or Remove, a separate Update is sent out to known servers to update the profile.

Also, I may have gone a little overboard with my own properties.

Occasionally a comment to a post is deleted. Or sometimes it hasn't reached my server for whatever reason. In this case, I can tell based on the next comment that it's from Bluesky and I really just don't care enough about that service to figure out why my server may not be pulling it down correctly.

But it sucks to have broken threads because one bit goes missing. So I added in this "Unavailable Comment" component just now.

I think that will help.

#Enigmatick #Threading

Occasionally a comment to a post is deleted. Or sometimes it hasn't reached my server for whatever reason. In this case, I can tell based on the next comment that it's from Bluesky and I really just don't care enough about that service to figure out why my server may not be pulling it down correctly.

But it sucks to have broken threads because one bit goes missing. So I added in this "Unavailable Comment" component just now.

I think that will help.

#Enigmatick #Threading

Because #Enigmatick is primarily built for me, there are a lot of niceties that I haven't gotten around to adding because "I can just look in the database." I also intentionally don't have any sort of notification mechanism (I may yield on that at some point). I don't even have a field for "email" in my database Actor model.

One of those was a popover UI for Likes and Reposts. I had included the data about who liked and reposted a post months ago in the JSON that the server provides to the client, but I hadn't followed through with utilizing it beyond a basic count.

This is much better than querying my database.

Because #Enigmatick is primarily built for me, there are a lot of niceties that I haven't gotten around to adding because "I can just look in the database." I also intentionally don't have any sort of notification mechanism (I may yield on that at some point). I don't even have a field for "email" in my database Actor model.

One of those was a popover UI for Likes and Reposts. I had included the data about who liked and reposted a post months ago in the JSON that the server provides to the client, but I hadn't followed through with utilizing it beyond a basic count.

This is much better than querying my database.

I recently added direct TLS certificate management in #Enigmatick using tls-alpn-01, so all that's needed to get TLS up and running is a public IP address and a domain name pointed at that address. This is managed by a separate reverse proxy process that's bundled in the main Enigmatick binary.

I think while I'm at #DEFCON, I'm going to experiment with Arti (or anything similar) to build a Tor reverse proxy as well. I can't do automatic TLS certificate provisioning for onion services from #LetsEncrypt, but that may be okay.

I recently added direct TLS certificate management in #Enigmatick using tls-alpn-01, so all that's needed to get TLS up and running is a public IP address and a domain name pointed at that address. This is managed by a separate reverse proxy process that's bundled in the main Enigmatick binary.

I think while I'm at #DEFCON, I'm going to experiment with Arti (or anything similar) to build a Tor reverse proxy as well. I can't do automatic TLS certificate provisioning for onion services from #LetsEncrypt, but that may be okay.

Late night progress. #LateNightDevs #ActivityPub #Enigmatick

Late night progress. #LateNightDevs #ActivityPub #Enigmatick

The changes I've made to #Enigmatick over the last couple of months are significant and I plan to write more about them soon. But one of my favorite is this small bit (supported by some larger structures) in my #ActivityPub crate.

impl ApCollection {
    /// Creates an async stream over all items in this collection and its linked pages.
    pub fn stream_all<F, Fut>(self, fetcher: F) -> ApCollectionStream
    where
        F: Fn(&str) -> Fut + Send + Sync + 'static,
        Fut: Future<Output = Result<ApCollection, Box<dyn std::error::Error + Send + Sync>>>
            + Send
            + 'static,
    {
        ApCollectionStream::new(self, fetcher)
    }
}

I wanted to be able to (using async code) take an ActivityPub Collection and iterate over all of its members without worrying about pulling in each page. This implementation accomplishes that.

The full source is here.

The changes I've made to #Enigmatick over the last couple of months are significant and I plan to write more about them soon. But one of my favorite is this small bit (supported by some larger structures) in my #ActivityPub crate.

impl ApCollection {
    /// Creates an async stream over all items in this collection and its linked pages.
    pub fn stream_all<F, Fut>(self, fetcher: F) -> ApCollectionStream
    where
        F: Fn(&str) -> Fut + Send + Sync + 'static,
        Fut: Future<Output = Result<ApCollection, Box<dyn std::error::Error + Send + Sync>>>
            + Send
            + 'static,
    {
        ApCollectionStream::new(self, fetcher)
    }
}

I wanted to be able to (using async code) take an ActivityPub Collection and iterate over all of its members without worrying about pulling in each page. This implementation accomplishes that.

The full source is here.

@mariusor That's along the lines of what I was working with as well (and that's still how this div is defined). The trouble I kept running into was placing the cursor after updating the display of a piece of text (e.g., adding the link automatically to a hashtag or updating the ActivityPub handle to indicate whether it was found and if it has the keys functionality that #Enigmatick users have). I could update the display, but the placement of the cursor was haphazard because the HTML content in the div would be across multiple lines and placing the cursor at the end was challenging.

But in retrospect, I think it's just more complicated than I need to worry about. I really like this new approach better. It's cleaner. But it does require more screen real-estate and won't be terribly useful on mobile screens. I can probably use CSS to do a switchable screen overlay for the preview.

@mariusor That's along the lines of what I was working with as well (and that's still how this div is defined). The trouble I kept running into was placing the cursor after updating the display of a piece of text (e.g., adding the link automatically to a hashtag or updating the ActivityPub handle to indicate whether it was found and if it has the keys functionality that #Enigmatick users have). I could update the display, but the placement of the cursor was haphazard because the HTML content in the div would be across multiple lines and placing the cursor at the end was challenging.

But in retrospect, I think it's just more complicated than I need to worry about. I really like this new approach better. It's cleaner. But it does require more screen real-estate and won't be terribly useful on mobile screens. I can probably use CSS to do a switchable screen overlay for the preview.

For a long time, I was trying to make the entry of text in my #Enigmatick Compose component behave as rich text like I've seen in solutions like Notion or Confluence. But I don't think I'm a strong enough front-end coder to make that work; it was always buggy and problematic.

fn main() {
    println!("Here's my new approach: a live representation of the text displayed as readers will see it");
}

For a long time, I was trying to make the entry of text in my #Enigmatick Compose component behave as rich text like I've seen in solutions like Notion or Confluence. But I don't think I'm a strong enough front-end coder to make that work; it was always buggy and problematic.

fn main() {
    println!("Here's my new approach: a live representation of the text displayed as readers will see it");
}

To save myself some hassle, I've taken to adding functions I want in a hurry to the server command itself.

/srv/enigmatick# ./enigmatick
A federated communication platform server

Usage: enigmatick <COMMAND>

Commands:
  init         Initialize folder structure for media files
  template     Generate .env.template file
  migrate      Run database migrations
  cache        Manage cached media files
  system-user  Create or ensure system user exists
  server       Start the web server and background tasks
  instances    Manage federated instances
  send         Send various activities
  muted-terms  Manage user muted terms
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version


/srv/enigmatick# ./enigmatick cache
Manage cached media files

Usage: enigmatick cache <COMMAND>

Commands:
  prune          Prune cached files older than specified duration (e.g., 30d, 2m, 1y)
  delete         Delete cached item by URL
  delete-server  Delete cached items from server/domain pattern
  help           Print this message or the help of the given subcommand(s)

Options:

  -h, --help  Print help

/srv/enigmatick# ./enigmatick send
Send various activities

Usage: enigmatick send <COMMAND>

Commands:
  actor-update  Send actor update to all known instances
  help          Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

The commands here run with direct database access. But my plan is to extend a lot of this (and much more) as a network-based CLI tool (e.g., maybe as ek in the enigmatick_wasm crate).

#Enigmatick #RustLang #ActivityPub

To save myself some hassle, I've taken to adding functions I want in a hurry to the server command itself.

/srv/enigmatick# ./enigmatick
A federated communication platform server

Usage: enigmatick <COMMAND>

Commands:
  init         Initialize folder structure for media files
  template     Generate .env.template file
  migrate      Run database migrations
  cache        Manage cached media files
  system-user  Create or ensure system user exists
  server       Start the web server and background tasks
  instances    Manage federated instances
  send         Send various activities
  muted-terms  Manage user muted terms
  help         Print this message or the help of the given subcommand(s)

Options:
  -h, --help     Print help
  -V, --version  Print version


/srv/enigmatick# ./enigmatick cache
Manage cached media files

Usage: enigmatick cache <COMMAND>

Commands:
  prune          Prune cached files older than specified duration (e.g., 30d, 2m, 1y)
  delete         Delete cached item by URL
  delete-server  Delete cached items from server/domain pattern
  help           Print this message or the help of the given subcommand(s)

Options:

  -h, --help  Print help

/srv/enigmatick# ./enigmatick send
Send various activities

Usage: enigmatick send <COMMAND>

Commands:
  actor-update  Send actor update to all known instances
  help          Print this message or the help of the given subcommand(s)

Options:
  -h, --help  Print help

The commands here run with direct database access. But my plan is to extend a lot of this (and much more) as a network-based CLI tool (e.g., maybe as ek in the enigmatick_wasm crate).

#Enigmatick #RustLang #ActivityPub

I've built #Enigmatick to manage threading using conversation IDs like Mastodon uses. But that's not aligned with #ActivityPub - or at least it wasn't the least time I looked. ActivityPub uses inReplyTo to indicate direct parent relationships and doesn't have a concept for linking multiple objects under a single ID.

Other systems don't use the conversation convention and that's caused some discord in my timeline. So now I'm learning about closure tables and concepts I probably would already know if I had studied computer science in college. But it's never too late to learn!

I've built #Enigmatick to manage threading using conversation IDs like Mastodon uses. But that's not aligned with #ActivityPub - or at least it wasn't the least time I looked. ActivityPub uses inReplyTo to indicate direct parent relationships and doesn't have a concept for linking multiple objects under a single ID.

Other systems don't use the conversation convention and that's caused some discord in my timeline. So now I'm learning about closure tables and concepts I probably would already know if I had studied computer science in college. But it's never too late to learn!

I reworked my image processing on #Enigmatick recently but haven't had a chance to try it out. Here's a picture of my son checking in with the neighbor llamas a few years ago.

I reworked my image processing on #Enigmatick recently but haven't had a chance to try it out. Here's a picture of my son checking in with the neighbor llamas a few years ago.

Migrated my followers from #Serendipitous to #Enigmatick. The process was pleasantly simple - even though I haven't implemented Move activities on the new system yet. I'll tackle that shortly as I see I have a backlog of those in my unprocessable queue.

It also seems like I need to expand my database connection pool; all of those Follow activities coming in at once was a little messy.

Migrated my followers from #Serendipitous to #Enigmatick. The process was pleasantly simple - even though I haven't implemented Move activities on the new system yet. I'll tackle that shortly as I see I have a backlog of those in my unprocessable queue.

It also seems like I need to expand my database connection pool; all of those Follow activities coming in at once was a little messy.

Using my amateur woodworking skills, I built myself a server rack using red oak and 72" metal rails last year. It lives in a dedicated corner of my basement where sound and heat are manageable challenges. It has since housed several enterprise switches (SONiC) and routers (VPP) I've been building around.

My venerable DS1513 NAS turned is now 12 years old, so I decided to leverage the rack to move to a Supermicro X11-based server running TrueNAS SCALE to replace it. 12 drives, 40 cores, 2 x 25Gb NICs, and 128GB of ECC RAM later, I may have over done it a little. But it's a very nice upgrade. I finished moving the NFS shares for this service over tonight and migrating to a new database run on the NAS server itself (which eliminates a long-standing pain point with iSCSI).

I'm considering just turning down my long-running Mastodon instance in favor of leaning in to #Enigmatick. As a result, I'm sending out lots of follow requests. Feel free to follow me back here, as this is what I'll be paying attention to moving forward.

Using my amateur woodworking skills, I built myself a server rack using red oak and 72" metal rails last year. It lives in a dedicated corner of my basement where sound and heat are manageable challenges. It has since housed several enterprise switches (SONiC) and routers (VPP) I've been building around.

My venerable DS1513 NAS turned is now 12 years old, so I decided to leverage the rack to move to a Supermicro X11-based server running TrueNAS SCALE to replace it. 12 drives, 40 cores, 2 x 25Gb NICs, and 128GB of ECC RAM later, I may have over done it a little. But it's a very nice upgrade. I finished moving the NFS shares for this service over tonight and migrating to a new database run on the NAS server itself (which eliminates a long-standing pain point with iSCSI).

I'm considering just turning down my long-running Mastodon instance in favor of leaning in to #Enigmatick. As a result, I'm sending out lots of follow requests. Feel free to follow me back here, as this is what I'll be paying attention to moving forward.

#Enigmatick's timeline retrieval performance was getting a little slow, so I spent some time optimizing my PostgreSQL queries yesterday. I was able to take the EXPLAIN ANALYZE loop from 17 seconds (an exaggeration of the actual experience in production of a delay of about 2 seconds) to 53ms. It was mostly down to a JOIN that employs a JSONB_AGG to pull in Actor records associated with an Activity to avoid having to make additional calls. The way I was matching against the attributed_to JSONB column was apparently slowing things down.

I also have some #Wasm calls that proactively transform #E2EE asymmetrically encrypted messages to symmetrically encrypted vault items. Those run on each load of the timeline, but I was able to use spawn_local to move the processing to a background task for timeline views that don't display encrypted content.

#Enigmatick's timeline retrieval performance was getting a little slow, so I spent some time optimizing my PostgreSQL queries yesterday. I was able to take the EXPLAIN ANALYZE loop from 17 seconds (an exaggeration of the actual experience in production of a delay of about 2 seconds) to 53ms. It was mostly down to a JOIN that employs a JSONB_AGG to pull in Actor records associated with an Activity to avoid having to make additional calls. The way I was matching against the attributed_to JSONB column was apparently slowing things down.

I also have some #Wasm calls that proactively transform #E2EE asymmetrically encrypted messages to symmetrically encrypted vault items. Those run on each load of the timeline, but I was able to use spawn_local to move the processing to a background task for timeline views that don't display encrypted content.

Integrating the shared jdt_activity_pub #RustLang crate used by the backend #Enigmatick server into the frontend #Wasm component. Necessarily broke a couple of Vodozemac functions, but those will be replaced by #OpenMLS shortly.

Integrating the shared jdt_activity_pub #RustLang crate used by the backend #Enigmatick server into the frontend #Wasm component. Necessarily broke a couple of Vodozemac functions, but those will be replaced by #OpenMLS shortly.

@alyxshang Most everything with #Enigmatick is in #Rust. The core is built on Rocket. The web interface is #Svelte, but it relies heavily on a client-side #WASM component built in Rust as well.

My #ActivityPub structs can be found here if they're of any use to you.

The Mitra and Lemmy projects are also Rust and might provide some useful inspiration.

One interesting thing I discovered about the RsaSignature2017 implementation in Mastodon is that rather than combining the options and message hashes as their native binary representation like: let combined = [options, message].concat() to use in the signature verification like verifying_key.verify(&combined, &signature), Mastodon converts the hashes to a hex string, smashes the textual representation together, and then performs the signature on the bytes of that (which means that we have to do the same in reverse for verification).

let options_hex = hex::encode(options);
let result_hex = hex::encode(result);
let combined_hex = format!("{options_hex}{result_hex}");
verifying_key.verify(combined_hex.as_bytes(), &signature);

It's inefficient; I suspect #Ruby may make operating on the bytes themselves challenging. But I'm just guessing, since it's been quite a while (10+ years) since I've done much in that ecosystem.

It'll be nice when the #Fediverse evolves past this bit of history.

#RustLang #Enigmatick #ActivityPub #RsaSignature2017

I took the time this evening to wade through the swamp of sadness that is #JsonLD and the idiosyncrasies of how it's implemented in Mastodon. However dubious the whole effort may be, I now have a working prototype that I'll add to #Enigmatick in the next few days.

My motivation is to allow me to connect my server to a relay with some confidence that I can verify the source of the messages that are being relayed in. There are newer standards that I want to build to eventually. But the reality is that this bastardized #RsaSignature2017 format is what's in use for the vast majority of messages today.

I'll add the routines to my #jdtactivitypub crate (on crates.io) so that other #RustLang folks have a working example. I'm also going to try to add my HTTP signature and verification methods to that crate.

I've become increasingly weary of the task of keeping my backend and #WASM structs aligned. So I decided to break #Enigmatick's #ActivityPub structs out into their own crate that I can use in both places. I have the Inbox and Outbox traits moved over into my core crate and just need to complete the removal of any dependencies on the data model structs.

Maybe this AP-focused crate can help other #Rust #Fediverse devs eventually.

I've become increasingly weary of the task of keeping my backend and #WASM structs aligned. So I decided to break #Enigmatick's #ActivityPub structs out into their own crate that I can use in both places. I have the Inbox and Outbox traits moved over into my core crate and just need to complete the removal of any dependencies on the data model structs.

Maybe this AP-focused crate can help other #Rust #Fediverse devs eventually.

I write posts here on #Enigmatick using #markdown. The web interface integrates the Showdown converter combined with highlight.js.

As such, my posts often have elements that might not look great on Mastodon and other platforms that strip a lot of the context out. They seem to do an okay job of making code blocks look different, but in a pretty terrible way.

If you want to see what I intend the posts to look like, use the "Open Original Page" on any of my posts in Mastodon to open the URL directly to my web server.

#Enigmatick has three sending modes today: Public, Direct, and Private. I've been working on tweaking the Public mode this morning. It was built so that it would compile inboxes from the list of a sender's followers, while deferring to the sharedInbox attribute where it exists. I've altered that so that Public posts are now sent to the sharedInbox of all instances that have communicated with my server in the last two weeks and any individual inboxes for followers on systems that do not use a sharedInbox. This is one of the queries that facilitates that change.

pub async fn get_instance_inboxes(conn: &Db) -> Vec<ApAddress> {
    let cutoff = Utc::now().naive_utc() - chrono::Duration::days(14);

    conn.run(move |c| {
        instances::table
            .filter(instances::blocked.eq(false))
            .filter(instances::shared_inbox.is_not_null())
            .filter(instances::last_message_at.gt(cutoff))
            .select(instances::shared_inbox.assume_not_null())
            .get_results::<String>(c)
    })
    .await
    .unwrap_or_default()
    .into_iter()
    .map(ApAddress::from)
    .collect()
}

I also made a significant change to the dispatching of these events, leveraging tokio to avoid having to wait for each connection to complete sequentially.

let handles: Vec<JoinHandle<LogMessage>> = inboxes
        .into_iter()
        .map(|inbox| {
            let client = client.clone();
            let body = body.clone();
            let profile = profile.clone();

            tokio::spawn(process_inbox(inbox, body, profile, client))
        })
        .collect();

    let mut logs = Vec::new();
    for handle in handles {
        if let Ok(log) = handle.await {
            logs.push(log);
        }
    }

I'm planning to add a fourth mode for Group Private #E2EE messages (i.e., encrypted for multiple group participants). I have the concept in my head; I just need to sit down and build it.

#ActivityPub #RustLang #WASM

I redesigned the sending function in #Enigmatick to use tokio::spawn. This message is the first test.

A beautiful thing about the #Fediverse is that you can interact with it however you want. If you want a dark interface with flashes of red, goldenrod, and blue - no one is going to stop you. And you want to do it all with #WASM in your browser? That's fine. And you want #E2EE? Go for it. You want to do it with #SvelteKit and #RustLang? Fine. Whatever.

None of it is a trivial undertaking, of course. Although if you just want to interact with #ActivityPub from a database (or JSON files) and curl, you could build something over a weekend (see my older #SofaPub code for an example).

Threads and Bluesky really hold no interest for me. I exist on Threads solely so that I can be there in there event they actually do something interesting with ActivityPub integration (they're pretty damn slow about it, though). Until I can tell my non-technical friends on Threads to follow me here on #Enigmatick and they don't have to jump through a bunch of hoops, I'm skeptical. Bluesky… meh.

JSONB support in Diesel (2.2.4) and Sqlite (3.47) is getting better, but it's still a bit fiddly. I can get diesel_cli (via diesel migration run) to create the database and tables with JSONB columns, but it fails to create the Rust schema.rs with:

Running migration 2024-12-23-034947_create_activities
Unsupported type: `jsonb`

I think diesel_cli just needs to catch up with the JSONB support in diesel. I'll set it aside in #Enigmatick for now and check back in a month or two.

#RustLang #Sqlite #Diesel

#Enigmatick displays posts in reverse chronological order. To facilitate that, I use a ORDER BY ... DESC directive in my timeline SQL query.

Comments to each post are displayed in forward chronological order and are threaded. I use the same query to retrieve a conversation (with some filters) as I do the timeline.

I've been struggling a little with the threading because of the order of the query. A reply to a reply might be in the list before the note it's replying to. I have some complex Typescript to handle the placement of notes and to queue orphan replies until their parent is placed.

For whatever reason, it occurred to me tonight that I could just flip the database query to ASC for the conversation query and solve all my troubles. A reminder, I suppose, that it's often helpful to take a step back and look at the problem from a larger perspective.

I'm growing more confident in being able to use #Enigmatick on a day-to-day basis, so expect to see more Follow requests from my user account here.

I'll keep my @justin account because it's always useful to have a Mastodon instance for testing. I won't be migrating as much as just duplicating. Hopefully that's okay with y'all.

I do plan to figure out account migrations at some point because it doesn't look all that difficult. It's just not a priority for me right now.

Re-establishing sqlite functionality is more in my sights. It looks like JSONB support has improved considerably since I did anything with it, so it may be easier than it was. The challenge I had before was managing two types of structs: one using serde_json::Value fields for Postgresql JSONB columns, and one using String fields for Sqlite TEXT columns. That's a real hassle; hopefully I can improve things.

Some time in the last week, I started seeing posts from a Norwegian law enforcement agency on my #Enigmatick personal server. It's not an account I (or anyone else on my server) followed.

I did some digging in my database and it appears that the service is sending to whatever #ActivityPub servers it knows about. Note the lack of any cc addresses in the JSON message.

I'm not complaining; I find it interesting. I wonder if we'll see more of this type of activity as the ecosystem grows, where services just target the sharedInbox on servers to send public messages.

#Fediverse

hey @jdt, I was just looking at an #Enigmatick post that is federated onto mastodon.social, but when I try to search for posts by their enigmatick.social url nothing comes up on either on mastodon.social or my server. Have you not implemented WebFinger-able posts yet?