#filespooler — Public Fediverse posts

@liiwi I already know and am a fan of #Rust! My #Filespooler program, for instance, is written in Rust: https://www.complete.org/filespooler/

@minoru Thanks for sharing that info!

It's practical because Syncthing's very nice distributed nature lets it be an underlying communications layer for a lot of other things. For instance, I talk about using #Filespooler over #Syncthing here: https://www.complete.org/using-filespooler-over-syncthing/ , and #NNCP over Syncthing here: https://changelog.complete.org/archives/10219-a-simple-delay-tolerant-offline-capable-mesh-network-with-syncthing-optional-nncp

That you can have a "roaming" device (eg, a phone) nicely sync between two locations that otherwise don't have connectivity to each other is pretty slick.

@alienghic @schlink Yes! This is a perfect use case for #NNCP. As @craigmaloney mentioned, #Syncthing can also do the job; if you name your files according to a set pattern, you can just add them to a directory on the sending side, and mv/rm them out of there on the receiving side when processed.

My #Filespooler tool https://www.complete.org/filespooler/ can provide some more queue-processing tools, esp. if you need the queue to be processed in order. It can run atop NNCP, Syncthing, or others.

I wrote "Building an #Asynchronous, Internet-Optional Instant Messaging System" today. It suggests using #DeltaChat (@delta) across an #email system that can run on non-IP networks to achieve #IM coverage without having to have an Internet backbone, using things like #NNCP, #Syncthing, #Filespooler, and such. I love the ways we can combine interesting tools like this!

https://changelog.complete.org/archives/10435-building-an-asynchronous-internet-optional-instant-messaging-system

@downey I am on board with the movement to #GiveUpGithub! I have already transitioned many (though not all of my older, semi-abandoned) repos to #Debian's #salsa service, which is based on #FLOSS #Gitlab and not limited to Debian projects. @Codeberg is another common choice and I would heartily recommend both.

My most recent #FreeSoftware project, #filespooler, has never had a Github presence of any kind. https://salsa.debian.org/jgoerzen/filespooler

3/

Introducing Fast, Ordered Unixy #Queues over #NNCP and #Syncthing with #Filespooler: https://floss.social/@jgoerzen/108387639329271399

"#Modem World" and modern communities that span interests: https://floss.social/@jgoerzen/108546787833118936

Why do I feel drawn to work from my #DEC #vt510 #terminal? Does it have advantages over my 27" LCD GUI? https://floss.social/@jgoerzen/108898543787831520

Growing up through the #PC and #Internet revolution in #rural America: https://floss.social/@jgoerzen/108909211718813036

@deadparrot Hello! I'm a #rust/#rustlang person. Been using it at work for several years, and also have several crates out there. #Filespooler https://www.complete.org/filespooler/ is my most recent project. I'm also behind #xbnet and #lorapipe, which can tunnel TCP/IP or other data down #XBee SX or #LoRA long-distance radios.

@deadparrot Hello! I'm a #rust/#rustlang person. Been using it at work for several years, and also have several crates out there. #Filespooler https://www.complete.org/filespooler/ is my most recent project. I'm also behind #xbnet and #lorapipe, which can tunnel TCP/IP or other data down #XBee SX or #LoRA long-distance radios.

@deadparrot Hello! I'm a #rust/#rustlang person. Been using it at work for several years, and also have several crates out there. #Filespooler https://www.complete.org/filespooler/ is my most recent project. I'm also behind #xbnet and #lorapipe, which can tunnel TCP/IP or other data down #XBee SX or #LoRA long-distance radios.

So that weird #pipe issue I saw with #Filespooler and used #strace to diagnose? Probably a #kernel #bug. https://changelog.complete.org/archives/10390-pipe-issue-likely-a-kernel-bug

@gwil Welcome, gwil! Glad to see you here. It looks like your interests intersect mine quite a bit; I've been interested in #NNCP and #Syncthing as decentralized, offline-first or offline-capable systems, and recently wrote #Filespooler in that space.

I hadn't heard of #Earthstar. After reading the website, it sounds kind of like Syncthing, but with a more-public way of joining a share? (Sharing a link without having the approve the request) I'd be interested in other comparisons.

Ever wanted to pass things like environment variables to commands in #ssh? #Filespooler without queues can help! Its metadata is perfect for this. https://www.complete.org/using-filespooler-without-queues-to-pass-more-metadata/

It seems that lately I’ve written several shell implementations of a simple queue that enforces ordered execution of jobs that may arrive out of order. After writing this for the nth time in bash, I decided it was time to do it properly. But first, a word on the why of it all.

My needs arose primarily from handling Backups over Asynchronous Communication methods – in this case, NNCP. When backups contain incrementals that are unpacked on the destination, they must be applied in the correct order.

In some cases, like ZFS, the receiving side will detect an out-of-order backup file and exit with an error. In those cases, processing in random order is acceptable but can be slow if, say, hundreds or thousands of hourly backups have stacked up over a period of time. The same goes for using gitsync-nncp to synchronize git repositories. In both cases, a best effort based on creation date is sufficient to produce a significant performance improvement.

With other cases, such as tar or dar backups, the receiving cannot detect out of order incrementals. In those situations, the incrementals absolutely must be applied with strict ordering. There are many other situations that arise with these needs also. Filespooler is the answer to these.

Before writing my own program, I of course looked at what was out there already. I looked at celeary, gearman, nq, rq, cctools work queue, ts/tsp (task spooler), filequeue, dramatiq, GNU parallel, and so forth.

Unfortunately, none of these met my needs at all. They all tended to have properties like:

• An extremely complicated client/server system that was incompatible with piping data over existing asynchronous tools
• A large bias to processing of small web requests, resulting in terrible inefficiency or outright incompatibility with jobs in the TB range
• An inability to enforce strict ordering of jobs, especially if they arrive in a different order from how they were queued

Many also lacked some nice-to-haves that I implemented for Filespooler:

• Support for the encryption and cryptographic authentication of jobs, including metadata
• First-class support for arbitrary compressors
• Ability to use both stream transports (pipes) and filesystem-like transports (eg, rclone mount, S3, Syncthing, or Dropbox)

Filespooler is a tool in the Unix tradition: that is, do one thing well, and integrate nicely with other tools using the fundamental Unix building blocks of files and pipes. Filespooler itself doesn’t provide transport for jobs, but instead is designed to cooperate extremely easily with transports that can be written to as a filesystem or piped to – which is to say, almost anything of interest.

Filespooler is written in Rust and has an extensive Filespooler Reference as well as many tutorials on its homepage. To give you a few examples, here are some links:

• Using Filespooler over Syncthing (and the most comprehensive tutorial)
• Using Filespooler over NNCP
• Compressing Filespooler Jobs
• Encrypting Filespooler Jobs with GPG or Age
• Guidelines for Writing To Filespooler Queues Without Using Filespooler

Filespooler is intentionally simple:

• The sender maintains a sequence file that includes a number for the next job packet to be created.
• The receiver also maintains a sequence file that includes a number for the next job to be processed.
• fspl prepare creates a Filespooler job packet and emits it to stdout. It includes a small header (