#clusterless — Public Fediverse posts

updated #clusterless subpop cli build to provide a Homebrew tap for easy installation.

https://github.com/ClusterlessHQ/subpop

subpop is an experimental tool for diffing datasets from the cli.

runs on #Linux and #macOS but sadly written in #java so no native binaries just yet.

while pondering my need for a remote compute environment, vs having random boxes littered about generating heat, I realized I could add a 'device' component concept to #Clusterless.

this concept not only compliments the current model types, it will be handy standalone.

consider an #AWS ec2/ecs instance doing some complex work and dropping files into S3 (over the new mount point feature) where a clusterless DAG takes over processing when the files arrive (via the S3 put boundary).

On that note, not using #clusterless for #dataengineering is a trap

https://writing.exchange/@ryansingel/111702630366999134

https://github.com/ClusterlessHQ

@seldo I don't know any firsthand,

but I spent the last couple weeks exploring what a #RAG pipeline would look like so I could write a sample application/pipeline using my #OpenSource #clusterless project

https://github.com/ClusterlessHQ

unfortunately the idea I had wasn't ultimately suitable for RAG and could be a simple BERT/BART summarizer pipeline without having a open/elasticsearch backend or other vector db.

still looking for a fun RAG based prototype I could build and share.

need to dig into this, but i've been doing replay (redrive) on #aws StepFunctions for years with my #data pipelines

https://aws.amazon.com/blogs/big-data/build-efficient-etl-pipelines-with-aws-step-functions-distributed-map-and-redrive-feature/

replay is one feature I haven't added back to #Clusterless yet, though all the metadata is there.

https://github.com/ClusterlessHQ

@c_chep

currently all my #clusterless examples (and scenario tester) use jsonnet, but it's got weak overall support.

CUE looks interesting, but no Java implementation for embedding (if that was a thing I was considering)

Tessellate is now on Docker Hub

https://hub.docker.com/r/clusterless/tessellate

Tessellate is a command line tool for reading and writing #data to/from multiple locations and across multiple formats.

#clusterless

Automating #AWS CloudWatch log export into S3 is no simple task.

Next #clusterless release will now have a new Component type called Activity that is simply a scheduled task..

The first Activity will be function that exports cloud watch logs created within the previous interval.

As they arrive, any arc can subscribe to the data drop and do things. To simplify that task, I'll update #tessellate

The cw log is a delimited text file with two columns, one is json. unlike all the others in aws!

ok, here's a new one for #aws users.

would anyone be interested in an automated way to extract CloudWatch logs (continuously) into an s3 bucket.

and have them converted into #parquet (/etc) for downstream custom processing. or simply partitioned with partition updates to AWS Athena/Glue?

the challenge for users is getting the `detail` json field exposed since it's app specific.

with #clusterless devs could then inject custom processing for custom app logs into the #data pipeline

I finally wrote up some documentation on using #clusterless Tessellate for #data stuff

https://docs.clusterless.io/tessellate/1.0-wip/index.html

I'm hosting my #clusterless docs on github, but #google is refusing to index the site fully.

it shows "currently not indexed" in the search console. which in turn claims they don't want to overload the site.

I get the sense that this is a shared problem for anyone hosting on github..

any suggestions on alternatives or hacks?

Released a new wip of #clusterless, for #cloud #data pipelines, last night that includes reporting on both arcs (workloads) and datasets.

https://github.com/ClusterlessHQ/clusterless

Below is a summary of the three datasets the #AWS s3 log sample app creates.

https://github.com/ClusterlessHQ/clusterless-aws-examples

Note we track the difference between intervals that have no data (empty, which may be intentional) vs a gap (the workload didn't run and create data).

getting closer...

here is a screenshot of the #clusterless cls command printing a summary table of workload (arc) completions since yesterday

I need to release 2.0 of the #java library mini-parsers into maven central before I can push this out and begin work on dataset status (think fsck for workload results)

hoping to make time to get another #clusterless release out this week.

I have commands to list deployed placements (regions etc), projects, and arcs (workloads). still need to get deployed datasets.

and, status reporting of both arcs and datasets.

that is, completed and failed arcs. and dataset completions, partials, empties, and gaps.

if a gap is found, the arc was skipped or failed, here is where you can re-run workloads deterministically. from the cli.

I'm thinking of resurrecting some code I have for Splunk like relative time adjusters

https://docs.splunk.com/Documentation/Splunk/9.1.1/Search/Specifytimemodifiersinyoursearch#Specify_relative_time_ranges

the #java library mini-parsers is due for an update, modern parboiled supports jdk17 now.

https://github.com/Heretical/mini-parsers

and #clusterless status reporting needs time range support on the cli and the splunk syntax is fairly concise.

anyone else interested in parser support?

Updated the #clusterless install docs to reflect the homebrew install option

https://docs.clusterless.io/guide/1.0-wip/install-quickstart.html

ok, finally! the #clusterless and #tessellate wip builds are published to #homebrew

https://github.com/ClusterlessHQ/homebrew-tap

I'll update all the install docs this week.

here is a little pre-announcement of a new #clusterless library clusterloss-commons

https://github.com/ClusterlessHQ/clusterless-commons

currently available in maven central.. but still under documented etc.

this project allows for sharing of some core libraries I find useful developing clusterless and tessellate. as well as some basics to help with #aws cdk development.

i'll make a bigger announcement as it matures.

probably time i sort out a real logo for #clusterless

https://github.com/ClusterlessHQ

is 99designs still a thing?

I've added a new how-to guide on creating a copy #data pipeline in #AWS s3 using only #clusterless intrinsic components. As files get uploaded, they get copied to a new location.

https://docs.clusterless.io/guide/1.0-wip/howtos/s3-copy-files.html

This roughly mirrors the example project, but has a bunch more explainers and examples on using the cls command to build a project file.

https://github.com/ClusterlessHQ

Long weekend of yard work ahead but look forward to completing a set of improved #clusterless documentation early next week.

Using jackson json views I can print out json for required properties and full json so configuring a simple pipeline doesn’t seem so daunting.

These in turn can be embedded directly in the docs online and help messages.

I added include/exclude filters to the S3PutListenerBoundary and S3CopyArc components to #clusterless

Now you can use ant like paths to exclude hidden files etc, in #AWS s3 buckets, like _SUCCESS with an exclude on **/_*

https://docs.clusterless.io/reference/1.0-wip/components/aws-core-s3-put-listener-boundary.html

https://docs.clusterless.io/reference/1.0-wip/components/aws-core-s3-copy-arc.html

How's this for a #clusterless tag line?

Think #cloud + #airflow without the airflow, but with a lot more trust and agility.

I've started publishing how-tos on using #clusterless to manage #cloud #data pipelines.

A little overkill, but the first is how to manage an #aws s3 bucket.

https://docs.clusterless.io/guide/1.0-wip/howtos/s3-bucket-resource.html

Having a scenario runner for #data flows in #clusterless is pretty cool for automated testing of dags of workloads.

especially if they are part of your ci/cd

https://github.com/ClusterlessHQ/clusterless/actions/runs/6319702471/job/17161069843

on every commit, a suite of scenarios are deployed, run, and destroyed against #aws

https://github.com/ClusterlessHQ/clusterless/tree/wip-1.0/clusterless-scenario/src/main/cls/scenarios

would be great to have time (be paid) to create a sample #clusterless app on #AWS to put data behind a #DuckDB WASM frontend

https://duckdb.org/2021/10/29/duckdb-wasm.html

So instead of Athena/Glue integration that can work against a complete corpus, have DuckDb over the last 30 days for investigations etc

https://github.com/ClusterlessHQ/aws-s3-log-pipeline

quick reminder, https://chris.wensel.net

I want to do this with #sqlite https://datasette.io as well (I did it in a previous role and it was awesome)

Just pushed a new #clusterless Tessellate release that updates the transform statement syntax to include intrinsic functions.

The first function is tsid, a unique long value generated by the https://github.com/f4b6a3/tsid-creator library.

More here: https://github.com/ClusterlessHQ/tessellate#intrinsic-functions

https://github.com/ClusterlessHQ/tessellate

@Cmastication depends on how you access it? If via a query, only partition on the most common predicates.

Repartitioning data for different access patterns is a key use case behind #clusterless and tessellate. See bio for links.

Otherwise yeah, partition via hash to get equal sized bits. Reminds me to add a hash transform to tessellate.

I'll make a bigger announcement later, but if you are following along with #clusterless development, note that we just added #AWS Glue/Athena support.

That means databases and tables can be deployed in tandem with a workload, and any new partitions that arrive will be added to the table.

https://github.com/ClusterlessHQ/clusterless

This example has been updated to show how it works and how simple it is (relatively).

https://github.com/ClusterlessHQ/aws-s3-log-pipeline

So imagine, every result dataset in the dag having a table to query.

just sayin', if you find chaining sql statements into a data processing dag a bit of a drag, I suggest you spend some time with #clusterless

https://github.com/ClusterlessHQ

declarative decentralized heterogeneous #data flows (in #aws today)