#kestra — Public Fediverse posts

Hi All! I started posting my data engineering learning journey and thought it would be great share here as well!

🚀 Week 3 of Data Engineering Zoomcamp by DataTalksClub complete! I'm really enjoying how hands on and practical the course is so far!

This week I focused on data warehousing with #Google #BigQuery. Coming from the world of #Microsoft Azure, it was a great experience to get familiar with BigQuery's serverless architecture and how it manages and processes big data at scale. Here's what I learned:

✅️ Created external tables from GCS bucket data sources (CSV/Parquet)
✅️ Use partitioning/clustering to save on cost & enhance speed of processing SQL queries
✅️ Used both #Docker & #Kestra to orchestrate the extraction, transfer, and loading 20+ million NYC taxi data at scale into a GCS bucket
✅️ Understand the advantages of columnar storage and query optimization

Check out my work here: https://github.com/ammartin8/data_engineering_zoom_camp/blob/main/modules/module_3/project_03/README.md

#googlecloud #dataengineering #microsoft #cloud #bigdata #dataanalytics #fedihire #linux #data

Stop fighting complex data pipelines. Kestra is an open-source, event-driven orchestrator that's infinitely scalable. 🧩 Define workflows with simple YAML or a no-code UI, then sync with Git. #opensource #kestra #orchestration #dataops #devops

We had a great @BeJUG event (full-house!) yesterday at Sweet Mustard in Kortrijk. Thanks, Loïc Mathieu and Julien Jakubowski 🥑 for the presentations about #Kestra and #ApachePulsar!

Next week @BeJUG in Kortrijk at #SweetMustard: "Lille invades Kortrijk: Kestra (Event Orchestrator) + Apache Pulsar (Messaging)" with @loicmathieu and Julien Jakubowski 🥑. There are still some places available. Register fast!

Loïc's Topic: #Kestra: an OpenSource #EventDriven Declarative #Orchestrator written in #Java

Julien's Topic: #ApachePulsar: Finally an Alternative to #kafka ?

https://www.meetup.com/belgian-java-user-group/events/298755642/

#java #jug #javausergroup