#datatools — Public Fediverse posts

🎉 Milestone Unlocked: Finished the Data Engineering Zoomcamp!

In 10 weeks, I moved from scripting to architecting systems. We built real production-grade infrastructure using Spark, Kafka, Airflow, and Kestra—not just hobby projects.

Capstone: A Storage Hard Drive Dashboard using real failure data from Backblaze
Stack: Terraform + Docker infra, Airflow orchestration, dbt modeling, Streamlit viz.

Key Lessons:
✅️ "It works on my laptop" isn't a strategy.
✅ Need IaC, partitioning, clustering, and strict error handling.
✅ dbt ensures reproducible, tested models.
✅ Infra is invisible work—if it breaks, your code fails.

Take the leap! It’s challenging but by week 10, pieces click into place. Seeing my pipeline run autonomously felt like crossing the finish line. 🏁

Thanks Data Talks Club team! On to the next challenge!

My project: https://github.com/ammartin8/hard_drive_analytics_dashboard

#mastodon #fediverse #data #spark #dataengineering #ai #technology #datatools #datapipelines #fedihire #thursday #sql #observability #etl #python #github

🎉 Milestone Unlocked: Finished the Data Engineering Zoomcamp!

In 10 weeks, I moved from scripting to architecting systems. We built real production-grade infrastructure using Spark, Kafka, Airflow, and Kestra—not just hobby projects.

Capstone: A Storage Hard Drive Dashboard using real failure data from Backblaze
Stack: Terraform + Docker infra, Airflow orchestration, dbt modeling, Streamlit viz.

Key Lessons:
✅️ "It works on my laptop" isn't a strategy.
✅ Need IaC, partitioning, clustering, and strict error handling.
✅ dbt ensures reproducible, tested models.
✅ Infra is invisible work—if it breaks, your code fails.

Take the leap! It’s challenging but by week 10, pieces click into place. Seeing my pipeline run autonomously felt like crossing the finish line. 🏁

Thanks Data Talks Club team! On to the next challenge!

My project: https://github.com/ammartin8/hard_drive_analytics_dashboard

#mastodon #fediverse #data #spark #dataengineering #ai #technology #datatools #datapipelines #fedihire #thursday #sql #observability #etl #python #github

🎉 Milestone Unlocked: Finished the Data Engineering Zoomcamp!

In 10 weeks, I moved from scripting to architecting systems. We built real production-grade infrastructure using Spark, Kafka, Airflow, and Kestra—not just hobby projects.

Capstone: A Storage Hard Drive Dashboard using real failure data from Backblaze
Stack: Terraform + Docker infra, Airflow orchestration, dbt modeling, Streamlit viz.

Key Lessons:
✅️ "It works on my laptop" isn't a strategy.
✅ Need IaC, partitioning, clustering, and strict error handling.
✅ dbt ensures reproducible, tested models.
✅ Infra is invisible work—if it breaks, your code fails.

Take the leap! It’s challenging but by week 10, pieces click into place. Seeing my pipeline run autonomously felt like crossing the finish line. 🏁

Thanks Data Talks Club team! On to the next challenge!

My project: https://github.com/ammartin8/hard_drive_analytics_dashboard

#mastodon #fediverse #data #spark #dataengineering #ai #technology #datatools #datapipelines #fedihire #thursday #sql #observability #etl #python #github

🎉 Milestone Unlocked: Finished the Data Engineering Zoomcamp!

In 10 weeks, I moved from scripting to architecting systems. We built real production-grade infrastructure using Spark, Kafka, Airflow, and Kestra—not just hobby projects.

Capstone: A Storage Hard Drive Dashboard using real failure data from Backblaze
Stack: Terraform + Docker infra, Airflow orchestration, dbt modeling, Streamlit viz.

Key Lessons:
✅️ "It works on my laptop" isn't a strategy.
✅ Need IaC, partitioning, clustering, and strict error handling.
✅ dbt ensures reproducible, tested models.
✅ Infra is invisible work—if it breaks, your code fails.

Take the leap! It’s challenging but by week 10, pieces click into place. Seeing my pipeline run autonomously felt like crossing the finish line. 🏁

Thanks Data Talks Club team! On to the next challenge!

My project: https://github.com/ammartin8/hard_drive_analytics_dashboard

#mastodon #fediverse #data #spark #dataengineering #ai #technology #datatools #datapipelines #fedihire #thursday #sql #observability #etl #python #github

🎉 Milestone Unlocked: Finished the Data Engineering Zoomcamp!

In 10 weeks, I moved from scripting to architecting systems. We built real production-grade infrastructure using Spark, Kafka, Airflow, and Kestra—not just hobby projects.

Capstone: A Storage Hard Drive Dashboard using real failure data from Backblaze
Stack: Terraform + Docker infra, Airflow orchestration, dbt modeling, Streamlit viz.

Key Lessons:
✅️ "It works on my laptop" isn't a strategy.
✅ Need IaC, partitioning, clustering, and strict error handling.
✅ dbt ensures reproducible, tested models.
✅ Infra is invisible work—if it breaks, your code fails.

Take the leap! It’s challenging but by week 10, pieces click into place. Seeing my pipeline run autonomously felt like crossing the finish line. 🏁

Thanks Data Talks Club team! On to the next challenge!

My project: https://github.com/ammartin8/hard_drive_analytics_dashboard

#mastodon #fediverse #data #spark #dataengineering #ai #technology #datatools #datapipelines #fedihire #thursday #sql #observability #etl #python #github

Diving deep into Spark batch processing!⚡️

Learned how to:
✅ Optimize data pipelines with filtering, repartitioning & grouping
✅ Design efficient ETL pipelines with Spark
✅ Understanding when and how to use partitioning strategies
✅ Use Google Cloud Storage (GCS) as a data source for Spark applications and configuring Spark to read Parquet or other formats from GCS
✅ Visualize execution plans for efficient coding
✅ Review the Spark UI for performance monitoring

💡 Key takeaway: One thing that amazes me about distributed computing is how we've transformed from struggling with massive datasets to generating insights in near real-time. As an analyst who has dealt with long wait times in processing data, spark saves so much time in getting results faster and make data-driven decisions more quickly.

Review my work here: https://github.com/ammartin8/data_engineering_zoom_camp/blob/main/modules/module_6/project_06/README.md

#mastodon #fediverse #data #spark #dataengineering #ai #technology #opensource #datatools #datapipelines #fedihire #wednesday #sql #observability #etl #python

Just completed a project building an end-to-end data pipeline for NYC taxi data using dlt 🚕📊! What a ride! 😅 The REST API extraction was particularly fun (in a challenging way) but dlt's modular design made it manageable. Here’s what I learned:

✅ Full life cycle: From REST API extraction to DuckDB loading, all in one framework
✅ Reproducibility: Tracked every transformation with dlt's lineage features
✅ Modular design: Defined reusable components for extracting and normalizing data
✅ Handles complexity: Seamlessly handled pagination from the API
Big takeaway: dlt isn't just tooling, it's a framework for thinking about data pipelines that emphasizes transparency and reproducibility which is essential for any modern data stack

#dlt #dataengineering #datapiplines #etl #fediverse #mastodon #opensource #oss #ai #data #linux #technology #duckdb #datatools

ICYMI: 3 days left: Amazon is locking the door on the data tools sellers depended on: With Amazon's BSA Agent Policy taking effect in 3 days, an industry expert argues the review cap and new rules are closing the third-party AI data pipeline for good. https://ppc.land/3-days-left-amazon-is-locking-the-door-on-the-data-tools-sellers-depended-on/ #Amazon #DataTools #BSAAgentPolicy #ThirdParty #AIData

ICYMI: 3 days left: Amazon is locking the door on the data tools sellers depended on: With Amazon's BSA Agent Policy taking effect in 3 days, an industry expert argues the review cap and new rules are closing the third-party AI data pipeline for good. https://ppc.land/3-days-left-amazon-is-locking-the-door-on-the-data-tools-sellers-depended-on/ #Amazon #DataTools #BSAAgentPolicy #ThirdParty #AIData

ICYMI: 3 days left: Amazon is locking the door on the data tools sellers depended on: With Amazon's BSA Agent Policy taking effect in 3 days, an industry expert argues the review cap and new rules are closing the third-party AI data pipeline for good. https://ppc.land/3-days-left-amazon-is-locking-the-door-on-the-data-tools-sellers-depended-on/ #Amazon #DataTools #BSAAgentPolicy #ThirdParty #AIData

Discover RQLBrowser from Logilab — a sleek way to explore and test RQL queries! Great demo for devs, data folks, and curious tinkerers. Check out the UI, examples, and tips to speed up your data workflow. #RQL #RQLBrowser #Logilab #DataTools #DevTools #FOSS #PeerTube #English
https://peertube.logilab.fr/videos/watch/7b487426-83a9-435f-b23c-cbdb93dce788

Building a Simple Tech Stack: Avoid Costly Mistakes and Boost Efficiency! #data #datatools

How to should you choose be best data tools to make an efficient stack? Keep it simple! source

https://quadexcel.com/wp/building-a-simple-tech-stack-avoid-costly-mistakes-and-boost-efficiency-data-datatools/

Một công cụ phân tích dữ liệu thời gian miễn phí mới đã ra mắt! **Prophetize** cho phép dự báo nhanh bằng thuật toán Holt-Winters trực tiếp trên trình duyệt, không cần tải lên đám mây. Hỗ trợ phát hiện mùa vụ tự động, xử lý ngày tháng phức tạp và xuất kết quả sang Excel. Dễ dàng, không cần đăng nhập. #Technology #TimeSeries #DataTools #PhầnMềmMiễnPhi #PhânTíchDữLiệu

https://www.reddit.com/r/SideProject/comments/1pu0v1d/i_made_a_browserbased_alternative_to_excel_for/

Parqeye – A CLI tool to visualize and inspect Parquet files

https://github.com/kaushiksrini/parqeye

#HackerNews #Parqeye #Parquet #CLI #DataVisualization #DataTools #OpenSource

🪿 When AI gets confused by geese - call DotDotGoose!

It’s a super cute open-source tool for counting things on images by hand.
Just click around and mark birds, bugs, butterflies - whatever you need 🐥🦋

Made by the American Museum of Natural History for real science -
but honestly, it’s so cozy I just wanna sit and count little geese all day 🤍

🔗 https://github.com/persts/DotDotGoose

#GooseGang #Python #OpenSource #DevGirl #DataTools

Need better insights from your data?
Talk to @eazyBI — they're at #BalticRuby in red shirts and ready to help! 📊
Thanks to our Bronze Sponsor for supporting the Ruby community with tools and good vibes! https://eazybi.com/

#eazyBI #RubyOnRails #RubyConference #DataTools

SQL Workbench – Republicans not welcome

https://www.sql-workbench.eu/

#HackerNews #SQLWorkbench #RepublicansNotWelcome #TechNews #SQLCommunity #DataTools

SQL Workbench – Republicans not welcome

https://www.sql-workbench.eu/

#HackerNews #SQLWorkbench #RepublicansNotWelcome #TechNews #SQLCommunity #DataTools

SQL Workbench – Republicans not welcome

https://www.sql-workbench.eu/

#HackerNews #SQLWorkbench #RepublicansNotWelcome #TechNews #SQLCommunity #DataTools

SQL Workbench – Republicans not welcome

https://www.sql-workbench.eu/

#HackerNews #SQLWorkbench #RepublicansNotWelcome #TechNews #SQLCommunity #DataTools

Democratizing AI access is our mission! That's why we've integrated Ollama with our (Un)Perplexed Spready software. See our comparison of which model—Mistral 7B or OLMo2 7B—might work best for your use case. https://matasoft.hr/qtrendcontrol/index.php/un-perplexed-spready/un-perplexed-spready-various-articles/146-comparative-analysis-of-mistral-7b-and-olmo2-7b-on-the-ollama-platform #AIForAll #Ollama #DataTools

I am so happy with the first own web application 🎉 I have developed: Tris, a simple and free web crawler 🕸️ 🕷️ !

You can try it for free online: https://tris.fly.dev, limited to 3 parallel crawls and 100 links of path depth of 3.

Next thing I will add will be a text input to set a target domain hhh, now I am making it hard! 🙈

#node #nodejs #web #webcrawler #crawler #seo #datatools #webscraper #scraping #seotools #seotool #tris #triswebcrawler #webapp #indie #indiedev

Our #DataTableRenderers 🈸 / 🈷️ for #VSCode Notebooks 📚 recently crossed 67K installs and 100K downloads. Thanks to all the devs and data scientists using them to view tabular data outputs and data summaries in Jupyter notebooks! 🤗

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-table

Sponsor our #DataTools 🛠️ work on GitHub, and sign up for our Pro sponsor tier to get notified about new #DataNotebook 📓 Pro Tools coming to VS Code IDE soon:

✍️ https://github.com/RandomFractals/pro-data-tools#data-notebook-pro-tools

#DataTable 中 / #DataNotebooks 📚 / #ProDataTools 🧙 ...

Running #SQL queries in our new #DataNotebook 📓 @code extension, rendering results with simple #DataTable, #DataSummary 🈷️ & #FlatDataGrid 中 from our #DataTableRenderers 🈸 + CSV #DataExport from VSCode Notebook cell output all in one go. We doubt #MalloyData is as flexible. 😎 #DataTools 🔬 ...

Our #DataTableRenderers 🈸 for #VSCode Notebooks 📚 has over 30,000 installs. It's one of the most widely used #dataNotebook 📓 extensions in VS marketplace. Extension includes scrollable #dataTable, #flatDataGrid & #dataSummary output renderers. Try it!
📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-table

#dataTools 🛠️ 💎💎💎...

🐤 Wow, just what we needed—a "lightweight" data framework that’s built on top of #DuckDB 🦆 and #3FS 🔧, because clearly, our data didn’t have enough buzzword sauces on it already. 🍛 But hey, at least now we have another tool to sit idly in our #GitHub repos, collecting digital dust while we pretend it’s the magic bullet for all data woes. 🎯✨
https://github.com/deepseek-ai/smallpond #lightweightdata #datatools #HackerNews #ngated

📣 Published new free and public #DuckDB #SQLTools Preview v1.6.0 for VS Code IDE with DuckDB v0.10.2 support.

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools

See our #ProDataTools repo for advanced #DuckDBPro schema, commands, and new features updates:

📰 https://github.com/RandomFractals/pro-data-tools/blob/main/duckdb-tools.md#pro-data-tools

#VSCode #DataTools 🛠️ 🧙‍♂️ ...

Announcing SSMS 20.2 ... and getting feedback for SSMS 21.

#sqlserver #ssms #database #data #datatools #azuresql
https://techcommunity.microsoft.com/t5/sql-server-blog/announcing-ssms-20-2-and-getting-feedback-for-ssms-21/ba-p/4187034

Our #DataPreview 🈸 extension recently crossed 500,000 installs in Visual Studio Code Marketplace.

We plan to release Data Preview Pro version with large data files support, new Prospective Data Viewer and better support of Arrow and Parquet data files next year for our Pro Sponsors on GitHub.

Meanwhile, try our older open source version from GitHub or VS Code Marketplace:

📰 https://github.com/RandomFractals/vscode-data-preview

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-preview

#VSCode #DataTools 🛠️ / #ProDataTools 🧙 ...

Started working on new #PGSqlTools plugin this month.

Similar to our #DuckDBPro and #SQLitePro #DataTools 🛠️ this extension will display all #Postgres DB objects in Connections Explorer, provide information schema, system catalogs and other standard PG view shortcut commands for Postgres, Cockroach and many other #pgwire compatible databases.

Aiming to support most of the popular PostgreSQL flavor databases from this list: https://wiki.postgresql.org/wiki/PostgreSQL_derived_databases

#VSCode #PostgreSQL #SQLTools #ProDataTools 🧙

Proposed this for new #SQLTools feature in #VSCode today:

#SQL / #DataTools 🛠️ ...

https://github.com/mtxr/vscode-sqltools/issues/1239

📢 We just released #DuckDBSqlTools v1.3.0 for #VSCode IDE with the latest #DuckDB v0.9.1 support.

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools

#SQL / #DataTools 🛠️

#DuckDB devs & users, we just hit 5K installs & over 7.6K downloads of our free & public #DuckDBSQLTools extension for #VSCode IDE.

DuckDB Labs & co. should just buy that extension to get more traction in the #DataTools field with #SQLTools devs, data scientists, data analysts, and other users using & exploring their DB in Code 😉

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools

#DuckDBTools 🛠️ #ProDataTools 🧙‍♀️ ...

Next up in #DataTools for #VSCode IDE ... #QuestDB? TBD 🤔

If you are still looking for some good simple #DataViewers to use in #VSCode IDE, try our #RandomFractalsInc viewers and #DataTools 🛠️ from the top row of tiles and extensions in VS Code marketplace search results for #DataViewer:

📰 https://marketplace.visualstudio.com/search?term=data%20viewer&target=VSCode&category=All%20categories&sortBy=Relevance

Use them for local and remote data preview 🈸, maps 🗺️, graphs 📈📊, tabular data display of flat data files and dataframes in Jupyter notebooks 📚.

#ProDataTools 🧙‍♂️ ...

For all the brave devs & #devCrews into building nifty #dataTools 🛠️ out there 😉 ...

Our #RandomFractalsInc works on @github circa October 2023 🤗

💖 https://github.com/RandomFractals

#RandomFractals #dataTools 🧙‍♂️ ...

We also wonder why 🤫. Maybe @OpenAtMicrosoft should open source their #DataWrangler if they want to compete with our dated OSS #dataTools 🛠️ in #VSCode and other 3rd party extensions that outrank them 🤗

🧐 https://github.com/microsoft/vscode-data-wrangler/issues/92

Gonna try to wrap up and release #DataNotebookPro 📓 #VSCode / #SQLTools extension this month for our #ProDataTools 🛠️ sponsors on GitHub ...

📰 https://github.com/RandomFractals/pro-data-tools#data-notebook-pro-tools

#DataNotebooks 📚 / #DataTools 🧙‍♂️ ...

A few remaining #ObservableJS 📓 to #Quarto doc conversion & docs tickets in our old Observable #DataTools 🛠️ repo to guide data analysts and data scientists exploring that space and tools ecosystem on how to use them in #VSCode IDE ...

📰 https://github.com/RandomFractals/observable-data-tools/issues

#DataNotebooks 📚

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

Users of #VSCodium IDE can also use our new #DuckDBSqlTools extension.

1⃣ Download duckdb-sql-tools-x.x.x.vsix 📦 from our extension releases on github:

📥 https://github.com/RandomFractals/duckdb-sql-tools/releases

2⃣ Use Install from #VSIX feature to install #DuckDB #SqlTools 📦👇

▶️ https://twitter.com/TarasNovak/status/1620073278743388160

#SQL 📜 #DataTools 🧙‍♂️ 🔬 💎 ...

Quick example of running #SQL query in our new #DataNotebook 📓 code extension we plan to release to our Pro sponsors on github soon ...

Query results are displayed using our public #DataTableRenderers 🈸 extension for VSCode IDE:

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.vscode-data-table

#DataNotebooks 📚 ... #DataTools 🔬 ...

Took me a few hours to whip my new #DataNotebook 📓 extension base code into shape today. Got #PRQL loaded into vscode notebook view using Open With option. Will wire SQL gen., query & results/output data rendering with our #DataTableRenderers this week for sure! #DataTools 🔬 ...

Gonna create new #DuckDBSqlNotebook 📓 code extension next month.

Who'd like to see it working in code notebooks by loading and executing Sql statements from plain Sql files?

DuckDB Sql Notebook extension will have new custom DuckDB connection explorer, use vscode notebook user interface, and integrate with our #DataTableRenderers for scrollable query results display.

#DuckDB #DataTools / #DataNotebooks 📚 ...

Taking down fully staffed @Google #dataTools in @code team in 2 weeks of coding & 4+ months in marketplace ... @lloydtabb & your #MalloyData crew been pwned in installs in @code:

📥 https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.duckdb-sql-tools

#ProDataTools 🧙‍♂️ ... 💎💎💎 ... 😎

P.S.: add #duckdb keyword to yours..

Talk about #DataTools 🪄 & #DuckDB in @code FTW: #MalloyData has been in dev w/fully staffed dev team since January 2022 & we put together our free trial #DuckDBSqlTools in a month in January 2023. One in VS marketplace for over a year & a half, the other only 4 months. 💎💎💎

After eval of #MalloyData tools, I am going back to #PRQL. Their repo, tools, docs & api look more mature to me, and should enable adding SQL queries to my data viewers next year.

PRQL playground: https://prql-lang.org/playground/

VSCode extension you can try: https://marketplace.visualstudio.com/items?itemName=prql-lang.prql-vscode

#dataTools 🛠️ ...

Explore more at: https://github.com/prql/prql#explore

Please tag your #dataExplores with Malloy using #MalloyData tag on #github. I'd like to see what other devs & data scientists create with that new #DAX language.

See ours here: https://github.com/topics/malloydata

+ https://github.com/RandomFractals/chicago-crimes#with-malloy-composer

+ https://github.com/RandomFractals/chicago-crimes#with-malloy-fiddle

#dataTools 🛠️ 2022 ...

📢 New 2022 #ChicagoCrimes #MalloyData #Fiddle app you can run in a browser to learn Malloy model and query semantics, nesting, and simple charting 📊📈 options:

📰 https://randomfractals.github.io/chicago-crimes/apps/malloy-fiddle

📓 https://github.com/RandomFractals/chicago-crimes#with-malloy-fiddle

🧐 https://github.com/RandomFractals/chicago-crimes/blob/main/docs/images/chicago-crime-reports-malloy-fiddle-app.gif

#DAX #dataApps #dataTools 🔬 ...

Updated #ChicagoCrimes with #MalloyData tools example & more info in docs:

🧐 https://github.com/RandomFractals/chicago-crimes#with-malloy-data

Clone that data repo & install Malloy #vscode extension to try it out:

📥 https://marketplace.visualstudio.com/items?itemName=malloydata.malloy-vscode

#dataTools 🔬 ...