#analyticsengineering — Public Fediverse posts

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

Are you using dbt?

I'm looking to learn more about the scale of dbt projects and the review process for modeling changes.

Can you answer 9 questions about your dbt usage?
(they're all multiple choice - yay!)

https://docs.google.com/forms/d/1O3korvArsxViiZbfatSw0RS9ocG5eJ4dh2D49zr5YSM

#dbt #DataEngineer #DataEngineering #AnalyticsEngineering #DataModeling #DataOps

dbt pipelines:

#dbt #DataEngineering #AnalyticsEngineering #Data

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

🔎 🗺️ The State of Data Engineering 2023 - a nice round up and review from Einat Orr of #dataengineering in 2023
https://lakefs.io/blog/the-state-of-data-engineering-2023/?utm_campaign=Social%20media%20activity&utm_source=mastodon&utm_medium=social&utm_content=blog_rm-SoDE23_230524
#datadon #analyticsengineering #opensource #data

Do you have a proper code review process for pushing changes to data projects?

With ELT and dbt, small changes to code can have a big impact on downstream models. The last thing you want is to push a breaking change to prod 😱

Why you need code review for you data projects
by Balu Rama Chandra

https://medium.com/inthepipeline/why-you-need-code-review-for-you-data-projects-95e039083df0

#DataEngineering #DataQuality #DataOps #CodeReview #dbt #AnalyticsEngineering

Looking for interesting datasets to practice with?
(and full end-to-end data pipelines to investigate)

Here are 5 (and a few extras) from the recent DataTalksClub Data Engineering Zoomcamp:

https://medium.com/inthepipeline/5-interesting-datasets-from-the-data-engineering-zoomcamp-f0d6dd4d5e15

Datasets Include:

- Daily data on Berlin bike thefts
- Air quality
- SF Eviction data
- Minneapolis 311
- NYC Restaurant Inspection

#DataEngineering #DataOps #Datasets #DataProjects #AnalyticsEngineering #DataPipelines #dbt #elt #DataViz

Knowing the data warehouse cost for building a dbt project has really been bugging us recently, so a colleague made a proof of concept to see the cost after running dbt.

It records the bytes billed from BigQuery to a file after dbt build finishes, and then calculates the cost based on the current cost of BigQuery ($5 per TB).

I’ve written a blog post about it:

https://medium.com/inthepipeline/how-to-track-your-dbt-projects-bigquery-data-warehouse-costs-in-dbt-a-poc-716068b1e2c3

Is this something folk are interested in?

#DataOps #BigQuery #DBT #DataEngineering #AnalyticsEngineering

dbt users - Do you consider data warehouse costs when building and maintaining projects?

If so, what steps you taking to keep costs down, or keep an eye on costs?

#dbt #DataQuality #DataBuildTool #DataEngineering #AnalyticsEngineering #DataOps

The essential component of any data toolkit: Duck[DB] Tape!

#datadon #duckdb #dataengineering #analyticsengineering

Enjoying the back catalogue of The Analytics Engineering #Podcast with @jthandy and @schottj - this episode from August 2021 with Erik Bernhardsson was really interesting.

🎧 https://overcast.fm/+w94WXfYA0
✍🏻 https://roundup.getdbt.com/p/erik-bernhardsson

#datadon #dataEngineering #analyticsEngineering

For those who might find me by hashtag (get ready for hashtag soup): I do lots of #dbt on #snowflake #data #dataengineering and #analytics #analyticsengineering. I enjoy #python and #pandas and #sql and #dax. I’ve been lucky to work on #azure and #aws, and I prefer #powerbi as my viz Swiss Army knife.

Worst part of being laid off "effective immediately" by far is the toll wrought by a dim model I hadn't quite cracked the code for yet being left completely unfinished.

#DataDon #AnalyticsEngineering #DataModeling

Hey #datadon, what are the main conferences that someone interested in#DataEngineering or #AnalyticsEngineering should be planning to attend next year? I'm assuming #dbt's #CoalesceConf, but what else?

A new blog, in which I explore #dbt, #FiveTran, #Airbyte, and try to generally make sense of the ELT landscape: https://rmoff.net/2022/11/08/data-engineering-in-2022-elt-tools/

#dataengineering #data #analyticsengineering