#analyticsengineering — Public Fediverse posts

Coalesce 2024 keynote announcements included among others:
* support for Apache Iceberg
* a DBT Copilot
* integrations with Power BI
* improved data quality and data freshness tools
* More visual editing support
* in general the vibe of making it easier to reference data across teams

#Coalesce #DBT #AnalyticsEngineering #Coalesce2024 #Data

https://coalesce.getdbt.com/

Just been doing a LinkedIn Learning course on dbt - a tool used in Analytics Engineering. Delighted that they were using DuckDB as the backend.
This kind of use is so huge for learners. Instead of faffing around to get credentials/access to a remote DB server just run locally for free and quickly get on with the learning of the app.

#DuckDB #DBT #AnalyticsEngineering #DB
https://www.linkedin.com/learning/data-engineering-with-dbt?trk=share_android_course_learning&shareId=%2Fejw28EGQSWFfcwj6qTIUg%3D%3D

Do you use a makefile with your dbt project?

I’m sure loads are already doing this, but I couldn’t find a lot of info online so wrote this short guide. (Boilerplate included)

Coming from a web dev background using npm/yarn, this clicked with me really quickly.

I’d love to know if others are doing this and if you have any makefile pro-tips to share??

https://medium.com/inthepipeline/how-to-use-a-makefile-to-speed-up-your-dbt-project-workflow-fb36eb676910

#DataEngineering #AnalyticsEngineering #data #CLI

How do you handle creating testing envs for your dbt PRs?

I wrote up a workflow for a more complex setup that would be suitable for projects with frequent ingestions and open PRs, by creating a static/immutable PR-specific environment to use as a base to compare dev to.

I'd love any feedback, or please share how you're doing it on your more complex projects:

https://medium.com/inthepipeline/youre-running-dbt-in-ci-now-what-f24c6717b9de

#DataEngineering #dbt #AnalyticsEngineering #data #BestPractices #DataScience

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

I hosted the PipeRider Community Office Hours yesterday and one cool feature we demoed was this online dbt manifest viewer.

Generate a Lineage Diff from just your dbt project's manifest files.

Try it out for yourself here:

https://staging.cloud.piperider.io/online-viewer#/?utm_source=mstdn

Watch it in action on Youtube:

https://www.youtube.com/watch?v=LHuTb3e_4O8

#DataEngineering #DataViz #dbt #DataTools #AnalyticsEngineering #OpenSource #PipeRider

Looking for interesting datasets to practice with?
(and full end-to-end data pipelines to investigate)

Here are 5 (and a few extras) from the recent DataTalksClub Data Engineering Zoomcamp:

https://medium.com/inthepipeline/5-interesting-datasets-from-the-data-engineering-zoomcamp-f0d6dd4d5e15

Datasets Include:

- Daily data on Berlin bike thefts
- Air quality
- SF Eviction data
- Minneapolis 311
- NYC Restaurant Inspection

#DataEngineering #DataOps #Datasets #DataProjects #AnalyticsEngineering #DataPipelines #dbt #elt #DataViz

dbt users - Do you consider data warehouse costs when building and maintaining projects?

If so, what steps you taking to keep costs down, or keep an eye on costs?

#dbt #DataQuality #DataBuildTool #DataEngineering #AnalyticsEngineering #DataOps

A new blog, in which I explore #dbt, #FiveTran, #Airbyte, and try to generally make sense of the ELT landscape: https://rmoff.net/2022/11/08/data-engineering-in-2022-elt-tools/

#dataengineering #data #analyticsengineering