#dataops — Public Fediverse posts

AI tools give a flash of speed, but only a system of linked layers delivers lasting motion. When a persona, automated pipeline, and analytics loop converse without manual hand-off, the workflow becomes a self-sustaining gear train that tolerates API changes and scales with market growth. Build the infrastructure, not the hype. 🚀 #AIMarketing #Automation #SystemsDesign #DataOps - Powered by FG

Strong data catalogs transform the way teams think. They shift work from guesswork to clear action. What’s your take? #DataCatalogs #SingleSourceOfTruth #DataQuality #DataTrust #DataOps #DataStrategy #AIReadyData #DataFuture
https://www.linkedin.com/pulse/building-data-catalogs-quiet-power-behind-single-truth-mohindroo--icosc

Flink Kubernetes operator: опыт построения стриминговой Big Data платформы

Всем привет! Меня зовут Артемий, я работаю SRE-инженером в команде RTP (real time processing) Clickstream в Авито . Сегодня мы хотим поделиться нашей историей о том, как мы переехали в Kubernetes, развернув Apache Flink с помощью Flink k8s operator.

https://habr.com/ru/companies/avito/articles/972412/

#data #dwh #DataOps #bigdata #flink #sql #realtime #analytics #analytical_engine

Flink Kubernetes operator: опыт построения стриминговой Big Data платформы

Всем привет! Меня зовут Артемий, я работаю SRE-инженером в команде RTP (real time processing) Clickstream в Авито . Сегодня мы хотим поделиться нашей историей о том, как мы переехали в Kubernetes, развернув Apache Flink с помощью Flink k8s operator.

https://habr.com/ru/companies/avito/articles/972412/

#data #dwh #DataOps #bigdata #flink #sql #realtime #analytics #analytical_engine

Flink Kubernetes operator: опыт построения стриминговой Big Data платформы

Всем привет! Меня зовут Артемий, я работаю SRE-инженером в команде RTP (real time processing) Clickstream в Авито . Сегодня мы хотим поделиться нашей историей о том, как мы переехали в Kubernetes, развернув Apache Flink с помощью Flink k8s operator.

https://habr.com/ru/companies/avito/articles/972412/

#data #dwh #DataOps #bigdata #flink #sql #realtime #analytics #analytical_engine

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://www.linkedin.com/pulse/data-observability-next-frontier-quality-management-mohindroo--lpgtc

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://www.linkedin.com/pulse/data-observability-next-frontier-quality-management-mohindroo--lpgtc

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://www.linkedin.com/pulse/data-observability-next-frontier-quality-management-mohindroo--lpgtc

FlinkSQL в Авито: что внутри и как нам это помогает

Привет! Меня зовут Артемий, я инженер в команде Processing и веду фичатрек по развитию FlinkSQL в Авито . Сегодня я расскажу про инструмент, который мы развиваем в Avito — FlinkSQL. Он использует потоковые данные для аналитики в реальном времени: метрики, временные срезы, A/B-расчеты, поиск аномалий. Все это на привычном SQL, чтобы команда видела картину и могла действовать сразу.

https://habr.com/ru/companies/avito/articles/959382/

#Flink #kubernetes #bigdata #k8s #operator #kubernetes_operator #flink_kubernetes_operator #Apache_Flink #DataOps #dataops_platform

FlinkSQL в Авито: что внутри и как нам это помогает

Привет! Меня зовут Артемий, я инженер в команде Processing и веду фичатрек по развитию FlinkSQL в Авито . Сегодня я расскажу про инструмент, который мы развиваем в Avito — FlinkSQL. Он использует потоковые данные для аналитики в реальном времени: метрики, временные срезы, A/B-расчеты, поиск аномалий. Все это на привычном SQL, чтобы команда видела картину и могла действовать сразу.

https://habr.com/ru/companies/avito/articles/959382/

#Flink #kubernetes #bigdata #k8s #operator #kubernetes_operator #flink_kubernetes_operator #Apache_Flink #DataOps #dataops_platform

FlinkSQL в Авито: что внутри и как нам это помогает

Привет! Меня зовут Артемий, я инженер в команде Processing и веду фичатрек по развитию FlinkSQL в Авито . Сегодня я расскажу про инструмент, который мы развиваем в Avito — FlinkSQL. Он использует потоковые данные для аналитики в реальном времени: метрики, временные срезы, A/B-расчеты, поиск аномалий. Все это на привычном SQL, чтобы команда видела картину и могла действовать сразу.

https://habr.com/ru/companies/avito/articles/959382/

#Flink #kubernetes #bigdata #k8s #operator #kubernetes_operator #flink_kubernetes_operator #Apache_Flink #DataOps #dataops_platform

FlinkSQL в Авито: что внутри и как нам это помогает

Привет! Меня зовут Артемий, я инженер в команде Processing и веду фичатрек по развитию FlinkSQL в Авито . Сегодня я расскажу про инструмент, который мы развиваем в Avito — FlinkSQL. Он использует потоковые данные для аналитики в реальном времени: метрики, временные срезы, A/B-расчеты, поиск аномалий. Все это на привычном SQL, чтобы команда видела картину и могла действовать сразу.

https://habr.com/ru/companies/avito/articles/959382/

#Flink #kubernetes #bigdata #k8s #operator #kubernetes_operator #flink_kubernetes_operator #Apache_Flink #DataOps #dataops_platform

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://medium.com/@sanjay.mohindroo66/data-observability-the-next-frontier-in-data-quality-management-d5d75ea375c7

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://medium.com/@sanjay.mohindroo66/data-observability-the-next-frontier-in-data-quality-management-d5d75ea375c7

Stop fighting complex data pipelines. Kestra is an open-source, event-driven orchestrator that's infinitely scalable. 🧩 Define workflows with simple YAML or a no-code UI, then sync with Git. #opensource #kestra #orchestration #dataops #devops

UPDATE: Hey everyone, there was a problem with the Restream-to-LinkedIn connector, and as we were attempting to restart the stream, the event on LinkedIn disappeared. We're not sure what caused this issue, but we are re-scheduling this event. Please stay tuned for the new date and time!

Today on #TheDrillDown, Ahmad and I are honored to be joined by a very special guest, @Jkerski! We'll be drilling down into the topic of integrating #DataOps tools and principles into your #MicrosoftFabric and #PowerBI workflows. Join us at 2PM Central time! https://lnkd.in/gbegXeAc

DevOps в 2025 году: отдельные дисциплины, машинное обучение и прогноз на будущее

Привет! Меня зовут Константин Полуэктов, я solution architect в Yandex Cloud и технический эксперт в онлайн-магистратуре от Яндекса и ИТМО

https://habr.com/ru/companies/yandex_praktikum/articles/915730/

#devops #llm #ai #dataops #devsecops #mlops

🚀 NEW on We ❤️ Open Source 🚀

Daniel Paes introduces Runink: a Go-native, open source platform that makes data pipelines fast, secure, and Kubernetes-free. Built with Linux primitives and Raft for strong governance.

https://allthingsopen.org/articles/runink-rethinking-data-pipelines-go-raft-linux-primitives

#WeLoveOpenSource #FOSS #GoLang #Linux #DataOps #DevTools

Tired of babysitting DIY scraping scripts that crash the moment you scale?
You’re not alone.

PromptCloud takes the pain out of large-scale data extraction with fully managed, reliable solutions — so you can focus on what really matters: insights.

🔗 https://shorturl.at/EApIO

#WebScraping #OpenData #DataEngineering #BigData #Automation #PromptCloud #TechForGood #DataOps

DBT: трансформация данных без боли

Привет! Меня зовут Кирилл Львов, я fullstack-разработчик в компании СберАналитика. В этой статье хочу рассказать про мощный инструмент трансформации данных — DBT (Data Build Tool). Сегодня любой средний и крупный бизнес хранит множество данных в разрозненных источниках (CRM, ERP, HRM, базы данных, файловые хранилища и т.д.). Каждая из этих систем самодостаточна и закрывает определённую боль бизнеса, но собрав данные из таких источников и стандартизировав их, нам открывается возможность анализировать данные, строить модели машинного обучения и принимать на основе этих данных управленческие решения. Для того чтобы реализовать такой подход строятся ELT (или ETL) процессы. ELT (Extract, Load, Transform) — это процесс, состоящий из трех этапов:

https://habr.com/ru/articles/907540/

#dbt #big_data #data_ingineering #аналитика_данных #трансформация_данных #elt #sql #dataops

❓Need clear “done” definitions when balancing code & data?

Align engineering & analytics acceptance criteria to avoid moving goalposts.

📋 Read how: https://newsletter.optimistengineer.com/p/your-engineering-team-just-met-data?r=nw9bj&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

#SoftwareEngineering #DataOps

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

A5: LOL, not sure we have empirically tested ‘known good patterns’ for #AI yet, let alone ‘best practices’! I’d say #DataOps & #MLOps for #data mgmt & algo auditability, validations to ensure provability & drive #trust are good starting places, but we need more ‘real world’ research.

#eWeekChat

A3. (cont) Also note #security issues are not just about #AI in #cybersecurity but also about cybersecurity for AI. Managing #penetration and #data leakage (cf. #DataOps), ensuring #audit and control for AI #algorithms (cf. #MLOps), filtering for malicious outputs, etc. 2-way street

#eWeekChat

Почему работа с данными так популярна: Data Scientist, Big Data и объектное хранение

По оценкам Cybersecurity Ventures, к 2025 году общий объем данных в облаках достигнет 100 зеттабайт, или 50% всех мировых данных на тот момент. Ландшафт хранения этой информации разнообразен: от недорогих обычных серверов до огромных хранилищ на сотни петабайт. При экспоненциальном росте информации возможности работы с данными расширяются. Отчет LinkedIn о новых вакансиях показывает, что мировой рынок работы в области науки о данных вырастет до 230,80 млрд долларов к 2026 году. Наука о данных является востребованной, интересной, и, казалось бы, очень сложной областью знаний. Однако хорошая новость заключается в том, что она доступна для новичков. В статье мы разберемся, с какими базовыми основами нужно познакомиться при работе с данными.

https://habr.com/ru/companies/cloud_mts/articles/781724/

#DataOps #data #данные #Data_Scientist #Big_Data #S3 #объектное_хранилище

HIRING: Senior Software Engineer in Security / Remote 👉 https://infosec-jobs.com/J44421/ #InfoSec #infosecjobs #CyberSecurity #CyberCareer #cyber #cloudnative #jobs #remotejob #techjobs #hiring #remotework #remote #opensalary #DataOps #CDKTF #Kubernetes #AWS #GCP #Azure

It's that times of the week for All Hands on #data 🥳

Whether you are trying to escape the summer heat, or hang out with Python's favorite bear 🐼 - this week's articles from across the data space have something for you!

Full articles 👇

(Subscribe to stay up to date 😊)

#dataorchestration #mlops #dataops #dataengineering

https://allhandsondata.substack.com/p/all-hands-on-data-61

🔥⏲️ Fudge Sunday "Shock the Chaos Monkey" A look at chaos engineering adoption within platform engineering teams

#chaos #chaosengineering #faulttolerance #faultfinding #faultinjection #securityengineering #dataops #dataoperations #ai #resiliency #devx #devops #devsecops #platformengineering #platformengineer #cloudinfrastructure #securityautomation #securitybydesign #shiftleft #shiftup #newsletter #newsletters

https://fudge.org/archive/shock-the-chaos-monkey/

🔥⏲️ Fudge Sunday "Shock the Chaos Monkey" A look at chaos engineering adoption within platform engineering teams

#chaos #chaosengineering #faulttolerance #faultfinding #faultinjection #securityengineering #dataops #dataoperations #ai #resiliency #devx #devops #devsecops #platformengineering #platformengineer #cloudinfrastructure #securityautomation #securitybydesign #shiftleft #shiftup #newsletter #newsletters

https://fudge.org/archive/shock-the-chaos-monkey/

🔥⏲️ Fudge Sunday "Shock the Chaos Monkey" A look at chaos engineering adoption within platform engineering teams

#chaos #chaosengineering #faulttolerance #faultfinding #faultinjection #securityengineering #dataops #dataoperations #ai #resiliency #devx #devops #devsecops #platformengineering #platformengineer #cloudinfrastructure #securityautomation #securitybydesign #shiftleft #shiftup #newsletter #newsletters

https://fudge.org/archive/shock-the-chaos-monkey/

🔥⏲️ Fudge Sunday "Shock the Chaos Monkey" A look at chaos engineering adoption within platform engineering teams

#chaos #chaosengineering #faulttolerance #faultfinding #faultinjection #securityengineering #dataops #dataoperations #ai #resiliency #devx #devops #devsecops #platformengineering #platformengineer #cloudinfrastructure #securityautomation #securitybydesign #shiftleft #shiftup #newsletter #newsletters

https://fudge.org/archive/shock-the-chaos-monkey/

🔥⏲️ Fudge Sunday "Shock the Chaos Monkey" A look at chaos engineering adoption within platform engineering teams

#chaos #chaosengineering #faulttolerance #faultfinding #faultinjection #securityengineering #dataops #dataoperations #ai #resiliency #devx #devops #devsecops #platformengineering #platformengineer #cloudinfrastructure #securityautomation #securitybydesign #shiftleft #shiftup #newsletter #newsletters

https://fudge.org/archive/shock-the-chaos-monkey/

I wrote about the Lineage Diff for dbt projects feature of PipeRider:

You can compare then lineage DAG from both and after making code changes in dbt. It's really useful for debugging issues/seeing impact etc:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataOps #DataLineage #DataViz #DataQuality #DataTesting #DataEngineering

I wrote about the Lineage Diff for dbt projects feature of PipeRider:

You can compare then lineage DAG from both and after making code changes in dbt. It's really useful for debugging issues/seeing impact etc:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataOps #DataLineage #DataViz #DataQuality #DataTesting #DataEngineering

I wrote about the Lineage Diff for dbt projects feature of PipeRider:

You can compare then lineage DAG from both and after making code changes in dbt. It's really useful for debugging issues/seeing impact etc:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataOps #DataLineage #DataViz #DataQuality #DataTesting #DataEngineering

I wrote about the Lineage Diff for dbt projects feature of PipeRider:

You can compare then lineage DAG from both and after making code changes in dbt. It's really useful for debugging issues/seeing impact etc:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataOps #DataLineage #DataViz #DataQuality #DataTesting #DataEngineering

I wrote about the Lineage Diff for dbt projects feature of PipeRider:

You can compare then lineage DAG from both and after making code changes in dbt. It's really useful for debugging issues/seeing impact etc:

https://medium.com/inthepipeline/dbt-data-lineage-diff-impact-analysis-visualized-bec9927b0c4e

#DataOps #DataLineage #DataViz #DataQuality #DataTesting #DataEngineering

Looking for interesting datasets to practice with?
(and full end-to-end data pipelines to investigate)

Here are 5 (and a few extras) from the recent DataTalksClub Data Engineering Zoomcamp:

https://medium.com/inthepipeline/5-interesting-datasets-from-the-data-engineering-zoomcamp-f0d6dd4d5e15

Datasets Include:

- Daily data on Berlin bike thefts
- Air quality
- SF Eviction data
- Minneapolis 311
- NYC Restaurant Inspection

#DataEngineering #DataOps #Datasets #DataProjects #AnalyticsEngineering #DataPipelines #dbt #elt #DataViz

#SPASQL is how our #VirtuosoRDBMS enables declarative operations on data organized as tables or graphs via #SPARQL inside #SQL.

What are the benefits?

[1] Ubiquitous Data Source Names (#DSNs), courtesy of #HTTP

[2] Fine-grained entity relationship graph comprising clickable edges and nodes that function as #SuperKeys

#ChatGPT knows SPASQL too!

https://community.openlinksw.com/t/howto-using-chat-gpt-to-generate-enhanced-sql-and-sparql-cheatsheets/3818/2#sparql-inside-sql-spasql-example-1

#SemanticWeb #LinkedData #GraphDatabase #DBMS #RDBMS #CDO #CIO #CTO #DataOps #DBpedia #LODCloud #KnowledgeGraph #HowTo

#SPASQL is how our #VirtuosoRDBMS enables declarative operations on data organized as tables or graphs via #SPARQL inside #SQL.

What are the benefits?

[1] Ubiquitous Data Source Names (#DSNs), courtesy of #HTTP

[2] Fine-grained entity relationship graph comprising clickable edges and nodes that function as #SuperKeys

#ChatGPT knows SPASQL too!

https://community.openlinksw.com/t/howto-using-chat-gpt-to-generate-enhanced-sql-and-sparql-cheatsheets/3818/2#sparql-inside-sql-spasql-example-1

#SemanticWeb #LinkedData #GraphDatabase #DBMS #RDBMS #CDO #CIO #CTO #DataOps #DBpedia #LODCloud #KnowledgeGraph #HowTo

#SPASQL is how our #VirtuosoRDBMS enables declarative operations on data organized as tables or graphs via #SPARQL inside #SQL.

What are the benefits?

[1] Ubiquitous Data Source Names (#DSNs), courtesy of #HTTP

[2] Fine-grained entity relationship graph comprising clickable edges and nodes that function as #SuperKeys

#ChatGPT knows SPASQL too!

https://community.openlinksw.com/t/howto-using-chat-gpt-to-generate-enhanced-sql-and-sparql-cheatsheets/3818/2#sparql-inside-sql-spasql-example-1

#SemanticWeb #LinkedData #GraphDatabase #DBMS #RDBMS #CDO #CIO #CTO #DataOps #DBpedia #LODCloud #KnowledgeGraph #HowTo

#SPASQL is how our #VirtuosoRDBMS enables declarative operations on data organized as tables or graphs via #SPARQL inside #SQL.

What are the benefits?

[1] Ubiquitous Data Source Names (#DSNs), courtesy of #HTTP

[2] Fine-grained entity relationship graph comprising clickable edges and nodes that function as #SuperKeys

#ChatGPT knows SPASQL too!

https://community.openlinksw.com/t/howto-using-chat-gpt-to-generate-enhanced-sql-and-sparql-cheatsheets/3818/2#sparql-inside-sql-spasql-example-1

#SemanticWeb #LinkedData #GraphDatabase #DBMS #RDBMS #CDO #CIO #CTO #DataOps #DBpedia #LODCloud #KnowledgeGraph #HowTo

#SPASQL is how our #VirtuosoRDBMS enables declarative operations on data organized as tables or graphs via #SPARQL inside #SQL.

What are the benefits?

[1] Ubiquitous Data Source Names (#DSNs), courtesy of #HTTP

[2] Fine-grained entity relationship graph comprising clickable edges and nodes that function as #SuperKeys

#ChatGPT knows SPASQL too!

https://community.openlinksw.com/t/howto-using-chat-gpt-to-generate-enhanced-sql-and-sparql-cheatsheets/3818/2#sparql-inside-sql-spasql-example-1

#SemanticWeb #LinkedData #GraphDatabase #DBMS #RDBMS #CDO #CIO #CTO #DataOps #DBpedia #LODCloud #KnowledgeGraph #HowTo

dbt users - Do you consider data warehouse costs when building and maintaining projects?

If so, what steps you taking to keep costs down, or keep an eye on costs?

#dbt #DataQuality #DataBuildTool #DataEngineering #AnalyticsEngineering #DataOps

[Part 5]
💭 If you're reading this & you've been involved with developing an ML Platform, how did you approach the "centralize vs distributed" discussion? What worked? What failed?

👇 Let me know in the comments below!

#mlops #mlplatform #ai #strategy #dataops #dataengineering #devops #platformengineering #platformdesign #mlengineer

Beyond the Buzz: SBOMs, AI, and DataOps for Organizational Resilience in a Post-Log4j World with Jessie Jamieson.

#BsidesNYC2023 #ai #dataops