#dataops — Public Fediverse posts

Learn how to install OpenMetadata on Ubuntu using Docker Compose. Deploy the complete stack with Airflow, PostgreSQL, and Elasticsearch.

Full guide here: https://ostechnix.com/install-openmetadata-ubuntu-docker-compose/

#Openmetadata #Metadata #DataGovernance #DataCatalog #Dataops #Devops #DataEngineering #Ubuntu #Postgresql #Docker #DockerCompose #Elasticsearch #ApacheAirflow

Learn how to install OpenMetadata on Ubuntu using Docker Compose. Deploy the complete stack with Airflow, PostgreSQL, and Elasticsearch.

Full guide here: https://ostechnix.com/install-openmetadata-ubuntu-docker-compose/

#Openmetadata #Metadata #DataGovernance #DataCatalog #Dataops #Devops #DataEngineering #Ubuntu #Postgresql #Docker #DockerCompose #Elasticsearch #ApacheAirflow

📅 :boost_love: New in #LabPlot [dev]: #Date and #Time functions in spreadsheets, including:

👉 today, now – current date and time.
👉 date, year, month, day, hour, minute, second – extract components from datetime values.
👉 datedif – calculate the difference between two dates.

Useful for time-based data analysis, logging and visualization.

#DataViz #OpenSource #FOSS #FLOSS #DataScience #DataAnalysis #Research #Productivity #OpenScience #Tech #Engineering #Education #SysAdmin #DevOps #DataOps

📅 :boost_love: New in #LabPlot [dev]: #Date and #Time functions in spreadsheets, including:

👉 today, now – current date and time.
👉 date, year, month, day, hour, minute, second – extract components from datetime values.
👉 datedif – calculate the difference between two dates.

Useful for time-based data analysis, logging and visualization.

#DataViz #OpenSource #FOSS #FLOSS #DataScience #DataAnalysis #Research #Productivity #OpenScience #Tech #Engineering #Education #SysAdmin #DevOps #DataOps

Strong data catalogs transform the way teams think. They shift work from guesswork to clear action. What’s your take? #DataCatalogs #SingleSourceOfTruth #DataQuality #DataTrust #DataOps #DataStrategy #AIReadyData #DataFuture
https://www.linkedin.com/pulse/building-data-catalogs-quiet-power-behind-single-truth-mohindroo--icosc

AI needs clean data 🤖

The quality of AI outputs depends on the quality of the data behind them.

📊 Better data
→ Better outcomes

⚠️ Poor data
→ Increased risk

MSPs implementing AI must continuously govern and maintain data quality.

𝐇𝐀𝐋𝐄𝐗𝐎 𝐏𝐎𝐕:

Data readiness is critical for AI success.

▸ Swipe

#DataQuality #AI #MSP #DataOps

AI needs clean data 🤖

The quality of AI outputs depends on the quality of the data behind them.

📊 Better data
→ Better outcomes

⚠️ Poor data
→ Increased risk

MSPs implementing AI must continuously govern and maintain data quality.

𝐇𝐀𝐋𝐄𝐗𝐎 𝐏𝐎𝐕:

Data readiness is critical for AI success.

▸ Swipe

#DataQuality #AI #MSP #DataOps

AI tools give a flash of speed, but only a system of linked layers delivers lasting motion. When a persona, automated pipeline, and analytics loop converse without manual hand-off, the workflow becomes a self-sustaining gear train that tolerates API changes and scales with market growth. Build the infrastructure, not the hype. 🚀 #AIMarketing #Automation #SystemsDesign #DataOps - Powered by FG

Strong data catalogs transform the way teams think. They shift work from guesswork to clear action. What’s your take? #DataCatalogs #SingleSourceOfTruth #DataQuality #DataTrust #DataOps #DataStrategy #AIReadyData #DataFuture
https://www.linkedin.com/pulse/building-data-catalogs-quiet-power-behind-single-truth-mohindroo--icosc

Flink Kubernetes operator: опыт построения стриминговой Big Data платформы

Всем привет! Меня зовут Артемий, я работаю SRE-инженером в команде RTP (real time processing) Clickstream в Авито . Сегодня мы хотим поделиться нашей историей о том, как мы переехали в Kubernetes, развернув Apache Flink с помощью Flink k8s operator.

https://habr.com/ru/companies/avito/articles/972412/

#data #dwh #DataOps #bigdata #flink #sql #realtime #analytics #analytical_engine

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://www.linkedin.com/pulse/data-observability-next-frontier-quality-management-mohindroo--lpgtc

FlinkSQL в Авито: что внутри и как нам это помогает

Привет! Меня зовут Артемий, я инженер в команде Processing и веду фичатрек по развитию FlinkSQL в Авито . Сегодня я расскажу про инструмент, который мы развиваем в Avito — FlinkSQL. Он использует потоковые данные для аналитики в реальном времени: метрики, временные срезы, A/B-расчеты, поиск аномалий. Все это на привычном SQL, чтобы команда видела картину и могла действовать сразу.

https://habr.com/ru/companies/avito/articles/959382/

#Flink #kubernetes #bigdata #k8s #operator #kubernetes_operator #flink_kubernetes_operator #Apache_Flink #DataOps #dataops_platform

Bad data breaks business. Data observability builds trust. Are you ready? #DataObservability #DataQuality #DataReliability #DataOps #CIO #CDO #CTO #DataTrust #DigitalTransformation #AI
https://medium.com/@sanjay.mohindroo66/data-observability-the-next-frontier-in-data-quality-management-d5d75ea375c7

Stop fighting complex data pipelines. Kestra is an open-source, event-driven orchestrator that's infinitely scalable. 🧩 Define workflows with simple YAML or a no-code UI, then sync with Git. #opensource #kestra #orchestration #dataops #devops

UPDATE: Hey everyone, there was a problem with the Restream-to-LinkedIn connector, and as we were attempting to restart the stream, the event on LinkedIn disappeared. We're not sure what caused this issue, but we are re-scheduling this event. Please stay tuned for the new date and time!

Today on #TheDrillDown, Ahmad and I are honored to be joined by a very special guest, @Jkerski! We'll be drilling down into the topic of integrating #DataOps tools and principles into your #MicrosoftFabric and #PowerBI workflows. Join us at 2PM Central time! https://lnkd.in/gbegXeAc

UPDATE: Hey everyone, there was a problem with the Restream-to-LinkedIn connector, and as we were attempting to restart the stream, the event on LinkedIn disappeared. We're not sure what caused this issue, but we are re-scheduling this event. Please stay tuned for the new date and time!

Today on #TheDrillDown, Ahmad and I are honored to be joined by a very special guest, @Jkerski! We'll be drilling down into the topic of integrating #DataOps tools and principles into your #MicrosoftFabric and #PowerBI workflows. Join us at 2PM Central time! https://lnkd.in/gbegXeAc

DevOps в 2025 году: отдельные дисциплины, машинное обучение и прогноз на будущее

Привет! Меня зовут Константин Полуэктов, я solution architect в Yandex Cloud и технический эксперт в онлайн-магистратуре от Яндекса и ИТМО

https://habr.com/ru/companies/yandex_praktikum/articles/915730/

#devops #llm #ai #dataops #devsecops #mlops

DataOpsに入門したい
https://qiita.com/yuta3okkie/items/4f5957952963f25eb0bb?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items

#qiita #データエンジニアリング #データ基盤 #DataOps

🚀 NEW on We ❤️ Open Source 🚀

Daniel Paes introduces Runink: a Go-native, open source platform that makes data pipelines fast, secure, and Kubernetes-free. Built with Linux primitives and Raft for strong governance.

https://allthingsopen.org/articles/runink-rethinking-data-pipelines-go-raft-linux-primitives

#WeLoveOpenSource #FOSS #GoLang #Linux #DataOps #DevTools

🚀 NEW on We ❤️ Open Source 🚀

Daniel Paes introduces Runink: a Go-native, open source platform that makes data pipelines fast, secure, and Kubernetes-free. Built with Linux primitives and Raft for strong governance.

https://allthingsopen.org/articles/runink-rethinking-data-pipelines-go-raft-linux-primitives

#WeLoveOpenSource #FOSS #GoLang #Linux #DataOps #DevTools

Tired of babysitting DIY scraping scripts that crash the moment you scale?
You’re not alone.

PromptCloud takes the pain out of large-scale data extraction with fully managed, reliable solutions — so you can focus on what really matters: insights.

🔗 https://shorturl.at/EApIO

#WebScraping #OpenData #DataEngineering #BigData #Automation #PromptCloud #TechForGood #DataOps

DBT: трансформация данных без боли

Привет! Меня зовут Кирилл Львов, я fullstack-разработчик в компании СберАналитика. В этой статье хочу рассказать про мощный инструмент трансформации данных — DBT (Data Build Tool). Сегодня любой средний и крупный бизнес хранит множество данных в разрозненных источниках (CRM, ERP, HRM, базы данных, файловые хранилища и т.д.). Каждая из этих систем самодостаточна и закрывает определённую боль бизнеса, но собрав данные из таких источников и стандартизировав их, нам открывается возможность анализировать данные, строить модели машинного обучения и принимать на основе этих данных управленческие решения. Для того чтобы реализовать такой подход строятся ELT (или ETL) процессы. ELT (Extract, Load, Transform) — это процесс, состоящий из трех этапов:

https://habr.com/ru/articles/907540/

#dbt #big_data #data_ingineering #аналитика_данных #трансформация_данных #elt #sql #dataops

❓Need clear “done” definitions when balancing code & data?

Align engineering & analytics acceptance criteria to avoid moving goalposts.

📋 Read how: https://newsletter.optimistengineer.com/p/your-engineering-team-just-met-data?r=nw9bj&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

#SoftwareEngineering #DataOps

❓Need clear “done” definitions when balancing code & data?

Align engineering & analytics acceptance criteria to avoid moving goalposts.

📋 Read how: https://newsletter.optimistengineer.com/p/your-engineering-team-just-met-data?r=nw9bj&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

#SoftwareEngineering #DataOps

Do you know of any #rstats teams who could use some help in 2025?

I'm a freelance consultant with 10yrs of experience in #DataSci, #DataEng, #DataOps, and providing #SysAdmin for related tools. I primarily work with R, but I have enough SQL and Python knowledge to get by. I know my way around a linux terminal and docker-compose file. I'm also handy as an interpreter between IT and analytics teams. Open to many types of engagements (project-based, temp staff, etc).

If you have a need or a lead, I'd love to chat!

boosts welcome :-)

A5: LOL, not sure we have empirically tested ‘known good patterns’ for #AI yet, let alone ‘best practices’! I’d say #DataOps & #MLOps for #data mgmt & algo auditability, validations to ensure provability & drive #trust are good starting places, but we need more ‘real world’ research.

#eWeekChat

A3. (cont) Also note #security issues are not just about #AI in #cybersecurity but also about cybersecurity for AI. Managing #penetration and #data leakage (cf. #DataOps), ensuring #audit and control for AI #algorithms (cf. #MLOps), filtering for malicious outputs, etc. 2-way street

#eWeekChat

"@BMCSoftware report examines #DataOps practices"

Interesting #research uncovers many positive practices to improve IT #data management, #automation, #analytics, ++

Sponsored, but conducted by @451Research, so should be pretty reliable.

https://www.infoworld.com/article/3477727/bmc-report-examines-dataops-practices.html

"@BMCSoftware Study Shows Opportunity for #Enterprise to Expand Use of Emergent #Data Sources for Initiatives Incl. #GenerativeAI, #LLM, #FinOps, #Sustainability"

Buzzword bingo but excellent original research into #DataOps via @451Research & @SPGlobal

https://www.prnewswire.com/news-releases/bmc-study-shows-opportunity-for-enterprises-to-expand-use-of-emergent-data-sources-for-initiatives-including-generative-ai-llms-finops-sustainability-302204791.html

From what I've seen, there are a number of data engineers who are super strong on the data science side of things but who could use more ops/devops skills. If this describes you, what are the skills that you hope to "fill in"?
The command line?
Git?
CI/CD?
APIs?
Cloud services?
#DataEngineering #DevOps #DataOps

"@DataRobot Introduces #AI #Observability with Real-Time Intervention for #GenerativeAI"

Cool #O11Y features for insight into AI assets like algorithms & data, adds important #MLOps & #DataOps capability to accelerate AI experiments & reduce #risk.
https://www.businesswire.com/news/home/20240502422285/en/DataRobot-Introduces-AI-Observability-with-Real-Time-Intervention-for-Generative-AI