#apacheflink — Public Fediverse posts

#Pinterest launched a next-gen CDC-based ingestion framework.

Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelines

Win: optimized cost & efficiency!

Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H

#SoftwareArchitecture #ChangeDataCapture

It's #qcon London!!
We're here with a booth and talks on #ZeroTrust and #apacheflink.
Also meeting old friends Nf colleagues!

Atlassian introduced Lithium - an in-house #ETL platform designed to meet the requirements of dynamic data movement.

Lithium simplifies cloud migrations, scheduled backups, and in-flight data validations with ephemeral pipelines and tenant-level isolation - ensuring efficiency, scalability & cost savings.

📢 InfoQ spoke with Niraj Mishra, Principal Engineer at Atlassian, about Lithium’s implementation and future.

🔗 Read more here: https://bit.ly/415RPYZ

#DataPipelines #KafkaStreams #ApacheKafka #ApacheFlink #SoftwareArchitecture

#InfoQ

It's lunch time at the datastream summit!
Leave the data, take the cannoli
#datastreaming #apacheflink #apachepulsar

I'm at the streaming data summit and Taylor Swift is the topic of the conversation.
What is Taylor Swift to us?
A traffic spike!
#apachepulsar #apacheflink

#CaseStudy - Discover how #Yelp reworked its data streaming architecture with #ApacheBeam & #ApacheFlink!

The company replaced a fragmented set of data pipelines for streaming transactional data into its analytical systems, like Amazon Redshift and in-house data lake, using Apache data streaming projects to create a unified and flexible solution.

Dive into the details: https://bit.ly/3WgkTL7

#InfoQ #SoftwareArchitecture #EventDrivenArchitecture #DataPipelines #Streaming

I find #scopecreep to be a real problem with #selfGuidedLearning, particularly online. Do others find the same and, if so, how do you deal with it?

As an example, I sat down to dig into #ApacheSuperset but that led to #ApacheAirflow, which lead to the Astronomer.io service, which led to #ApacheFlink. By the end of the day I’d had a great overview of some really neat stuff but was not better positioned to be productive in Superset than I was at the start of the day.

Curious to hear from folks, besides #apacheKafka, what are your favorite #streaming technologies? 🤔

#streamingdata #eventstreaming #streamingtechnology #apacheFlink #apacheSpark #apachePulsar

#Current23 is the next generation of #KafkaSummit, so expect some #apacheKafka talks. But there's room for everyone! #apacheDruid, #apacheFlink, #ETL, #dataMesh, #machineLearning, #IoT, #databases...

We have room for talks on the unique #devops challenges of monitoring and managing any of these systems; talks on #sociotechnical impact of rearchitecting your business for streaming; #streaming in academia; and just plain fun and geeky sessions.

The list goes on and on!

You can then use #LogicalDecoding - #Postgres’ change data capture capability - to retrieve the messages from write-ahead log (WAL), process them, and relay them to external consumers.

In this #InfoQ article, Gunnar Morling explores how to take advantage of this feature for implementing three different use cases:
✅ Propagating data between microservices via the #OutboxPattern
✅ #ApplicationLogging
✅ Enriching #AuditLogs with metadata

Learn more: http://bit.ly/3ZiMZnH

#Debezium #ApacheFlink #Microservices

No more centralized schema committees or custom code to maintain, just the data you need in the format you want, when you need it.

https://bit.ly/33Nayxx #ApachePulsar #ApacheKafka #ApacheFlink