#apacheflink — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #apacheflink, aggregated by home.social.
-
Schema proliferation builds slowly and gets expensive fast.
One schema per event type seems reasonable - until you're:
• Managing 10+ tables
• Writing union queries across all of them
• Propagating a single field rename everywhereThe Alternative❓ Discriminator-based schema consolidation.
It reduces schema count, simplifies downstream consumption, and makes schema evolution manageable. New variants become additive changes instead of breaking existing consumers.
🔗 Check out the #InfoQ article for a practical look at the pattern and its trade-offs ⇨ https://bit.ly/4uC023n
-
Schema proliferation builds slowly and gets expensive fast.
One schema per event type seems reasonable - until you're:
• Managing 10+ tables
• Writing union queries across all of them
• Propagating a single field rename everywhereThe Alternative❓ Discriminator-based schema consolidation.
It reduces schema count, simplifies downstream consumption, and makes schema evolution manageable. New variants become additive changes instead of breaking existing consumers.
🔗 Check out the #InfoQ article for a practical look at the pattern and its trade-offs ⇨ https://bit.ly/4uC023n
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
#Pinterest launched a next-gen CDC-based ingestion framework.
Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelinesWin: optimized cost & efficiency!
Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H
-
Flink Stateful Functions https://nightlies.apache.org/flink/flink-statefun-docs-stable/ hasn't had an update in several years and is still compiled against an old version of Apache Flink.
I spent some time during the holidays uplifting it to Flink 2.2.0 https://github.com/fransking/flink-statefun/commit/25b2dfcc3feafd701eb2fec32d413fab56e1cfe6
Notionally it appears to be working as expected.
-
This new feature ensures seamless schema updates for CDC data sources, enhancing flexibility and data consistency. https://hackernoon.com/final-project-report-1-schema-evolution-support-on-apache-seatunnel-flink-engine #apacheflink
-
This new feature ensures seamless schema updates for CDC data sources, enhancing flexibility and data consistency. https://hackernoon.com/final-project-report-1-schema-evolution-support-on-apache-seatunnel-flink-engine #apacheflink
-
Join us in #Lucerne on October 15, 2025 for an exciting talk by Viktor Gamov about #StreamProcessing with #Apache #Flink!
Stream processing has come a long way. Today’s systems handle strict correctness guarantees, low latency, and terabytes of state. In this talk, Viktor will introduce #ApacheFlink, explain concepts like stateful and event-time stream processing, and explore its APIs and ecosystem.
More info & registration: https://www.jug.ch/html/events/2025/stateful_stream_processing.html
-
Join us in #Lucerne on October 15, 2025 for an exciting talk by Viktor Gamov about #StreamProcessing with #Apache #Flink!
Stream processing has come a long way. Today’s systems handle strict correctness guarantees, low latency, and terabytes of state. In this talk, Viktor will introduce #ApacheFlink, explain concepts like stateful and event-time stream processing, and explore its APIs and ecosystem.
More info & registration: https://www.jug.ch/html/events/2025/stateful_stream_processing.html
-
Explore the latest Apache Flink SQL & Table API updates, enabling real-time AI, advanced joins, UDFs, and cloud-native state management. https://hackernoon.com/what-you-need-to-know-about-the-latest-updates-to-apache-flinks-sql-and-table-apis #apacheflink
-
Explore the latest Apache Flink SQL & Table API updates, enabling real-time AI, advanced joins, UDFs, and cloud-native state management. https://hackernoon.com/what-you-need-to-know-about-the-latest-updates-to-apache-flinks-sql-and-table-apis #apacheflink
-
It's #MLPrague today! I'm here with @stereosky
Teaching a workshop streaming in the machine learning with #apacheflink and #python!
Come along and find us at 14:00 where we'll run through streaming, ML and how to put it all together! -
It's #MLPrague today! I'm here with @stereosky
Teaching a workshop streaming in the machine learning with #apacheflink and #python!
Come along and find us at 14:00 where we'll run through streaming, ML and how to put it all together! -
It's #qcon London!!
We're here with a booth and talks on #ZeroTrust and #apacheflink.
Also meeting old friends Nf colleagues! -
It's #qcon London!!
We're here with a booth and talks on #ZeroTrust and #apacheflink.
Also meeting old friends Nf colleagues! -
Atlassian introduced Lithium - an in-house #ETL platform designed to meet the requirements of dynamic data movement.
Lithium simplifies cloud migrations, scheduled backups, and in-flight data validations with ephemeral pipelines and tenant-level isolation - ensuring efficiency, scalability & cost savings.
📢 InfoQ spoke with Niraj Mishra, Principal Engineer at Atlassian, about Lithium’s implementation and future.
🔗 Read more here: https://bit.ly/415RPYZ
#DataPipelines #KafkaStreams #ApacheKafka #ApacheFlink #SoftwareArchitecture
-
Atlassian introduced Lithium - an in-house #ETL platform designed to meet the requirements of dynamic data movement.
Lithium simplifies cloud migrations, scheduled backups, and in-flight data validations with ephemeral pipelines and tenant-level isolation - ensuring efficiency, scalability & cost savings.
📢 InfoQ spoke with Niraj Mishra, Principal Engineer at Atlassian, about Lithium’s implementation and future.
🔗 Read more here: https://bit.ly/415RPYZ
#DataPipelines #KafkaStreams #ApacheKafka #ApacheFlink #SoftwareArchitecture
-
Spent this morning digging into a #ApacheFlink job in production. I just love this technology, both in development and operation. Give it a try!
-
🎁 An early Christmas present for y'all: a new blog in which I explore Flink CDC. Building data pipelines declaratively is pretty nice, but is it ready for prime-time? Let's find out!
-
This role will let me build on my recent work, dive into new tech, and support the next wave of #openSource innovation. While I may be shifting gears, I’m looking forward to continued collaboration with the wonderful #apacheKafka 🦦 and #apacheFlink 🐿 communities.
-
🎃The October issue of #CheckpointChronicle is now out 🌟
It covers Ververica's Fluss, #ApacheFlink 2.0, Iggy.rs, Strimzi's support for #ApacheKafka 4.0, tons of OTF material from @vanlightly, Christian Hollinger's write up of ngrok's data platform, nice detail of how SmartNews use #ApacheIceberg with Flink and #ApacheSpark, a good writeup from Sudhendu Pandey on #ApachePolaris, notes from Kir Titievsky on Kafka's Avro serialisers, and much more!
-
It's lunch time at the datastream summit!
Leave the data, take the cannoli
#datastreaming #apacheflink #apachepulsar -
I'm at the streaming data summit and Taylor Swift is the topic of the conversation.
What is Taylor Swift to us?
A traffic spike!
#apachepulsar #apacheflink -
It's the formal opening of @FlinkFoward!
If you're here, the opening talks are starting!! #apacheflink #datastreaming #realtime! -
I'm a sucker for a nice plushy. And these two are from #current24 and reprisent my favourite raft powered distributed systems, #TiDB and #Redpanda!
Both which work great with @ververicadata and #apacheflink -
Part two of the journey to #current24, getting off the plane...
I'll see most of you all tomorrow, though some will be at the meet up this evening.
Ping me to chat nonsense about
#apacheflink, #apachePaimon and #apachekafka -
Latest version of my Whisky clustering using Apache projects talk:
https://speakerdeck.com/paulk/groovy-whiskey
Tickets are still available for CoCEU. #apachecon #communityovercode #apachewayang #ApacheFlink #ApacheSpark #ApacheBeam #ApacheIgnite #ApacheCommons @ApacheGroovy #opensource #machinelearning #groovylang -
does anyone have suggestions for #ApacheFlink podcasts?
I am not looking for documentation or YouTube videos. I am looking for audio RSS feeds I can add to my podcatcher. If the RSS feed has embedded video, that is fine, but YouTube does not provide a RSS feed with an embed.
-
#CaseStudy - Discover how #Yelp reworked its data streaming architecture with #ApacheBeam & #ApacheFlink!
The company replaced a fragmented set of data pipelines for streaming transactional data into its analytical systems, like Amazon Redshift and in-house data lake, using Apache data streaming projects to create a unified and flexible solution.
Dive into the details: https://bit.ly/3WgkTL7
#InfoQ #SoftwareArchitecture #EventDrivenArchitecture #DataPipelines #Streaming
-
Do you know SQL? Exactly!
Most databases and data processing tools support SQL for exactly that reason. And we see a strong movement for all of them to get closer to the standard, day by day.
In this weeks episode of the Cloud Commute podcast, our host @noctarius2k talks with @gunnarmorling from #Decodable about the benefits of #SQL, how #CDC (change data capture) works and why Decodable uses #ApacheFlink as the underlying technology for its #StreamProcessing offering.
-
Just another day on the #ApacheFlink user mailing list 🙃
👉For better or worse, understanding JARs is an unavoidable first step to using Flink if you're running it yourself.
👇 That's why I wrote about it a lot 😁
1️⃣ https://www.decodable.co/blog/flink-sql-and-the-joy-of-jars
2️⃣ https://www.decodable.co/blog/flink-sql-misconfiguration-misunderstanding-and-mishaps -
✍️Blogged: Flink SQL—Misconfiguration, Misunderstanding, and Mishaps
🫖 Pull up a comfy chair, grab a mug of tea, and settle in to read about my adventures troubleshooting some gnarly #ApacheFlink problems ranging from the simple to the ridiculous…
🔗 https://dcbl.link/troubleshooting-flinksql2
👉 Topics include:
🤔 What's Running Where? (Fun with Java Versions)
🤨 What's Running Where? (Fun with JAR dependencies)
😵 What's Running Where? (Not So Much Fun with Hive MetaStore) -
If you enjoyed my recent #ApacheFlink blogs on catalogs (https://www.decodable.co/blog/catalogs-in-flink-sql-a-primer & https://www.decodable.co/blog/catalogs-in-flink-sql-hands-on) and JARs (https://www.decodable.co/blog/flink-sql-and-the-joy-of-jars) you're going to *love* the new post later this week that goes deep into the weeds of troubleshooting some real-world gnarly problems…
-
Everything you wanted to know (and some you didn't) about Catalogs in Flink SQL https://www.decodable.co/blog/catalogs-in-flink-sql-hands-on
-
-
🥳 Woo, #apacheFlink 1.19 has been released!
Details in the release blog 👇🏻
https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
-
✨Here is a list of all the **#ApacheFlink** talks (16 of them!) at **#kafkasummit** London next week:
🔗 https://rmoff.net/2024/03/15/apache-flink-talks-at-kafka-summit-london-2024/
**#ksl24** **#FlinkAtKSL #openSource**