#apachekafka — Public Fediverse posts

flask-confluent-kafka

Extensão Flask totalmente tipada (PEP 561) e testada para integrar producers e consumers do Confluent Kafka em poucas linhas — configuração direto pelo app.config, autenticação SASL opcional e suporte a múltiplos clientes nomeados, independentes do par padrão. Publicada no PyPI, com CI ativo e changelog versionado.

🔗 https://github.com/Riverfount/flask-confluent-kafka

#Python #Flask #Kafka #ApacheKafka

Kafka CLI options driving you crazy? AI might finally have a solution.
Grzegorz Kocur tested Confluent’s mcp-confluent server with Claude Code on local Kafka clusters running mTLS and SASL/SCRAM.
Read to find out what happened: https://softwaremill.com/managing-kafka-with-ai-and-mcp/

#Kafka #ApacheKafka #AI #MCP #DevOps #ClaudeAI #Confluent

𝗞𝗮𝗳𝗸𝗮𝗛𝗤:

#ApacheKafka #GUI #KafkaHQ

https://thewhale.cc/posts/kafkahq

KafkaHQ is an open source GUI for Apache Kafka. Kafka GUI for topics, topics data, consumers group and more...

Apache Kafka concepts can be tricky; visualizing them helps.

The SoftwareMill Kafka Visualization tool breaks down the fundamentals interactively so you can see how things work in real time.

🆕 Now updated to support Kafka Share Groups.

Give it a spin: https://softwaremill.com/kafka-visualisation/

#ApacheKafka #Kafka #DistributedSystems #SoftwareEngineering

any #kafka committers around here that could review this? https://github.com/apache/kafka/pull/22826

#apachekafka

I need to work on an #apachekafka project. Thinking of doing something for #netlabelday or maybe @friendsofccmusic more generally.

Any ideas on what would be useful for #ccmusic and #netlabels ?

Event-driven architecture promises scalability, but the real tradeoffs in Java-based real-time systems only show up in production.

Drawing on a Java/Kafka contact center platform handling 80k BHCC across 10k agents, Sagar Deepak Joshi's new #InfoQ article explores exactly where things break down:
🔹 State management & partition limits
🔹 Message deduplication
🔹 JVM tuning challenges
🔹 Cascading consumer failures

Discover the Redis-backed patterns used to solve them and keep the system resilient.

🔗 Read now for more insights: https://bit.ly/4bmaRPb

#Java #SpringBoot #ApacheKafka #Redis #Microservices #SoftwareArchitecture

Are Apache Kafka follower replicas synchronous or asynchronous?
The confusing answer is: both.
This article clears it up:
https://softwaremill.com/apache-kafka-replica-followers-synchronous-or-asynchronous/

#ApacheKafka #Confluent

Повторная обработка сообщений в Kafka Consumer

Привет! Меня зовут Дмитрий Михеев, я ведущий разработчик в MAGNIT OMNI — бизнес-группе ритейлера «Магнит», которая отвечает за развитие омниканального опыта для клиентов. В своих сервисах для межсервисных коммуникаций помимо gRPC-запросов мы используем брокер сообщений Kafka. Если описывать его в двух словах, Kafka — это распределённый журнал событий (event log), через который сервисы обмениваются данными в реальном времени. Не буду подробно останавливаться на устройстве Kafka — это хорошо описано в документации. В этой статье хочу подсветить один неочевидный момент, который может привести к проблемам при работе с consumer’ами — повторную обработку сообщений (retry).

https://habr.com/ru/companies/magnit/articles/1043068/

#kafka #apachekafka #kafkaconsumer #retry #повторнаяобработка #идемпотентность #высоконагруженныесистемы #java

Schema proliferation builds slowly and gets expensive fast.

One schema per event type seems reasonable - until you're:
• Managing 10+ tables
• Writing union queries across all of them
• Propagating a single field rename everywhere

The Alternative❓ Discriminator-based schema consolidation.

It reduces schema count, simplifies downstream consumption, and makes schema evolution manageable. New variants become additive changes instead of breaking existing consumers.

🔗 Check out the #InfoQ article for a practical look at the pattern and its trade-offs ⇨ https://bit.ly/4uC023n

#Java #ApacheKafka #ApacheFlink

This #InfoQ article examines how #ApacheKafka is evolving toward a #CloudNative architecture through tiered storage, elastic consumers, virtual clusters, and diskless storage proposals.

Read now: https://bit.ly/4u0hZr0

#AI #CloudComputing #kafkastreams #softwarearchitecture

#Confluent introduces a new approach in #ApacheKafka that moves schema IDs from message payloads to record headers.

✅ Simplify schema governance & evolution.
✅ Improve compatibility across serialization formats
✅ Reduce coupling between data & metadata in event-driven architectures

Read the deep dive on #InfoQ ⇨ https://bit.ly/4tF7Fot

#ML #EventStreamProcessing #ProtocolBuffers #DataPipelines #DataAnalytics

Grafana Labs announced #Grafana 13, introducing a new Loki Kafka-backed ingestion architecture and AI Observability in Grafana Cloud for real-time monitoring and evaluation of AI systems.

The release also includes GCX, a new CLI designed to surface Grafana Cloud data inside agentic development environments.

Details here 👉 https://bit.ly/4mPUqz8

#Observability #AI #DevOps #ApacheKafka #InfoQ

Our experts, Krzysztof Grajek and Grzegorz Kocur, will be speaking at the Data Streaming Meetup in Kraków. View the agenda and join the event: https://meetup.com/krakow-kafka/events/314177442/

#ApacheKafka #DataStreaming #Confluent #Krk #Meetup #StreamProcessing #Rustlang #DevOps

Kafka has reached commodity status — and that's actually its greatest achievement. Here's what it means for where streaming innovation goes next. https://hackernoon.com/kafka-has-become-the-postgres-of-streaming-and-that-changes-everything #apachekafka

Kafka in-cluster replication won't save you from a regional outage. You need a battle-tested Multi-Region strategy.
Read this to master:
- RPO/RTO trade-offs for global scale
- Active-Active vs. Stretched Clusters (3-DC & 2.5-DC)
- Solving the offset translation nightmare
- Real-world failover testing
https://softwaremill.com/guide-to-apache-kafka-disaster-recovery-and-multi-region-architectures/

#ApacheKafka #SystemArchitecture #DisasterRecovery #DataEngineering

Meet Tansu!

An #opensource, Kafka-compatible broker that’s stateless, leaderless, and scales to zero.

• Pluggable storage: S3, SQLite, Postgres
• Broker-side schema validation
• Direct writes to Iceberg & Delta Lake

Built in #RustLang, - 20MB RAM, ~10ms startup

Learn more ⇨ https://bit.ly/4d119nc

#EventDrivenArchitecture #ApacheKafka #SoftwareDevelopment

Check out what's new in the #Spring community 👉 https://bit.ly/3NMwcbY

The third milestone releases of: Spring Boot, Spring Security, Spring Integration, Spring AI and Spring AMQP; along with the second milestone releases of Spring Data and Spring for Apache Kafka.

#Java #SpringBoot #SpringData #SpringSecurity #SpringAI #SpringVault #ApacheKafka

Apache Kafka has gotten rather long in the tooth, is Apache Iggy the successor? https://hackernoon.com/what-the-heck-is-apache-iggy #apachekafka

Is your Kafka lag dashboard lying? Learn how log compaction and retention silently manipulate reported timestamps to understate consumer delays and how to identify these inaccuracies and ensure your monitoring reflects actual system performance.
https://softwaremill.com/compaction-and-retention-edge-cases-that-make-your-kafkas-lag-metrics-inaccurate/

#ApacheKafka #DataEngineering #DistributedSystems #EventStreaming #Observability

#Pinterest launched a next-gen CDC-based ingestion framework.

Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
• Latency cut from 24+ hours to 15 minutes
• Processing of only changed records
• Support for incremental updates & deletions
• Petabyte-scale data across 1,000+ pipelines

Win: optimized cost & efficiency!

Read the architectural deep dive on InfoQ 👉 https://bit.ly/4rMJB2H

#SoftwareArchitecture #ChangeDataCapture

Apache Kafka 4.2.0 is here – and it’s a big one. The headliner? Queues for Kafka (KIP-932) is officially GA. This means you can finally use Kafka for classic "competing consumer" queuing patterns without the rigid constraints of partition assignments.
But there is much more:

#ApacheKafka #Kafka42 #EventStreaming #DataEngineering #DistributedSystems

#Uber has open-sourced #uForwarder - a push-based #ApacheKafka consumer proxy built to handle trillions of messages & multiple petabytes of data daily.

uForwarder introduces:
🔹 Context-aware routing
🔹 Head-of-line blocking mitigation
🔹 Adaptive auto-rebalancing
🔹 Partition-level delay processing

The result ❓ Read #InfoQ to find out: https://bit.ly/4u9T2Lg

#SoftwareArchitecture #EventDrivenArchitecture #Microservices #DistributedSystems #OpenSource

Check out what's new in the #Spring community 👉 https://bit.ly/4kZSJyw

The second milestone releases of: Spring Boot, Spring Security, Spring Integration, Spring Modulith and Spring AMQP; along with the first milestone releases of Spring Session, Spring for Apache Kafka and Spring LDAP.

#Java #SpringBoot #SpringData #SpringSecurity #SpringAMQP #ApacheKafka

𝗞𝗮𝗳𝗸𝗮𝗛𝗤:

#ApacheKafka #GUI #KafkaHQ

https://thewhale.cc/posts/kafkahq

KafkaHQ is an open source GUI for Apache Kafka. Kafka GUI for topics, topics data, consumers group and more...

#LinkedIn just upgraded its legacy ZooKeeper service discovery platform to enhance scalability & performance.

How❓ By leveraging #ApacheKafka & the xDS protocol.
The result❓ The new architecture enables eventual consistency, supports multiple languages, and allows migration without downtime.

Details here ⇨ https://bit.ly/3NRZFRP

#SoftwareArchitecture #InfoQ

Becoming a Confluent Elite Partner is an important milestone for SoftwareMill.

We are one of the few companies in Europe and the only one in Poland to hold this title. It showcases our extensive experience in helping organizations build scalable data platforms and fully harness the potential of continuous data flow using the Confluent platform and Apache Kafka-powered solutions.

👉 Learn more about our experience and services: sml.io/confluent-partner

#apachekafka #confluent #kafkaconsulting