home.social

#apacheiceberg — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #apacheiceberg, aggregated by home.social.

fetched live
  1. Next Wednesday: the engineer who built ColdFront live on the architecture.

    Most #Postgres databases pay SSD prices for data nobody queries. ColdFront gives you #PostgreSQL to #ApacheIceberg using the same #SQL and the same table names with writable cold tier.

    On August 19, 8 AM PST, catch @vyruss on the engineering + TLA+-verified distributed writes. Paul Rothrock will give a live demo.

    Q&A at the end.

    📅 us02web.zoom.us/webinar/regist

    🔗 github.com/pgEdge/ColdFront

    #DuckDB #DataEngineering #OpenSource

  2. Next Wednesday: the engineer who built ColdFront live on the architecture.

    Most #Postgres databases pay SSD prices for data nobody queries. ColdFront gives you #PostgreSQL to #ApacheIceberg using the same #SQL and the same table names with writable cold tier.

    On August 19, 8 AM PST, catch @vyruss on the engineering + TLA+-verified distributed writes. Paul Rothrock will give a live demo.

    Q&A at the end.

    📅 us02web.zoom.us/webinar/regist

    🔗 github.com/pgEdge/ColdFront

    #DuckDB #DataEngineering #OpenSource

  3. Open source transparent #Postgres tiering to Apache Iceberg. Same table names, same SQL, OLTP + analytics + AI in one database. Cold data is still writable - one DELETE handles a GDPR request, no restore-to-hot cycle.

    In-process #DuckDB (no sidecar, no daemon). Stock upstream #PostgreSQL. Up to 90% cheaper storage. @vyruss (who built it) + Paul Rothrock live demo - August 19, 8 AM PST.

    🔗 github.com/pgEdge/ColdFront
    📅 us02web.zoom.us/webinar/regist

    #Cybersecurity #ApacheIceberg #DuckDB #FOSS #Postgres

  4. Open source transparent #Postgres tiering to Apache Iceberg. Same table names, same SQL, OLTP + analytics + AI in one database. Cold data is still writable - one DELETE handles a GDPR request, no restore-to-hot cycle.

    In-process #DuckDB (no sidecar, no daemon). Stock upstream #PostgreSQL. Up to 90% cheaper storage. @vyruss (who built it) + Paul Rothrock live demo - August 19, 8 AM PST.

    🔗 github.com/pgEdge/ColdFront
    📅 us02web.zoom.us/webinar/regist

    #Cybersecurity #ApacheIceberg #DuckDB #FOSS #Postgres

  5. Most Postgres databases store 80-90% of their rows on expensive SSD that nobody's queried in months.

    ColdFront is pgEdge's open source answer: moves cold data to Apache Iceberg on object storage (up to 90% cheaper) while keeping the same table names, SQL, and connection string. No application changes. No vendor lock-in.

    Jimmy Angelakos (who built it) & Paul Rothrock demo it live - 8/19, 8 AM PST. Register: 📅 hubs.la/Q04rdpHQ0

    #PostgreSQL #ApacheIceberg #Database #OpenSource #Postgres

  6. Most Postgres databases store 80-90% of their rows on expensive SSD that nobody's queried in months.

    ColdFront is pgEdge's open source answer: moves cold data to Apache Iceberg on object storage (up to 90% cheaper) while keeping the same table names, SQL, and connection string. No application changes. No vendor lock-in.

    Jimmy Angelakos (who built it) & Paul Rothrock demo it live - 8/19, 8 AM PST. Register: 📅 hubs.la/Q04rdpHQ0

    #PostgreSQL #ApacheIceberg #Database #OpenSource #Postgres

  7. Learn about Cloudflare's Town Lake, its unified internal data platform, where billing workloads account for 53% of all platform queries.

    Built alongside Skipper, an AI-powered analytics agent, Town Lake unifies access to operational, billing, security, and business data that was previously spread across fragmented systems.

    Details here ➤ bit.ly/4bvWOGR

    #Cloudflare #ApacheIceberg #Postgres #DataAnalytics #DataPlatform #DataGovernance #PlatformEngineering #InfoQ

  8. Learn about Cloudflare's Town Lake, its unified internal data platform, where billing workloads account for 53% of all platform queries.

    Built alongside Skipper, an AI-powered analytics agent, Town Lake unifies access to operational, billing, security, and business data that was previously spread across fragmented systems.

    Details here ➤ bit.ly/4bvWOGR

  9. In most tiering systems, cold data is read-only. A GDPR deletion on archived rows means restore-delete-rearchive: a half-day job.

    ColdFront: UPDATE or DELETE archived rows with one SQL statement.
    HFS Research analyst Ashish Chaturvedi, in Anirban Ghoshal's InfoWorld piece today. ColdFront appears alongside Databricks, Snowflake & EDB on the OLTP/OLAP divide. The only 100% #OpenSource option, with #Postgres as the interface.

    📖 hubs.la/Q04mQNSw0

    #PostgreSQL #ApacheIceberg #DataEngineering

  10. In most tiering systems, cold data is read-only. A GDPR deletion on archived rows means restore-delete-rearchive: a half-day job.

    ColdFront: UPDATE or DELETE archived rows with one SQL statement.
    HFS Research analyst Ashish Chaturvedi, in Anirban Ghoshal's InfoWorld piece today. ColdFront appears alongside Databricks, Snowflake & EDB on the OLTP/OLAP divide. The only 100% #OpenSource option, with #Postgres as the interface.

    📖 hubs.la/Q04mQNSw0

    #PostgreSQL #ApacheIceberg #DataEngineering

  11. Cold data is read-only - that's the assumption baked into basically every tiering solution on the market. ColdFront breaks it.

    UPDATE & DELETE on archived rows work through standard SQL. A GDPR deletion on five-year-old events is a single DELETE statement. No restore cycle.

    DuckDB in-process, Apache Iceberg on any S3. Stock unpatched #PostgreSQL 16/17/18. Beta now, PostgreSQL License. Led by Jimmy Angelakos.

    📖 github.com/pgEdge/coldfront

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  12. Cold data is read-only - that's the assumption baked into basically every tiering solution on the market. ColdFront breaks it.

    UPDATE & DELETE on archived rows work through standard SQL. A GDPR deletion on five-year-old events is a single DELETE statement. No restore cycle.

    DuckDB in-process, Apache Iceberg on any S3. Stock unpatched #PostgreSQL 16/17/18. Beta now, PostgreSQL License. Led by Jimmy Angelakos.

    📖 github.com/pgEdge/coldfront

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  13. pgEdge ColdFront: #PostgreSQL data tiering. Hot data in the heap, cold to Apache Iceberg on S3 - up to 90% lower storage cost.

    The cold tier is writable. UPDATE & DELETE on cold rows work in standard SQL. No restore cycle, no rehydration. No app changes.

    DuckDB runs in-process. No daemon, no sidecar. PostgreSQL License, beta now. Led by @vyruss.

    Press release: 👉 pgedge.com/press-releases/pged

    GitHub: 🔗 github.com/pgEdge/coldfront

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  14. pgEdge ColdFront: #PostgreSQL data tiering. Hot data in the heap, cold to Apache Iceberg on S3 - up to 90% lower storage cost.

    The cold tier is writable. UPDATE & DELETE on cold rows work in standard SQL. No restore cycle, no rehydration. No app changes.

    DuckDB runs in-process. No daemon, no sidecar. PostgreSQL License, beta now. Led by @vyruss.

    Press release: 👉 pgedge.com/press-releases/pged

    GitHub: 🔗 github.com/pgEdge/coldfront

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  15. I'lll spare you the "I did a thing" cliché and just say I'm really proud to share what I've been building lately: ColdFront is live.

    Figuring out how to move data to while keeping it queryable and updatable as if it were still local, without requiring any application code changes, is exactly the kind of hard problem I enjoy.

    Check out the link below to see how it all works under the hood.

    hubs.la/Q04lS0fb0

  16. I'lll spare you the "I did a thing" cliché and just say I'm really proud to share what I've been building lately: #pgEdge ColdFront is live.

    Figuring out how to move #PostgreSQL data to #S3 while keeping it queryable and updatable as if it were still local, without requiring any application code changes, is exactly the kind of hard problem I enjoy.

    Check out the link below to see how it all works under the hood.

    hubs.la/Q04lS0fb0

    #Postgres #OpenSource #ApacheIceberg #Analytics #LLM #AI

  17. 🧊 pgEdge ColdFront beta is out - transparent data tiering for #PostgreSQL. Fully writable cold tier.

    Hot data stays in the heap. Cold data moves to Iceberg on S3 at up to 90% lower cost. UPDATE & DELETE on archived rows, same SQL. No rehydration. No code changes.

    DuckDB runs in-process - no daemon, no RPC. C extension routes DML to the correct tier transparently.

    Development by Jimmy Angelakos. Blog by Antony Pegg: 📖 pgedge.com/blog/introducing-co

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  18. 🧊 pgEdge ColdFront beta is out - transparent data tiering for #PostgreSQL. Fully writable cold tier.

    Hot data stays in the heap. Cold data moves to Iceberg on S3 at up to 90% lower cost. UPDATE & DELETE on archived rows, same SQL. No rehydration. No code changes.

    DuckDB runs in-process - no daemon, no RPC. C extension routes DML to the correct tier transparently.

    Development by Jimmy Angelakos. Blog by Antony Pegg: 📖 pgedge.com/blog/introducing-co

    #OpenSource #DataEngineering #ApacheIceberg #DuckDB

  19. DuckDB Labs released #DuckLake 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.

    Key features:
    • catalog-stored small updates
    • improved sorting and partitioning
    • compatibility with Iceberg-style data features

    Learn more ⇨ bit.ly/48PsPIS

    #InfoQ #DuckDB #ApacheIceberg #AI #DataLake #DataStorage

  20. DuckDB Labs released 1.0 - a data lake format that stores table metadata in a SQL database, rather than spreading it across object storage files.

    Key features:
    • catalog-stored small updates
    • improved sorting and partitioning
    • compatibility with Iceberg-style data features

    Learn more ⇨ bit.ly/48PsPIS

  21. Lakehouse architectures allow multiple engines to run on shared data through open table formats like #ApacheIceberg.

    But #SQL identifier resolution and catalog naming rules differ across engines - creating hidden interoperability failures.

    In this #InfoQ article, Maninder Parmar explains why enforcing consistent naming conventions and cross-engine validation is critical.

    📰 Read now: bit.ly/4902zeH

    #RelationalDatabases #DataLake

  22. Lakehouse architectures allow multiple engines to run on shared data through open table formats like .

    But identifier resolution and catalog naming rules differ across engines - creating hidden interoperability failures.

    In this article, Maninder Parmar explains why enforcing consistent naming conventions and cross-engine validation is critical.

    📰 Read now: bit.ly/4902zeH

  23. #Pinterest launched a next-gen CDC-based ingestion framework.

    Using #ApacheKafka, #ApacheFlink, #ApacheSpark & #ApacheIceberg, they achieved:
    • Latency cut from 24+ hours to 15 minutes
    • Processing of only changed records
    • Support for incremental updates & deletions
    • Petabyte-scale data across 1,000+ pipelines

    Win: optimized cost & efficiency!

    Read the architectural deep dive on InfoQ 👉 bit.ly/4rMJB2H

    #SoftwareArchitecture #ChangeDataCapture

  24. launched a next-gen CDC-based ingestion framework.

    Using , , & , they achieved:
    • Latency cut from 24+ hours to 15 minutes
    • Processing of only changed records
    • Support for incremental updates & deletions
    • Petabyte-scale data across 1,000+ pipelines

    Win: optimized cost & efficiency!

    Read the architectural deep dive on InfoQ 👉 bit.ly/4rMJB2H

  25. #AWS announced 2 new capabilities for #S3Tables!

    🔹 Intelligent-Tiering storage class that automatically optimizes costs based on access patterns
    🔹 Replication support that keeps Apache Iceberg table replicas consistent across AWS regions and accounts - no manual syncing required

    Find out more: bit.ly/4qgRn3Y

    #CloudComputing #S3 #ApacheIceberg #InfoQ

  26. announced 2 new capabilities for !

    🔹 Intelligent-Tiering storage class that automatically optimizes costs based on access patterns
    🔹 Replication support that keeps Apache Iceberg table replicas consistent across AWS regions and accounts - no manual syncing required

    Find out more: bit.ly/4qgRn3Y

  27. Преодоление разрыва между озерами данных и хранилищами данных

    ​Системы хранения данных типа «озера данных» сочетают в себе гибкость озер данных с надежностью, производительностью и возможностями управления, характерными для хранилищ данных.

    В современных аналитических системах компании в значительной степени полагаются на озера данных...

    #DST #DSTGlobal #ДСТ #ДСТГлобал #озёраданных #хранилищаданных #lakehouse #ApacheIceberg #Метаданные #Кэширование

    Источник: dstglobal.ru/club/1144-preodol

  28. #DuckDB now supports end-to-end interaction with Iceberg REST Catalogs directly in the browser - no infrastructure setup required.

    With DuckDB-Wasm, users can query, read, and write Iceberg tables seamlessly.

    Learn more: bit.ly/4qCTYoF

    #DataAnalytics #WebAssembly #ApacheIceberg #AI #InfoQ

  29. now supports end-to-end interaction with Iceberg REST Catalogs directly in the browser - no infrastructure setup required.

    With DuckDB-Wasm, users can query, read, and write Iceberg tables seamlessly.

    Learn more: bit.ly/4qCTYoF

  30. Cloudflare has just launched the open beta of its Cloudflare Data Platform - a managed service for ingesting, storing & querying analytical data tables using open standards like Apache Iceberg.

    🔍 Dive into the key insights on #InfoQbit.ly/49y1tIa

    #CloudComputing #DataLake #DataAnalytics #ApacheIceberg #Cloudflare

  31. Cloudflare has just launched the open beta of its Cloudflare Data Platform - a managed service for ingesting, storing & querying analytical data tables using open standards like Apache Iceberg.

    🔍 Dive into the key insights on bit.ly/49y1tIa

  32. #Netflix scaled 𝐌𝐮𝐬𝐞 to handle 𝐭𝐫𝐢𝐥𝐥𝐢𝐨𝐧-𝐫𝐨𝐰 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬!

    ➡️ Muse helps teams see which artwork & videos resonate with audiences.
    ➡️ To keep up with demand, Netflix 𝐫𝐞𝐝𝐞𝐬𝐢𝐠𝐧𝐞𝐝 𝐭𝐡𝐞 𝐝𝐚𝐭𝐚 𝐥𝐚𝐲𝐞𝐫, cutting query latencies by ~50% while keeping results accurate and responsive.

    🔗 Learn more: bit.ly/4gG3HGU

    #SoftwareArchitecture #DataBase #ApacheIceberg #InfoQ

  33. scaled 𝐌𝐮𝐬𝐞 to handle 𝐭𝐫𝐢𝐥𝐥𝐢𝐨𝐧-𝐫𝐨𝐰 𝐝𝐚𝐭𝐚𝐬𝐞𝐭𝐬!

    ➡️ Muse helps teams see which artwork & videos resonate with audiences.
    ➡️ To keep up with demand, Netflix 𝐫𝐞𝐝𝐞𝐬𝐢𝐠𝐧𝐞𝐝 𝐭𝐡𝐞 𝐝𝐚𝐭𝐚 𝐥𝐚𝐲𝐞𝐫, cutting query latencies by ~50% while keeping results accurate and responsive.

    🔗 Learn more: bit.ly/4gG3HGU

  34. Watching the re-indexing of an archival catalog backup of AtoM, I realized:

    Indices populated with 18751 documents in 164.84 seconds.

    19k Objects?
    Thats /nothing/ for a regular #bigDATA tech-tool. This is peanuts.

    400.000 Objects?
    Millions?! - According to documentation of #ApacheIceberg #ObjectStore #Redis #KeyDB, etc: **easy**

    #DLTP & #GLAM: Storing and using those "objects" in key/value annotated filesystems with bigDATA tools:

    **FUN!!**

  35. Amazon #S3 now supports sort and z-order compaction for #ApacheIceberg tables, promising reduced scan times & lower engine costs.

    Available for both S3 Tables and traditional S3 buckets via AWS Glue Data Catalog optimization.

    Dive into the details: bit.ly/3GyjxWQ

    #InfoQ #AWS #DataAnalytics

  36. Amazon now supports sort and z-order compaction for tables, promising reduced scan times & lower engine costs.

    Available for both S3 Tables and traditional S3 buckets via AWS Glue Data Catalog optimization.

    Dive into the details: bit.ly/3GyjxWQ

  37. 📢 Behold, the earth-shattering breakthrough of Nimtable: a web UI to *click* on Apache Iceberg tables! 🙄 Presumably because using command line tools is an insurmountable task for mere mortals. Or maybe it’s just a clever way to make clicking around a web interface the new rocket science. 🚀
    github.com/nimtable/nimtable #Nimtable #ApacheIceberg #WebUI #Innovation #TechNews #ClickAndGo #HackerNews #ngated

  38. 📢 Behold, the earth-shattering breakthrough of Nimtable: a web UI to *click* on Apache Iceberg tables! 🙄 Presumably because using command line tools is an insurmountable task for mere mortals. Or maybe it’s just a clever way to make clicking around a web interface the new rocket science. 🚀
    github.com/nimtable/nimtable #Nimtable #ApacheIceberg #WebUI #Innovation #TechNews #ClickAndGo #HackerNews #ngated

  39. "Centralize Your Data Lake: Apache Polaris Supports Apache Iceberg and Now Delta Lake"

    BTW 'Polaris' used to be the name of the UK nuclear deterrent pre 1996. 😬

    snowflake.com/en/engineering-b

    #ApacheIceberg #ApachePolaris #DataLake

  40. "Centralize Your Data Lake: Apache Polaris Supports Apache Iceberg and Now Delta Lake"

    BTW 'Polaris' used to be the name of the UK nuclear deterrent pre 1996. 😬

    snowflake.com/en/engineering-b

    #ApacheIceberg #ApachePolaris #DataLake

  41. What happens when you marry database with ? you could query huge datasets fast and with 10x cheaper storage. Sounds promising, right?

    Join me tomorrow on the live stream to find out!

    May 20th, 11am PT / 20:00 CET:
    youtube.com/watch?v=VeyTL2JlWp0

  42. What happens when you marry #ClickHouse database with #ApacheIceberg? you could query huge datasets fast and with 10x cheaper storage. Sounds promising, right?

    Join me tomorrow on the live stream to find out!

    May 20th, 11am PT / 20:00 CET:
    youtube.com/watch?v=VeyTL2JlWp