home.social

#dataingestion — Public Fediverse posts

Live and recent posts from across the Fediverse tagged #dataingestion, aggregated by home.social.

fetched live
  1. Meles – công cụ mã nguồn mở AGPL bằng Java cho thu thập & lưu trữ dữ liệu. Hỗ trợ nguồn đa dạng (serial, TCP, I2C), xử lý bằng script XML, tính toán lambda, gửi qua MQTT, lưu vào Postgres/SQLite, hiển thị Grafana, cảnh báo qua draw.io (email/matrix). Chưa có GUI, nhưng có thể điều khiển bằng telnet, matrix, email. #OpenSource #Meles #DataIngestion #DAQ #Java #IoT #MãNguồnMở #DữLiệu

    reddit.com/r/opensource/commen

  2. The Hadoop ecosystem comprises various tools and frameworks designed to handle large-scale data processing and analytics. Let's discuss the core components, namely Hadoop, HBase, and Hive, along with other significant tools such as Pig, Sqoop, Flume, Oozie, and Zookeeper.

    linuxexpert.org/so-you-wanna-d

    #Hadoop #HBase #Hive #BigData #HadoopEcosystem #HDFS #MapReduce #YARN #Pig #Sqoop #Flume #Oozie #Zookeeper #DataProcessing #DataAnalytics #DataWarehousing #ETL #DataIngestion #Security

  3. I discovered a crazy bug today trying to import data from an Excel file into a Laravel app.

    Modern Excel files are actually zip files containing several XML files in a specific directory structure, but it seems that within the zip file the "directory structure" is not real, so each XML file just has some directory structure like stuff prepended to its file name.

    In the particular Excel file I'm importing, the XML file that knows about the different worksheets in the Excel workbook was saved with the "/" characters in its directory structure as backslashes. This seems very unexpected, none of the Excel packages I tried for PHP could read it.

    In the end I used PHP's built-in ZipArchive class to rename the XML file in the Excel archive, then it could be imported properly.

    That wasn't how I expected to spend my afternoon!

    #UntrustedData #PHP #DataIngestion #WeirdBugs #DeveloperLife

  4. Equalum grabs $18M Series B to help companies ingest data faster - Equalum, an Israeli startup, which helps companies gather data from a variety of enterprise sources,... more: feedproxy.google.com/~r/Techcr #planveninvestments #dataingestion #enterprise #startups #funding #equalum #tc