#pandas — Public Fediverse posts

Profile raw CSVs from ETL or APIs: this 282-line pandas script flags outliers via z-score (std thresh) and outputs an HTML report. Intermediate level, ready to run. https://www.valtersit.com/python/pandas-dataframe-profiler-and-outlier-detector/ #python #data #pandas

If I had to explain my job... #IOER_FDz .. I'd just show this workflow from the last two days. It's a great example of building reproducible, automated documentation. 🚀

1. Start: A project schema, drafted by colleagues in Excel.

2. Problem: Excel isn't reproducible and is hard to use as a single source of truth for a database.
Solution: I migrated the schema to the universal Protobuf format, making it version-controlled and language-neutral. [1]

3. Problem: Protobuf definitions aren't easy for everyone to read.
Solution: Using an existing #CI/#CD workflow from [3] (cheers @mcnesium), I put the schema on a documentation website. Now it's accessible and legible. [4]

4. Problem: Colleagues still needed a familiar template for data collection.
Solution: Two Python scripts using #pandas now auto-generates both `.xlsx` and open-source `.ods` schema [5] and templates [6] directly from the Protobuf single-point-of-truth, and make all of these available for download, too.

5. Problem: The text-based schema wasn't visual enough.
Solution: Added a script [7] to the CI that transforms the Protobuf files into a #MermaidJS class diagram, rendered directly on the docs site. [8]

6. Problem: The diagram was too narrow in the site's layout.
Solution: The script now generates a second, wide-format version of the diagram for a dedicated fullscreen view. [9]

7. Problem: The static diagram was hard to edit or restyle.
Solution: The CI now generates a shareable link to the Mermaid Live Editor [10]. It reads our latest diagram file, compresses it (using pako/zlib), and bakes it into the URL. Thanks to a code snippet from a friendly stranger on GitHub [11], anyone can now open the *latest* version of the schema in the editor with a single click.

From a static Excel file to version-controlled, multi-format, visual, and interactive documentation, all fully automated. That's the job.

#Automation #GitLab #Python #Protobuf #OpenData #Documentation

@ioer

[1]: https://gitlab.vgiscience.de/casereports/rwlschema
[2]: https://docs.casereports.fdz.ioer.info/
[3]: https://lbsn.vgiscience.org
[4]: https://docs.casereports.fdz.ioer.info/protobuf/
[5]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_schema.py?ref_type=heads
[6]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_template.py?ref_type=heads
[7]: https://gitlab.vgiscience.de/casereports/rwlschema/-/blob/main/resources/generate_proto_uml.sh?ref_type=heads
[8]: https://docs.casereports.fdz.ioer.info/konzept/
[9]: https://docs.casereports.fdz.ioer.info/uml/fullscreen.html
[10]: https://mermaidlive.com/play
[11]: https://github.com/mermaid-js/mermaid-live-editor/discussions/1291#discussioncomment-6837936

#Climatechange is depleting #plant #nutrition, threatening #herbivores from #grasshoppers to giant #pandas

Rising #CO2 levels are driving faster plant growth but reducing #nutrient density, with alarming impacts across #ecosystems.

https://www.nationofchange.org/2024/12/30/climate-change-is-depleting-plant-nutrition-threatening-herbivores-from-grasshoppers-to-giant-pandas/

A zoo in the city of Kobe said Wednesday that it will delay the return of a giant female panda to China by another year as the animal undergoes medical treatment. https://www.japantimes.co.jp/news/2023/12/28/japan/kobe-zoo-panda-return/?utm_content=buffer156c6&utm_medium=social&utm_source=mastodon&utm_campaign=bffmstdn #japan #kobe #pandas #tantan

#Introduction: This account is all about #earlymodern #diplomacy. We toot about the ups & downs of editing a #handbook on #emdiplomacy & give insights into the #LifeOfTheEditors. We introduce our great authors, reveil #emdiplomacysSecrets, present #emdiplomacysFaces, discuss #emdiplomacysSources. We share reading recommendations & #emdiplomacy related research. We love to connect (with) #researchers & #histodons from the field of #emdiplomacy. And there will also be some #Pandas, of course.

#Introduction: This account is all about #earlymodern #diplomacy. We toot about the ups & downs of editing a #handbook on #emdiplomacy & give insights into the #LifeOfTheEditors. We introduce our great authors, reveil #emdiplomacysSecrets, present #emdiplomacysFaces, discuss #emdiplomacysSources. We share reading recommendations & #emdiplomacy related research. We love to connect (with) #researchers & #histodons from the field of #emdiplomacy. And there will also be some #Pandas, of course.

#Introduction: This account is all about #earlymodern #diplomacy. We toot about the ups & downs of editing a #handbook on #emdiplomacy & give insights into the #LifeOfTheEditors. We introduce our great authors, reveil #emdiplomacysSecrets, present #emdiplomacysFaces, discuss #emdiplomacysSources. We share reading recommendations & #emdiplomacy related research. We love to connect (with) #researchers & #histodons from the field of #emdiplomacy. And there will also be some #Pandas, of course.

#Introduction: This account is all about #earlymodern #diplomacy. We toot about the ups & downs of editing a #handbook on #emdiplomacy & give insights into the #LifeOfTheEditors. We introduce our great authors, reveil #emdiplomacysSecrets, present #emdiplomacysFaces, discuss #emdiplomacysSources. We share reading recommendations & #emdiplomacy related research. We love to connect (with) #researchers & #histodons from the field of #emdiplomacy. And there will also be some #Pandas, of course.

#Introduction: This account is all about #earlymodern #diplomacy. We toot about the ups & downs of editing a #handbook on #emdiplomacy & give insights into the #LifeOfTheEditors. We introduce our great authors, reveil #emdiplomacysSecrets, present #emdiplomacysFaces, discuss #emdiplomacysSources. We share reading recommendations & #emdiplomacy related research. We love to connect (with) #researchers & #histodons from the field of #emdiplomacy. And there will also be some #Pandas, of course.