#datawrangling — Public Fediverse posts

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

I've been starting to add my scripts onto #codeberg for processing my wildcounts biodiversity monitoring data. I've added an R script I just finished that adds the recording date and time to audio recordings made with an Apple Watch.

I needed this for a survey I had to do last month using my Apple Watch when my phone was malfunctioning. I started my GPS track (with my Garmin GPS) then spoke each of my observations into the voice recorder app on my watch, one recording per observation.

To geotag all those observations, I needed to know exactly when each recording was made, to match it with my GPS track. This was a good deal more complicated than I expected (because a lot of modern Apple software is too dumb to be really useful).

(Since my watch couldn't connect to my phone, it stored up the recordings then synced them to iCloud later. I was surprised to discover that all the file creation dates and media creation dates in the audio files were the times they synced to iCloud, not when they were recorded. That required an unexpected journey into the innards of the VoiceMemos app sqlite database using Terminal to extract the real recording times.)

Anyway, I now have an R script that automates all this.

https://codeberg.org/wildcounts/utilities/

#wildcounts #RStats #datawrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

Missing [Survey, etc] Data Can Be A Geographic Phenomenon
--
https://doi.org/10.1080/24694452.2026.2640220 <-- shared paper
--
#GIS #mapping #spatial #DataScience #missing #data #spatial #AAG #autocorrelation #geographicallyweightedregression #GWR #imputation #missingdata #survey #surveynonresponse #incomplete #surveyquestions #ethnicity #income #spatialdata #alldataisspatial #UK #FinancialLives #geography #spatialanalysis #geostatistics #location #imputing #statistics #dataset #DataImputation #MissingData #DataCleaning #DataPreprocessing #DataWrangling #DataQuality #DataEngineering #FinancialData #FinancialAnalytics #FinincialModeling #FinDataScience

I'm reviewing the content and computational skills taught in a Research Data Skills module.
#data #DataWrangling

Apparently my knowledge of perl is pretty outdated nowadays.
#perl #perl4datascience

But am I crazy to want to teach them to use the *nix command line, and grep, awk and sed???
#DataScience #ChangeMyMind

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

Turns out, quick coding tasks are not actually quick but can eat up all day. Who would have thought... Not me, apparently... And why is this model so slow? #R #stats #DataWrangling

These six Pandas functions power most data wrangling: read_csv to load data, info to inspect, isnull for missing values, drop to clean, groupby to summarize, and apply to customize logic. Master these and you control your dataset.

📕 https://ebokify.com/python

#Python #Pandas #DataWrangling #DataAnalytics #MachineLearning #DataScience #CodingTips

#TIL about #querychat (https://posit-dev.github.io/querychat/), it is simply amazing to get quickly a #Shiny app that allows people to explore data using a chatbot interface. There are #Python and #RStats implementations.

#LLM #AI #Data #EDA #DataWrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

It's not hard to bill-lief that our technically-inclined volunteers stay afloat using @duckdb.org for their #DataWrangling needs

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

Turning NASA Wake-up Calls into data

by @beet_keeper

For a while back then I was into space flight again. Scientists, science communicators, and engineers were all excited for a new era of rocket launches and the potential unification of the human race as we look towards the future.

During that time I discovered Colin Fries’ work in the NASA History Division to document all NASA “Wake-up calls”. A wake-up call is simply a piece of music used to wake astronauts on missions, a different piece of music, daily, for the duration of the flight.

Take, for example, the last Space Shuttle mission (Space Transportation System) STS-135; it was in flight for 13 days, and the wake-up call on day one was Coldplay’s Viva la Vida, while on day 13 it was Kate Smith singing God Bless America.

As a huge music buff who has the radio or music television on 18 hours a day, I really wanted to delve into this further. While Colin’s work is great, it’s just a PDF file (@wtfpdf). A PDF is not an ideal file format for querying data and gleaning new insights. So, while I wanted to explore it, I first decided to turn it into a true dataset. The result was a set of resources, a website, a JSON, a CSV, and an SQLite database which are each more functional and more maintainable over time.

Lets take a look at the results and https://nasawakeupcalls.github.io below!

#ApacheTika #Code #Coding #DataWrangling #Datasette #DatasetteLite #DH #DigitalHumanities #glam #harkive #NASA #NASAWakeUpCall #NASAWakeUpCalls #OpenData #PersonalProjects #Science #Space #SpaceHistory #Twitter #WakeUpCall

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

Oh, the endless saga of trying to bridge the gap between #Excel and #Markdown 😩. Apparently, the internet gods have decided that converting tables is a task only for the truly worthy—or those with a spare afternoon to wrestle with server errors 🤦‍♂️. Meanwhile, your data waits, trapped in spreadsheet purgatory.
https://thisdavej.com/copy-table-in-excel-and-paste-as-a-markdown-table/ #Conversion #Struggles #DataWrangling #SpreadsheetPurgatory #TechFrustration #HackerNews #ngated

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

"Why Python for Data Analysis?"
https://wesmckinney.com/book/preliminaries

from "#Python for #DataAnalysis: #DataWrangling with #pandas, #NumPy & #Jupyter"

#Python for #DataAnalysis: #DataWrangling with #pandas, #NumPy, and #Jupyter 3rd edition openly available at https://wesmckinney.com/book/

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

@datadon

Unfilled cells influence models.
"Handling Missing Data in Machine Learning": https://ml-nn.eu/a1/51.html by Calin Sandu @mlnn

#missingData #bias #wealth #dataQuality #complexity #dataDev #machineLearning #dataPrep #EDA #dataWrangling

📢 Join us for a hands-on tutorial on effective data wrangling with R!

🌟 Learn essential techniques using tidyverse packages.

🗓️When: September 30, 2024 at 7PM CET
💥RSVP: 🔗 https://www.meetup.com/rladies-rome/events/303481357

#DataWrangling #RStats #DataScience
@fgazzelloni @silacos @rafagrlucas

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling

a simple script to trim a set of audio files to a maximum length (e.g. 2 minutes each): https://gist.github.com/danstowell/84eff5456ce6b3c4d1c82f29de374790 #python #soundscape #datawrangling

Say goodbye to the frustrations of copying and pasting data to and from R with Datapasta from @milesmcbain! Get the package now: https://milesmcbain.github.io/datapasta/ #DataWrangling #Rstats

Sometimes you get data in less than optimal format, e.g. as a png of a figure 😭... In that case https://cran.r-project.org/web/packages/metaDigitise/vignettes/metaDigitise.html might be the rescue. #rstats #ohno #datawrangling