#siegfried — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #siegfried, aggregated by home.social.
-
Declarative all the way down: Building PRONOM signatures with JSONID
by @beet_keeperPRONOM signatures are a form of declarative language, you describe the anticipated behavior in PRONOM’s regular expression syntax and tools like DROID, FIDO, and Siegfried will interpret those instructions and attempt to match them against different files to return a file format identification.
Normally, you will write PRONOM signatures by hand but doing so for file formats based on other file format building blocks can lead to inconsistencies. Bertrand Caron previously also recognized this in the XML formats that are described with PRONOM signatures on Wikidata.
XML can use single quotes ‘ (hex: 0x27) and double quotes ” (hex: 0x22) for attribute data, and so, do we make a PRONOM signature with multiple sequences anticipating the use of either?
The answer is more often than not likely to be yes, because the appearance of these values are often helpful for identifying boundaries for strings that we know must exist.
But the more file formats that we need to add to PRONOM that are based on foundational formats like XML, or JSON, or similar, the more inconsistencies will creep in, such as sequences that are looking specifically for one byte sequence over another.
The issue extends further if file formats allow data to appear at the beginning of file, or we need to account for a variable amount of white-space, or we want to start thinking about multi-byte character encoding.
We can, and in the XML issue described by myself and Caron, I think the recommendation is very much to create editorial standards for signatures for file formats based on other baseline, structured data formats like XML, JSON, YAML, and so on.
And standards are well and good, but what if tooling could help us?
For JSON this is exactly what I have tried to do in JSONID.
What does this feature look like? And what does it get us? Let’s take a look.
#declarativeProgramming #digipres #DigitalPreservation #DROID #FIDO #FileFormatIdentification #FileFormats #JSON #jsonid #JSONL #NTTW #NTTW9 #PRONOM #RDM #ResearchData #siegfried #StructuredData #structuredText #TOML #YAML -
Ich bin wegen einer Internetpanne zurzeit ohne Wifi und ohne Festnetz und ich kann meine Arbeiten nicht mehr von meinem Computer aus versenden. Klar, das Smartphone bedient sich bei G4. Mir ist diese digitale Abhängigkeit nicht geheuer. Das erinnert mich an das Lindenblatt auf Siegfrieds Rücken. Hagen freut sich...
#wifi #Internet #hagen #siegfried #cartoon #digital -
Ok, Boomer „Siegfried“! 🤭🙌🌱
@NationalerFortschritt @Bugspriet @Terranerin
-
Wow! Er lige kommet igennem trilogien #Siegfried af #AlexAlice. En superfedt cocktail af #NordiskMytologi og historien om nibelungens ring. Super fed stil. Oh the feels.
-
Wagner-Zeit im Staatstheater Saarbrücken. Gleich beginnt #Siegfried
-
Hagen von Nestlé,
vormals Hagen von Tronje her, Hüter der heiligen Quellen
#nestle #siegfried #hagen #nibelungen #cartoons #carton #trinkwasser -
wikidata + mediawiki = wikidata + provenance == wikiprov
by @beet_keeperToday I want to showcase a Wikidata proof of concept that I developed as part of my work integrating Siegfried and Wikidata.
That work is wikiprov a utility to augment Wikidata results in JSON with the Wikidata revision history.
For siegfried it means that we can showcase the source of the results being returned by an identification without having to go directly back to Wikidata, this might mean more exposure for individuals contributing to Wikidata. We also provide access to a standard permalink where records contributing to a format identification are fixed at their last edit. Because Wikidata is more mutable than a resource like PRONOM this gives us the best chance of understanding differences in results if we are comparing siegfried+Wikidata results side-by-side.
I am interested to hear your thoughts on the results of the work. Lets go into more detail below.
#Code #CreativeCommons #Data #digitalLiteracy #Golang #mediawiki #OpenData #OpenSource #provenance #reification #reify #siegfried #SPARQL #wikibase #wikidata
-
simpledroid: completing the circle
It’s nearing the end of 2024 and that must mean a PRONOM hackathon as part of the World Digital Preservation Day (#WDPD2024).
My contribution is a follow-up on my work earlier in the year to produce a valid DROID signature file from Wikidata in wddroidy.
simpledroid is available on GitHub and creates a simple DROID signature file from PRONOM itself, creating a scripted pathway to create a signature file using official PRONOM data that doesn’t require the current PRONOM database and its legacy stored procedures.
It also does away with a lot of the excess data in the current DROID signature file which was previously an optimization for its Boyer Moore Horspool search algorithm, as described by Matthew Palmer.
The primary reason for simpledroid was to complete the circle on my previous efforts and to prove that it was possible to create a simplified signature file and for it to work with DROID. The result is about 80-90% there, with only a few skeleton files that remain unidentified – it should only require a small amount of forensic research to determine the reason.
The output provides a way for simplifying the signature file generation process, offering new opportunities to create alternative versions, or filtering what’s already there, e.g. filtering out any signatures that aren’t explicitly for image identification, e.g. in a digitization workflow.
It may provide another way into PRONOM data for those who might look at DROID first as well as opening up different ways to modify and test signatures.
It is possible to see in the reference output, that the signatures are much easier to understand via this simplified DROID file.
simpledroid outputs a file with a smaller footprint than the current file:
1.2M DROID_SignatureFile_Simple_2024-11-11T12-29-22Z.xml3.4M DROID_SignatureFile_V118.xmlIt also contains all of the file classification data e.g.
FormatType="Video"from PRONOM that will be added into DROID in a future release (and is already available in Siegfried).Unlike the wddroidy work, priorities have also been added to the signature file so the mechanics of the signature file are pretty close to the official version (DROID uses the signature sequence and offsets to identify a file, but it then uses a priority to determine what results to display to the user where there may otherwise be positive matches for formats that provide the foundation for another, e.g. how XML forms the basis of SVG or XHTML.
It might be possible to remove some data around minimum and maximum offsets in the new file after discovering that simplified droid syntax requires curly bracket syntax at the beginning and end of sequences to mimic the same behavior, e.g.
With a
BOFoffset,min_offset = 2, and signature =BADF00D1, the signature needs to become{2}BADF00D1to work.The code is pretty straightforward and uses a few tricks to output XML sensibly without having to build the document’s tree (DOM) in a more verbose way. There are probably a few other shortcuts I’d fix with time if the code was ever useful, including improving variable naming and adding tests.
I’m not sure this code will ever be needed, or used by anyone, but for a quick hack and a quick proof of concept, it felt good to put it out there. Maybe someone will look at this or the wddroidy work and see there may be a way to federate different sources of signature information together into something DROID can use. Or it might be a useful demonstration to the DROID team that allows them to simplify PRONOM’s database and output mechanisms in a way that remains compatible with existing tools.
Previous research week work
My previous work for PRONOM research week includes a dashboard and API for getting more information out of PRONOM, including listings of those records still requiring descriptions or signatures. You may find that work interesting and it is available at https://pronom.ffdev.info and https://api.pronom.ffdev.info.
And if you want to get in on the signature development work, signature development utility 2.0 (https://ffdev.info) was also a previous effort of mine for research week 2020 and will hopefully also benefit from outputting DROID’s simplified syntax.
A week of file formats
Of course with World Digital Preservation Day, file formats were pretty popular.
Andrew Jackson attempted to calculate how many distinct formats might be out there using methods used to calculate ecological diversity.
Amanda Tome described the scope of their work and shared a number of useful resources including useful links to the PRONOM starter pack and to the PRONOM drop-in sessions.
You might also find out a bit more about yourself by playing this File Format Dating Game from Lotte Wijsman and colleagues: Susanne van den Eijkel, Anton van Es, Elaine Murray, Francesca Mackenzie, Ellie O’Leary, and Sharon McMeekin. (I ended up on a date with FASTA (FDD000622) in my first play-through!)
Not specifically for WDPD, but in the same week I also enjoyed this presentation from Ange Albertini looking at different ways of identifying file formats. One big take away for me was thinking about how to get more forensic information out of a file format identification. DROID doesn’t tell us a lot, but is there a world in which one day it could?
Let me know if you find any of this work useful at all; and good luck on your file format endeavors this week.
#digipres #DigitalPreservation #DROID #FileFormats #PRONOM #Python #siegfried #SkeletonTestCorpus #WDPD #WDPD2024
-
I just added a wrapper for Google's magika detector to Apache Tika.
You can now get detection from `file`, `siegfried` and `magika` (and of course, Tika) in a single parse.