#pronom — Public Fediverse posts

Hi #PRONOM folks, do you know if that resource https://www.nationalarchives.gov.uk/aboutapps/fileformat/pdf/pronom_4_info_model.pdf is still up to date, or if there is one that replaces it?

#digipres

A couple exciting updates in the #fileformat space this week. First #PRONOM released a new signature, v121, with 29 new PUIDs, 29 new signatures and 18 updates. Including an APK signature! @BertrandCaron #digipres https://www.nationalarchives.gov.uk/aboutapps/pronom/release-notes.xml 1/2

simpledroid: completing the circle

It’s nearing the end of 2024 and that must mean a PRONOM hackathon as part of the World Digital Preservation Day (#WDPD2024).

My contribution is a follow-up on my work earlier in the year to produce a valid DROID signature file from Wikidata in wddroidy.

simpledroid is available on GitHub and creates a simple DROID signature file from PRONOM itself, creating a scripted pathway to create a signature file using official PRONOM data that doesn’t require the current PRONOM database and its legacy stored procedures.

It also does away with a lot of the excess data in the current DROID signature file which was previously an optimization for its Boyer Moore Horspool search algorithm, as described by Matthew Palmer.

The primary reason for simpledroid was to complete the circle on my previous efforts and to prove that it was possible to create a simplified signature file and for it to work with DROID. The result is about 80-90% there, with only a few skeleton files that remain unidentified – it should only require a small amount of forensic research to determine the reason.

The output provides a way for simplifying the signature file generation process, offering new opportunities to create alternative versions, or filtering what’s already there, e.g. filtering out any signatures that aren’t explicitly for image identification, e.g. in a digitization workflow.

It may provide another way into PRONOM data for those who might look at DROID first as well as opening up different ways to modify and test signatures.

It is possible to see in the reference output, that the signatures are much easier to understand via this simplified DROID file.

simpledroid outputs a file with a smaller footprint than the current file:

1.2M DROID_SignatureFile_Simple_2024-11-11T12-29-22Z.xml
3.4M DROID_SignatureFile_V118.xml

It also contains all of the file classification data e.g. FormatType="Video" from PRONOM that will be added into DROID in a future release (and is already available in Siegfried).

Unlike the wddroidy work, priorities have also been added to the signature file so the mechanics of the signature file are pretty close to the official version (DROID uses the signature sequence and offsets to identify a file, but it then uses a priority to determine what results to display to the user where there may otherwise be positive matches for formats that provide the foundation for another, e.g. how XML forms the basis of SVG or XHTML.

It might be possible to remove some data around minimum and maximum offsets in the new file after discovering that simplified droid syntax requires curly bracket syntax at the beginning and end of sequences to mimic the same behavior, e.g.

With a BOFoffset, min_offset = 2, and signature = BADF00D1, the signature needs to become {2}BADF00D1 to work.

The code is pretty straightforward and uses a few tricks to output XML sensibly without having to build the document’s tree (DOM) in a more verbose way. There are probably a few other shortcuts I’d fix with time if the code was ever useful, including improving variable naming and adding tests.

I’m not sure this code will ever be needed, or used by anyone, but for a quick hack and a quick proof of concept, it felt good to put it out there. Maybe someone will look at this or the wddroidy work and see there may be a way to federate different sources of signature information together into something DROID can use. Or it might be a useful demonstration to the DROID team that allows them to simplify PRONOM’s database and output mechanisms in a way that remains compatible with existing tools.

Previous research week work

My previous work for PRONOM research week includes a dashboard and API for getting more information out of PRONOM, including listings of those records still requiring descriptions or signatures. You may find that work interesting and it is available at https://pronom.ffdev.info and https://api.pronom.ffdev.info.

And if you want to get in on the signature development work, signature development utility 2.0 (https://ffdev.info) was also a previous effort of mine for research week 2020 and will hopefully also benefit from outputting DROID’s simplified syntax.

A week of file formats

Of course with World Digital Preservation Day, file formats were pretty popular.

Andrew Jackson attempted to calculate how many distinct formats might be out there using methods used to calculate ecological diversity.

Amanda Tome described the scope of their work and shared a number of useful resources including useful links to the PRONOM starter pack and to the PRONOM drop-in sessions.

You might also find out a bit more about yourself by playing this File Format Dating Game from Lotte Wijsman and colleagues: Susanne van den Eijkel, Anton van Es, Elaine Murray, Francesca Mackenzie, Ellie O’Leary, and Sharon McMeekin. (I ended up on a date with FASTA (FDD000622) in my first play-through!)

Not specifically for WDPD, but in the same week I also enjoyed this presentation from Ange Albertini looking at different ways of identifying file formats. One big take away for me was thinking about how to get more forensic information out of a file format identification. DROID doesn’t tell us a lot, but is there a world in which one day it could?

Let me know if you find any of this work useful at all; and good luck on your file format endeavors this week.

#digipres #DigitalPreservation #DROID #FileFormats #PRONOM #Python #siegfried #SkeletonTestCorpus #WDPD #WDPD2024

Last for today, but certainly not least: Ange Albertini presenting "Understanding file type identifiers"
@Ange
Warning: contains raw bytes

#Magika

https://github.com/google/magika

#hacklu2024 @hack_lu #Yara
#LibMagic #TrID #Yara #Magika #PeID #Pronom #FDD #ShareMime #DiE

Last for today, but certainly not least: Ange Albertini presenting "Understanding file type identifiers"
@Ange
Warning: contains raw bytes

#Magika

https://github.com/google/magika

#hacklu2024 @hack_lu #Yara
#LibMagic #TrID #Yara #Magika #PeID #Pronom #FDD #ShareMime #DiE

Last for today, but certainly not least: Ange Albertini presenting "Understanding file type identifiers"
@Ange
Warning: contains raw bytes

#Magika

https://github.com/google/magika

#hacklu2024 @hack_lu #Yara
#LibMagic #TrID #Yara #Magika #PeID #Pronom #FDD #ShareMime #DiE

Last for today, but certainly not least: Ange Albertini presenting "Understanding file type identifiers"
@Ange
Warning: contains raw bytes

#Magika

https://github.com/google/magika

#hacklu2024 @hack_lu #Yara
#LibMagic #TrID #Yara #Magika #PeID #Pronom #FDD #ShareMime #DiE

Last for today, but certainly not least: Ange Albertini presenting "Understanding file type identifiers"
@Ange
Warning: contains raw bytes

#Magika

https://github.com/google/magika

#hacklu2024 @hack_lu #Yara
#LibMagic #TrID #Yara #Magika #PeID #Pronom #FDD #ShareMime #DiE

Wikidata is a good service, Wikibase (on which Wikidata is built) is a better platform.

I have spoken before about its potential to be added into the file-format registry ecosystem in a federated model.

If we are to use it as a registry that can perhaps complement the pipelines going into PRONOM, e.g. in vendor’s digital preservation platforms such as the Rosetta Format Library, a Wikidata should be able to output different serializations of signature file for tools such as Siegfried, DROID or FIDO.

• Siegfried ✅: https://github.com/richardlehane/siegfried/wiki/Wikidata-identifier
• Fido ❌: I’ll need to revisit this!

And what about DROID?

Conversion to DROID

It’s not straightforward to say to a Wikibase/Wikidata Query Service, “output XML in the shape of a DROID signature file”, but it is straightforward to write a converter script.

I had this very thought last week while presenting with colleagues at a File Format Workshop at iPRES in Ghent.

It dawned on me that the conversion script would actually be simple thanks to a change in format to DROID whereby it can process all its own signatures, where previously it required DROID to pre-process them. It’s a long story, a more simple rendition is that DROID no longer requires DROID byte-code to record information about an identification pattern, and can instead store signatures in the attribute of a byte sequence element as-is, i.e. a PRONOM formatted regular expression from PRONOM itself, or Wikidata.

This realization resulted in my writing a conversion script (it took just over a half-day) during some down-time on the train home this past weekend.

The script is called wddroidy (after WD-40 🙄🥁) and can be found here.

Results

We can see using the skeleton suite from Richard Lehane’s Builder that we can positively identify files using the new signature file.

Links can also be made to work with Wikidata identifiers by modifying the PUID URL pattern in the DROID configuration, e.g. to:

http://wikidata.org/entity/%s

The screenshot below shows where in the dialog that setting is:

Reference signature file

A reference signature file can be found in the wddroidy repository here. There are approximately 8119 file formats listed and 8195 file format signatures for those.

NB. We know there are different issues with Wikidata including how to identify a “format” and the quality of the signatures. We capture some of these in a global repository: https://github.com/ffdev-info/wikidp-issues/issues

DROID simplified format

The real headline here might be how easy it was to create the output using the DROID simplified format.

I have spoken about it briefly before but not in any detail.

In-short DROID no longer uses its own byte-code encoding that included strange terms such as DefaultShift, Shift Byte, and SubSequence (instructions to DROID about how to perform Boyer Moore Horspool search). See below and note especially how the bytes are split in Shift Byte attributes and elements:

<?xml version="1.0" encoding="UTF-8"?><FFSignatureFile xmlns="http://www.nationalarchives.gov.uk/pronom/SignatureFile" Version="1" DateCreated="2024-09-23T18:16:09+00:00">  <InternalSignatureCollection>    <InternalSignature ID="1" Specificity="Specific">      <ByteSequence Reference="BOFoffset">        <SubSequence MinFragLength="0" Position="1" SubSeqMaxOffset="0" SubSeqMinOffset="0">          <Sequence>255044462D312E34</Sequence>          <DefaultShift>9</DefaultShift>          <Shift Byte="25">8</Shift>          <Shift Byte="50">7</Shift>          <Shift Byte="44">6</Shift>          <Shift Byte="46">5</Shift>          <Shift Byte="2D">4</Shift>          <Shift Byte="31">3</Shift>          <Shift Byte="2E">2</Shift>          <Shift Byte="34">1</Shift>        </SubSequence>      </ByteSequence>    </InternalSignature>  </InternalSignatureCollection>  <FileFormatCollection>    <FileFormat ID="1" Name="Development Signature" PUID="dev/1" Version="1.0" MIMEType="application/octet-stream">      <InternalSignatureID>1</InternalSignatureID>      <Extension>ext</Extension>    </FileFormat>  </FileFormatCollection></FFSignatureFile>

The updated format was made possible via Matt Palmer via his ByteSeek work, and can now except a regularly encoded PRONOM formatted regular expression (regex) in an attribute in the ByteSequence element. See here for a signature file equivalent to the above:

<?xml version="1.0" encoding="UTF-8"?><FFSignatureFile      xmlns="http://www.nationalarchives.gov.uk/pronom/SignatureFile" Version="1" DateCreated="2024-09-23T18:16:09+00:00">  <InternalSignatureCollection>    <InternalSignature ID="1" Specificity="Specific">      <ByteSequence Reference="BOFoffset" Sequence="255044462D312E34" Offset="0" />    </InternalSignature>  </InternalSignatureCollection>  <FileFormatCollection>    <FileFormat ID="1" Name="Development Signature" PUID="dev/1" Version="1.0" MIMEType="application/octet-stream">      <InternalSignatureID>1</InternalSignatureID>      <Extension>ext</Extension>    </FileFormat>  </FileFormatCollection></FFSignatureFile>

The format is much easier to read, and after a bit of time sitting with the DROID signature file format you realize it is fairly easy to output as well. I use some very rudimentary templates in wddroidy using  Python’s f-strings.

It means other sources of PRONOM encoded signatures can output much simpler signature files and they can be used by DROID. I myself need to add it to the signature development utility – this would allow the utility to run standalone on anyone’s PC.

One next step for this approach might be to confirm that it does work entirely as expected by extracting all of PRONOM’s signatures proper and performing a mapping to the simplified format – if we can match against all the skeleton files in the latest Builder release then we should be looking good!

Priorities

I am always reminded, but always forget about priorities! This is part of how DROID resolves a file format into a single identifier, e.g. where SVG can match XML, we often want the more specific format returned, and so a priority is used to prioritize that one over the other, resulting in a single unambiguous identification for the DROID user. It manifests in the signature file as:

Using the results

Using the results can be done for two things:

1. (Probably) There are a greater number of patterns in the Wikidata output than in PRONOM. If you have a file that remains unidentified, you can try the reference file for clues as to what it may be. I’d only use caution and investigate the exact byte sequence used for a match and understand its properties. I’d also check that the mapping also looks accurate, I’ve tried one or two runs using the identifier and it looks good, but there may still be mistakes.
2. For improving the quality of the sources in Wikidata. As you can see from the Skeleton suite there are a lot of gaps. We a) have a rough idea what these are, and b) know the identification doesn’t work via Wikidata. Why is that? Is the signature in Wikidata simply not good enough? Are patterns missing? Is there another error or issue we can help with given our expertise in file format identification?

Hacking wddroidy

You can hack wddroidy. Currently it allows you to limit the number of results returned, and also modify the ISO language code used by the tool. You can see this in the command line arguments:

python wddroidy.py --helpusage: wddroidy [-h] [--definitions DEFINITIONS] [--wdqs] [--lang LANG] [--limit LIMIT] [--output OUTPUT] [--output-date] [--endpoint ENDPOINT]create a DROID compatible signature file from Wikidataoptions: -h, --help show this help message and exit --definitions DEFINITIONS   use a local definitions file, e.g. from Siegfried --wdqs, -w live results from Wikidata --lang LANG, -l LANG change Wikidata language results --limit LIMIT, -n LIMIT   limit the number of resukts --output OUTPUT, -o OUTPUT   filename to output to --output-date, -t output a default file with the current timestamp --endpoint ENDPOINT, -url ENDPOINT   url of the WDQSfor more information visit https://github.com/ross-spencer/wddroidy

The actual SPARQL query used can be manually edited in the src folder. E.g. you can limit the query by format or family or classification. I provide some more inspiration in the Siegfried Wiki.

Let me know if it’s useful!

This is really just a quick hack and it needs a lot more testing to improve the quality of the output. Most can be dealt with on the Wikidata side I am sure, but some might need to be done in the tool. If it’s useful, reach out, and let’s discuss what can be changed or how it can be used in your work.

Data quality

It will quickly become apparent the data quality isn’t what it is with PRONOM and that is why a curated and authoritative service such as PRONOM is always going to be needed. As mentioned in previous talks, this can in theory be complemented with downstream data in federated databases. This might mean curating Wikidata better using some of the tools available, or curating data into a Wikibase (the platfom Wikidata is built upon). Both options bring different benefits and advantages such as creating a bigger tent of signature developers on Wikidata, or, another example, more expressive signatures being made available via federated Wikibases.

And a word on Wikiba.se

A reminder too, that setting up a Wikibase can take some effort (I was once running three at the same time 😬) but a service called https://wikiba.se/ exists. wikiba.se could form an excellent scratch pad to begin thinking about mapping PRONOM like data to a Wikibase and also begin solving some of the other issues around mapping container signatures and outputting those in a way that is compatible for DROID. Let me know if you give it a whirl, or want to collab on any of that.

Otherwise, thanks in advance! And enjoy wddroidy!

https://exponentialdecay.co.uk/blog/making-droid-work-with-wikidata/

#Code #Coding #digipres #DigitalPreservation #DROID #FileFormat #FileFormats #OpenData #PRONOM #siegfried #SoftwareDevelopment #wikidata

@Thorsted highlighted the TCDBx database with 58,921 entries of type/creator codes (with/related to 19,737 file extensions). This is a huge number, especially compared to #pronom or even #TrID! #ipres2024 (sorry for redrafting and spamming people)

@nkrabben on AV Formats in #PRONOM. (Also discussing DROID and #Siegfried.) Example of the difference between how PRONOM looks for an mp4 signature vs how #MediaInfo identifies an mp4. #NTTW7