10. Transactional mechanisms and provenance

[jhpoelen]

I was pleasantly surprised to see the topic of “Transactional mechanisms and provenance” in this “Digital/Extended Specimen” consultation.

@hardistyar

In a transactional data publishing model, each edit to a data record, is represented through a transaction published to a linked list. Similar to block chains, transactions in the list are cryptographically related to all prior transactions ensuring data integrity and immutability and preservation of version history. Yet, unlike block chains, transactions do not need to be bundled into blocks nor are computationally intensive models of consensus necessary. When summed, transactions can be used to derive a given view of a data item.

@hardistyar 's quoted description above of this transactions-provenance topic neatly matches ideas centered around Preston (https://preston.guoda.bio / GitHub - bio-guoda/preston: a biodiversity dataset tracker), a transactional git-like dataset tracker, that has been in use since 2018 and that I’ve referenced, and discussed with many of you in various meetings, discussions, prototypes, and email exchanges since 2018. (see GitHub - bio-guoda/preston: a biodiversity dataset tracker for some examples of publications, conference presentations, data publications, and forum/twitter posts)

So, I’d suggest to add a reference to Preston (GitHub - bio-guoda/preston: a biodiversity dataset tracker) and related publication (e.g., MJ Elliott, JH Poelen, JAB Fortes (2020). Toward Reliable Biodiversity Dataset References. Ecological Informatics. Redirecting) in the section “Information Resources” at the start of this discussion topic.

Please bear with me as I attempt to address @hardistyar excellent discussion prompts point-by-point:

@hardistyar

1. What use cases can transactional publishing solve that can’t be solved by existing mechanisms?

Transactional publishing (or data tracking using reliable content-based identifiers) allows for systematic tracking (and referencing) of datasets (and records in them), irregardless of were they happened to be located. Existing mechanisms (e.g., web APIs, data portals) usually rely on trusting a single web entities to follow hard to maintain/verify “Cool URIs” (Cool URIs for the Semantic Web) principles . For a suite of use cases enabled by systematic dataset content tracking, GitHub - bio-guoda/preston: a biodiversity dataset tracker might be of interest.

@hardistyar

2. Is transactional publishing a solution to the existing problem of data integration and attribution?

Yes, transactional publishing provides a solid basis for reliable data references, an essential ingredient of reliable and reproducible data integration and attribution. You’ll find that Preston uses standards based (rdf / Prov-O, sha-2 hashing, triple implosion) techniques to implement transactions through linking provenance logs of biodiversity datasets in a decentralized storage architecture (pretty much following the git architecture beyond the file system). We’ve been running the Preston infrastructure using existing run-of-the-mill storage solutions (Zenodo, Internet Archive, Software Heritage Library, rsync-ed mirrors, commodity external hard-disk) to systematically track and version datasets/records registered in GBIF/iDigBio (and more) since 2018 .

In other words, we’ve shown that 404s (aka linkrot) or (un)expected changes (aka content drift) are no longer as much of an issue by introducing transactional provenance without having to significantly change existing biodiversity infrastructures.

@hardistyar

3. Any transactional mechanism is going to rely on unique identifiers for all data objects. How do the issues surrounding Topic 7 relate to transactional publishing? Are identifiers necessary (at least in the way we normally think of them)?

I’d say that commonly used unique identifiers like UUIDs/DOIs/PURLs, while being aspirationally unique at best, are useful as long as the provenance of the context in which they appear can be reliably referenced and accessed.

So, I’d say that “unique” identifiers are useful, but not necessary.

@hardistyar

4. What pieces of information do we need to capture to Extend specimen records - annotations, georeferences, publications, Genbank sequences, Nagoya related information (loans, gifts, accessions? Who made these (agents)? When were they made (dates)?

I think that a well-designed provenance / transaction mechanism can be independent of the kind of domain-specific content that is being tracked. I found that Prov-O (PROV-O: The PROV Ontology) entity-activity-actor model provides basic information elements/model to implement a content/format-agnostic (or as the original git author Linus Torvalds put it: “dumb”) git-like provenance-transaction system.

@hardistyar

5. What infrastructure components are missing and needed to make this a reality?

I see mostly conceptual challenges in moving from a location-based data perspective (the Berner-Lee “Cool-URI” dilemma) to a content-based perspective (information-centric networking similar to Ted Nelson’s DocuVerse as proposed in his visionary Computer Lib/Dream Machines Computer Lib/Dream Machines - Wikipedia).

Also, I see some socio-economical coordination challenges in deciding how to make sure to keep many copies of relevant digital data around. But similar conservation/curation challenge have been taken up by collection managers, curators, and librarians for centuries, and some of this institutional know-how can be taken from the physical (e.g., books, specimen, artworks) and applied to the digital realm (e.g., files, digital datasets, digital images).

@hardistyar

6. What changes to existing infrastructure (CMSs, aggregators, publishers, data flow, etc.) are needed to make this a reality? Can the existing pieces support transactional publishing? Why? Why not?

Over the last years, we’ve successfully tracked data across major biodiversity networks without having to ask for any changes to be made. Admittedly, more efficient approaches can be imagined, but from what I can tell versioned snapshots of existing APIs can go a long way.

@hardistyar

7. To accommodate the full breadth of digital assets, data storage would likely need to be separate from the transaction chain? What are the implications of “off-chain” data storage and what mechanisms need to be in place to accommodate this?

Preston uses content-addressed storage to allow for provenance and content to be kept separately in decentralized manner without compromising the integrity of the versioned data corpus. We have examples of Zenodo data publications only including the provenance (and their translation chains) and storing (heavier) data objects elsewhere (see e.g., http://doi.org/10.5281/zenodo.3852671)

@hardistyar

8. The goal is to have an open network enabling researchers to publish derivative data at will even if rejected by primary data sources. What measures need to be in place to ensure primary publishers maintain the authoritative views for a given record?

Primary publishers / curators can reliably reference trusted datasets in their (cryptographically signed) provenance logs similar to how tradition publishers bundle received manuscripts in journals. In other words, trusted institutions or publishers can curate high quality data collections by publishing reliable references (and related provenance) to datasets they deem worthy to their cause and quality standards. They may even choose to keep copies of these referenced datasets around to help keep the data around for a little longer.

@hardistyar

9. How do specifics of Topic 9 (regulatory, legal, ethical topics) affect the open sharing of data and exposure of transactions in this fashion? How do we obscure data that should not be seen (threatened/endangered/CITES species, other sensitive data)?

Because of ability to store data “off-chain”, sensitive data can be reliably referenced in scientific papers or composite datasets without making the sensitive dataset openly available. An additional layer of access control can be introduced by providing a reference to an encrypted copy of the sensitive dataset. But, this poses long term archiving issues (e.g., losing private keys will make data inaccessible forever) that may outweight the benefits of added access controls.

I hope my comments show that transactions mechanisms and provenance are a pragmatic and proven way to overcome many of the data integration issues we all encounter daily without having to move our digital heaven and earth.

Thank you for your attention and I hope my contributions to this topic will help advance our methods to make the most out of our digital biodiversity corpora now and in the future.

Curious to hear your thoughts and comments.

-jorrit

Jorrit H. Poelen