I manage the data for a few different collections at The Ohio State University Museum of Biological Diversity. We use Specify 7 as our DBMS, and we also have a GBIF IPT installation hosted at our institution.
Specify 7 has an RSS feed that can be used as an endpoint for GBIF uploads, but it lacks the capability to export data in the correct format for certain fields. For example, it cannot export data in dwc’s expected format for eventdate because dates are stored in two separate fields (startDate and endDate) and Specify’s export query cannot combine them internally. Specify is also limited when it comes to formatting the EML file associated with the upload.
We used our IPT to push an initial data dump to GBIF last year, but have held off on uploading more data because transforming it to the right format involves a manual process that I have to run. I could write a script to do the transformation in one step, but that doesn’t solve the issue of automating the export.
My question is: Is there an option to set up a custom import script that performs the necessary data transformations from Sp7’s raw export, either in our IPT or on GBIF’s end importing directly from our Sp7 RSS feed?
I’d really like to automate updates but I’d like to avoid having to set up a standalone webservice/cron job that watches the RSS feed, runs the script, and pushes it to GBIF through the IPT on our end, as that would be another moving piece for us to manage and our IT team is quite small and not just dedicated to managing our DBs. I could set up a cron job locally, but I don’t want the exports to be dependent on my individual machine being online, as the pipeline should work independently of me.
If you sent up endpoints for the Darwin Core Archives and register them on GBIF, you only need to make sure that the archive are up to date. GBIF checks all the endpoints weekly and ingest the archives if they have been refreshed.
Hi @mgrosjean thanks for the reply! Unfortunately it does not help.
We already have a DWCA endpoint, the issue is not how to do that, but rather whether there is any way to transform the export that we serve at OSUM Invertebrate Zoology DarwinCore RSS Feed to DWC compliant format in our IPT endpoint without setting up an intermediate/external service. This basically involves merging a few columns and adding text prefixes to a few fields. From your initial answer it seems like the answer is probably not.
I haven’t used Specify much, but this is what I’d do.
I’d create a minimal script that does these basic transformations from the RSS feed to a number of csv files and have that script run on a schedule on something like Github Actions.
Then I’d point my IPT to these files, and set the dataset to auto update on the same (or a slower) schedule than my script.
If you feel the need you could setup your Github Action (or similar) to place a pull request with a review request towards yourself so you can manually check the quality of the mapping before merging it into the files that are used by the IPT and then ingested by GBIF.
This is a similar workflow that we use to publish a set of occurrence datasets that depend on external database connections that we do not control, and can only access via APIs.
It might seem a hassle to setup, and adds an extra layer of tooling, but it has worked dependently, and without extra costs for us.