Hi All,
We created a fast GBIF parquet to RDF converter[0] and some people like the capabilities that gives. We have one user asking for some information not available in the GBIF parquet data [1].
What is the recommended way to download all GBIF data? should we retrieve each DWC archive one by one via API or is there a cheaper way to do this?
Regards,
Jerven
PS. The LLMs suggest DWC-A download, but that would be expensive if it is a download of all of GBIF.
If you want the raw data, you can use the Monthly snapshot datasets in our Occurrence snapshots in the Darwin Core Archive format. There are several options with slightly different queries (with coordinates, occurrenceStatus: present, etc), but if you want everything you can look for the ‘No filters used’ Query: https://doi.org/10.15468/dl.8yrbe7 that will also have all the extensions and you can use that for your converter.
It is a big file of 2TB, but that is just because of the size of the data, even the simple download is 465 GB.
Let us know if this solves your question or do you want to know something else
Thanks the complete 2TB no filter option is what we will need. Good to know we wont be overloading some API endpoint!