# Data blog **URL:** https://discourse.gbif.org/c/data-blog/16.md?page=1 [Latest](https://discourse.gbif.org/latest.md) · [Categories](https://discourse.gbif.org/categories.md) · [Tags](https://discourse.gbif.org/tags.md) **Page:** 2 --- ## [Parquet data schema](https://discourse.gbif.org/t/parquet-data-schema/4083)
**Author:** [@dmi3k](https://discourse.gbif.org/u/dmi3k)\ **Replies:** 4\ **Last updated:** [August 14, 2023, 2:52pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083 "2023-08-14T14:52:27Z")
Hi, I am not sure where is the best place to ask this question, so forgive me if this belongs elsewhere (e.g. gbif repo). Who decides on the AWS schema? I seems like the stateprovince is not systematically recorded an… --- ## [How to choose a dataset class on GBIF? - GBIF Data Blog](https://discourse.gbif.org/t/how-to-choose-a-dataset-class-on-gbif-gbif-data-blog/1513)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 1\ **Last updated:** [August 7, 2023, 6:31am UTC](https://discourse.gbif.org/t/how-to-choose-a-dataset-class-on-gbif-gbif-data-blog/1513 "2023-08-07T06:31:20Z")
If you are a (first time) publisher on GBIF and you are trying to decide which type of dataset would best fit your data, this blogpost is for you. This is a companion discussion topic for the original entry at https://… --- ## [Identifying potentially related records - How does the GBIF data-clustring feature work? - GBIF Data Blog](https://discourse.gbif.org/t/identifying-potentially-related-records-how-does-the-gbif-data-clustring-feature-work-gbif-data-blog/3083)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 18\ **Last updated:** [June 1, 2023, 10:16pm UTC](https://discourse.gbif.org/t/identifying-potentially-related-records-how-does-the-gbif-data-clustring-feature-work-gbif-data-blog/3083 "2023-06-01T22:16:21Z")
The data available in GBIF include many so-called “duplicate” records. This is something that users might be familiar with. You download data from GBIF, analyze them and realize that some records have the same date, scie… --- ## [Exploring Related Records in the Flowering Plant Genus Senegalia in Brazil - GBIF Data Blog](https://discourse.gbif.org/t/exploring-related-records-in-the-flowering-plant-genus-senegalia-in-brazil-gbif-data-blog/3825)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [January 30, 2023, 3:51pm UTC](https://discourse.gbif.org/t/exploring-related-records-in-the-flowering-plant-genus-senegalia-in-brazil-gbif-data-blog/3825 "2023-01-30T15:51:19Z")
In 2020 GBIF released a news item “New data-clustering feature aims to improve data quality and reveal cross-dataset connections.” Basically, we run an algorithm across datasets shared with GBIF to search for similaritie… --- ## [Checklist publishing on GBIF - some explanations on taxonID, scientificNameID, taxonConceptID, acceptedNameUsageID, nameAccordingTo - GBIF Data Blog](https://discourse.gbif.org/t/checklist-publishing-on-gbif-some-explanations-on-taxonid-scientificnameid-taxonconceptid-acceptednameusageid-nameaccordingto-gbif-data-blog/3758)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [December 12, 2022, 10:41am UTC](https://discourse.gbif.org/t/checklist-publishing-on-gbif-some-explanations-on-taxonid-scientificnameid-taxonconceptid-acceptednameusageid-nameaccordingto-gbif-data-blog/3758 "2022-12-12T10:41:07Z")
When data publishers publish checklists, they will use a a Darwin Core Archive Taxon Core. And although the taxon core terms are already described here, what exactly to put in which field can sometimes be confusing. And … --- ## [Country Centroids - GBIF Data Blog](https://discourse.gbif.org/t/country-centroids-gbif-data-blog/679)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 5\ **Last updated:** [November 26, 2022, 7:32pm UTC](https://discourse.gbif.org/t/country-centroids-gbif-data-blog/679 "2022-11-26T19:32:06Z")
Country Centroids are a known data quality issue within the GBIF network. This is a companion discussion topic for the original entry at https://data-blog.gbif.org/post/country-centroids/ --- ## [Which data can be shared through GBIF and what cannot - GBIF Data Blog](https://discourse.gbif.org/t/which-data-can-be-shared-through-gbif-and-what-cannot-gbif-data-blog/3704)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [November 17, 2022, 11:05am UTC](https://discourse.gbif.org/t/which-data-can-be-shared-through-gbif-and-what-cannot-gbif-data-blog/3704 "2022-11-17T11:05:41Z")
Preparing a dataset to be shared on GBIF.org can be a daunting task and many publishers realize that not all their data fits in the Darwin Core standard (DwC) and extensions GBIF uses to structure, standardize and displa… --- ## [The World Checklist of Vascular Plants (Fabaceae) - GBIF Data Blog](https://discourse.gbif.org/t/the-world-checklist-of-vascular-plants-fabaceae-gbif-data-blog/3206)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 2\ **Last updated:** [May 13, 2022, 6:05am UTC](https://discourse.gbif.org/t/the-world-checklist-of-vascular-plants-fabaceae-gbif-data-blog/3206 "2022-05-13T06:05:44Z")
The World Checklist of Vascular Plants (WCVP): Fabaceae is a new GBIF mediated checklist that drastically increases the coverage of the family Fabacea in the GBIF backbone. This is a companion discussion topic for the … --- ## [Finding data gaps in the GBIF backbone taxonomy - GBIF Data Blog](https://discourse.gbif.org/t/finding-data-gaps-in-the-gbif-backbone-taxonomy-gbif-data-blog/3330)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [April 25, 2022, 2:46pm UTC](https://discourse.gbif.org/t/finding-data-gaps-in-the-gbif-backbone-taxonomy-gbif-data-blog/3330 "2022-04-25T14:46:08Z")
When publishers supply GBIF with a scientific name, this name is sometimes not found in the GBIF taxonomic backbone. In these cases, the occurrence record gets a data quality flag called taxon match higher rank. This mea… --- ## [Proper handling of missing values through predicates (rgbif)](https://discourse.gbif.org/t/proper-handling-of-missing-values-through-predicates-rgbif/3272)
**Author:** [@MarcRiera](https://discourse.gbif.org/u/MarcRiera)\ **Replies:** 5\ **Last updated:** [April 1, 2022, 4:09am UTC](https://discourse.gbif.org/t/proper-handling-of-missing-values-through-predicates-rgbif/3272 "2022-04-01T04:09:37Z")
Hi! I’m using the R package rgbif to request data to GBiF, and in order to get a smaller and more manageable download, I am using predicates to remove from the start observations with specific establishmentMeans or basis… --- ## [Using apache-arrow and parquet with GBIF mediated occurrences - GBIF Data Blog](https://discourse.gbif.org/t/using-apache-arrow-and-parquet-with-gbif-mediated-occurrences-gbif-data-blog/3221)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [February 18, 2022, 3:31pm UTC](https://discourse.gbif.org/t/using-apache-arrow-and-parquet-with-gbif-mediated-occurrences-gbif-data-blog/3221 "2022-02-18T15:31:02Z")
As written about in a previous blog post, GBIF now has database snapshots of occurrence records on AWS. This allows users to access large tables of GBIF mediated occurrence records from Amazon s3 remote storage. This acc… --- ## [GBIF API beginners guide - GBIF Data Blog](https://discourse.gbif.org/t/gbif-api-beginners-guide-gbif-data-blog/2991)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 5\ **Last updated:** [November 25, 2021, 9:35am UTC](https://discourse.gbif.org/t/gbif-api-beginners-guide-gbif-data-blog/2991 "2021-11-25T09:35:05Z")
This a GBIF API beginners guide. The GBIF API technical documentation can be found here. This documentation is nice, but you might be a bit confusing if you have never used an API before. This guide is will walk you thro… --- ## [Common things to look out for when post-processing GBIF downloads - GBIF Data Blog](https://discourse.gbif.org/t/common-things-to-look-out-for-when-post-processing-gbif-downloads-gbif-data-blog/2504)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 7\ **Last updated:** [October 20, 2021, 11:15am UTC](https://discourse.gbif.org/t/common-things-to-look-out-for-when-post-processing-gbif-downloads-gbif-data-blog/2504 "2021-10-20T11:15:37Z")
Here I present a checklist for filtering GBIF downloads. In this guide, I will assume you are familar with R. This guide is also somewhat general, so your solution might differ. This guide is intended to give you a check… --- ## [Search, download, analyze and cite (repeat if necessary) - GBIF Data Blog](https://discourse.gbif.org/t/search-download-analyze-and-cite-repeat-if-necessary-gbif-data-blog/1376)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 15\ **Last updated:** [September 15, 2021, 7:47pm UTC](https://discourse.gbif.org/t/search-download-analyze-and-cite-repeat-if-necessary-gbif-data-blog/1376 "2021-09-15T19:47:46Z")
There is a lot of GBIF-mediated data available. More than 1.3 B occurrence records covering hundreds of thousands of species in all part of the worlds. All free, open and available at the touch of a button. Users can dow… --- ## [Did you know that...? - some of the lesser known functionalities around GBIF.org - GBIF Data Blog](https://discourse.gbif.org/t/did-you-know-that-some-of-the-lesser-known-functionalities-around-gbif-org-gbif-data-blog/2990)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [August 11, 2021, 8:46am UTC](https://discourse.gbif.org/t/did-you-know-that-some-of-the-lesser-known-functionalities-around-gbif-org-gbif-data-blog/2990 "2021-08-11T08:46:08Z")
During the first-ever virtual GBIF 2021 Global Nodes Meeting, GBIFS hosted a “game show”: a one-hour “battle of Nodes vs. helpdesk”. The not-so-hidden goal was to demonstrate some of the lesser known functionalities of G… --- ## [GBIF and Apache-Spark on AWS tutorial - GBIF Data Blog](https://discourse.gbif.org/t/gbif-and-apache-spark-on-aws-tutorial-gbif-data-blog/2785)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [June 2, 2021, 9:24am UTC](https://discourse.gbif.org/t/gbif-and-apache-spark-on-aws-tutorial-gbif-data-blog/2785 "2021-06-02T09:24:52Z")
GBIF now has a snapshot of 1.3 billion occurrences✝ records on Amazon Web Services (AWS). This guide will take you through running Spark notebooks on AWS. This guide is similar to the one written previously about Microso… --- ## [The GBIF Registry of Scientific Collections (GRSciColl) in 2021 - GBIF Data Blog](https://discourse.gbif.org/t/the-gbif-registry-of-scientific-collections-grscicoll-in-2021-gbif-data-blog/2569)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [March 12, 2021, 2:34pm UTC](https://discourse.gbif.org/t/the-gbif-registry-of-scientific-collections-grscicoll-in-2021-gbif-data-blog/2569 "2021-03-12T14:34:11Z")
The GBIF Registry of Scientific Collections, also known as GRSciColl, has been available on GBIF.org since 2019 but it recently got some more attention when we connected it to GBIF occurrences. Now is the perfect time to… --- ## [GBIF occurrence license processing - GBIF Data Blog](https://discourse.gbif.org/t/gbif-occurrence-license-processing-gbif-data-blog/1813)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 2\ **Last updated:** [February 7, 2021, 8:53am UTC](https://discourse.gbif.org/t/gbif-occurrence-license-processing-gbif-data-blog/1813 "2021-02-07T08:53:27Z")
GBIF will now process occurrence licenses record-by-record. Previously all occurrence licenses defaulted to their dataset license (provided by the publisher). This was in line with what as stated in our terms of use: T… --- ## [Not a bird download - GBIF Data Blog](https://discourse.gbif.org/t/not-a-bird-download-gbif-data-blog/1138)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 1\ **Last updated:** [November 25, 2020, 3:10pm UTC](https://discourse.gbif.org/t/not-a-bird-download-gbif-data-blog/1138 "2020-11-25T15:10:47Z")
Recently we were asked on GitHub whether there was a way to get all animal occurrences that are not a bird. This seems like an easy enough request, but unfortunately, there is currently no way to exclude groups from a do… --- ## [(Almost) everything you want to know about the GBIF Species API - GBIF Data Blog](https://discourse.gbif.org/t/almost-everything-you-want-to-know-about-the-gbif-species-api-gbif-data-blog/2371)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 1\ **Last updated:** [November 19, 2020, 9:03am UTC](https://discourse.gbif.org/t/almost-everything-you-want-to-know-about-the-gbif-species-api-gbif-data-blog/2371 "2020-11-19T09:03:24Z")
Today, we are talking about the GBIF Species API. Although you might not use it directly, you probably encountered it while using the GBIF web portal: This is a companion discussion topic for the original entry at http… --- ## [Outlier Detection Using DBSCAN - GBIF Data Blog](https://discourse.gbif.org/t/outlier-detection-using-dbscan-gbif-data-blog/2251)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 2\ **Last updated:** [September 1, 2020, 6:48pm UTC](https://discourse.gbif.org/t/outlier-detection-using-dbscan-gbif-data-blog/2251 "2020-09-01T18:48:51Z")
Geographic outliers at GBIF are a known problem. Outliers can be errors, coordinates with high uncertainty, or simply occurrences from an undersampled region. This is a companion discussion topic for the original entry… --- ## [GBIF Regional Statistics - 2020 - GBIF Data Blog](https://discourse.gbif.org/t/gbif-regional-statistics-2020-gbif-data-blog/2182)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 5\ **Last updated:** [May 27, 2020, 7:14pm UTC](https://discourse.gbif.org/t/gbif-regional-statistics-2020-gbif-data-blog/2182 "2020-05-27T19:14:20Z")
Iceland Index. A plot of raw species counts from equal-area hexagons approximately the size of Iceland. This plot excludes international waters, but includes countries EEZ areas. We see here that while Iceland has \>5K sp… --- ## [Which tools can I use to share my data on GBIF? - GBIF Data Blog](https://discourse.gbif.org/t/which-tools-can-i-use-to-share-my-data-on-gbif-gbif-data-blog/1845)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 2\ **Last updated:** [April 9, 2020, 6:32am UTC](https://discourse.gbif.org/t/which-tools-can-i-use-to-share-my-data-on-gbif-gbif-data-blog/1845 "2020-04-09T06:32:13Z")
As you probably already know, GBIF.org doesn’t host any data. The system relies on each data provider making their data available online in a GBIF-supported format. It also relies on organization letting GBIF know where … --- ## [Does Biodiversity Informatics 💘 Wikidata? - GBIF Data Blog](https://discourse.gbif.org/t/does-biodiversity-informatics-wikidata-gbif-data-blog/1799)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [February 21, 2020, 2:24pm UTC](https://discourse.gbif.org/t/does-biodiversity-informatics-wikidata-gbif-data-blog/1799 "2020-02-21T14:24:59Z")
Open online APIs are fantastic! You can use someone else’s infrastructure to create workflows, do research and create products without giving anything in return, except acknowledgement. But wait a minute! Why is everyone… --- ## [Frictionless Data and Darwin Core - GBIF Data Blog](https://discourse.gbif.org/t/frictionless-data-and-darwin-core-gbif-data-blog/1628)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 2\ **Last updated:** [January 16, 2020, 8:38am UTC](https://discourse.gbif.org/t/frictionless-data-and-darwin-core-gbif-data-blog/1628 "2020-01-16T08:38:56Z")
Frictionless data is about removing the friction in working with data through the creation of tools, standards, and best practices for publishing data using the Data Package standard, a containerization format for any ki… --- ## [Understanding basis of record - a living specimen becomes a preserved specimen - GBIF Data Blog](https://discourse.gbif.org/t/understanding-basis-of-record-a-living-specimen-becomes-a-preserved-specimen-gbif-data-blog/1349)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 0\ **Last updated:** [September 18, 2019, 8:55am UTC](https://discourse.gbif.org/t/understanding-basis-of-record-a-living-specimen-becomes-a-preserved-specimen-gbif-data-blog/1349 "2019-09-18T08:55:54Z")
Recently a user noticed that there were Asian Red Pandas (Ailuridae) occurring in North America, and wondered if someone had made a mistake. When an occurrence observation comes from a zoo or botanical garden, it is usua… --- ## [Six questions answered about the GBIF Backbone Taxomomy - GBIF Data Blog](https://discourse.gbif.org/t/six-questions-answered-about-the-gbif-backbone-taxomomy-gbif-data-blog/1334)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 4\ **Last updated:** [October 5, 2019, 9:17pm UTC](https://discourse.gbif.org/t/six-questions-answered-about-the-gbif-backbone-taxomomy-gbif-data-blog/1334 "2019-10-05T21:17:52Z")
This past week our informatics team has been updating the Backbone taxonomy on GBIF.org. This is a fairly disruptive process which sometimes involves massive taxonomic changes but DON’T PANIC. This is a companion discu… --- ## [Citizen Science on GBIF - 2019 - GBIF Data Blog](https://discourse.gbif.org/t/citizen-science-on-gbif-2019-gbif-data-blog/1234)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 4\ **Last updated:** [September 17, 2019, 9:55am UTC](https://discourse.gbif.org/t/citizen-science-on-gbif-2019-gbif-data-blog/1234 "2019-09-17T09:55:03Z")
Citizen Science datasets on GBIF plotted with all other (gray) GBIF datasets (\>100K occurrences). There are many citizen science datasets with millions of occurrences (eBird, (Swedish) Artportalen), and the top 3 dataset… --- ## [Exploring es50 for GBIF - GBIF Data Blog](https://discourse.gbif.org/t/exploring-es50-for-gbif-gbif-data-blog/1154)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 4\ **Last updated:** [August 11, 2019, 3:33am UTC](https://discourse.gbif.org/t/exploring-es50-for-gbif-gbif-data-blog/1154 "2019-08-11T03:33:20Z")
Roderic Page has suggested that GBIF could make es50 maps similar to what organizations like OBIS are already doing. I decided to make one for land animals (graph above). This is a companion discussion topic for the or… --- ## [Big National Checklists - GBIF Data Blog](https://discourse.gbif.org/t/big-national-checklists-gbif-data-blog/1106)
**Author:** [@dodobot](https://discourse.gbif.org/u/dodobot)\ **Replies:** 13\ **Last updated:** [July 3, 2019, 11:19am UTC](https://discourse.gbif.org/t/big-national-checklists-gbif-data-blog/1106 "2019-07-03T11:19:40Z")
Here I plot the total names in checklists published on GBIF linked to a single country. A checklist dataset is a term for any dataset that contains primarily a list of taxonomic names. National species checklists are lis… [Previous page](https://discourse.gbif.org/c/data-blog/16.md) [Next page](https://discourse.gbif.org/c/data-blog/16.md?page=2)