# Parquet data schema

**URL:** <https://discourse.gbif.org/t/parquet-data-schema/4083>\
**Category:** Data blog\
**Created:** [August 9, 2023, 1:41pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083 "2023-08-09T13:41:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dmi3k](https://sea1.discourse-cdn.com/gbif1/user_avatar/discourse.gbif.org/dmi3k/32/832_2.png) [@dmi3k](https://discourse.gbif.org/u/dmi3k)\
**Post date:** [August 9, 2023, 1:41pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083/1 "2023-08-09T13:41:19Z")

</div>

Hi,  
I am not sure where is the best place to ask this question, so forgive me if this belongs elsewhere (e.g. gbif repo).  
Who decides on the [AWS schema](https://github.com/gbif/occurrence/blob/master/aws-public-data.md)?  
I seems like the `stateprovince` is not systematically recorded and is generally worse than `level1Gid`, which is missing from the schema. I understand it might be an inferred field in GBIF, but querying `stateprovince` is a major pain compared to well-structured GADM subdivisions.

I think the most valuable fields are the taxonomy keys and GADM keys.

---

<div class="post-metadata">

**Author:** ![MattBlissett](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@MattBlissett](https://discourse.gbif.org/u/MattBlissett)\
**Post date:** [August 9, 2023, 3:05pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083/2 "2023-08-09T15:05:51Z")

</div>

Hi,

The schema is the same as the “Simple” download format available on [GBIF.org](http://GBIF.org) — this is our (GBIF’s) responsibility, not Amazon’s, though we did discuss it with them.

`level1Gid` is inferred from the coordinates, if present. `stateprovince` is not altered at all, so it varies widely depending on the data provided to us.

---

<div class="post-metadata">

**Author:** ![sformel](https://sea1.discourse-cdn.com/gbif1/user_avatar/discourse.gbif.org/sformel/32/1179_2.png) [@sformel](https://discourse.gbif.org/u/sformel)\
**Post date:** [August 14, 2023, 1:13pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083/3 "2023-08-14T13:13:25Z")

</div>

@MattBlissett related to this, I’ve always been curious, why are these fields changed to all lower case instead of the camelCase of Darwin Core? It seems like such a trivial change that could be avoided, but I’m guessing it serves a purpose.

---

<div class="post-metadata">

**Author:** ![MattBlissett](https://avatars.discourse-cdn.com/v4/letter/m/58956e/32.png) [@MattBlissett](https://discourse.gbif.org/u/MattBlissett)\
**Post date:** [August 14, 2023, 2:11pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083/4 "2023-08-14T14:11:26Z")

</div>

I think it was to be more SQL-like, and was possibly also due to a limitation in Apache HBase which we used to use to hold the occurrence data.

---

<div class="post-metadata">

**Author:** ![sformel](https://sea1.discourse-cdn.com/gbif1/user_avatar/discourse.gbif.org/sformel/32/1179_2.png) [@sformel](https://discourse.gbif.org/u/sformel)\
**Post date:** [August 14, 2023, 2:52pm UTC](https://discourse.gbif.org/t/parquet-data-schema/4083/5 "2023-08-14T14:52:27Z")

</div>

Gotcha. Thanks for the quick answer.