# Getting ALA data to the usable stage

**URL:** <https://discourse.gbif.org/t/getting-ala-data-to-the-usable-stage/5413>\
**Category:** Data Use\
**Created:** [October 13, 2024, 6:26am UTC](https://discourse.gbif.org/t/getting-ala-data-to-the-usable-stage/5413 "2024-10-13T06:26:18Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![datafixer](https://sea1.discourse-cdn.com/gbif1/user_avatar/discourse.gbif.org/datafixer/32/956_2.png) [@datafixer](https://discourse.gbif.org/u/datafixer)\
**Post date:** [October 13, 2024, 6:26am UTC](https://discourse.gbif.org/t/getting-ala-data-to-the-usable-stage/5413/1 "2024-10-13T06:26:18Z")

</div>

Working with biodiversity data often involves a fair amount of filtering and cleaning (these are [different things](https://discourse.gbif.org/t/filtering-isnt-cleaning/4108)). We don’t often have to reformat a Darwin Core dataset, though, thanks to GBIF’s _Integrated Publishing Toolkit_. Occurrence datasets that have been through the IPT typically come out as simple tab-separated tables in UTF-8 encoding, with a single header line containing Darwin Core field names.

This isn’t the case for datasets shared with GBIF by the _Atlas of Living Australia_. A Darwin Core archive from the ALA with an occurrence core has an **occurrence.txt** file with 182 fields but no header line, and with every data item surrounded by quotes. The accompanying **meta.xml** file has 182 field names, but many of the field name references are defective:

- 8 with "[http://rs.ala.org.au/terms/1.0/](http://rs.ala.org.au/terms/1.0/)…" don’t resolve

- 4 with "[ABCD - Base Ontology Terms](http://rs.tdwg.org/abcd/terms/abcd)…" should instead go to "[ABCD - Base Ontology Terms](https://abcd.tdwg.org/terms/)…"

- 3 have DwC terms (_eventAttributes, locationAttributes, occurrence Attributes_) which TDWG says are deprecated and should not be used

- 2 with "[http://hiscom.chah.org.au/hispid/terms/](http://hiscom.chah.org.au/hispid/terms/)…" don’t resolve

- 3 with "[GGBN Data Standard](http://data.ggbn.org/schemas/ggbn/terms/)…" don’t resolve

- 1 has no URL or explainer: "taxonRankID"

I can’t do anything about the **meta.xml** , but to make ALA occurrence data usable as a single table I can use the BASH shell function shown below. I extract **occurrence.txt** and **meta.xml** from the ALA archive, navigate to the directory containing those files and enter “alaprep”. The function generates a new TSV “ocala.txt” with the 182 field names in a header line and no quotes around data items.

`alaprep() { cat <(sed 's|taxonRankID|/taxonRankID|' meta.xml | awk -v FS="/|\"/>" '/field index="0"/ {f=1} f {printf("%s\t",$(NF-1))} /field index="181"/ {exit}' | sed 's/\t$/\n/') occurrence.tsv | sed 's/^"//g;s/"\t"/\t/g;s/"$//' > ocala.txt; }`

Please email me directly if you have questions about the function.

–

_Robert Mesibov (“datafixer”); [robert.mesibov@gmail.com](mailto:robert.mesibov@gmail.com)_