# Filtering isn't cleaning

**URL:** <https://discourse.gbif.org/t/filtering-isnt-cleaning/4108>\
**Category:** Data Use\
**Created:** [August 22, 2023, 8:33am UTC](https://discourse.gbif.org/t/filtering-isnt-cleaning/4108 "2023-08-22T08:33:57Z")\
**Posts on this page:** 1\
**Showing post:** 16

<div class="post-metadata">

**Author:** ![datafixer](https://sea1.discourse-cdn.com/gbif1/user_avatar/discourse.gbif.org/datafixer/32/956_2.png) [@datafixer](https://discourse.gbif.org/u/datafixer)\
**Post date:** [September 1, 2023, 8:43pm UTC](https://discourse.gbif.org/t/filtering-isnt-cleaning/4108/16 "2023-09-01T20:43:25Z")

</div>

@sharif.islam, recording individual annotations (and corrections) in a structured way is of course a good idea, but while you are thinking about it, please consider

- suggested corrections that apply to multiple records (\> 1000s)
- suggested corrections that involve multiple fields at the same time
- suggested corrections that involve multiple records at the same time (“These records contradict each other”)
- suggested corrections that identify [strict duplicate](https://discourse.gbif.org/t/duplicate-occurrence-records/3735) records
- suggestions without explicit corrections (“This can’t be right, please check”; “There is a missing-but-expected value here”; “One or more of these values is incorrect, please check”; see [here](https://discourse.gbif.org/t/field-disagreements-in-darwin-core/3370))

Atomic annotations for individual values in individual fields are suited to TDWG-type “assertions” but not for the real-world checking that I and other data auditors do, and presumably will also do when DiSSCo and participants begin their own checking.

An alternative type of record-keeping is to have an original dataset version, a corrected or queried dataset version and a diff that applies to the whole dataset. Each diff could be given a PID, and the new versions would be generated by both the data publisher (DiSSCo participant or DiSSCo staff) and data checkers.

Yes, this is information-dense and requires substantial storage, but it is less complicated than tracking individual annotations/corrections.

---

_[View the full topic](https://discourse.gbif.org/t/filtering-isnt-cleaning/4108)._