Bug in occurrence/download/request API when using axios-retry

1

The /occurrence/download/request API route is no longer behaving correctly. It is now always returning a download ID of 0041096-231120084113126 regardless of the query. This is an 862Gb file.

The problem is not the polygon passed to the ‘within’ predicate because I have tried other polygons (e.g. the one from the documentation).

The problem may well happen for all requests e.g. I tried the geoDistance example from the docs:

"predicate": {
    "type": "geoDistance",
    "latitude": "90",
    "longitude": "100",
    "distance": "5km",
  }

I have also tried a separate user account and the same problem occurs.

I’d be grateful if the dev team could look into it.

Many thanks,
Neil

2

Hi @neil thank you for reporting the issue.

It looks like the geo distance example in the documentation is incorrect (the trailing comma is the issue), we will correct it, thanks!

The downloads seem to work as far as I can see. Would it be possible to send your full script to helpdesk@gbif.org so we can investigate a bit more? Thank you!

3

Thank you for the reply. I created a minimal example to reproduce the problem. It appears that the problem happens when using axios-retry but not when using plain axios. Full details in an email to the helpdesk.

I can work around the issue by using plain axios but I still suspect that something strange is happening server side because returning an old download ID (0041096-231120084113126) doesn’t seem right. If axios-retry was messing up the request somehow then I’d expect the server to give an error.

4

In case it helps work out what’s happening, 0041096-231120084113126 is this month’s all-data DWCA-format download. Download requests for all data are pointed to this download, rather than recreating a ~900GB file (this takes hours). It should mean the request is arriving without a predicate value.

5

Thank you Matt - that helps. The bug was on our end - passing the POST body to the axios-retry with an extra layer of nesting in the JSON i.e. the usual payload was wrapped in { data: theUsualPayload }.

You’re right Matt, that the request arrived without a predicate but there doesn’t appear to have been any other validation of the request body. Detecting that data was an unexpected attribute and generating an error would have helped track down the error more swiftly.

Thanks again,
Neil

6

Glad to hear that Neil.

The general design of the API is to ignore unknown parameters. A predicate of {} is the way to request a full download, and {"data": …} is therefore seen the same way.

Similar cases have happened before; I’ve reopened Invalid queries in occurrence downloads are silently ignored · Issue #273 · gbif/occurrence · GitHub so we can see if this could be improved.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.