Scratchpads scratched?

Many will be familiar with the Scratchpad system which were created by the EU Network of Excellence (EDIT) in 2006 and continued to be support funded by GBIF, Biodiversity Informatics Group & EoL. The final partner was Natural History Museum who recently announced that the project was to close, all sites reverting to “read-only” according to Vince Smith, Head of Digital, Data & Informatics at NHM
Much to the consternation of the many of us who actively maintain these important sites (e.g. my European Micropezids & Tanypezids and Barry Warrington’s Agromyzidae)
The FAQ section of the Scratchpad website stated:

Ultimately the longevity of the Scratchpads will be dependent upon whether they remain useful and actively used. Without community support for infrastructure and content, the Scratchpads project will be short lived. Every indication is that the Scratchpads’ popularity will make the project as secure as any informatics project in the foreseeable future.

And indeed they have been useful, active and popular until last week’s announcement.
Consternation is now widespread amongst all users, in particular those who are actively maintaining them. We have been seeking support for their continuation amongst a range of partners, Dipterists Forum (~8 of our recording schemes use them), UKCEH and NBN (UK’s GBIF node who operate the NBN Atlas). No great success so far, one Scratchpad manager currently debating some solutions which would involve digging deeply into personal funds.

  • Perhaps the GBIF community are able to suggest means by which this valuable project might continue.
  • Perhaps you operate one of your own and have similar concerns
    Do get in touch if so
    Darwyn Sumner
3 Likes

@Darwyn Thank you for sharing your post re: Scratchpads.

In my (short) 10 years of being active in the biodiversity informatics community, I’ve seen datasets, custom websites, and entire platforms come and go [1] because human getting older, a change of heart, or funds running low. I am sure I am not the only one to have seen this.

I’d be happy to work with your to offer my expertise (within reason) to help review the architecture and finds ways to (1) independently create snapshots of one or more scratchpads projects and archive them, (2) suggest any path forward in supporting the platform in its current form, or (3) find a way to move valuable knowledge captured in Scratchpads and move them into some new home.

As far as I can tell, there’s a desire to keep the some of some active project alive, and, as a maintainer of a project myself, I am eager to assist colleagues and share my skills in software engineering, software architecture, data publications, and designing knowledge sharing methods resilient against funding droughts or (temporary) inactivity…

Please feel to reach out, or share your thoughts below.

-jorrit

references

[1] Poelen, JH. 2017. Lifestages of Species Interaction Datasets. Accessed at https://www.globalbioticinteractions.org/2017/01/24/lifestages-of-species-interaction-datasets/ on 2023-05-05 .

I wonder whether there could be a way to use the Scratchpad-Pensoft integration to create something like a special “farewell” issue in the Biodiversity Data Journal to archive Scratchpad content that people still care about at this stage.
Pinging @vsmithuk who is familiar with both ends. CC @Lyubomir .

1 Like

Indeed, this decision has come as a blow to several people who have been active in maintaining their Scratchpads sites - even having projects based on such initiatives. What is more, these sites and their structure have acted as a depository of information, data and references with a somewhat “volatile plasticity” if I may say, which allowed an up-to-date information dissemination, combining data from published works, museum specimens and even personal communications… It is unfortunate but understandable that this comes to an end.
What we need to do is find the best alternative, which doesn’t necessarily mean that it will be the best for all! Unfortunately not all users are hi-tec savvy (myself included…) to the extend that transferring their sites with all the complexity, attached files and set module relationships through lines of code is feasible to them. I don’t know if there are amongst us individuals with the expertise to create manuals or provide step-by-step guidelines to the group in order to help out in such a transition.
Nevertheless, I remain optimistic that as Jorrit said that in the past software and depositories have come and gone, new tools and even opportunities might arise.
I am following the thread and talks with a positive attitude and hope for the best…

1 Like

I’m glad to see some sympathetic replies to Darwyn’s post. I’m one of the active Scratchpadders now scratching my head as to what to do and where to go. The problem is that Scratchpads are written in Drupal 7 which is going end-of-life in November of this year. The site design is showing its age, for instance with lack of responsive design for phone and tablet screens. It’s still a great concept, but it needs updating, refreshing and new capabilities added. The obvious thing to do is to port the Drupal 7 code to Drupal 10. This the NHM was unwilling/unable to do. The interest and the money have gone elsewhere. Clearly, even in a best-case scenario, nothing like the original concept of free sites from a central server is going to happen again. Of course it’s far easier and cheaper now than it’s ever been to host one’s own site using a cloud service. We have formed a “Save or Scratchpads” (SOS) group and had our first meeting this week. We are discussing our options. We have one Scratchpad-savvy Drupal developer who is motivated to port Scratchpads to Drupal 10, but his employment may take him in other directions. If we could get salary for him and maybe another for a year or two and our user group could become the steering committee for the future of this project, I think we could rescue the existing sites realize some of the heretofore unrealized potential of Scratchpads. If anyone reading this is a biodiversity informatics and Drupal-savvy website developer interested in this project, please get in touch. Of course all ideas are welcome.

~ John

https://mormyrids.myspecies.info/en

Dear Darwyn and colleagues,

Many thanks for starting this thread. I really appreciate this, and the whole NHM Scratchpad team (past and present) is very grateful for the appreciation of the Scratchpads. First, I wanted to reiterate a few points to clear up any misconceptions, and then I will say a few words on possible solutions. I know there are several fora where comments about the Scratchpads have been posted. However, I will try to reply to queries here, with some links to the Scratchpad blog and Twitter where appropriate, so I would encourage you and others to post to this forum to keep the queries together.

Firstly, we are making the sites read-only because contributions have dramatically reduced over the past few years; the sites have become much harder to maintain; finding capable developers to do that maintenance has become very difficult; and we have been unable to secure any significant funding to undertake the upgrades necessary to maintain the security across the many thousands of sites.

The timing for the decision relates to the fact that Drupal version 7 reaches ‘end of life’ in November 2023, so recognising that the small number of active Scratchpad users will need time to come up with alternative plans, we wanted to maximise the notice available for all concerned.

Also, to reiterate, after September 2023, all sites, unless requested by the site maintainer, will remain accessible on the web, and the structured data behind each site will be preserved, essentially in perpetuity. Only write access will cease. We will continue to provide support to maintain read access and for site maintainers to get access to that structured data, should they request it.

Regarding alternatives, some have suggested that upgrading the Scratchpads to Drupal version 10 is the best course of action and that a “Save our Scratchpads” (SOS) group has been formed with this in mind. I encourage community self-support but be under no illusion about the magnitude of this task. We looked extensively at this (and other options) over the past four years, carefully examining a range of options, but ruled this out based on time, cost and complexity.

Scratchpads use over 200 Drupal modules. Most of these are contributed by Drupal users, and most are not available for Drupal 10, so re-writing these modules to version 10 would be a massive technical and social undertaking. In addition, there are many tens of custom modules (a little under half) that we have written in the Scratchpad team to make the Scratchpads functional. Upgrading these would be another massive technical undertaking, and its worth bearing in mind that our current full-time Scratchpad developer (with us since 2021) has been exclusively focused on community management and bug fixing rather than writing any Scratchpad modules or adding new functionality. To give you a sense of the effort involved, previous version upgrades have taken two full-time lead developers and 2.5 community support staff over two years. On top of this, there is much technical complexity in sustainably hosting the sites (c. 2000 different sites at the moment, though a small fraction of these remains active), and then there will be ongoing issues and bug fixing, plus community management. By the time this is done, there will doubtless be demands to move to Drupal 11, which is already in development by the Drupal community. I say this not to dissuade any attempt to move to Drupal 10 but to be realistic about the resources involved. Migrating and hosting a single site is much more straightforward than moving all the sites. However, this would still be a considerable technical undertaking, especially as modules would need re-writing, in many cases from scratch, because of the changes in how Drupal 10 works.

Given this complexity, a more straightforward solution would be to move to a more sustainable platform. As part of the review, we looked at many options, including developing a new simpler system that would be more sustainable and address some of the broader issues around the Scratchpad data model. Our review did not reveal any system that remotely approaches the breadth of functionality offered by Scratchpads. The one system that came close is TaxonWorks, and this is why we have worked with the TaxonWorks team in Illinois to help make this transition for those interested and able. However, we also recognise that TaxonWorks is not suitable for everyone. TaxonWorks is quite different from Scratchpads in several respects, and it focuses on some of the more traditional (taxon-based) use cases that Scratchpad deals with. As a result, some communities exclusively focused on particular categories of data (e.g. species observations, taxonomic literature or geographically constrained sites) will find that move more challenging. Likewise, some heavily customised sites are going to find any more difficult.

In some cases, collections management systems might be more appropriate for these data. We looked at Specify and the Living Atlas framework as part of the review, and for a small number of sites, these would be a good fit depending on the distribution of their contributor community. However, that also makes it difficult to provide scalable support for every site to export their content because there is no simple way to automate this. The best we can do is provide access to that structured data and guidance to the community, whether with Taxonworks or another system.

The last option was to develop a new platform. Technology has moved on since we launched the Scratchpads on Drupal 15 years ago, so this is not quite as much of an undertaking as it sounds. However, it would still require considerable new financial support. Despite our best efforts and some small offers from the community, it is improbable that we can realise the required funding level, especially when so many other pressing issues draw upon our digital expertise.

Much of this relates to approaches in how we integrate data as a community, either through new data models, community curation and advances in new technologies like AI, which are transforming what we can do and how we do it.

To this end, I will be at the upcoming SPNHC meeting in San Francisco (May 28 – June 2) if any of you are planning to attend, and I would be happy to discuss this further. Likewise, if you have further questions or comments, post them here so I or another team member can respond.

Thanks again for your interest
Vince Smith, NHM UK.

4 Likes

FYI that some from the Scratchpads community have reached out to Symbiota Support Hub (https://symbiota.org/) and we’re responding and will explore what may work. Open to more inquiries.

1 Like

I’ll note that I had a Zoom call with Vince last week that helped me understand a bit better how we got to where we are and I thank him for that. Still, I am a bit amazed that reproducing more or less what a Scratchpad could do in 2011 is such a heavy lift in 2023. Perhaps their flaw was being too far ahead of their time. I still need help understanding why there is so little felt need in the biodiversity informatics community for something like them at the present time. I review proposals for a small foundation that awards grants to biodiversity informatics projects in Africa. Every proposal says they will mobilize data and put them on a website. What website? Built from scratch? I’ve seen some of these websites and it’s not pretty. Some use Symbiota Portals, but currently those don’t reproduce many Scratchpad functions. Why isn’t developing something like next-gen Scratchpads a priority for GBIF, DISSCO, NSF and everyone else? Free hosting and IT was nice, but not necessary now that cloud hosting is relatively inexpensive. But we need a taxonomy-oriented, extensible, customizable, content management system employing responsive web design (i.e. will look good on a phone) that doesn’t require an IT department to set up.

Fascinating to read the developing thread on the development of scratchpads and attempts to help rescue data held in the scratchpad coffins.

If only there was a way to just export all the data. . . and preserve them as is.

To me, this is a reminder that current/future platforms might want to consider thinking about migration / archiving now, especially because a fluid archiving / export workflow takes time to mature . . .

For instance, how to archive/cite versions of GBIF, the source data they used, the taxonomic interpretations applied to their mediated/interpreted data products.

Similar questions can be asked about TaxonWorks, Symbiota, Arctos, etc.

Thanks again for sharing such details on the context in which Scratchpads went from actively managed to a static resource, and how their communities responded to this.

All the Scratchpad sites are fully Open Access with no APC (article processing fees) either as we’re all volunteer expert naturalists
Solutions should be OA minus APC else we’re at variance with GBIF’s principles of Open Data

@Darwyn Great to hear that all the Scratchpad sites are fully Open Access. Do you have any procedures in place to access that open data in bulk/batch?

Sensu publications as in https://www.frontiersin.org/articles/10.3389/fnins.2018.00656/full plus the movie https://paywallthemovie.com/ together with various reports and reviews in Nature and New Scientist in 2018 - a libararian’s perspective too at The Use of Electronic-Only Journals in Scientific Research. | Issues in Science and Technology Librarianship Still currently the subject of campaigns and actions such as mass walkout by the entire board of a science journal (‘Too greedy’: mass walkout at global science journal over ‘unethical’ fees | Peer review and scientific publishing | The Guardian)
So in our context they’re valuable information sources which no-one’s going to charge you to read or charge authors to submit.
Duplicate terminology can be an unfortunate barrier, “Open Access” is also used to describe levels of access to parts of the countryside.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.