Dear Darwyn and colleagues,
Many thanks for starting this thread. I really appreciate this, and the whole NHM Scratchpad team (past and present) is very grateful for the appreciation of the Scratchpads. First, I wanted to reiterate a few points to clear up any misconceptions, and then I will say a few words on possible solutions. I know there are several fora where comments about the Scratchpads have been posted. However, I will try to reply to queries here, with some links to the Scratchpad blog and Twitter where appropriate, so I would encourage you and others to post to this forum to keep the queries together.
Firstly, we are making the sites read-only because contributions have dramatically reduced over the past few years; the sites have become much harder to maintain; finding capable developers to do that maintenance has become very difficult; and we have been unable to secure any significant funding to undertake the upgrades necessary to maintain the security across the many thousands of sites.
The timing for the decision relates to the fact that Drupal version 7 reaches ‘end of life’ in November 2023, so recognising that the small number of active Scratchpad users will need time to come up with alternative plans, we wanted to maximise the notice available for all concerned.
Also, to reiterate, after September 2023, all sites, unless requested by the site maintainer, will remain accessible on the web, and the structured data behind each site will be preserved, essentially in perpetuity. Only write access will cease. We will continue to provide support to maintain read access and for site maintainers to get access to that structured data, should they request it.
Regarding alternatives, some have suggested that upgrading the Scratchpads to Drupal version 10 is the best course of action and that a “Save our Scratchpads” (SOS) group has been formed with this in mind. I encourage community self-support but be under no illusion about the magnitude of this task. We looked extensively at this (and other options) over the past four years, carefully examining a range of options, but ruled this out based on time, cost and complexity.
Scratchpads use over 200 Drupal modules. Most of these are contributed by Drupal users, and most are not available for Drupal 10, so re-writing these modules to version 10 would be a massive technical and social undertaking. In addition, there are many tens of custom modules (a little under half) that we have written in the Scratchpad team to make the Scratchpads functional. Upgrading these would be another massive technical undertaking, and its worth bearing in mind that our current full-time Scratchpad developer (with us since 2021) has been exclusively focused on community management and bug fixing rather than writing any Scratchpad modules or adding new functionality. To give you a sense of the effort involved, previous version upgrades have taken two full-time lead developers and 2.5 community support staff over two years. On top of this, there is much technical complexity in sustainably hosting the sites (c. 2000 different sites at the moment, though a small fraction of these remains active), and then there will be ongoing issues and bug fixing, plus community management. By the time this is done, there will doubtless be demands to move to Drupal 11, which is already in development by the Drupal community. I say this not to dissuade any attempt to move to Drupal 10 but to be realistic about the resources involved. Migrating and hosting a single site is much more straightforward than moving all the sites. However, this would still be a considerable technical undertaking, especially as modules would need re-writing, in many cases from scratch, because of the changes in how Drupal 10 works.
Given this complexity, a more straightforward solution would be to move to a more sustainable platform. As part of the review, we looked at many options, including developing a new simpler system that would be more sustainable and address some of the broader issues around the Scratchpad data model. Our review did not reveal any system that remotely approaches the breadth of functionality offered by Scratchpads. The one system that came close is TaxonWorks, and this is why we have worked with the TaxonWorks team in Illinois to help make this transition for those interested and able. However, we also recognise that TaxonWorks is not suitable for everyone. TaxonWorks is quite different from Scratchpads in several respects, and it focuses on some of the more traditional (taxon-based) use cases that Scratchpad deals with. As a result, some communities exclusively focused on particular categories of data (e.g. species observations, taxonomic literature or geographically constrained sites) will find that move more challenging. Likewise, some heavily customised sites are going to find any more difficult.
In some cases, collections management systems might be more appropriate for these data. We looked at Specify and the Living Atlas framework as part of the review, and for a small number of sites, these would be a good fit depending on the distribution of their contributor community. However, that also makes it difficult to provide scalable support for every site to export their content because there is no simple way to automate this. The best we can do is provide access to that structured data and guidance to the community, whether with Taxonworks or another system.
The last option was to develop a new platform. Technology has moved on since we launched the Scratchpads on Drupal 15 years ago, so this is not quite as much of an undertaking as it sounds. However, it would still require considerable new financial support. Despite our best efforts and some small offers from the community, it is improbable that we can realise the required funding level, especially when so many other pressing issues draw upon our digital expertise.
Much of this relates to approaches in how we integrate data as a community, either through new data models, community curation and advances in new technologies like AI, which are transforming what we can do and how we do it.
To this end, I will be at the upcoming SPNHC meeting in San Francisco (May 28 – June 2) if any of you are planning to attend, and I would be happy to discuss this further. Likewise, if you have further questions or comments, post them here so I or another team member can respond.
Thanks again for your interest
Vince Smith, NHM UK.