"Download All Responses" button and API endpoint for retrieving guestbook responses time out when used to get responses from certain collections

[jggautier]

What steps does it take to reproduce the issue?

• When does this issue occur?
  When trying to get all guestbook responses for a collection by:

  • Going to the collection's "Manage Dataset Guestbooks" page and clicking "Download All Responses"
  • Using the Retrieve Guestbook Responses for a Dataverse Collection endpoint described at https://guides.dataverse.org/en/6.9/api/native-api.html#retrieve-guestbook-responses-for-a-dataverse-collection

• Which page(s) does it occurs on?

  • On the "Manage Dataset Guestbooks" pages of certain collections in Harvard Dataverse
  • In terminals and IDE's and where ever else users might use the API endpoint to get guestbook responses of certain collections in Harvard Dataverse

• What happens?

  • After clicking "Download All Responses" and waiting for a bit, a "504 Gateway Time-out" page appears

  • After trying to use the API endpoint and waiting for a bit, the endpoint returns "504 Gateway Time-out"
    Endpoint runs for about a minute...
    

    ... and times out
    

  Sometimes users are able to get the responses, using the "Download All Responses" button or the API endpoint, after a few tries.

• To whom does it occur (all users, curators, superusers)?
  Superusers and managers of collections

• What did you expect to happen?

• The guestbook always downloads as a CSV file, at least for collections with fewer file downloads
• The "Retrieve Guestbook Responses for a Dataverse Collection endpoint" always returns the guestbook responses instead of a 504 timeout error, even for guestbooks with many file downloads. We created this endpoint for cases where users can't view or download their guestbook respones using their browsers because there are two many responses. See IQSS/dataverse #7767

Which version of Dataverse are you using?
v6.9 - v6.11

Any related open or closed issues to this bug report?

• Unable to download certain guestbooks, possibly related to number of responses (~150,000 responses) #7767

[jggautier]

For collections with too many guestbook responses, their managers aren't able to get the collections' guestbook responses through their browsers, because the Download All Responses button on the Manage Dataset Guestbooks page doesn't work.

So we created the Retrieve Guestbook Responses for a Dataverse Collection endpoint so that they could use the API to get their collections' guestbook responses. See #7767 and its PR at #7931.

The endpoint times out when trying to get guestbook responses of collections in Harvard Dataverse with the most guestbook responses:

• https://dataverse.harvard.edu/dataverse/pkgw
• https://dataverse.harvard.edu/dataverse/Kencorpus
• https://dataverse.harvard.edu/dataverse/held-extracted-data
• https://dataverse.harvard.edu/dataverse/the_review
• https://dataverse.harvard.edu/dataverse/qje
• https://dataverse.harvard.edu/dataverse/data_ncov
• https://dataverse.harvard.edu/dataverse/JPE
• https://dataverse.harvard.edu/dataverse/IFPRI
• https://dataverse.harvard.edu/dataverse/restat

The endpoint does work for collections that have fewer responses. It seems like if the collection has two million or more responses, the endpoint times out, but I can't be sure about this or in general about how related the number of responses is to the time outs.

[ekraffmiller]

While we are working on this, it would also be good to support paging. We will need it to display guestbook responses in the SPA.

[jggautier]

I was wondering how this might affect the SPA

[jggautier]

Just also noting that for at least a few months now it's also felt like this endpoint takes longer to run in general, even when trying to get the responses of guestbooks with way fewer than 2 million responses. I have no hard evidence of this, just a feeling.

[cmbz]

2026-06-09: Added 6.12 milestone. @ekraffmiller, please add to the Sprint Planning doc and advocate for sizing this issue during our next Sprint Planning meeting.