feat(prepro): basic validation of raw read submissions

[maverbiest]

Partially resolves #6758

This PR adds general functionality to validate user-submitted files in the nextclade preprocessing pipeline. How files are processed is determined based on the FileCategory and a simple switch statement.

The PR introduces a function to validate a submission of raw sequencing reads as a first use case:

def validate_raw_reads_submission(
    files: list[FileIdAndName],
) -> tuple[list[ProcessingAnnotation], list[ProcessingAnnotation]]:

The validation is very basic for the moment, and only considers the number of submitted files and their files extensions. This could be built out in the future to include more robust checks.

Alternative approaches

Rather than having a switch statement on FileCategory in prepro to determine how different files are processed, we could define this in the values.yaml, similar to how we do for metadata fields. E.g.:

    submissionDataTypes:
      files:
        enabled: true
        categories:
          - name: raw_reads
            displayName: Raw reads
            preprocessing:
              function: validate_raw_reads_submission
          ...

I decided against this for now to get a working version with a minimal diff, but I would be happy to implement this if people feel it's better.

PR Checklist

• All necessary documentation has been adapted.
• The implemented feature is covered by appropriate, automated tests.
• Any manual testing that has been done is documented (i.e. what exactly was tested?)

🚀 Preview: https://simple-fq-validation.loculus.org

[claude]

Claude finished @maverbiest's task in 2m 46s —— View job

---

Code Review

• Gather context
• Read changed files
• Review code quality, security, performance, testing, and documentation
• Post inline comments
• Summarize findings

---

Overall this is a clean, well-scoped PR. The architecture choices (typed enum instead of plain string, dispatching via match, integrating into both aligned/unaligned paths) are solid. One actual bug and a few minor issues noted below.

One bug to fix before merge:

file.name.endswith(extension) without a leading dot gives false positives. A file named reads.notfastq passes because "reads.notfastq".endswith("fastq") is True. Fix: use "." + extension in the check (see inline comment on line 1969).

Other observations (inline comments posted):

• The "too many files" annotation uses a semicolon-joined string as the source field name via from_single. Using from_fields with individual file names listed separately would be more consistent with the per-file annotation style used below it.
• The error message displays the extension list without leading dots (e.g. ['fastq', ...]), which won't match what users see in filenames — should show .fastq, etc. once the dot fix is applied.
• FileCategory.ANNOTATIONS is used for preprocessing output, not user input — the case _ branch will silently log a warning if it ever appears in submitted files, which may be confusing. Worth a short comment to document the distinction.
• No test covers the single-valid-file case (only the two-file case is tested as valid input).

[claude]

FileCategory.ANNOTATIONS is currently in the enum (added for internal use by upload_flatfiles), so if the backend ever sends annotations as an input category, this branch would log a warning. Worth noting for future maintainers that ANNOTATIONS is preprocessing-generated output, not a user-submittable input category — so it should never appear in unprocessed.files.

[maverbiest]

Yes currently when we add a new input file category, this match statement will need to be updated to handle it. That is as intended (see discussion of alternative in the PR description)

[tombch]

I'm wondering if this should emit a hard exception, as (if my understanding is correct) a file category could be added to the values.yaml, and so will be accepted in the backend, but then if its missing in this match statement, it would pass through without validation?

[tombch]

I guess a hard exception would prevent other categories being allowed for other users of loculus which might not be a good thing

[maverbiest]

This file caught some stray updates from running the formatter

[tombch]

Tested it out and this looks good to me!

[corneliusroemer]

Maybe worth moving the file related functions (this one and validate raw reads) to a new file to not grow this 2k line monstrosity even further

[anna-parker]

I think this name is a bit confusing as it implies this is a file submitted by the user but annotations are created by the prepro pipeline, I think just "File" is ok

[anna-parker]

This is a message for submitters so I would actually state we want to have paired-end or single-end raw reads, and thus accept a max of 2 files.

[anna-parker]

some small improvements but also looks good overall!

[anna-parker]

Regarding the alternative suggestion, I like it but I would hold off for now - its not required for the beta and we can make the config a bit nicer in a later step, you can create an issue for this though to track it :-)