Skip to content

Bounded RIS Reference Import

Outcome

Researchers can manually select and import a RIS reference file without first converting it to SyRF CSV or EndNote XML. The first delivery is intentionally limited to a safe, reviewable RIS path through the existing search-upload workflow.

This brief replaces the broad proposal in PR #2628. That proposal is retained in Git history as research input, but its architecture decisions and delivery claims are not authoritative.

Current State

PR #2971 delivered the bounded RIS slice on 2026-08-30. PR #3004 then closed the study-export formula-safety gap exposed during final review:

  • the upload dialog offers explicit EndNote XML, CSV, and RIS choices; RIS is manually selected and there is no format auto-detection;
  • LibraryFileType.Ris is append-only value 5; historical persisted values remain unchanged;
  • RIS travels through the existing signed-upload, object-storage notification, parser-registry, and Study persistence path;
  • validation is based on RIS structure rather than extension or MIME claims;
  • supported input encodings are strict UTF-8, UTF-8 BOM, UTF-16 LE/BE with BOM, and Windows-1252;
  • the original upload is retained, while successful imports produce a bounded augmented RIS object for the current workflow;
  • fatal validation, persistence, storage, and cancellation paths do not intentionally expose partially imported studies;
  • warnings are bounded and sanitised;
  • study CSV/TSV exports use SyRF's reversible marker for dangerous spreadsheet prefixes while preserving generated finite negative numbers as numeric values;
  • TY maps only to the current Study reference type and never to PRISMA acquisition-source provenance;
  • NBIB, BibTeX, CSL-JSON, external CSV profiles, and format auto-detection remain unimplemented follow-ups;
  • the approved future data direction uses system-scoped pmPublication records and immutable project-scoped Citation values, but that model is not yet implemented.

Delivered MVP Boundary

The first independently shippable slice is manual RIS import only.

Included

  • a new append-only LibraryFileType.Ris value;
  • an explicit RIS option in the existing upload dialog;
  • complete web, API, object-storage metadata, notifier, parser-registry, and current Study persistence wiring;
  • structural RIS validation based on file content, independent of extension and MIME claims;
  • bounded file, record, line, field, diagnostic, and in-memory behaviour;
  • documented support for UTF-8, UTF-8 BOM, UTF-16 BOM, and Windows-1252 input;
  • conservative mapping into fields already available on Study;
  • bounded, sanitised diagnostics without raw reference content or personal data;
  • cancellation and all-or-nothing failure behaviour;
  • retention of the original uploaded file so a future Citation migration can reprocess it;
  • safe CSV generation for any imported values later emitted to spreadsheet software;
  • automated fixtures and regression coverage for existing EndNote XML and CSV import.

The delivered default ceilings are 10 MiB for input and generated output, 50,000 records, 32 Ki characters per physical line, 128 Ki characters per field, 512 fields and 4,096 physical lines per record, 128 continuation lines per field, 100 stored diagnostics, and 8 MiB of mapped UTF-8 text per Study. Generated output uses an approximately 64 KiB bounded producer/consumer queue.

Not Delivered by This Slice

  • NBIB, BibTeX, or CSL-JSON import;
  • automatic format detection or a format-choice wizard;
  • external Web of Science, Scopus, EBSCO, or other CSV profiles;
  • PMID, MeSH, language, volume, issue, page, or other schema expansion;
  • pmPublication/Citation migration or global identifier reconciliation;
  • retirement or renumbering of historical file-type values;
  • staged or partially visible imports for files above the bounded MVP limits.

Architecture and Data Rules

Compatibility

Existing persisted file-type values remain readable. RIS is appended after all historical values; existing numeric assignments must never be reordered or reused.

The MVP may adapt parsed RIS records into the current Study sink. It must not introduce the obsolete pmReference/ImportRecord proposal or pretend the approved pmPublication/Citation direction is already implemented.

PRISMA Semantics

RIS publication types such as TY describe the cited work. They must not populate PRISMA acquisition-source provenance.

PRISMA sourceType and sourceName describe where a systematic search was conducted and remain properties of the SystematicSearch/search-import context. Generic RIS content cannot safely infer them.

Retention Gate

The gate was verified on 2026-08-30: current object-storage lifecycle rules do not expire original objects under the project-scoped Imported Search Libraries prefix, and the completed SearchImportJob retains its OriginalFileUrl. The implementation must preserve both behaviours. Any future lifecycle change that could expire originals before Citation backfill requires the minimal pmPublication/Citation persistence kernel first.

Validation and Resource Safety

The parser must reject malformed or mismatched content even when the extension, MIME type, or manual selection says RIS. Production uses fixed ceilings for:

  • uploaded bytes;
  • records per file;
  • characters per physical line and per field;
  • fields and continuation lines per record;
  • stored diagnostics;
  • total parsed studies held before commit.

Unknown tags may produce bounded warnings and be omitted from the current Study; structural errors are fatal. Fatal errors and cancellation leave no partially imported studies. Logs and API diagnostics identify the record and a sanitised reason category; malformed physical-line errors may also identify a line number. They do not echo titles, abstracts, notes, author contact details, or arbitrary raw values.

Conservative Field Mapping

The MVP maps only fields supported by the current Study model:

RIS data Current SyRF field Rule
TI/T1 Title Prefer TI, then T1; do not concatenate conflicting titles
AU/A1 Authors Preserve author order; join using the existing Study representation
AB/N2 Abstract Prefer AB, then N2
PY/Y1 Year Accept a valid four-digit year only
JO/JF/T2 Journal Use a deterministic documented precedence
DO DOI Strip a recognised DOI URL/prefix and validate conservatively
UR URL Accept an absolute HTTP(S) URL only
KW Keywords Preserve order and repeated tags within current field limits
TY Reference type Keep publication type separate from search-source provenance

Unsupported values remain available only in the retained original file until the approved Citation model can represent them.

Delivered Acceptance Criteria

  1. A user can select a valid RIS file and complete the existing import workflow into current Study records.
  2. Content that is not structurally valid RIS is rejected even if its filename or MIME type claims RIS.
  3. Valid fixtures from at least three independent exporters pass, covering LF/CRLF, supported encodings, repeated tags, continuation lines, and unknown tags.
  4. Empty, truncated, malformed, oversized, overlong-field, over-record-limit, and excessive-diagnostic fixtures fail safely and predictably.
  5. Cancellation or fatal parsing produces no partial imported studies.
  6. Historic enum values and EndNote XML/CSV import behaviour remain covered by regression tests.
  7. No raw bibliographic or personal values appear in logs or user-facing diagnostics.
  8. Study CSV/TSV exports neutralise spreadsheet formula prefixes for imported user-controlled values.
  9. The original RIS object remains project-authorised and available for future backfill for the agreed retention period.
  10. Focused unit, integration, UI, API-contract, notifier, resource-bound, and exact-head CI checks pass before merge; staging evidence is recorded separately in the delivery pull request.

Delivered Critical Path

  1. Confirm original-object retention and authorisation behaviour.
  2. Append the shared enum and wire the explicit format choice through every service boundary.
  3. Implement bounded decoding, structural validation, parsing, and conservative mapping.
  4. Add fixtures and focused integration/regression tests.
  5. Verify the complete upload-to-persistence path and staging behaviour.

Sequenced Follow-ups

  1. Audit new-upload handling for historical LivingSearchJson and add direct PubMed XML fixtures.
  2. Define PMID identity and Citation persistence, then add NBIB.
  3. Add confidence-based format detection with a manual override.
  4. Add known external CSV profiles after formula-safety coverage.
  5. Add CSL-JSON.
  6. Reassess BibTeX libraries and its macro/nesting threat model before implementation.
  7. Implement the approved pmPublication/Citation migration, backfill, provenance, privacy, and cross-project authorisation rules.

Resolved Delivery Gate

  • Original project-scoped reference uploads are retained without a configured expiry, and completed import jobs keep the original object URL for future Citation backfill.