File Service

The legacy systems had accumulated a specific category of failure: silent errors. Files would disappear from a depositor's workspace without explanation. Uploads would fail, but the user was not notified — file metadata would appear in the deposit workspace as if the upload had succeeded, and the depositor would discover the error only when attempting to publish, sometimes months later. Processing multiple large deposits simultaneously could create backlogs lasting more than 24 hours, with no visibility into why.

The failures weren't random — they had a structural cause. File handling was built into each product individually rather than operating as an independent service. The architecture made it impossible to monitor, upgrade, or scale file processing without touching every product that depended on it.

The File Service solved this at the infrastructure level: a single, modular service accessed by all The Archive products. Files upload directly from the browser to S3 — bypassing intermediate servers that introduced timeout failures. A configurable processing pipeline handles virus scanning, metadata extraction, and storage checks independently, so a failure in one stage doesn't corrupt another. Every stage is logged and visible to both staff and depositors in real time.

User spectrum: Designing for the Full Range

The Archive's depositor population spans an enormous range of technical sophistication and motivation. The design couldn't optimize for one end of that range without failing the other.

‍

The Two-Axis Display Problem

The core UX insight from design working sessions: file list behavior is governed by two independent axes that must be treated separately.

My solution: metadata priority tiers that assign display conditions based on both axes together. A field earns its place based on context importance and screen size simultaneously, not independently.

The File Service had to handle use cases that exposed the limits of the previous system in the most concrete terms. One project brought more than 11TB of video files digitized from 1,500 VHS tapes — a single deposit larger than anything the legacy system could process. Another organization regularly receives deposits with tens of thousands of files and strict publication deadlines. A third delivers a large quarterly file via SFTP that needed automated ingestion without manual intervention.

Each of these wasn't just a performance problem — it was a UX problem. When a large upload is running for hours, the user needs to know it's working. When a batch of 10,000 files is ingesting in parallel, the status display needs to communicate aggregate progress without becoming a wall of individual status indicators. I designed the progress and status experience for these edge cases as primary scenarios, not exceptions.

The File Service shipped and is available for production use since March 2025. It replaced two legacy systems and became the shared file infrastructure for all The Archive products.

The File Service taught me that infrastructure design is user experience design. The pipeline behind the upload button is invisible to the user — until it fails. When it fails silently (as the legacy system did), it creates a specific kind of distrust that is very hard to recover from: the user stops believing the system is telling them the truth. Making the pipeline visible — its stages, its progress, its errors — transforms infrastructure from an invisible assumption into a legible, trustworthy contract.