Genome Sequencing Pipeline

A genomics lab's sequencers produce raw data files of several hundred gigabytes per sample, uploaded to S3 as each run finishes. Each sample goes through an alignment and variant-calling pipeline of containerised tools that need dozens of CPU cores and a lot of memory for several hours, with hundreds of samples queued at busy times. The tools read the same reference genome and intermediate files thousands of times, so they need a shared, very high-throughput file system rather than fetching from object storage over and over. Jobs should run on the cheapest spare capacity available and restart automatically if an instance is reclaimed, and the lab does not want to run a scheduler or cluster itself.