Things only get interesting at scale.
Now we have 100’s of files, lots of jobs, and we’re overwhelming the EC2 worker.