My first foray into big data processing was managing this old-school Hortonworks Hadoop cluster, at this company that was fiercely adamant about running all of their servers on-prem, as if cloud computing were taboo. One of my first tasks was to determine what kind of RAID configuration we should use for our new fleet of clusters. We were to use Apache Spark, a new-ish big data processing framework -- it was technically around since the early 2010s but had only recently been promoted to an Apache top-level project -- to process trillions of records of ad auction data from a stream into nicely-sized HDFS blocks, in the form of snappy-compressed parquet files.

That was a little over 10 years ago. Nowadays, most of the data engineers I know would raise a confused eyebrow if I mentioned any of those terms above -- though, they may also raise an eyebrow at yet another explain-y tangent of mine, but I digress. Gone are the days of debating which compression algorithm works with which file format, or navigating the delicate balancing act of tuning shuffle partitions — skew on one side, small HDFS blocks on the other.

Nowadays, almost all of that is abstracted away. We've finally moved away from Hadoop -- turns out, Kubernetes is actually a decent scheduler, who'd've thought? -- and with the advent of highly-available (and ridiculously cheap) cloud object storage, we've been able to separate compute and storage to the point where we barely have to think about cluster management anymore. Sure, we pay a network transfer cost every single time we run a job, but that provides a great balancing mechanism and the dreaded Skew Bogeyman rarely shows itself.

But what's most interesting for me has been this new generation of data processing technologies that's cropped up in the past few years: DuckDB, Polars, and DataFusion. They all leverage Apache Arrow, gear towards everything in volatile memory, and bias towards local execution. One could (glibly) jump to the conclusion that RAM is cheap, but outside of being terribly wrong, they would also miss an interesting trend that I think is starting to take shape: 1. You don't need big data as much as you think. By leaning into local processing, you optimize for local reproducibility and ease of user experience -- the same DSL can run against a single node or a multi-cluster topology. 2. In-memory serialization (i.e., Arrow) matters more than reliability. If a node in a cluster goes down, there's a real cost to spinning up another node and reloading the data lost from volatile memory -- but optimizing the topology for computation is worth that occasional spot loss. 3. The emergence of web/client-side processing (the WASM is seriously powerful) fundamentally alters the shape of your data. Offline processing doesn't need to be a big bulky batch job anymore. As big as your cluster can get, it's still dwarfed by the aggregate compute your clients have available. Browsers today aren't even hiding the fact that they're essentially a thin wrapper over your CPUs, so offloading to the client reduces workload and increases flexibility in analyzing your semi-raw aggregated data.

Looking back, it's a strange kind of full circle. 10 years ago, my job was squeezing more performance out of physical disks in a room I could actually walk into. Today, the frontier is squeezing more performance out of memory I'll never see, running in a browser tab I don't control. The scale changed, the abstractions changed, but the underlying question -- how do you move data to compute as cheaply and predictably as possible -- hasn't. I just think we're finally getting more creative about where "compute" even lives. Interested to see where the next 10 years takes us.