Case Study · Systems Performance

Distributed MapReduce Framework

A distributed MapReduce framework in C/C++ that runs across EC2 instances with gRPC control, Protobuf-based shuffle records, and HDFS-backed output.

This is the clearest example of how I debug systems: measure first, find the actual bottleneck, and then change the design instead of guessing.

View Repository Get in touch

Throughput

7x improvement after isolating reducer lock contention

Deployment

Distributed across EC2 mapper, reducer, and master nodes

Key insight

More mapper concurrency slowed the job once the reducer became the bottleneck

What I built

Designed and implemented the framework end to end as part of graduate systems work, including evaluation across multiple cluster and buffer configurations.

C++gRPCProtobufAWS EC2HDFSpthreads

A master node coordinates workers and reducer instances over gRPC and loads runtime cluster configuration.

Mapper workers run multi-threaded map logic, batch intermediate records, and stream them to the reducer over custom TCP shuffle channels.

The reducer merges intermediate streams, invokes user-defined reduce functions, and writes final output to HDFS.

Results

Identified that barrier-off runs could become 6–7x slower than barrier-on runs because the reducer serialized producer progress behind a shared lock.

Found that adding more mapper units did not improve throughput once the single reducer saturated, which changed how I thought about scaling the system.

Narrowed the best-performing configuration toward balanced mapper parallelism and moderate shuffle batch sizes instead of maxing out every concurrency knob.

Hard problems

Reducer lock contention in non-barrier mode

The reducer held the same mutex used by producer RPC handlers while parsing batches and calling reduce logic. That serialized the entire pipeline and made non-barrier mode dramatically slower on large runs.

Scaling mapper count without overloading the reducer

Increasing mapper units from 1 to 4 increased shuffle pressure, socket activity, and reducer-side contention. The evaluation showed the reducer, not the mappers, was the real throughput limit.

Buffer sizing and burst behavior

Larger buffers reduced flush frequency but also made the system burstier. Moderate buffer sizes delivered steadier flow once reducer lock scope and backpressure behavior were accounted for.

Reliability and verification

Ran repeated evaluation sweeps across mapper counts, thread counts, buffer sizes, and barrier modes to compare means and variance rather than trusting one-off timings.

Used bounded buffers and producer-consumer synchronization to avoid unbounded memory growth during shuffle pressure.

Documented a concrete reducer-side fix: copy under lock, unlock, then parse and reduce outside the critical section.

Tradeoffs and lessons

A single reducer simplified ordering and implementation, but it became the dominant scalability limit.

TCP shuffle reduced overhead for bulk data transfer, but required explicit backpressure and concurrency management.

Barrier mode improved stability under pressure, but reduced overlap between mappers and the reducer.