sqrtspace-experiments/case_studies/distributed_computing/README.md

Distributed Computing: Space-Time Tradeoffs at Scale

Overview

Distributed systems make explicit decisions about replication (space) vs computation (time). Every major distributed framework embodies these tradeoffs.

1. MapReduce / Hadoop

Shuffle Phase - The Classic Tradeoff

// Map output: Written to local disk (space for fault tolerance)
map(key, value):
    for word in value.split():
        emit(word, 1)

// Shuffle: All-to-all communication
// Choice: Buffer in memory vs spill to disk
shuffle.memory.ratio = 0.7  // 70% of heap for shuffle
shuffle.spill.percent = 0.8 // Spill when 80% full

Memory Settings Impact:

• High memory: Fast shuffle, risk of OOM
• Low memory: Frequent spills, 10x slower
• Sweet spot: √(data_size) memory per node

Combiner Optimization

// Without combiner: Send all data
map: (word, 1), (word, 1), (word, 1)...

// With combiner: Local aggregation (compute for space)
combine: (word, 3)

// Network transfer: 100x reduction
// CPU cost: Local sum computation

2. Apache Spark

RDD Persistence Levels

// MEMORY_ONLY: Fast but memory intensive
rdd.persist(StorageLevel.MEMORY_ONLY)
// Space: Full dataset in RAM
// Time: Instant access

// MEMORY_AND_DISK: Spill to disk when needed
rdd.persist(StorageLevel.MEMORY_AND_DISK)
// Space: Min(dataset, available_ram)
// Time: RAM-speed or disk-speed

// DISK_ONLY: Minimal memory
rdd.persist(StorageLevel.DISK_ONLY)
// Space: O(1) RAM
// Time: Always disk I/O

// MEMORY_ONLY_SER: Serialized in memory
rdd.persist(StorageLevel.MEMORY_ONLY_SER)
// Space: 2-5x reduction via serialization
// Time: CPU cost to deserialize

Broadcast Variables

// Without broadcast: Send to each task
val bigData = loadBigDataset() // 1GB
rdd.map(x => doSomething(x, bigData))
// Network: 1GB × num_tasks

// With broadcast: Send once per node
val bcData = sc.broadcast(bigData)
rdd.map(x => doSomething(x, bcData.value))
// Network: 1GB × num_nodes
// Memory: Extra copy per node

3. Distributed Key-Value Stores

Redis Eviction Policies

# No eviction: Fail when full (pure space)
maxmemory-policy noeviction

# LRU: Recompute evicted data (time for space)
maxmemory-policy allkeys-lru
maxmemory 10gb

# LFU: Better hit rate, more CPU
maxmemory-policy allkeys-lfu

Memcached Slab Allocation

• Fixed-size slabs: Internal fragmentation (waste space)
• Variable-size: External fragmentation (CPU to compact)
• Typical: √n slab classes for n object sizes

4. Kafka / Stream Processing

Log Compaction

# Keep all messages (max space)
cleanup.policy=none

# Keep only latest per key (compute to save space)
cleanup.policy=compact
min.compaction.lag.ms=86400000

# Compression (CPU for space)
compression.type=lz4  # 4x space reduction
compression.type=zstd # 6x reduction, more CPU

Consumer Groups

• Replicate processing: Each consumer gets all data
• Partition assignment: Each message processed once
• Tradeoff: Redundancy vs coordination overhead

5. Kubernetes / Container Orchestration

Resource Requests vs Limits

resources:
  requests:
    memory: "256Mi"  # Guaranteed (space reservation)
    cpu: "250m"      # Guaranteed (time reservation)
  limits:
    memory: "512Mi"  # Max before OOM
    cpu: "500m"      # Max before throttling

Image Layer Caching

• Base images: Shared across containers (dedup space)
• Layer reuse: Fast container starts
• Tradeoff: Registry space vs pull time

6. Distributed Consensus

Raft Log Compaction

// Snapshot periodically to bound log size
if logSize > maxLogSize {
    snapshot = createSnapshot(stateMachine)
    truncateLog(snapshot.index)
}
// Space: O(snapshot) instead of O(all_operations)
// Time: Recreate state from snapshot + recent ops

Multi-Paxos vs Raft

• Multi-Paxos: Less memory, complex recovery
• Raft: More memory (full log), simple recovery
• Tradeoff: Space vs implementation complexity

7. Content Delivery Networks (CDNs)

Edge Caching Strategy

# Cache everything (max space)
proxy_cache_valid 200 30d;
proxy_cache_max_size 100g;

# Cache popular only (compute popularity)
proxy_cache_min_uses 3;
proxy_cache_valid 200 1h;
proxy_cache_max_size 10g;

Geographic Replication

• Full replication: Every edge has all content
• Lazy pull: Fetch on demand
• Predictive push: ML models predict demand

8. Batch Processing Frameworks

Apache Flink Checkpointing

// Checkpoint frequency (space vs recovery time)
env.enableCheckpointing(10000); // Every 10 seconds

// State backend choice
env.setStateBackend(new FsStateBackend("hdfs://..."));
// vs
env.setStateBackend(new RocksDBStateBackend("file://..."));

// RocksDB: Spill to disk, slower access
// Memory: Fast access, limited size

Watermark Strategies

• Perfect watermarks: Buffer all late data (space)
• Heuristic watermarks: Drop some late data (accuracy for space)
• Allowed lateness: Bounded buffer

9. Real-World Examples

Google's MapReduce (2004)

• Problem: Processing 20TB of web data
• Solution: Trade disk space for fault tolerance
• Impact: 1000 machines × 3 hours vs 1 machine × 3000 hours

Facebook's TAO (2013)

• Problem: Social graph queries
• Solution: Replicate to every datacenter
• Tradeoff: Petabytes of RAM for microsecond latency

Amazon's Dynamo (2007)

• Problem: Shopping cart availability
• Solution: Eventually consistent, multi-version
• Tradeoff: Space for conflict resolution

10. Optimization Patterns

Hierarchical Aggregation

# Naive: All-to-one
results = []
for worker in workers:
    results.extend(worker.compute())
return aggregate(results)  # Bottleneck!

# Tree aggregation: √n levels
level1 = [aggregate(chunk) for chunk in chunks(workers, sqrt(n))]
level2 = [aggregate(chunk) for chunk in chunks(level1, sqrt(n))]
return aggregate(level2)

# Space: O(√n) intermediate results
# Time: O(log n) vs O(n)

Bloom Filters in Distributed Joins

// Broadcast join with Bloom filter
BloomFilter filter = createBloomFilter(smallTable);
broadcast(filter);

// Each node filters locally
bigTable.filter(row -> filter.mightContain(row.key))
        .join(broadcastedSmallTable);

// Space: O(m log n) bits for filter
// Reduction: 99% fewer network transfers

Key Insights

1. Every distributed system trades replication for computation

2. The √n pattern appears in:

   • Shuffle buffer sizes
   • Checkpoint frequencies
   • Aggregation tree heights
   • Cache sizes

3. Network is the new disk:

   • Network transfer ≈ Disk I/O in cost
   • Same space-time tradeoffs apply

4. Failures force space overhead:

   • Replication for availability
   • Checkpointing for recovery
   • Logging for consistency

Connection to Williams' Result

Distributed systems naturally implement √n algorithms:

• Shuffle phases: O(√n) memory per node optimal
• Aggregation trees: O(√n) height minimizes time
• Cache sizing: √(total_data) per node common

These patterns emerge independently across systems, validating the fundamental nature of the √(t log t) space bound for time-t computations.