Spark sends the next operation to the node that already holds its data in memory.
The cluster manager does not do the placing. It keeps nodes alive, adds nodes and removes nodes. Spark has a management library, which sends operations where they belong:
- A data set goes to node S1 and the computation completes. The result, RDD2, is in the memory of S1.
- RDD2 goes through operation 2. S1 has RDD2 in memory, so operation 2 goes to S1, and no bandwidth is used to send results to another server.
- An add operation on DS1 gives DS2. The next operation is mult on DS2. The map says DS2 exists in S1, so mult is assigned to S1.
After a crash a new node is kept in the same rack, else the same availability zone, else the same region.
An RDD is a read-only partitioned collection of records. Its preferred locations are a list of nodes, which are tagged. Kubernetes or Mesos is the cluster manager, and ZooKeeper is used for leader election.