A Spark node keeps its data sets, the RDDs, in a least recently used cache in memory.
A node, S1, has 10 GB of memory and 100 GB of data to bring in. The data is streamed, and the data set being worked on is kept in memory.
S1 runs three jobs, J1, J2 and J3, and the memory cannot hold all their data:
- The disk. Data that does not fit moves to disk and can be pulled from disk.
- The order. First in, first out keeps J3, the latest job, and moves the RDDs of J1 to disk. Least recently used gives preference to a more active J1.
- The level. Data is evicted on the level of the data set, not the job.
A cache holds DS1 to DS6, which belong to J1, J2 and J3. When DS7 arrives and the memory has run out, the least recently used data set is kicked out. It could be DS5, DS3 or DS2.
Recently used data stays in memory the longest, which reduces the number of disk operations.