The map and reduce phases of MapReduce split up the computing job
into chunks that standard machines can process in a short time. The various
steps of MapReduce are shown in Figure 4.5 and an example on a collection
consisting of two documents is shown in Figure 4.6. First, the input data,in our case a collection of web pages, are split into n splits where the size of
the split is chosen to ensure that the work can be distributed evenly (chunks
should not be too large) and efficiently (the total number of chunks we need
to manage should not be too large); 16 or 64 MB are good sizes in distributed
indexing. Splits are not preassigned to machines, but are instead assigned
by the master node on an ongoing basis: As a machine finishes processing
one split, it is assigned the next one. If a machine dies or becomes a laggard
due to hardware problems, the split it is working on is simply reassigned to
another machine.