CSC 2/458:  Parallel and Distributed Systems

Spring 2017.

Assignment 4:  MPI and Hadoop

This is a (hopefully) small assignment to be completed concurrently with the end-of-semester project.  Its goal is to familiarize you with the basics of MPI, the package of choice for cluster-based high-performance computing, and Hadoop, the open-source package for Map-Reduce cluster-based data analytics. 

For MPI, your assignment is to solve the same Gaussian elimination problem that you tackled in Assignment 3.  For Hadoop, your assignment is to generate a histogram of word lengths found in a set of input documents. 

With the help of department lab staff, we have created a small dedicated cluster on the csug network, with installations of both MPI and Hadoop.  All students in the class (including those who have previously worked on the cs network) will need to work at csug for this assignment. 

Documentation for MPI can be found at the MPI Forum.  Documentation for Hadoop can be found at the Apache Foundation.  Be sure to check out the tutorial and the commands reference.  Instructions for using both packages on the local cluster will be posted shortly to the course site on Blackboard


Your MPI program should implemented a blocked (banded) solution.  If you are registered for 458, you should implement smart pivoting (extra credit for 258 students).  Your README file should compare performance to what you saw in assignment 3.  Be sure to test your code using the input files from that assignment. 


Your Hadoop program should take a set of text files as input, and should produce a histogram of word lengths found in those documents—something along the following lines:

  1: ****               1234
  2: *******            2345
  3: ********           2500
  4: *************      4003
  5: **************     4212
  6: ************       3752
  7: *******            2302
  8: ********           2416
  9: ******             1987
 10: *****              1612
 11: *****              1520
 12: **                  829
 13: ***                1006
 14: **                  775
 15:                      40
 16: *                   312
 17:                      11
 18:                       0
 19:                       9
20+: ***            1113

The idea here is that the number of stars should be approximately proportional to the number of words of the given length (specified precisely in the right-hand column).  Please adjust the scaling to keep the width of the output below 80 characters. 

To simplify grading, please adopt the convention that words are separated by white space and that the length of a word is the number of letters it contains.  So “mother-of-pearl” has length 13 (the dashes don’t count).  Zero-length words (those consisting entirely of non-letters) should not be reported.  So “Plan 9” is two words—one of length 4 and a second that you should ignore. 

A set of test input files will be posted shortly to Blackboard. 


As in past assignments, follow the turnin directions.  Questions can be posted to the discussion board.  We will be grading the assignment on a mix of completeness and correctness, programming style, and quality of write-up. 

As in the past two assignments, be sure to leave plenty of time for experiments and debugging.  If you wait too long, you won’t be able to get stand-alone time on the cluster. 

Due Date: Friday, April 28, at end of day.

Last Change:14 April 2017 / Michael Scott's email address