This is a (hopefully) small assignment to be completed concurrently with the end-of-semester project. Its goal is to familiarize you with the basics of MPI, the package of choice for cluster-based high-performance computing, and Hadoop, the open-source package for Map-Reduce cluster-based data analytics.
For MPI, your assignment is to solve the same Gaussian elimination problem that you tackled in Assignment 3. For Hadoop, your assignment is to generate a histogram of word lengths found in a set of input documents.
With the help of department lab staff, we have created a small dedicated
cluster on the
csug network, with installations of both MPI
All students in the class (including those who have previously worked on
cs network) will need to work at
Documentation for MPI can be found at the MPI Forum. Documentation for Hadoop can be found at the Apache Foundation. Be sure to check out the tutorial and the commands reference. Instructions for using both packages on the local cluster will be posted shortly to the course site on Blackboard.
Your MPI program should implemented a blocked (banded) solution.
If you are registered for 458, you should implement smart pivoting
(extra credit for 258 students).
README file should compare performance to what you saw
in assignment 3.
Be sure to test your code using the input files from that
Your Hadoop program should take a set of text files as input, and should produce a histogram of word lengths found in those documents—something along the following lines:
1: **** 1234 2: ******* 2345 3: ******** 2500 4: ************* 4003 5: ************** 4212 6: ************ 3752 7: ******* 2302 8: ******** 2416 9: ****** 1987 10: ***** 1612 11: ***** 1520 12: ** 829 13: *** 1006 14: ** 775 15: 40 16: * 312 17: 11 18: 0 19: 9 20+: *** 1113
The idea here is that the number of stars should be approximately proportional to the number of words of the given length (specified precisely in the right-hand column). Please adjust the scaling to keep the width of the output below 80 characters.
To simplify grading, please adopt the convention that words are separated by white space and that the length of a word is the number of letters it contains. So “mother-of-pearl” has length 13 (the dashes don’t count). Zero-length words (those consisting entirely of non-letters) should not be reported. So “Plan 9” is two words—one of length 4 and a second that you should ignore.
A set of test input files will be posted shortly to Blackboard.
As in past assignments, follow the turnin directions. Questions can be posted to the discussion board. We will be grading the assignment on a mix of completeness and correctness, programming style, and quality of write-up.
As in the past two assignments, be sure to leave plenty of time for experiments and debugging. If you wait too long, you won’t be able to get stand-alone time on the cluster.