Computer Science 248/448


Fall 2009

Homework 1
Due Fri Sep 4, 5pm
Homework 2
Due Fri Sep 18, 5pm
Homework 3
Due Wed Oct 14, 5pm
Homework 4
Due Wed Nov 18, 5pm

Your assignment is to implement IBM Model 1. You will train parameters using Expectation Maximization on a parallel French-English corpus, and evaluating the results on held-out test data in terms of model perplexity. In particular, your implementation should include:

Training data can be found here: /u/cs448/data/hw4/. This directory contains parallel French-English text from the Canadian Parliament. Both sides (French and English) have been run through a tokenizer to split off punctuation from words.

You should floor all probabilities to a low number, say 1e-07, to avoid numerical problems as well as dead-ends in the EM training. Similarly, you may need to prune low-valued parameters in order to make memory usage and file sizes manageable.

Please turn in:

This is a big data set, and training is time- and memory-intensive.

gildea @ cs rochester edu
November 10, 2009