Yuncheng Li

Computer Science, University of Rochester

Apr. 30, 2014, BST512 Final Project

- Latent Dirichlet Allocation
- Algorithm
- Results on real data
- Simulations

Example articles from Associated Press

- 2,246 news articles
- 10,473 unique words
- 342,533 tokens

- … went
**committee**leaders … receipts**members**four … sits**senate**appropriations … - …
**house**thursday … damage**bill**tally … democrats**vote**followed … - … state
**senate**seat … january**senate**two … get**bill**passed … vetoed**bill**made … - … public
**members**shared … windfall**house**members … house**members**accepted … - … new
**congress**discuss … rights**bill**push … early**bill**defeated … - … brief
**house**voted … passed**house**earlier … key**vote**senate … - … campaign
**members**family … nine**members**contributed … family**members**made … - …
**house**tuesday … rejected**legislation**protect … men**house**defeated … - … dan
**bill**give … lautenberg**bill**bradley … contend**bill**door …

- … communist
**party**milan … communist**party**premier - … polls
**opposition**alliance … municipal**elections**leading … percent**vote**percent … - … leader
**opposition**labor … tony**party**conference … leftist**party**moderate … - … man
**national**front … front**party**leader … attended**political**meeting … - … association
**national**made … majority**elections**refuses - … second
**election**complete … four**political**parties … political**parties**opposing … - … participate
**election**runoff … alberto**political**favored … months**election**campaigning … - … parliamentary
**elections**held … march**government**stopped … voting**elections**held … - … democratic
**party**labor … labor**minister**december … prime**minister**noboru … - … prime
**minister**noboru … accepted**political**donations … named**political**supreme …

- … acquired
**million**shares … singer**company**officials … acquired**company**million … - … ran
**company**dismantled - … merger
**offer**banca … bank**new**york … outstanding**offer**banca … - … dividend
**share**common … common**stock**santa … pacific**share**shares … - … hostile
**billion**tender … tender**offer**food … nabisco**share**best … - … back
**million**kraft … possible**company**excited … partner**new**kohlberg … - … business
**million**cash … last**company**announced … generated**company**… - … buy
**stock**coastamerica … coastamerica**share**american … tender**offer**five … - … macmillan
**new**publishing … services**company**offered … acquire**billion**week … - … holding
**company**subsidiary … premiums**million**last … revenues**billion**personal …

- … raided
**police**believe … tuesday**found**searched … last**two**weeks … - … fled
**police**speeds … state**police**boy … satisfactory**police**names … - … patrolman
**two**children … away**found**hour … tuesday**two**ages … - … looking
**police**officer … parked**two**weeks … computer**found**notion … - … story
**two**accidentally … playing**police**girl … told**police**two … - … nine
**police**victims … sunday**night**central … taking**police**arrested … - …
**man**stabbed … family**police**michael … slashed**arrested**investigation … - … occurred
**man**service … handgun**shot**nelson … assistance**man**police … - … allegedly
**two**rented … wounded**police**unidentified … unidentified**man**treated … - … gunman
**shot**two … shot**two**people … hostage**police**man … police**man**shot …

- … four
**people**leaving … opened**police**regional … march**killed**three … - … three
**government**southern … back**military**convoy … liberation**army**captured … - …
**people**died … five**police**hospital … clashes**army**demonstrators … - … dubbed
**people**fired … armed**army**capable … funds**government**allegations … - … british
**military**barracks … british**military**officials … republican**army**blamed … - …
**government**supporters … tuesday**people**witnesses … reports**two**big … - … killing
**people**injuring … hundreds**people**died - … british
**army**reinforcements … terrorist**army**army … army**army**headquarters … - …
**police**clashed … theater**security**source … student**killed**four … - … occupied
**army**barracks … border**killed**undisclosed … responsibility**reported**civilians …

From Wikipedia, LDA, …, allows sets of observations to be explained by **unobserved groups** that explain why some **parts** of the data are similar.

- Discover latent topic space (explain the topic space and how it related to semantic affinity between documents)
- Identify keywords

Variables | Meaning | Descriptions |
---|---|---|

\(M \in \mathbb{N}_{+}\) | number of documents | 3,000 |

\(N_m \in \mathbb{N}_{+}\) | number of words in \(m\)-th document | 100 |

\(J\) | number of unique words | 20,000 |

\(K\) | number of topics (predefined) | 50 |

\(y_{m,n} \in 1:J\) | \(n\)-th word of the \(m\)-th document | 300K vector |

\(z_{m,n} \in 1:K\) | topic assigned to \(y_{m,n}\) | 300K vector |

\(\theta_m \in [0,1]^K\) | topic distribution for document \(m\) | 3,000 * 50 matrix, 150K |

\(\phi_k \in [0,1]^J\) | word distribution for topic \(k\) | 20,000 * 50 matrix, 1M |

\(\alpha \in \mathbb{R}_{+}^K\) | Dirichlet prior for \(\theta_m\) | 0.01 |

\(\beta \in \mathbb{R}_{+}^J\) | Dirichlet prior distributions for \(\phi_k\) | 0.05 |

- Topic is assigned to
**individual words**, and document is a mixture of topics. - Two key probabilities,
- \(\theta_{k,m} := p(z_{m,n} = k), \forall n\)
- \(\phi_{k,j} := p(y_{m,n} = j | z_{m,n} = k), \forall (m,n)\)

- Therefore, \[ \begin{array}{rcl} p(y_{m,n} = j) &=& \sum_k p(y_{m,n} = j, z_{m,n} = k) \\ &=& \sum_k p(y_{m,n} = j | z_{m,n} = k) p(z_{m,n} = k) \\ &=& \sum_k \phi_{k,j} \theta_{k,m} \end{array} \]
- For a specific word in a document, the topic assigned to it depends on
- the word itself, i.e., \(\phi_k\)
- the overall topic distribution for the document, i.e., \(\theta_m\)

- Draw word distribution \(\phi_k \sim \text{Dir}(\beta)\) for each topic \(k\)
- Draw topic distribution \(\theta_m \sim \text{Dir}(\alpha)\) for each document \(m\)
- For each
**SLOT**\(n\) in document \(m\), draw topic \(z_{m,n} \sim \text{Multi}(\theta_m)\). - For each
**SLOT**\(n\) in document \(m\), draw word \(y_{m,n} \sim \text{Multi}(\phi_{z_{m,n}})\)

Inference goal: given \(y_{m,n}\), fit \(\phi_k\), \(\theta_m\) and \(z_{m,n}\)

Some important counts,

- \(c_{k,j}\) denotes, in any documents, how many times, word \(j\) assigned to topic \(k\)
- \(c_{k,m}\) denotes, in document \(m\), how many words assigned to topic \(k\)
- \(c_k\) denotes, throughout entire corpus, how many words assigned to topic \(k\). \[ c_k = \sum_j c_{k,j} = \sum_m c_{k,m} \]

Once we have a draw of the vector \(z_{m,n}\), we can compute these counts efficiently.

- Joint posterior distribution,

\[ p(\phi, \theta, z | y, \alpha, \beta) \propto p(\phi|\beta) p(\theta|\alpha) p(z|\theta) p(y|\phi, z) \]

Conditional posteriors, (conjugacy, independency, and assume uniform Dirichlet prior \(\alpha\) and \(\beta\)) \[ p(\theta_m \in [0,1]^K| z, \alpha) = Dir(c_{k,m} + \alpha) \] \[ p(\phi_k \in [0,1]^J| z, \beta) = Dir(c_{k,j} + \beta) \] \[ \begin{array}{rcl} p(z_{m,n} = \mathbb{k} | \phi_k, \theta_m, y_{m,n}) & \propto & p(\mathbb{k} | \theta_m) p(y_{m,n} | \mathbb{k}, \phi_k) \\ &=& \phi_{\mathbb{k},y_{m,n}} \times \theta_{m, \mathbb{k}} \\ \end{array} \]

A gibbs sampler can built based on these equations.

For many reasons, people usually infer such kind of posteriors using

*collapsed gibbs sampling*.Note that the vector \(z_{m,n}\) is a sufficient statistics for \(\phi_k\) and \(\theta_m\).

- Sample each dimension of the big vector \(z_{m,n}\) one by one.
- \(z_{-(m,n)}\) denotes topic assignment for all words except \(z_{m,n}\). Note \(z = z_{m,n} \cup z_{-(m,n)}\) \[ \begin{array}{rcl} z_{m,n} & \sim & p(z_{m,n} | z_{-(m,n)}, y, \alpha, \beta) \\ & \propto & p(z_{m,n}, z_{-(m,n)}, y|\alpha, \beta) \\ & = & p(z,y| \alpha, \beta) \end{array} \]
- Note that \(\phi_k\) and \(\theta_m\) is missing in the above posterior, because they are integrated out from the joint posterior as following, (
**collapsed**) \[ p(z,y|\alpha, \beta) = \int \int p(y, z, \theta, \phi | \alpha, \beta) d\theta d\phi \] - By sophisticated derivations and keep only the terms relevant to \(z_{m,n}\), we can simplify the integration as follows. The counts superscripted with \(-(m,n)\) means the counts excluding \(z_{m,n}\),

\[ p(z_{m,n} = \mathbb{k}| z_{-(m,n)}, y, \alpha, \beta) \propto \frac{(c_{\mathbb{k}, m}^{-(m,n)} + \alpha) \times (c_{\mathbb{k}, y_{m,n}}^{-(m,n)} + \beta)}{c_{\mathbb{k}}^{-(m,n)} + J * \beta} \]

- Intuitively, estimate \(\phi_k\) and \(\theta_m\) using \(z_{-(m,n)}\), then draw \(z_{m,n}\) from \(\theta_{m, k} \times \phi_{k,y_{m,n}}\)

\[ p(z_{m,n} = \mathbb{k}| z_{-(m,n)}, y, \alpha, \beta) \propto \frac{(c_{\mathbb{k}, m}^{-(m,n)} + \alpha) \times (c_{\mathbb{k}, y_{m,n}}^{-(m,n)} + \beta)}{c_{\mathbb{k}}^{-(m,n)} + J * \beta} \]

- Input: \(y_{m,n}\)
- Random init \(z_{m,n}\)
- Init the counts \(c_{k,m}, c_{k,j}, c_{k}\) from \(z_{m,n}\)
- for i = 1:niter
- for each (m, n) do,

- \(j \leftarrow y_{m,n}\)
- \(\mathbb{k} \leftarrow z_{m,n}\)
- decrement counts \(c_{\mathbb{k},m}\) -= 1, \(c_{\mathbb{k},j}\) -= 1, \(c_{\mathbb{k}}\) -= 1, (which are just \(c_{k,m}^{-(m,n)}, c_{k,j}^{-(m,n)}, c_k^{-(m,n)}\))
- compute \(p(z_{m,n}|...)\) using the above equation
- draw \(z_{m,n} \rightarrow \mathbb{k}\)
- increment counts \(c_{\mathbb{k},m}\) += 1, \(c_{\mathbb{k},j}\) += 1, \(c_{\mathbb{k}}\) += 1.

- Output: \(z_{m,n}, c_{k,m}, c_{k,j}\) and \(c_{k}\)

- Computation complexity is \(O(\text{niter}*MN*K)\).
- In one typical experiment,
*niter*= 20000,*MN*= 342,000 and*K*= 50. Roughly 34 Billion floating point operations. - Core sampling implemented with C++ (20 times faster than pure R implementation)
- 2 hours on average for single run.
- Simulations using clusters (72 nodes) in 24 hours.

News articles from Associated Press, used by the original LDA paper.

- 2,246 news articles \(M\)
- 10,473 unique words \(J\)
- 342,533 tokens \(MN\)

```
source("load.data.R")
source("./gibbs.vis.stat.R")
```

```
## Loading required package: foreach
## Loading required package: iterators
## Loading required package: snow
## Loading required package: methods
```

```
dataset.ap <- load.ap.data()
A <- gibbs.vis.stat(dataset.ap, do.save = F, param.id = 100, do.example = T)
```

`## data/perp.res-38c9d69773382496b55398092ba43b28-90-1000-4-0.9.rdb`

```
## Loading required package: Rcpp
## Loading required package: RcppArmadillo
## Loading required package: lattice
```

```
##
##
## Top words for top topics
## [,1] [,2] [,3] [,4] [,5] [,6]
## [1,] "government" "new" "i" "police" "i" "court"
## [2,] "officials" "years" "years" "two" "think" "trial"
## [3,] "president" "year" "people" "people" "dont" "charges"
## [4,] "two" "million" "two" "killed" "people" "judge"
## [5,] "official" "last" "like" "city" "going" "case"
## [6,] "meeting" "people" "day" "man" "just" "attorney"
## [7,] "last" "two" "just" "three" "know" "prison"
## [8,] "told" "program" "first" "shot" "get" "years"
## [9,] "states" "officials" "time" "officials" "say" "convicted"
## [10,] "foreign" "national" "home" "injured" "time" "guilty"
## [,7] [,8] [,9] [,10]
## [1,] "percent" "party" "dukakis" "company"
## [2,] "year" "government" "bush" "million"
## [3,] "rate" "communist" "campaign" "stock"
## [4,] "rates" "political" "jackson" "billion"
## [5,] "economy" "opposition" "democratic" "offer"
## [6,] "prices" "minister" "convention" "inc"
## [7,] "report" "new" "presidential" "corp"
## [8,] "economic" "leader" "vice" "share"
## [9,] "increase" "republics" "republican" "new"
## [10,] "price" "democratic" "president" "agreement"
##
##
## Top documents for topics
## [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
## [1,] 38 1666 2026 573 1586 428 821 559 316 1477
## [2,] 1653 2171 914 1434 1748 1557 370 2098 2029 1091
## [3,] 675 949 1505 1878 227 985 1550 2160 2066 561
## [4,] 1127 66 5 973 333 82 944 132 919 47
## [5,] 1281 266 1903 138 2094 1414 219 103 866 2183
## [6,] 1809 1858 828 1508 1290 713 2196 929 2065 537
## [7,] 611 2058 54 1629 1301 644 1322 832 421 915
## [8,] 1259 592 480 152 1959 1677 15 2143 517 165
## [9,] 1741 218 1320 1265 1663 213 1452 753 8 1854
## [10,] 2049 1968 1217 209 2195 665 1621 595 1960 1379
```

Following observations,

- Topics are ordered by ‘popularity’, i.e., \(c_k\)
- Trivial topics may be ordered very high
- Mostly politics or business for top 10 topics
- Some words are selected as keywords for many topics, such as ‘i’, ‘government’ and ‘people’ …

Extensive list of example documents for discovered topics with keywords highlighted, (Note that these topics can **not be automatically named** by this method.)

- Topic # 0
- … united
**states**state … quoted**foreign**ministry … united**states**severed … - …
**president**mikhail … reported**official**soviet - … week
**president**soviet … questions**foreign**ministry … interviews**foreign**reporters … - … evidence
**government**sudan - …
**foreign**minister … trip**foreign**minister … comes**two**visits … - … soviet
**president**mikhail … britain**president**bush … difficult**president**gorbachev … - …
**government**thursday … soviet**president**mikhail … prevented**two**countries … - … peace
**official**spoke … summit**meeting**june … attend**meeting**north … - … leftist
**government**contra … sandinistas**government**appeared … principle**last**week … - …
**foreign**minister … iran**foreign**minister … geneva**meeting**mediator …

- … united
- Topic # 1
- … concerns
**new**congress - … coming
**officials**see … see**people**discharged … medicaid**program**million … - … important
**new**panel … watkins**officials**collider … congress**last**year … - … proposed
**national**constitution … existed**two**years … two**years**china … - … epa
**new**regulations … done**program**value - … installing
**new**weather … obtaining**new**acknowledges … acknowledges**program**fallen … - … rules
**officials**say … restrictions**new**dump … emergency**people**going … - … service
**people**questions … particular**officials**increased … obviously**people**get … - …
**two**american … elected**officials**urged … lines**people**examine …

- … concerns
- Topic # 2
- … downtown
**people**think … doing**i**think … worth**i**pay … eat**like**admits … - … car
**two**wonderful … spent**time**concerned … scared**i**am … - … message
**day**volunteers … word**people**world … estimated**people**get … - …
**years**john … asked**i**says … issue**people**mom - … elliott
**i**busfield … tv**first**i … first**i**know … know**i**make … - … jump
**i**thought … jump**first**river … thinks**i**guys … age**just**sitting … - … says
**like**time … like**time**like … time**like**actually … skills**i**practiced … - … leaves
**home**bunch … five**years**reach … figure**i**percent … - … month
**years**north … wanted**like**cooking … cooking**i**like …

- … downtown
- Topic # 3
- … anger
**police**responding … twelve**people**arrested … crowd**people**gathered … - … gunman
**shot**two … shot**two**people … two**people**morning … - … guards
**shot**trying … yugoslav**police**saying - … protest
**police**handling … reported**city**people … city**people**lake … - … occurred
**man**service … handgun**shot**nelson … assistance**man**police … - … four
**people**fatally … fatally**shot**eight … day**shot**head … - … six
**two**authorities … left**three**woman … woman**two**young … - … story
**two**accidentally … accidentally**killed**sister … playing**police**girl … - …
**people**attended … department**city**county - … allegedly
**two**rented … wounded**police**unidentified … unidentified**man**treated …

- … anger
- Topic # 4
- …
**think**americans … middle**i**think … i**think**advised … prepared**say**travel … - … got
**get**users … change**i**saw … saw**i**saw … entertainment**people**involved … - … release
**i**definition - … economic
**going**calm … thing**going**comment … levels**say**substantial … - … happens
**i**think … i**think**really - … anybody
**i**think … i**think**thought … story**people**magazine … - … rude
**i**mayor … joseph**just**kind … attempt**get**question … - … fitzwater
**just**talked … sure**know**exactly - … really
**say**accepted … monday**i**little … mother**think**means … - … second
**time**seven … long**i**get … i**get**home … main**just**hope …

- …
- Topic # 5
- … pleaded
**guilty**stealing … federal**judge**district … district**judge**thomas … - … second
**trial**child … acquitted**charges**peggy … total**charges**prosecution … - …
**judge**sentenced … patrolman**years**life … life**prison**throwing … - … racketeering
**trial**today … defense**case**corruption … district**case**public … - … pleaded
**guilty**connection … six**years**federal … pleaded**guilty**conspiracy … - … pleaded
**guilty**manslaughter … months**prison**placed … circuit**judge**daniel … - … woman
**convicted**tuesday … fraud**charges**five … stand**trial**convicted … - … racketeering
**trial**two … district**court**judge … court**judge**george … - … monday
**charges**killing … last**prison**officials … supreme**court**upheld … - … proof
**case**new … district**attorney**general … went**trial**last …

- … pleaded
- Topic # 6
- … interest
**rates**treasury … discount**rate**percent … rate**percent**percent … - … retail
**prices**britain … britain**percent**august … levels**year**highest … - … unemployment
**rate**held … steady**percent**last - … shot
**percent**biggest … increases**percent**january … january**percent**march … - … annual
**rate**just … clothing**prices**fell … food**prices**first … - … climbed
**percent**healthy … healthy**increase**economists … edged**percent**november … - … plunged
**percent**nine … followed**percent**increase … percent**increase**revised … - … operating
**rate**mines … today**report**likely … points**percent**last … - … main
**economic**forecasting … plunged**percent**drop … leading**economic**indicators … - … moderate
**percent**government … mortgage**rates**make … annual**rate**million …

- … interest
- Topic # 7
- … session
**communist**party … communist**party** - … prominent
**communist**secretary … yugoslav**communist**communist … - … excerpts
**new**regulations … banning**political**activity … major**opposition**… - … succeeded
**communist**party … communist**party**centrist … radicals**party**organize … - … session
**communist**party … communist**party**third - … won
**political**gaining … gaining**communist**party … communist**party**approval … - … give
**opposition**parties … spokesman**opposition**decision … negotiations**communist**party … - … prime
**minister**head … lead**new**government … new**government**replace … - … interior
**minister**job … interior**minister**connection … stemming**political**gorbachev … - … friday
**new**city … parliamentary**opposition**progressive … member**communist**supporter …

- … session
- Topic # 8
- …
**president**carter … described**democratic**michael … michael**dukakis**tuesday … - … shows
**jackson**percent … michael**dukakis**third … found**vice**george … - … marked
**campaign**commercials … unprecedented**presidential**dukakis … - … clear
**democratic**beating … jesse**jackson**albert … necessarily**vice**presidential … - … illinois
**republican**james … came**convention**city … michael**dukakis**lloyd … - … michael
**dukakis**wednesday … republicans**convention**trip … dan**republican**vice … - … michael
**dukakis**back … join**democratic**jesse … jesse**jackson**says … - … candidate
**democratic**caucuses … michael**dukakis**eventual … union**democratic**candidates … - …
**democratic**presidential … democratic**presidential**contender …

- Topic # 9
- … merger
**offer**banca … bank**new**york … outstanding**offer**banca … - … dividend
**share**common … common**stock**santa … pacific**share**shares … - … percent
**billion**net … entire**company**just … record**billion**tender … - … macmillan
**new**publishing … services**company**offered … acquire**billion**week … - … launch
**billion**cash … tender**offer**macmillan … subject**billion**bid … - … say
**company**lost … offered**share**take … take**company**offer … - … texas
**billion**hostile … tender**offer**million … offer**million**shares … - …
**billion**buyout … texas**new** - …
**company**seeking … cents**share**close … consolidated**new**york … - … pay
**billion**cash … assume**million**cain … gain**million**chase …

- … merger

We can make following observations,

- Random select which documents \(m\) (words \(j\)) to trace \(\theta_{m,k}\) (\(\phi_{k,j}\))
- Only top values are displayed.
- Note the \(\phi_{k,j}\) at the bottom. The top values for this word are still near zero, which indicated this word rarely contributed to any topic.
- Out of the 5 randomly selected words, none of them are strong keywords.
- The convergence is not very well, but acceptable.

One strange trace, whose parameters are larger than 1.

`A <- gibbs.vis.stat(dataset.ap, do.save = F, param.id = 100, do.trace = T)`

`## data/perp.res-38c9d69773382496b55398092ba43b28-90-1000-4-0.9.rdb`

Another name for likelihood, but many different definitions for LDA model. I computed perplexity as follows, \[
\text{Perplexity}(y^{un.obs}_{m,n} | \phi, \theta) = exp\left( -\frac{\sum_{m, n} \log p(y_{m,n}^{un.obs} | \phi, \theta)}{MN}\right)
\] \[
p(y_{m,n} | \phi, \theta) = \sum_{k=1}^K \phi_{k, y_{m,n}} \theta_{m,k}
\] For each document, keep a number of words as unobserved and the rest as observed. The observed words are used to infer \(\phi_k\) and \(\theta_m\). Using some point estimates of \(\theta_m\) and \(\phi_k\) from the gibbs draws, we can compute the *perplexity* for unobserved words.

Note that we can control the *ratio* of observed to unobserved words to see what the percentage of words we need to make good fit, which are evaluated in next page.

As the definition indicates, smaller perplexity means the fitted model can better explain unobserved data. Following is a plot of perplexity over iterations.

`A <- gibbs.vis.stat(dataset.ap, do.save = F, param.id = 100, do.plot = T)`

`## data/perp.res-38c9d69773382496b55398092ba43b28-90-1000-4-0.9.rdb`

As we can see, perplexity decreases very fast to the converging point. 1000 iterations is enough

Following is of our interests,

- How is the performance over different number of topics \(K\).
- How about different
*ratio*of observed words.

`A <- gibbs.vis.stat(dataset.ap, do.save = T)`

`## loading data/multi.run-38c9d69773382496b55398092ba43b28.rdb`

Here are our observations,

- Larger number of topics and more observed data can better explain the data.
- As \(K\) and
*ratio*increase, the effects decrease.

- Draw true \(\theta\) and \(\phi\) from the Dirichlet prior, according to predefined \(K\), \(M\) and \(N\).
- Generate topics and words according to \(\theta\) and \(\phi\)
- Infer \(\hat{\theta}\) (the last draw)
- Compare \(\theta\) and \(\hat{\theta}\), and report the bias.

A problem with comparing \(\theta\) and \(\hat{\theta}\) directly: **non-identifiablity**. \(\hat{\theta}\) can be arbitrary permutation of \(\theta\).

In mixture model, one common practice is to find a optimal relabeling. Another approach is to measure **affinity correlations**.

(Not a formal term, idea comes from a mixture model homework)

Given \(\theta\), we can compute distance between each pair of documents \(d(m_1, m_2)\) by \[ d(m_1, m_2)_\theta = \sqrt{\sum_{k=1}^{K} \|\theta_{m_1, k} - \theta_{m_2, k}\|^2}, \] which can be seen as a distance between \(m_1\) and \(m_2\) in the topic space.

*Affinity correlations*: \(\text{cor} (d_\theta, d_\hat{\theta})\)

Basic idea is if two documents is close in the group truth topic space \(\theta\), they should be close in the estimated topic space \(\hat{\theta}\).

Use a GMM to explain the idea.

It is treated as *pesudo bias* to evaluate simulations.

The number of topics is 50, which is relatively large.

```
numTopics <- 50
sizeVocab <- 10000
numDocs <- 5000
averageDocumentLength <- 100
averageWordsPerTopic <- 10
n.rep <- 5
for (rep.i in 1:n.rep) {
dataset.sim.50 <- load.sim.data(numTopics, sizeVocab, numDocs, averageDocumentLength,
averageWordsPerTopic, rep.i)
A <- gibbs.vis.stat(dataset.sim.50, n.save = 100, do.save = T)
}
```

`## loading data/multi.run-18dfdad9950cad221634b5ff19d37787.rdb`

`## loading data/multi.run-bc560150094a12f4ecdf523deb9d832a.rdb`

`## loading data/multi.run-fb9310067831d871a0959f68b6335e60.rdb`

`## loading data/multi.run-803b43a568a33810d0369d1036260aca.rdb`

`## loading data/multi.run-c316c230871bf7f26b5dc79ce6b1ade8.rdb`

A smaller number of topics and model size is chosen.

```
numTopics <- 5
sizeVocab <- 1000
numDocs <- 500
averageDocumentLength <- 100
averageWordsPerTopic <- 10
n.rep <- 5
for (rep.i in 1:n.rep) {
dataset.sim.5 <- load.sim.data(numTopics, sizeVocab, numDocs, averageDocumentLength,
averageWordsPerTopic, rep.i)
A <- gibbs.vis.stat(dataset.sim.5, n.save = 100, do.save = T)
}
```

`## loading data/multi.run-34f3ffc230fdf34401f6c4e17cf29ce8.rdb`

`## loading data/multi.run-6acf960f82d2d04fe59624a5ef8bb09f.rdb`

`## loading data/multi.run-bd977582a30dacde9f5cce173bb8892a.rdb`

`## loading data/multi.run-9a52e401ff5844a78f034fcab80be4d3.rdb`

`## loading data/multi.run-2f923423610e24933d0e1519d83d6f08.rdb`

- Sensitive to the choice of \(K\). However, problems with \(\hat{K} < K\) is more severe than the cases with \(\hat{K} > K\).
- Perplexity, as a evaluation measurement, may or may not be related to real performance.

- Apply Metropolis Hasting with good proposal variance to make larger jumps
- Diagnostics for this kind of
*Big Model* - Richer models

- Rcpp for C++ integration
- snow, foreach for parallezation
- knitr and pandoc for documentation
- Stackoverflow
- Wikipedia
- Original LDA-C code from the author,

All source code for this project are posted online. https://github.com/raingo/topicmodel

- Sally for the wonderful classes and feedbacks.