Unsupervised and Weakly-Supervised Probabilistic Modeling of Text
Instructor: Ivan Titov
Time: Friday, 2.15 - 3.45 pm
Location: Building C 7.2, room 2.11
Office hours: Friday, 4 pm - 5 pm in Ivan's office (C7.4, 3.22), or send me a message by e-mail
In this seminar, we will focus on generative (mostly Bayesian) models of texts.
We will start with the most basic topic models (Latent Dirichlet Allocation)
but then we will proceed to considering more recent and advanced generative
models which induce topic segmentation, topic hierarchies, shallow semantic
and syntactic representations and use some form of supervision.
We are planning both consider inference techniques for these models
(Expectation Maximization, more general variational methods, Markov chain Monte Carlo methods,
belief propagation) and their application to various natural language problems
(e.g., segmentation, summarization, sentiment analysis, grounded language acquisition).
The goals of this seminar is to both to understand the methodology (classes of models considered in NLP,
approximation techniques for learning and inference) and to learn about interesting
applications of generative methods which use little or know supervision.
The term paper is due on September 26 (see requirements below).
- Present a paper to the class (35-40 minutes long presentation)
- Write 3 critical reviews (surveys) of 3 selected papers (1.5 - 2 pages each)
- Write a term paper (12 - 15 pages) (If you are registered for 4 points you do not need to write the term paper)
- Read papers before the talks and participate in discussion
- Class participation grade: 60 %
- Your talk and discussion after the talk
- Participation in discussion of other papers
- 3 reviews (5 % each )
- Term paper grade: 40 %
- Only if you get 7 points, otherwise class participation constitutes 100 %
You can skip ONE class without giving any explanation to me (if it is not the class on which you are presenting).
If you need to skip more, you will need to write an additional critical review for every paper presented while you were absent.
- Present the chosen paper in an accessible way
- Present sufficient background, do not expect the audience to know much about Machine Learning or Natural Language Processing, except for the material already covered in the class (according to surveys
there is a good number of people who have no ML background)
- Have a critical view on the paper: discuss shortcomings, possible future work, etc
- To give a good presentation in most of the cases you will need to read one or two additional papers (e.g., those referenced in the paper)
- You should have a look into material on how to give a good presentation compiled by Alexander Koller
- The language for talks and discussions will be English
- Given the number of students now, we are planning to have 35 minutes long presentations, on some days we may decide to have 2 presentations
- Send me your slides (preferably in PDF) 4 days before the talk by 6 pm (the first 2 presenters can send me slides 2 days before the talk)
- If we keep the class on Friday, it means that the deadline is Mon, 6 pm
- I will give my feedback 2 days before the seminar (on Wed)
- A short critical (!) essay reviewing one of the papers in the list
- One or two paragraphs presenting the essence of the paper
- Other parts underlying both positive sides (what you like) of the paper and shortcomings
- You need to submit 3 reviews. There will be 3 reviewers for each presentation.
- The review should be submitted (by email in pdf) before the presentation of the paper in class (Exception is the additional reviews submitted for the classes you missed: you should submit such an additional review within 2 weeks of the corresponding class and before the end of the term)
- No copy-paste from the paper. It should be all your words.
- Length: 1.5 - 2 pages each
- Describe the paper you presented in class.
- It should be written in a style of a research paper, the only difference is that in this paper most of the work you present here is not your own
- Your ideas, analysis, comparison
- It should be written in English
- Comparison of the methods used in the paper with other material presented in the class or any other related work
- Any ideas on improvement of the approach
- Any alternative interpretation or analysis
- Paper organization
- Technical correctness
- Style (written in research style without inappropriate speculations, correct citations, etc)
- Your ideas are meaningful and interesting
Length: 12 - 15 pages
Deadline: September 26. I would recommend to submit it soon after your presentation, as it would probably be easy.
Submitted in PDF to my email
- Basic topic models (LDA / PLSA) and inference/learning techniques (EM / MCMC)
- By April 23 please choose a paper to present AND 3 other papers you will review. Use the link to Google Docs I sent to you (Do not send your choice to me by email)
- Vote in Doodle poll for the date of future seminars (sent on April 20)
- If did not receive the links from me on Tue April 20, send me a message urgently. It probably means that (1) you haven't sent email to me and (2) you did not fill email field in the survey
- If you have not filled the survey please do it. Also, when selecting papers to present and review, please read the comments column in the Google Document (a couple of the papers require some ML background to read them)
[Slides (with added references)]
- No class - AISTATS conference
- No class - ACL / CoNLL conference
Full List of Papers
The papers are approximately in the order they are going to be presented. There may be some changes though, watch the schedule above for more details. The list in Google Docs (you should have a link)
is supposed to be more up to date.
- Bayesian Unsupervised Topic Segmentation, Eisenstein and Barzilay, EMNLP 08
- Topic Modeling: Beyond Bag of Words, Wallach, ICML 06
- Global Models of Document Structure Using Latent Permutations, Chen et al., NAACL 09
- Supervised Topic Models, Blei and McAuliffe, NIPS 07
- Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression David Mimno and Andrew McCallum. UAI, 2008
- Mixtures of Hierarchical Topics with Pachinko Allocation, Mimno et al., ICML 07
- Hierarchical topic models and the nested Chinese restaurant process, Blei et al., NIPS 03
- Integrating Topics and Syntax, Griffiths et al., NIPS 04
- A fully Bayesian approach to unsupervised part-of-speech tagging. Goldwater and Griffiths, ACL 07
- Adaptor Grammars: a framework for specifying compositional nonparametric Bayesian models. Johnson et al, NIPS 07
- Unsupervised Multilingual Learning for Part-of-Speech Tagging", Naseem et al, JAIR 09
- Gestural Cohesion for Topic Segmentation, Eisenstein et al, ACL 08
- Joint Models of Text and Aspect Ratings for Sentiment Summarization, Titov and McDonald, ACL 08
- Learning semantic correspondences with less supervision, Liang et al., ACL 09
- Topic modeling with network regularization, Mei et al., WWW 08
- Reading to Learn: Constructing Features from Semantic Abstracts. Eisenstein et al., EMNLP 09