10708 (CMU) Probabilistic Graphical Models (2018)
Overview
Many of the problems in artificial intelligence, statistics, computer systems, computer vision, natural language processing, and computational biology, among many other fields, can be viewed as the search for a coherent global conclusion from local information. The probabilistic graphical model’s framework provides a unified view of this wide range of problems, enabling efficient inference, decisionmaking and learning in problems with a very large number of attributes and huge datasets. This graduatelevel course will provide you with a strong foundation for both applying graphical models to complex problems and for addressing core research topics in graphical models. The class will cover three aspects: The core representation, including Bayesian and Markov networks, and dynamic Bayesian networks; probabilistic inference algorithms, both exact and approximate; and, learning methods for both the parameters and the structure of graphical models. Students entering the class should have a preexisting working knowledge of probability, statistics, and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate. It is expected that after taking this class, the students should have obtained sufficient working knowledge of multivariate probabilistic modeling and inference for practical applications, should be able to formulate and solve a wide range of problems in their own domain using GM and can advance into more specialized technical literature by themselves. Students are required to have successfully completed 10701 or 10715, or an equivalent class.
Where and When
 Time: Tuesday, Thursday 12:00 – 1:20 pm
 Location: GatesHillman Center 4307
 Piazza Link: https://piazza.com/cmu/spring2018/10708/home
 GradeScope Link: https://gradescope.com/courses/14138
People
Instructor:
 Kayhan Batmanghelich
Office Hour: GatesHillman Center 8228 on Tuesdays 1:30 – 2:30 pm
Administrative Assistant:
 Noreen Doloughty
Teaching Assistants:
 Yifeng Tao (yifengt@andrew.cmu.edu)
 Xiongtao Ruan (xruan@andrew.cmu.edu)
 Yuanning Li (yuanninl@andrew.cmu.edu)
If you wish to email only the instructors, the email is 10708Spring18@gmail.com.
Announcements
 If you have any questions about class policies or course material, please use piazza or you can email all of the instructors at 10708Spring18@gmail.com. Please use this list instead of individual email addresses to ensure a prompt response.
 The Homework 0 is out. The deadline is 7:00 pm Jan 23. No late HW will be accepted.
 Please find the scribe template here.
 The project proposal will be due by 7 PM on Friday, February 16th, and should be submitted via Gradescope.
 Homework 1 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is Feb 18 (11:59 pm).  Homework 2 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is Mar 21 (11:59 pm).  The project midway report will be due 11:59 pm, April 13th. See Piazza for more detail.
 Homework 3 is released. You can view it on the course page: https://piazza.com/cmu/
spring2018/10708/resources . The due date is April 20 (11:59 pm).
Lectures
Date  Lecture  Scribe  Readings  Announcements  

Jan 16, 2018  Lecture 1 (Kayhan) – Slide, Video
Probabilistic Graphical Model: 
None  None  HW0 is out.  
Module 1: Representation  
Jan 18, 2018  Lecture 2 (Kayhan) – Slide, Annotated, Video
Directed GMs: Bayesian Networks 
Notes (by Sumedha Singla)  1. Koller and Friedman Textbook, Ch. 3 2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 3 

Jan 23, 2018  Lecture 3 (Kayhan) – Slide, Annotated, Video
Undirected Graphic Model 
Notes (by Arjun Sharma)  1. Koller and Friedman Textbook, Ch. 4 2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 4 
HW 0 is due today.  
Jan 25, 2018  Lecture 4 (Kun Zhang) – Slide, Annotated, Video
Causal Graphic Model 
Notes (by M. Malik and N. Shajarisales)  None  The reading summary is due in a week.  
Module 2: Classical Methods of Inference & Learning  
Jan 30, 2018  Lecture 5 (Kayhan) – Slide, Annotated, Video
Algorithms for Exact Inference 
Notes (by MF. Chang and D. Rajagopal)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6
Other resources: (1) 

Feb 1, 2018  Lecture 6 (Kayhan) – Slide, Annotated, Video
Factor graph, message passing, and Junction Tree 
Notes (S. Greg)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6
Other resources: (1) 

Feb 6, 2018  Lecture 7 (Kayhan) – Slide, Annotated, My Notes, Class Notes, Video
Exponential families and friends: Learning the parameters of a fully observed BN 
Notes (A. Kamath)  Kevin Murphy, Machine Learning A Probabilistic Perspective, Ch. 9
or Jordan Textbook, Ch. 8, Ch. 9 

Feb 8, 2018  Lecture 8 (Kayhan) – Slide, Annotated, My Notes, Class Notes, Video
Learning the parameters of UGM 
Notes (C. Zhou and C. Zhang)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9, Section 6  
Feb 13, 2018  Lecture 9 (Kayhan) – Slide, Video
EM and partially observed GM 
None  Jordan Textbook, Ch. 10
or David Barber, Bayesian Reasoning, and Machine Learning, Ch. 11 
The reading summary is due in a week.  
Module 3: Graphical Model in Application & Learning  
Feb 15, 2018  Lecture 10 (Kayhan) – Slide, Annotated, Video
HHM and CRF 
Notes (B. Lengerich and M. Kleyman)  H. Wallach, Conditional Random Fields: An Introduction
J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data 

Feb 20, 2018  Lecture 11 (Kayhan) – Slides, Annotated, Video
CRF (Cont’d) + Intro to Topic Models 
Notes (A. Yang and J. He)  D. Blei et al., Latent Dirichlet Allocation (Sections 14)  
Feb 22, 2018  Lecture 12 (Kayhan) – Slides, Annotated, Class Notes, Video
Intro to Topic Models (Cont’d), Factor Analysis and (maybe State Space) 
Notes (G. Plumb and A. Rumack)  Jordan Textbook, Ch. 14
Jordan Textbook, Ch. 15 (if we finished State Space) 

Feb 27, 2018  No Class  HW 2 is out.  
Mar 1, 2018  Lecture 13 (Kun Zhang) – Slides, Video
Learning Structure of a Graphical Model 
None  This short overview paper
and David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9.5 
The reading summary is due in a week.  
Module 4: Approximate Inference & Learning  
Mar 6, 2018  Lecture 14 (Kayhan) – Slides, Annotated, Video
Loopy Belief Propagation 
Notes (K. Xiong, C. Malaviya)  M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1  
Mar 8, 2018  Lecture 15 (Kayhan) – Slides, Annotated, Video
Mean field Approximation 
Notes (Y. Hung, H. Tsai)  D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 126  
Mar 20, 2018  Lecture 16 (Kayhan) – Slides, Annotated, Video
Stochastic Gradient Descent, SVI, and scalability 
Notes (Y. Feng, J. Wu)  D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 126  
Mar 22, 2018  Lecture 17 (Kayhan) – Slides, Jupyternotebook, Video
Approximate Inference Monte Carlo Methods 
Notes (B. Paria, P. Chikersal)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27  
Mar 27, 2018  Lecture 18 (Kayhan) – Slides, Annotated, My Notes, Video
MCMC and Gibbs sampling 
None  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27  
Mar 29, 2018  Lecture 19 (Kayhan) – Slides, Annotated, Video
Hamiltonian Monte Carlo 
Notes (B. Lyu)  Slice Sampling: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Section 29.7  The reading summary is due in a week.  
Module 5: Deep Learning and Graphical Models & Learning  
Apr 3, 2018  Lecture 20 (Kayhan) – Slides, Annotated, Video
Introduction to Deep Learning 
Notes(A. Alavi, Yu Chen)  Deep Learning Book, Ch. 6.25, 20.34
Optional:


Apr 5, 2018  No Class  
Apr 10, 2018  Lecture 21 (Kayhan) – Slides, Annotated, Video
A Hybrid: Deep Learning and Graphical Models 
Notes (P. Liang, A. Rayasam)  Diederik P. Kingma, Variational Inference & Deep Learning: A New Synthesis, Ch. 2
Generative Adversarial Nets: Goodfellow et al., 2014. 

Apr 12, 2018  Lecture 22 (Kayhan, Mingming Gong) – Slides, Annotated, Video
A Hybrid DL and GM (cont’d) + 
Notes (S. Bai, C.K. Yeh)  None  
Apr 17, 2018  Lecture 23 (Mingming Gong, Kayhan) – Slides, Video
Applications in Computer Vision (cont’d) + Gaussian Process 
None  None  The reading summary is due in a week.  
Module 6: Spectral and nonparametric view & Learning  
Apr 24, 2018  Lecture 24 (Kayhan) – Slides, JupyterNotebook, Annotated, Video
Gaussian Process 
Notes (A. Siddhant, S. Ghosh, C. Nagpal)  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 19
Optional: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2 

Apr 26, 2018  Lecture 25 (Forough Arabshahi) – Slides, Annotated, Video
Spectral Methods 
None  None 