Teaching

10-708 (CMU) Probabilistic Graphical Models (2018)

Probabilistic Graphical Models

Overview

Many of the problems in artificial intelligence, statistics, computer systems, computer vision, natural language processing, and computational biology, among many other fields, can be viewed as the search for a coherent global conclusion from local information. The probabilistic graphical model’s framework provides a unified view of this wide range of problems, enabling efficient inference, decision-making and learning in problems with a very large number of attributes and huge datasets. This graduate-level course will provide you with a strong foundation for both applying graphical models to complex problems and for addressing core research topics in graphical models.  The class will cover three aspects: The core representation, including Bayesian and Markov networks, and dynamic Bayesian networks; probabilistic inference algorithms, both exact and approximate; and, learning methods for both the parameters and the structure of graphical models. Students entering the class should have a pre-existing working knowledge of probability, statistics, and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate.  It is expected that after taking this class, the students should have obtained sufficient working knowledge of multivariate probabilistic modeling and inference for practical applications, should be able to formulate and solve a wide range of problems in their own domain using GM and can advance into more specialized technical literature by themselves.  Students are required to have successfully completed 10701 or 10715, or an equivalent class.

Where and When

People

Instructor:

  • Kayhan Batmanghelich
    Office Hour: Gates-Hillman Center 8228 on Tuesdays  1:30 – 2:30 pm

Administrative Assistant:

  • Noreen Doloughty

Teaching Assistants:

If you wish to email only the instructors, the email is 10708Spring18@gmail.com.

Announcements

  • If you have any questions about class policies or course material, please use piazza or you can email all of the instructors at 10708Spring18@gmail.com. Please use this list instead of individual email addresses to ensure a prompt response.
  • The Homework 0 is out. The deadline is 7:00 pm Jan 23. No late HW will be accepted.
  • Please find the scribe template here.
  • The project proposal will be due by 7 PM on Friday, February 16th, and should be submitted via Gradescope.
  • Homework 1 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Feb 18 (11:59 pm).
  • Homework 2 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Mar 21 (11:59 pm).
  • The project midway report will be due 11:59 pm, April 13th. See Piazza for more detail.
  • Homework 3 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is April 20 (11:59 pm).

Lectures

 

Date Lecture Scribe Readings Announcements
Jan 16, 2018 Lecture 1 (Kayhan) – Slide, Video

Probabilistic Graphical Model:
A view from the moon

 None None HW0 is out.
Module 1: Representation
Jan 18, 2018 Lecture 2 (Kayhan) – Slide, Annotated, Video

Directed GMs: Bayesian Networks

 Notes (by Sumedha Singla) 1. Koller and Friedman Textbook, Ch. 3
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 3
Jan 23, 2018 Lecture 3 (Kayhan) – Slide, Annotated, Video

Undirected Graphic Model

 Notes (by Arjun Sharma) 1. Koller and Friedman Textbook, Ch. 4
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 4		 HW 0 is due today.
Jan 25, 2018 Lecture 4 (Kun Zhang) – Slide, Annotated, Video

Causal Graphic Model

 Notes (by M. Malik and N. Shajarisales) None The reading summary is due in a week.
Module 2: Classical Methods of Inference & Learning
Jan 30, 2018 Lecture 5 (Kayhan) – Slide, Annotated, Video

Algorithms for Exact Inference

 Notes (by MF. Chang and D. Rajagopal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)
Feb 1, 2018 Lecture 6 (Kayhan) – Slide, Annotated, Video

Factor graph, message passing, and Junction Tree

 Notes (S. Greg) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)
Feb 6, 2018 Lecture 7 (Kayhan) – Slide, Annotated, My Notes, Class Notes, Video

Exponential families and friends: Learning the parameters of a fully observed BN

 Notes (A. Kamath) Kevin Murphy, Machine Learning A Probabilistic Perspective, Ch. 9

or

Jordan Textbook, Ch. 8, Ch. 9
Feb 8, 2018 Lecture 8 (Kayhan) – Slide, Annotated, My Notes, Class Notes, Video

Learning the parameters of UGM

 Notes (C. Zhou and C. Zhang) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9, Section 6
Feb 13, 2018 Lecture 9 (Kayhan) – Slide, Video

EM and partially observed GM

 None Jordan Textbook, Ch. 10

or  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 11

 The reading summary is due in a week.
Module 3: Graphical Model in Application & Learning
Feb 15, 2018 Lecture 10 (Kayhan) – Slide, Annotated, Video

HHM and CRF

 Notes (B. Lengerich and M. Kleyman) H. Wallach, Conditional Random Fields: An Introduction

J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Feb 20, 2018 Lecture 11 (Kayhan) – Slides, Annotated, Video

CRF (Cont’d) + Intro to Topic Models

 Notes (A. Yang and J. He) D. Blei et al., Latent Dirichlet Allocation (Sections 1-4)
Feb 22, 2018 Lecture 12 (Kayhan) – Slides, Annotated, Class Notes, Video

Intro to Topic Models (Cont’d), Factor Analysis and (maybe State Space)

 Notes (G. Plumb and A. Rumack) Jordan Textbook, Ch. 14

Jordan Textbook, Ch. 15 (if we finished State Space)
Feb 27, 2018  No Class HW 2 is out.
Mar 1, 2018 Lecture 13 (Kun Zhang) – Slides, Video

Learning Structure of a Graphical Model

 None This short overview paper

and

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9.5

 The reading summary is due in a week.
Module 4: Approximate Inference & Learning
Mar 6, 2018 Lecture 14 (Kayhan) – Slides, Annotated, Video

Loopy Belief Propagation

 Notes (K. Xiong, C. Malaviya) M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1
Mar 8, 2018 Lecture 15 (Kayhan) – Slides, Annotated, Video

Mean field Approximation

 Notes (Y. Hung, H. Tsai) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26
Mar 20, 2018 Lecture 16 (Kayhan) – Slides, Annotated, Video

Stochastic Gradient Descent, SVI, and scalability

 Notes (Y. Feng, J. Wu) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26
Mar 22, 2018 Lecture 17 (Kayhan) – Slides, Jupyter-notebook, Video

Approximate Inference Monte Carlo Methods

 Notes (B. Paria, P. Chikersal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27
Mar 27, 2018 Lecture 18 (Kayhan) – Slides, Annotated, My Notes, Video

MCMC and Gibbs sampling

 None David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27

Convergence Diagnostics For MCMC
Mar 29, 2018 Lecture 19 (Kayhan) – Slides, Annotated, Video

Hamiltonian Monte Carlo

 Notes (B. Lyu) Slice Sampling: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Section 29.7

HMC: A Conceptual Introduction to Hamiltonian Monte Carlo

 The reading summary is due in a week.
Module 5: Deep Learning and Graphical Models & Learning
Apr 3, 2018 Lecture 20 (Kayhan) – Slides, Annotated, Video

Introduction to Deep Learning

 Notes(A. Alavi, Yu Chen) Deep Learning Book, Ch. 6.2-5, 20.3-4

Optional:
Apr 5, 2018  No Class
Apr 10, 2018 Lecture 21 (Kayhan) – Slides, Annotated, Video

A Hybrid: Deep Learning and Graphical Models

 Notes (P. Liang, A. Rayasam) Diederik P. Kingma, Variational Inference & Deep Learning: A New Synthesis,  Ch. 2

Generative Adversarial Nets: Goodfellow et al., 2014.
Apr 12, 2018 Lecture 22 (Kayhan, Mingming Gong) – Slides, Annotated, Video

A Hybrid DL and GM (cont’d) +
Applications in Computer Vision

 Notes (S. Bai, C.-K. Yeh) None
Apr 17, 2018 Lecture 23 (Mingming Gong, Kayhan) – Slides, Video

Applications in Computer Vision (cont’d) + Gaussian Process

 None None The reading summary is due in a week.
Module 6: Spectral and non-parametric view & Learning
Apr 24, 2018 Lecture 24 (Kayhan) – Slides, Jupyter-Notebook, Annotated, Video

Gaussian Process

 Notes (A. Siddhant, S. Ghosh, C. Nagpal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 19

Optional: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2
Apr 26, 2018 Lecture 25 (Forough Arabshahi) – Slides, Annotated, Video

Spectral Methods

 None None