Teaching

10-708 (CMU) Probabilistic Graphical Models

Probabilistic Graphical Models

Overview


Many of the problems in artificial intelligence, statistics, computer systems, computer vision, natural language processing, and computational biology, among many other fields, can be viewed as the search for a coherent global conclusion from local information. The probabilistic graphical model’s framework provides a unified view of this wide range of problems, enabling efficient inference, decision-making and learning in problems with a very large number of attributes and huge datasets. This graduate-level course will provide you with a strong foundation for both applying graphical models to complex problems and for addressing core research topics in graphical models.  The class will cover three aspects: The core representation, including Bayesian and Markov networks, and dynamic Bayesian networks; probabilistic inference algorithms, both exact and approximate; and, learning methods for both the parameters and the structure of graphical models. Students entering the class should have a pre-existing working knowledge of probability, statistics, and algorithms, though the class has been designed to allow students with a strong numerate background to catch up and fully participate.  It is expected that after taking this class, the students should have obtained sufficient working knowledge of multivariate probabilistic modeling and inference for practical applications, should be able to formulate and solve a wide range of problems in their own domain using GM and can advance into more specialized technical literature by themselves.  Students are required to have successfully completed 10701 or 10715, or an equivalent class.

Where and When

People


Instructor:

  • Kayhan Batmanghelich
    Office Hour: Gates-Hillman Center 8228 on Tuesdays  1:30 – 2:30 pm

Administrative Assistant:

  • Noreen Doloughty

Teaching Assistants:

If you wish to email only the instructors, the email is 10708Spring18@gmail.com.

Announcements


  • If you have any questions about class policies or course material, please use piazza or you can email all of the instructors at 10708Spring18@gmail.com. Please use this list instead of individual email addresses to ensure a prompt response.
  • The Homework 0 is out. The deadline is 7:00 pm Jan 23. No late HW will be accepted.
  • Please find the scribe template here.
  • The project proposal will be due by 7 PM on Friday, February 16th, and should be submitted via Gradescope.
  • Homework 1 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Feb 18 (11:59 pm).
  • Homework 2 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is Mar 21 (11:59 pm).
  • The project midway report will be due 11:59 pm, April 13th. See Piazza for more detail.
  • Homework 3 is released. You can view it on the course page: https://piazza.com/cmu/spring2018/10708/resources. The due date is April 20 (11:59 pm).

Lectures


 

Date Lecture Scribe Readings Announcements
Jan 16, 2018 Lecture 1 (Kayhan) – SlideVideo

Probabilistic Graphical Model:
A view from the moon

None None HW0 is out.
Module 1: Representation
Jan 18, 2018 Lecture 2 (Kayhan) – SlideAnnotatedVideo

Directed GMs: Bayesian Networks

Notes (by Sumedha Singla) 1. Koller and Friedman Textbook, Ch. 3
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 3
Jan 23, 2018 Lecture 3 (Kayhan) – SlideAnnotatedVideo

Undirected Graphic Model

Notes (by Arjun Sharma) 1. Koller and Friedman Textbook, Ch. 4
2. David Barber, Bayesian Reasoning, and Machine Learning, Ch. 4
HW 0 is due today.
Jan 25, 2018 Lecture 4 (Kun Zhang) – Slide, Annotated, Video

Causal Graphic Model

Notes (by M. Malik and N. Shajarisales) None The reading summary is due in a week.
Module 2: Classical Methods of Inference & Learning
Jan 30, 2018 Lecture 5 (Kayhan) – SlideAnnotatedVideo

Algorithms for Exact Inference

Notes (by MF. Chang and D. Rajagopal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)

Feb 1, 2018 Lecture 6 (Kayhan) – SlideAnnotatedVideo

Factor graph, message passing, and Junction Tree

Notes (S. Greg) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 5 and 6

Other resources: (1)

Feb 6, 2018 Lecture 7 (Kayhan) – SlideAnnotated, My Notes, Class NotesVideo

Exponential families and friends: Learning the parameters of a fully observed BN

Notes (A. Kamath) Kevin Murphy, Machine Learning A Probabilistic Perspective, Ch. 9

or

Jordan Textbook, Ch. 8, Ch. 9

Feb 8, 2018 Lecture 8 (Kayhan) – SlideAnnotatedMy NotesClass NotesVideo

Learning the parameters of UGM

Notes (C. Zhou and C. Zhang) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9, Section 6
Feb 13, 2018 Lecture 9 (Kayhan) – SlideVideo

EM and partially observed GM

None Jordan Textbook, Ch. 10

or  David Barber, Bayesian Reasoning, and Machine Learning, Ch. 11

The reading summary is due in a week.
Module 3: Graphical Model in Application & Learning
Feb 15, 2018 Lecture 10 (Kayhan) – SlideAnnotated, Video

HHM and CRF

Notes (B. Lengerich and M. Kleyman) H. Wallach, Conditional Random Fields: An Introduction

J. Lafferty et al., Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Feb 20, 2018 Lecture 11 (Kayhan) – SlidesAnnotatedVideo

CRF (Cont’d) + Intro to Topic Models

Notes (A. Yang and J. He) D. Blei et al., Latent Dirichlet Allocation (Sections 1-4)
Feb 22, 2018 Lecture 12 (Kayhan) – SlidesAnnotatedClass NotesVideo

Intro to Topic Models (Cont’d), Factor Analysis and (maybe State Space)

Notes (G. Plumb and A. Rumack) Jordan Textbook, Ch. 14

Jordan Textbook, Ch. 15 (if we finished State Space)

Feb 27, 2018  No Class HW 2 is out.
Mar 1, 2018 Lecture 13 (Kun Zhang) – SlidesVideo

Learning Structure of a Graphical Model

None This short overview paper

and

David Barber, Bayesian Reasoning, and Machine Learning, Ch. 9.5

The reading summary is due in a week.
Module 4: Approximate Inference & Learning
Mar 6, 2018 Lecture 14 (Kayhan) – SlidesAnnotatedVideo

Loopy Belief Propagation

Notes (K. Xiong, C. Malaviya) M. Wainwright and M. Jordan, Graphical Models, Exponential Families, and Variational Inference, Sec. 4.1
Mar 8, 2018 Lecture 15 (Kayhan) – SlidesAnnotatedVideo

Mean field Approximation

Notes (Y. Hung, H. Tsai) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26
Mar 20, 2018 Lecture 16 (Kayhan) – SlidesAnnotatedVideo

Stochastic Gradient Descent, SVI, and scalability

Notes (Y. Feng, J. Wu) D. M. Blei, A. Kucukelbir, J. D. McAuliffe, Variational Inference: A Review for Statisticians, Pages 1-26
Mar 22, 2018 Lecture 17 (Kayhan) – SlidesJupyter-notebook, Video

Approximate Inference Monte Carlo Methods

Notes (B. Paria, P. Chikersal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27
Mar 27, 2018 Lecture 18 (Kayhan) – SlidesAnnotatedMy Notes, Video

MCMC and Gibbs sampling

None David Barber, Bayesian Reasoning, and Machine Learning, Ch. 27

Convergence Diagnostics For MCMC

Mar 29, 2018 Lecture 19 (Kayhan) – SlidesAnnotatedVideo

Hamiltonian Monte Carlo

Notes (B. Lyu) Slice Sampling: David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Section 29.7

HMC: A Conceptual Introduction to Hamiltonian Monte Carlo

The reading summary is due in a week.
Module 5: Deep Learning and Graphical Models & Learning
Apr 3, 2018 Lecture 20 (Kayhan) – SlidesAnnotatedVideo

Introduction to Deep Learning

Notes(A. Alavi, Yu Chen) Deep Learning Book, Ch. 6.2-5, 20.3-4

Optional:

Apr 5, 2018  No Class
Apr 10, 2018 Lecture 21 (Kayhan) – SlidesAnnotatedVideo

A Hybrid: Deep Learning and Graphical Models

Notes (P. Liang, A. Rayasam) Diederik P. Kingma, Variational Inference & Deep Learning: A New Synthesis,  Ch. 2

Generative Adversarial Nets: Goodfellow et al., 2014.

Apr 12, 2018 Lecture 22 (Kayhan, Mingming Gong) – SlidesAnnotatedVideo

A Hybrid DL and GM (cont’d) +
Applications in Computer Vision

Notes (S. Bai, C.-K. Yeh) None
Apr 17, 2018 Lecture 23 (Mingming Gong, Kayhan) – SlidesVideo

Applications in Computer Vision (cont’d) + Gaussian Process

None None The reading summary is due in a week.
Module 6: Spectral and non-parametric view & Learning
Apr 24, 2018 Lecture 24 (Kayhan) – SlidesJupyter-NotebookAnnotatedVideo

Gaussian Process

Notes (A. Siddhant, S. Ghosh, C. Nagpal) David Barber, Bayesian Reasoning, and Machine Learning, Ch. 19

Optional: L. Song. Learning via Hilbert space embedding of distributions, Sec. 2.1, 2.2, 3.1, 3.2

Apr 26, 2018 Lecture 25 (Forough Arabshahi) – SlidesAnnotatedVideo

Spectral Methods

None None