Schedule

Introduction and Background

Aug 29

No Class

Aug 31

Lecture Introduction, Course Overview, & How-to Read/Review/Present (on Zoom)

Sep 5

Lecture Crash Course on Classical ML - Part 1 (on Zoom)

Sep 7

Lecture Crash Course on Classical ML - Part 2 (on Zoom)

Generalization Mysteries

Sep 12

Discussion (on Zoom)

• Paper A: Understanding Deep Learning Requires Rethinking Generalization
• (optional) Understanding Deep Learning (Still) Requires Rethinking Generalization

• Paper B: Fantastic Generalization Measures and Where to Find Them

Sep 14

Lecture (on Zoom)

• Implicit bias of SGD / Algorithmic regularization

Discussion (on Zoom)

• Paper A: Reconciling modern machine-learning practice and the classical bias-variance tradeoff
• (optional) Deep Double Descent: Where Bigger Models and More Data Hurt

Sep 19

Lecture (on Zoom)

• Benign Overfitting
• Unresolved Mysteries

Optimization Dynamics

Sep 21

Lecture (on Zoom)

• Convex Optimization
• Non-convex Landscape
• Tractability

Sep 26

Discussion Neural Tangent Kernel (NTK) (on Zoom)

• Paper A: Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• (optional) A Note on Lazy Training in Supervised Differentiable Programming

Guest Lecture Beyond NTK (on Zoom)

• Eran Malach, Postdoc, Harvard
• (optional) Learning Parities with Neural Networks

Sep 28

No Class ICLR Deadline

Oct 3

Discussion Optimization Surprises/Mysteries

• Paper A: Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit
• (optional) SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics

• Paper B: Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• (optional) Learning threshold neurons via the “edge of stability”

Unsupervised Learning

Oct 5

Lecture A primer on semi-supervised, active, and self-supervised learning

Oct 10

Discussion Self-supervised Learning

• Paper A: Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• (optional) Understanding Contrastive Learning Requires Incorporating Inductive Biases

• Paper B: Masked Autoencoders Are Scalable Vision Learners

Oct 12

No Class Fall Break

Foundation Models

Oct 17

Lecture LLMs/Transformers - History, Basics

Oct 19

Dicussion LLMs/Transformers - Reasoning

• Paper A: Transformers Learn Shortcuts to Automata
• (recommended) Blog about Transformers Learn Shortcuts to Automata
• (optional) Self-Attention Networks Can Process Bounded Hierarchical Languages

• Paper B: Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Oct 24

Dicussion LLMs/Transformers - In-context Learning

• Paper A: What Can Transformers Learn In-Context? A Case Study of Simple Function Classes

• Paper B: An Explanation of In-context Learning as Implicit Bayesian Inference
• (optional) Transformers learn in-context by gradient descent

Oct 26

Lecture Score-based and Diffusion Models - Overview and Technical Derivation

Oct 31

Dicussion Score-based and Diffusion Models - Improvements

• Paper A: Score-Based Generative Modeling through Stochastic Differential Equations
• (recommended) Blog

• Paper B: Consistency Models

OOD and Adversarial Robustness

Nov 2

Lecture Robustness - Background

Nov 7

Dicussion Robustness - Adversarial

• Paper A: Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

• Paper B: Certified Adversarial Robustness via Randomized Smoothing

Nov 9

Lecture Robustness - Out-of-distribution

Dicussion Robustness - Out-of-distribution

• Paper A: Understanding the Failure Modes of Out-of-Distribution Generalization

Challenges

Nov 14

Discussion Scale

• Paper A: Training Compute-Optimal Large Language Models

• Paper B: Beyond neural scaling laws: beating power law scaling via data pruning

Nov 16

Dicussion Bias

• Paper A: Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings

• Paper B: On the dangers of stochastic parrots: can language models be too big?

Nov 21

Lecture Miscellaneous Topics

Nov 23

No Class Thanksgiving Break

Guest Lectures and Paper Presentations

Nov 28

Guest Lecture Reinforcement Learning

• Akshay Krishnamurthy, Principal Research Manager, Microsoft Research, New York City

Nov 30

Guest Lecture

• Eric Wong, Assistant Professor, UPenn CIS

Dec 5

Project Presentation

Dec 7

Project Presentation