Transformers

Speaker

Helen Qu | Flatiron Institute

Build a decoder-only transformer (GPT-style) from scratch in PyTorch. Train it on Tiny Shakespeare for character-level language modeling and generate text.

Topics covered:

• Self-attention as a learned, data-dependent mixing operator
• Causal (masked) self-attention for autoregressive modeling
• Building a GPT-style Transformer block from scratch
• Token and positional embeddings
• Training a small autoregressive language model
• Text generation with temperature and top-k sampling

Lecture tutorial materials and jupyter notebooks can be found here: https://tingyuansen.github.io/NASA_AI_ML_STIG/#schedule

We'll be using this slido link for Q&A during the talk: https://app.sli.do/event/kx2WYpZtkHYtbWMntJoFFj 

The link to join the meeting is here: https://science.nasa.gov/astrophysics/programs/cosmic-origins/community/ai-ml-stig-lecture-series-9-feb-2026/