Transformers

Speaker

Helen Qu, Flatiron

Build a decoder-only transformer (a small GPT-like language model) from scratch in PyTorch. Train it on the Tiny Shakespeare dataset for character-level language modeling and use it to generate text, understanding every component along the way.

Topics Covered:

• Self-attention as a learned, data-dependent mixing operator
• Causal (masked) self-attention for autoregressive modeling
• Building a GPT-style Transformer block from scratch
• Token and positional embeddings
• Training a small autoregressive language model
• Text generation with temperature and top-k sampling

Meeting Connection

Join the Meeting