CS336: Language Modeling from Scratch

Logistics

• Lectures: Monday/Wednesday 3:00-4:20pm in Skilling Auditorium
• Office hours:
  • Tatsu Hashimoto: 1x per week
  • Percy Liang: 1x per week
  • Marcel Rød: 2x per week
  • Herman Brunborg: 2x per week
  • Steven Cao: 2x per week
• Contact: Students should ask all course-related questions in public Slack channels. All announcements will also be made in Slack. For personal matters, email cs336-spr2526-staff@lists.stanford.edu.

Coursework

Assignments

• Assignment 1: Basics (version from 2025)
• Implement all of the components (tokenizer, model architecture, optimizer) necessary to train a standard Transformer language model.
• Train a minimal language model.
• Assignment 2: Systems (version from 2025)
• Profile and benchmark the model and layers from Assignment 1 using advanced tools, optimize Attention with your own Triton implementation of FlashAttention2.
• Build a memory-efficient, distributed version of the Assignment 1 model training code.
• Assignment 3: Scaling (version from 2025)
• Understand the function of each component of the Transformer.
• Query a training API to fit a scaling law to project model scaling.
• Assignment 4: Data (version from 2025)
• Convert raw Common Crawl dumps into usable pretraining data.
• Perform filtering and deduplication to improve model performance.
• Assignment 5: Alignment and Reasoning RL (version from 2025)
• Apply supervised finetuning and reinforcement learning to train LMs to reason when solving math problems.
• Optional Part 2 (version from 2025): implement and apply safety alignment methods such as DPO.

All (currently tentative) deadlines are listed in the schedule.

Honor code

Like all other classes at Stanford, we take the student Honor Code seriously. Please respect the following policies:

• Collaboration: Study groups are allowed, but students must understand and complete their own assignments, and hand in one assignment per student. If you worked in a group, please put the names of the members of your study group at the top of your assignment. Please ask if you have any questions about the collaboration policy.
• AI tools: Prompting LLMs such as ChatGPT is permitted for low-level programming questions or high-level conceptual questions about language models, but using it directly to solve the problem is prohibited. We strongly encourage you to disable AI autocomplete (e.g., Cursor Tab, GitHub CoPilot) in your IDE when completing assignments (though non-AI autocomplete, e.g., autocompleting function names is totally fine). We have found that AI autocomplete makes it much harder to engage deeply with the content.
• Existing code: Implementations for many of the things you will implement exist online. The handouts we'll give will be self-contained, so that you will not need to consult third-party code for producing your own implementation. Thus, you should not look at any existing code unless when otherwise specified in the handouts.

Submitting coursework

• All coursework are submitted via Gradescope by the deadline. Do not submit your coursework via email.
• If anything goes wrong, please ask a question in Slack or contact a course assistant.
• You can submit as many times as you'd like until the deadline: we will only grade the last submission.
• Partial work is better than not submitting any work.

Late days

• Each student has 6 late days to use. A late day extends the deadline by 24 hours.
• You can use up to 3 late days per assignment.

Regrade requests

If you believe that the course staff made an objective error in grading, you may submit a regrade request on Gradescope within 3 days after the grades are released.

Sponsor

We would like to thank Together AI for sponsoring the compute for this class.