Build a Large Language Model (From Scratch)
Original price was: $180.99.$29.00Current price is: $29.00.
🔒 Secure Checkout ⚡ Instant Digital Download
How to implement LLM attention mechanisms and GPT-style transformers.
In Build a Large Language Model (from Scratch) bestselling author Sebastian Raschka guides you step by step through creating your own LLM. Each stage is explained with clear text, diagrams, and examples. You’ll go from the initial design and creation, to pretraining on a general corpus, and on to fine-tuning for specific tasks.
Build a Large Language Model (from Scratch) teaches you how to:
• Plan and code all the parts of an LLM
• Prepare a dataset suitable for LLM training
• Fine-tune LLMs for text classification and with your own data
• Use human feedback to ensure your LLM follows instructions
• Load pretrained weights into an LLM
Build a Large Language Model (from Scratch) takes you inside the AI black box to tinker with the internal systems that power generative AI. As you work through each key stage of LLM creation, you’ll develop an in-depth understanding of how LLMs work, their limitations, and their customization methods. Your LLM can be developed on an ordinary laptop, and used as your own personal assistant.
Purchase of the print book includes a free eBook in PDF and ePub formats from Manning Publications.
About the technology
Physicist Richard P. Feynman reportedly said, “I don’t understand anything I can’t build.” Based on this same powerful principle, bestselling author Sebastian Raschka guides you step by step as you build a GPT-style LLM that you can run on your laptop. This is an engaging book that covers each stage of the process, from planning and coding to training and fine-tuning.
About the book
Build a Large Language Model (From Scratch) is a practical and eminently-satisfying hands-on journey into the foundations of generative AI. Without relying on any existing LLM libraries, you’ll code a base model, evolve it into a text classifier, and ultimately create a chatbot that can follow your conversational instructions. And you’ll really understand it because you built it yourself!
What’s inside
• Plan and code an LLM comparable to GPT-2
• Load pretrained weights
• Construct a complete training pipeline
• Fine-tune your LLM for text classification
• Develop LLMs that follow human instructions
About the reader
Readers need intermediate Python skills and some knowledge of machine learning. The LLM you create will run on any modern laptop and can optionally utilize GPUs.
About the author
Sebastian Raschka is a Staff Research Engineer at Lightning AI, where he works on LLM research and develops open-source software.
The technical editor on this book was David Caswell.
Table of Contents
1 Understanding large language models
2 Working with text data
3 Coding attention mechanisms
4 Implementing a GPT model from scratch to generate text
5 Pretraining on unlabeled data
6 Fine-tuning for classification
7 Fine-tuning to follow instructions
A Introduction to PyTorch
B References and further reading
C Exercise solutions
D Adding bells and whistles to the training loop
E Parameter-efficient fine-tuning with LoRA
Product Details & What’s Included:
-
📖 eBook: PDF + EPUB (370 Pages)
-
🎥 Video Course: 77 MP4 Lessons (8 Hours 12 Mins)
-
💻 Code Files: Ready-to-use Python Scripts + Datasets
-
👤 Author: Sebastian Raschka
-
🏢 Publisher: Manning / O’Reilly
-
⚡ Access: Instant Digital Download
-
🔄 Updates: Free Lifetime Updates
23 reviews for Build a Large Language Model (From Scratch)
Related products
-
Building LLMs for Production: Enhancing LLM Abilities and Reliability with Prompting, Fine-Tuning, and RAG
Original price was: $49.99.$9.99Current price is: $9.99. Add to cart -
Computer Programming And Cyber Security for Beginners: This Book Includes: Python Machine Learning, SQL, Linux, Hacking with Kali Linux, Ethical Hacking. Coding and Cybersecurity Fundamentals
Original price was: $111.97.$19.00Current price is: $19.00. Add to cart -
Microsoft Excel 365 Bible 2nd Edition
Original price was: $155.00.$23.00Current price is: $23.00. Add to cart -
AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch
Original price was: $99.99.$23.05Current price is: $23.05. Add to cart
Stephan Miller (verified owner) –
Though I don’t think I be building my own LLM any time soon, it does help to know how they work when you deal with them. Some of the quirks and unexpected responses start to make more sense. I have worked with classification and deep learning before, but LLMs were a black box to me. Now, at least, I have a clearer idea of what they do and actually keep up how the technology is changing.
TanelP (verified owner) –
This book was perfect for me. I’m a computer performance specialist, but haven’t yet gotten serious about ML and language models. I’ve read occasional overview articles, so have an idea what things like “vectors” and “matrix multiplication” are, but I didn’t see the full picture. I had bought some other machine learning books before that tried to cover everything about everything and never got even half-way through reading them. This book covers not only the practical examples (and source code) with all the steps for training your own toy language models (Python/pytorch code), but also it explained how all the training layers work together in unison. On the training architecture topic, this book did a better job in a handful of pages than all the deep papers I had read in the past, so I should probably have started from this book, not the other way around.
Also, the book does a good job incrementally building the knowledge by adding a new layer after another as you progress through the book. Highly recommended!
BRIAN (verified owner) –
What an amazing book detailing how each component of the language models components fit together and work synchronously. It is not too difficult to read / follow along if you have previous coding experience with Neural Networks and PyTorch on Machine learning projects. It definitely was a great purchase to understand what it takes to build a local LLM. I had to remove 1 star because the book already tore a bit on the front cover on day 3 of reading.
Dennis Davis (verified owner) –
I’m still reading the book, and completed coding everything in Chapter 2. So far the approach of breaking down the concepts into fundamental parts and then showing how those parts are built into more complex implementations – that can then be better understood because of the author’s presentation is perfect for how I learn.
For the benefit of others with NVIDIA GPU configuring CUDA:
1 Find the CUDA support level of your GPU – on Windows NVIDIA Control Panel -> System Information(at the bottom) -> Components tab – installed driver software SUPPORT LEVEL is listed – Not the actual software!
2 Install MS Visual Studio (2022) Needed by NVIDIA CUDA software
3 Install the version of the NVIDIA CUDA software supported by info from step 1 AND PyTorch (for example my HW supported CUDA up to 12.7, but PyTorch software support tops out at 12.4 (as of today 1/13/2025), so I went with 12.4 NVIDIA CUDA software.
4. During the driver custom install (not the default simplified install) deselect NVIDIA GeForce Experience – caused errors for me
5. Reboot after NVIDIA CUDA software installation
6. On the NVIDIA CUDA installation page there are deviceQuery and bandwidthTest exe’s that will validate the CUDA HW/SW interface is functioning
7. Run the PyTorch installer – I use Anaconda environment- so ran the conda install command coped from the PyTorch installation web page (shown in the book), from a command line inside my conda target environment – restart anaconda, I use vs code restarted both when the install was completed
8. On the NVIDIA CUDA installation page it states to install – conda install cuda -c nvidia to the conda target environment – when the book says run – torch.cuda.is_available() it should return True
I don’t consider this a defect of the book – there is already enough hand-holding by the author – imho some work still needs to be done by the reader!!
So far getting a great appreciation/comprehension about what is behind Large Lanquage Models – Thank You!!
Wael Mohsen (verified owner) –
I appreciated the book for its thoroughness and attention to detail. However, I believe it would benefit from being printed in color, as many images on the O’Reilly website are more vibrant and clearer when viewed in color. Additionally, enhancing the resolution of some images would improve the overall experience. For these reasons, I would rate the book 4 out of 5. With these adjustments, I think it could easily earn a perfect score of 5 out of 5.
Manfred Kremer (verified owner) –
This book shows step by step all ingredients which are put together in order to build a GPT-2 model from scratch. All functions are explained explicitely in python, before the equivalent functions of pytorch are used. I really liked to follow the book to the end.
There is also a discussion forum about the book on github, where readers can ask questions, which are promptly answered by the author.
That said, there remain many questions about WHY the method works, and why some steps are made. E.g. why use multihead attention: to my understanding this completely scrambles the embedding vectors, and it is like a miracle that the method works so well. But there were page limits for the book, and and going deeper into this kind of questions would pprobably have doubled the size of the book.
Christoph Kuhn (verified owner) –
Gute begleitende Praxisbeispiele mit Code
Higgs meets Boson (verified owner) –
I’ve bought tons of ML, DE, programming, cloud architecture books, etc…
This book is absolutely fantastic! Especially combined by the current YouTube series published by the author (March 2025).
Sebastian’s Packt books are also excellent but I must say this book stands on its own. This book is extremely well written and clear, builds each component in the Transformer Architecture piece by piece, it makes me feel like I can actually build an LLM on my own.
At a minimum this book will help you understand the Transformer Architecture (Attention Mechanism, Feed Forward, Layer Norm, etc…) rather than importing models from HugginFace and not really know what’s going on in the background.
If you are like me and are not satisfied with just building RAGs/LLM applications without understanding the model architecture, this book is for you!
I’ll keep buying from this author as long as the quality of his content is as good as this.
Hayes (verified owner) –
Good book to get a up close look at LLM development in a rudimentary level. Glosses over some sections so you may need to supplement with some searching but you can make a starter chat bot with what’s in book.
Eugene Veselov (verified owner) –
The book is amazing. Much better than I expected. I was minimally familiar with neural networking techniques (finished 6-months course on Coursera, and by now have forgotten most of it). So, I had a vague idea about forward and backward propagation, remembered such terms as dropout, normalization etc. without actually remembering their meaning. From the Andrew Ng course I remembered the term “transformer” (since he had a few good introductory explanations of it), but by now I completely forgot how it works. My knowledge of Python was very limited (and mostly forgotten). I knew nothing about PyTorch. When I saw the references to the book on Facebook, I decided that it might be helpful for me to recall these concepts, and especially interesting was to learn the concept of transformers and self-attention which I knew belong to the foundation of modern LLMs.
The book exceeded my expectations. It is written in an excellent methodical style. Introduces concepts one by one, helps experimenting with them in the real code. It provided an excellent introduction to PyTorch (in Appendix A, which the author recommended to consume before reading the rest of the book). The introduction is short, not overwhelming the reader with millions potential concepts of the huge ecosystem of Python and PyTorch, and still sufficient for productive consuming the entire book that uses both. All the concepts are defined in easy-to-consume steps, leading eventually to a complete overall understanding of GPT model. I am not naive to think that I can develop LLMs by myself now, but I definitely got more than expected. And enjoyed the material a lot.
I did not use the code from GitHub (by the book’s reference). Instead, I meticulously re-entered all the examples from the book’s text into several Jupyter Notebooks in VSCode. This way I moved a bit slower but understood material better. Even found a few minor (typo-level) issues in the code.
I am working on an ordinary Surface Book (no GPU), and all examples work instantaneously so far (obviously, it will change when I come to training). I am now in the position after chapter 4: Built the untrained GPT model and cannot wait when I will start training and using it.
Highly recommend the book to everyone who wants to make their hands “dirty” with the AI.
Robert Huntsman (verified owner) –
This book is the most comprehensive yet accessible text I’ve read covering the foundation of Generative AI. There is no better way to master this field than to develop a model from scratch. Sebastian takes you through each step, not skipping detail, until you are confident in coding a model yourself. I highly recommend this book.
Alvaro Brange (verified owner) –
I loved the details and step-by-step approach used in the book. If it is possible, I suggest to also purchasing the videos in Manning which help to understand the concepts.
I have paid much more for some online curses from MIT about DNN, and this book alone have a lot of content also in DNN, although it is not their focus
carson (verified owner) –
I was able to instantly take what I learned from this book and find out why past AI projects failed and how to apply it the the project I’m currently working. Even with up to date college classes, the subject area is so vast.
Rafiq (verified owner) –
This book is an absolute masterpiece. The writer knows how to present complex concepts in simple, absorbable ways. From concepts to labs/demoes, he makes you feel like you’re sitting in an ivy league class. The companion YouTube channel is the icing on the cake. I highly recommend this for anyone interested in learning the fundamentals of ML
Xu Han (verified owner) –
The best LLM book that I have read so far! Crystal clear.
Never Stop Learning (verified owner) –
I think this book is an excellent introduction to the practical aspects of building a Large Language Model (LLM). It illustrates the use of the fundamental components of building LLMs by having the reader follow along by constructing a “toy LLM”; basic linear algebra (e.g. matrix multiplication, matrix transposes), gradient decent optimization, random number generation, and ad hoc aspects of training a model effectively, and the large quantities of digital data (e.g. text, video, images, sound) needed to build a model. My key takeaway from reading this book is that scaling this approach to ever larger datasets combined with ad hoc architecture and parameter tweaks have nothing in common with human intelligence which is far beyond the simplistic toolset used in constructing LLMs. To think otherwise is naive at best and foolish at worst. Relying on this stochastic-by-nature approach to important human endeavors is setting oneself up for some unexpected catastrophic failures.
george (verified owner) –
The best book I have ever read on generative AI and machine learning. It is essential reading for anyone interested in working in the AI field. Even the appendix is highly valuable and well worth reading.
jose david angarita pertuz (verified owner) –
The best book to put ahead on 2026 for AI topics
Heath Hunnicutt (verified owner) –
This is a great book. It’s full of diagrams. The text is well-written.
Manning Press has a neat system, in which the print copy of the book contains a license key for the PDF of the book. I downloaded that and let my LLMs read it, too.
James C. (verified owner) –
Clear Explanations Without Hiding the Hard Parts
What I like most is that it doesn’t treat LLMs like magic. It breaks the process down step by step—from tokenization and embeddings to attention, training, and generation—so I could see how all the pieces connect.
Hands-On Learning That Actually Sticks
The examples make a huge difference. Instead of only reading theory, I was able to follow along and understand how each part works in code. That made concepts like transformers and self-attention feel much less intimidating.
Great for Going Beyond Surface-Level AI
This isn’t just a “what is AI?” book. It helped me understand the mechanics behind modern language models and gave me a much stronger foundation for experimenting on my own.
Challenging, But Worth It
Some sections take focus, but that’s part of why I liked it. It pushes you to really understand the material.
Ting (verified owner) –
Solid and practical! Start from small, But enable you to expand further and gain confidence
Jinshan Liu (verified owner) –
I had grown weary of books that focus solely on how to use pre-built packages, so this was a refreshing change. It gave me a genuinely deep understanding of LLMs — not just how to use them, but how they actually work. The bonus materials provided by the author are the icing on the cake. Highly recommended!
cyee (verified owner) –
This is a very well written book