[1] is a good start, although if you want to train from scratch on CPU, you'll have to downscale as transformers need quite a bit of data before they learn to use position embeddings. For example try a single-layer RNN on Shakespeare texts [2] or a list of movie titles from IMDB [3]. You'll have to fill in the blanks because things have evolved quite a bit since those were used for language models, but you can find some tutorials [4] and examples [5].
It doesn't appear to have an online component, nor is there any mention of OCW, so I'm guessing not. Perhaps they'll record it and post later, though. I was curious to see why this was posted if it's not something that is open to remote participation or viewing, but I guess it was posted for the syllabus/readings?
The linked courseware (required readings) is open and pretty good to get started.
Is it available to be taken remotely at own pace?
Doesn't look like it but the slides and recommended reading are posted and should be able to give you some color.
From the course page - Classes are in person on Wednesdays from 2:00PM - 4:00PM Eastern Standard Time in E14-526
I want a course that can simulate LLM using Small language model instead. It could run fast on a CPU machine, for the sake of learning.
[1] is a good start, although if you want to train from scratch on CPU, you'll have to downscale as transformers need quite a bit of data before they learn to use position embeddings. For example try a single-layer RNN on Shakespeare texts [2] or a list of movie titles from IMDB [3]. You'll have to fill in the blanks because things have evolved quite a bit since those were used for language models, but you can find some tutorials [4] and examples [5].
[1] https://jaykmody.com/blog/gpt-from-scratch/ [2] https://github.com/demmojo/text-rnn/blob/master/datasets/sha... [3] https://datasets.imdbws.com/ [4] https://pytorch.org/tutorials/beginner/chatbot_tutorial.html [5] https://github.com/pytorch/examples/tree/main/word_language_...
Is it possible to view the lectures online?
It doesn't appear to have an online component, nor is there any mention of OCW, so I'm guessing not. Perhaps they'll record it and post later, though. I was curious to see why this was posted if it's not something that is open to remote participation or viewing, but I guess it was posted for the syllabus/readings?