I wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/independent researchers/startups, and I think with a 175B parameter model, not having the trained model itself isn't necessarily the biggest hurdle for most people (having hardware with x hundred GB memory is - well maybe not for startups.)
Although there seems to be hope:
>In prior work, OpenAI found that the outputs from models trained with RLHF were preferred to those from 100x larger models trained without human feedback.
Yeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B models are able to compete with GPT task [0] so I guess it might be using it but probably isn't, so there's further room for improvement.
More relevantly, it means that the model parameter count, dataset size, and training procedure are balanced such that they can achieve equal or better performance as GPT-3 while being smaller.
This is planned to be 70B but trained in the chinchilla-optimal way (more data + training). Scaling laws suggest this should outperform the base 175B GPT-3. Then release the base model as well as the RLHF-tuned models.
Just to get OTP up for inference would require a very large spend. To use GPT-NeoX (20B parameters) for inference requires 45GB of vRAM minimally. Its hard for me to imagine using a 170B model for fun somewhere unless one has a large GPU farm or lots of money.
GPT-NeoX-20B was specifically targeted to fit on A40s, A6000s, and a pair of 3090 Tis. Anything larger than that is going to be a real struggle for people who don’t own computing clusters to use.
I would urge them to build something good that fits on 24GB maybe 48GB VRAM, otherwise there will be slow opensource progress. I need Codex alternative like that desperately to fix programming.
I've been waiting for this ever since they trained BLOOM and even GPT Neo. Without instruction tuning they are almost worthless. If all you want is classification then it's better to use BERT, but if you need the generative part then you have to scale up to these big guys.
What I need is the instruction following skills of GPT-3, running on a single (large) GPU, and not having to pay by token.
I am very much wishing for the same. I have used T0_3B (https://github.com/bigscience-workshop/t-zero) on my M40 with 24GB of vRAM. It was fun to get it working, but its not remotely comparable to GPT-3 (instructGPT).
I wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/independent researchers/startups, and I think with a 175B parameter model, not having the trained model itself isn't necessarily the biggest hurdle for most people (having hardware with x hundred GB memory is - well maybe not for startups.)
Although there seems to be hope:
>In prior work, OpenAI found that the outputs from models trained with RLHF were preferred to those from 100x larger models trained without human feedback.
Yeah, they say it will be "chinchilla-optimal", which means that it probably will be < 70B, might be actually much less as I've seen some recent work that 20B models are able to compete with GPT task [0] so I guess it might be using it but probably isn't, so there's further room for improvement.
[0]https://www.reddit.com/r/MachineLearning/comments/y4tp4b/r_u...
Seems like that 20B model is available on HuggingFace [0]
[0] https://huggingface.co/docs/transformers/model_doc/ul2
What's "chinchilla"? is that a codename for a specific CPU/GPU?
EDIT: no, it's a competitor with GPT-3 [1]
[1] https://analyticsindiamag.com/deepmind-launches-gpt-3-rival-...
More relevantly, it means that the model parameter count, dataset size, and training procedure are balanced such that they can achieve equal or better performance as GPT-3 while being smaller.
This is planned to be 70B but trained in the chinchilla-optimal way (more data + training). Scaling laws suggest this should outperform the base 175B GPT-3. Then release the base model as well as the RLHF-tuned models.
> CarperAI will release a chinchilla-optimal large language model
Next-level terminology
https://www.deepmind.com/publications/an-empirical-analysis-...
for now I'd checkout T0 or bloom, lots of models can be instructed already.
I created https://text-generator.io and have a whole bunch of examples on there of instruction following style prompts
How do you instruct BLOOM? Does it need fine-tuning?
have you considered SQL generation with two inputs:
1. data from the information_schema of the database
2. a natural language question
I've played around with gpt3 but not providing a full schema for the database is a glaring issue with accuracy of the SQL generated.
I've done something similar with gpt3 (codex) and had good results
Has anyone tested OTP 170B from Meta? Seems like it's public
Just to get OTP up for inference would require a very large spend. To use GPT-NeoX (20B parameters) for inference requires 45GB of vRAM minimally. Its hard for me to imagine using a 170B model for fun somewhere unless one has a large GPU farm or lots of money.
GPT-NeoX-20B was specifically targeted to fit on A40s, A6000s, and a pair of 3090 Tis. Anything larger than that is going to be a real struggle for people who don’t own computing clusters to use.
I would urge them to build something good that fits on 24GB maybe 48GB VRAM, otherwise there will be slow opensource progress. I need Codex alternative like that desperately to fix programming.
I've been waiting for this ever since they trained BLOOM and even GPT Neo. Without instruction tuning they are almost worthless. If all you want is classification then it's better to use BERT, but if you need the generative part then you have to scale up to these big guys.
What I need is the instruction following skills of GPT-3, running on a single (large) GPU, and not having to pay by token.
I am very much wishing for the same. I have used T0_3B (https://github.com/bigscience-workshop/t-zero) on my M40 with 24GB of vRAM. It was fun to get it working, but its not remotely comparable to GPT-3 (instructGPT).
Can anyone explain what’s new here? I couldn’t quite follow the article.
I wonder if this relates to DeepCTRL that came out recently.