points by benjismith 3 years ago

It's really not so complicated. This is just an issue with text tokenization, and the fact that the learning model never actually sees the raw input bytes.

All modern LLMs use a tokenizer to convert a sequence of bytes into a sequence of tokens. Short, common words like "the" and "why" are represented as single tokens, while longer and less-common words are represented by multiple tokens. For example, the word "fantastic" is three tokens ("f", "ant", "astic").

Each of these tokens is assigned an arbitrary integer value ("fantastic" becomes [69, 415, 3477]) and then those integer values are used to lookup embedding vectors for each word.

Each embedding vector represents the MEANING of the tokens, by plotting them into a 4096-dimensional vector-space. At runtime, the model looks up each token ID in a dictionary and finds its embedding vector.

For the word "fantastic", those embedding vectors might look something like this:

  "f"        (69) = [  0.123,  0.456, ...etc...  0.789, -0.890 ]
  "ant"     (415) = [  0.111, -0.222, ...etc...  0.333, -0.444 ]
  "astic"  (3477) = [ -0.101,  0.202, ...etc... -0.303,  0.404 ]

All of these vectors are assembled into a matrix, and then passed into the layers of neural network, where the actual training/inference occurs.

So the language-model has NO IDEA how any of the words are spelled, because the tokenization (and embedding vector lookup) happens as a pre-processing step, outside the bounds of the learning algorithm.

If you want a LLM to understand spelling, you have to include exhaustive spelling information in its training data. For example:

  "The word 'fantastic' is spelled f-a-n-t-a-s-t-i-c."
  "The word 'FANTASTIC' is spelled F-A-N-T-A-S-T-I-C."
  ...etc...

And even then, even with 100k+ English words all spelled out in your training data, you'd be hard-pressed to infer any ROT-13 tokens in your output data, because the learning model has probably never seen a token like "qvq" or "pebff".

You can play with the GPT tokenizer directly here:

https://beta.openai.com/tokenizer

It will show you the tokenization of any block of text, and the token IDs of the resultant tokens. It's very handy if you spend much time working with GPT-3 (or any other modern language-model!)

yesenadam 3 years ago

So.. how does it do this? :

  Me: To "smert" a word, I tell you every letter in it, and its place in the word, in order. For example, to smert "wertyu" I'd write:
  The first letter is "w".
  The second letter is "e".
  The third letter is "r".
  The fourth letter is "t".
  The fifth letter is "y".
  Lastly, the sixth letter is "u".
  
  Please smert "sdfkjhsndf" for me.
  
  ChatGPT: To "smert" the word "sdfkjhsndf", I would write the following:
  
  The first letter is "s".
  The second letter is "d".
  The third letter is "f".
  The fourth letter is "k".
  The fifth letter is "j".
  The sixth letter is "h".
  The seventh letter is "s".
  The eighth letter is "n".
  The ninth letter is "d".
  Lastly, the tenth letter is "f".
  
  This is a way of spelling out the letters in a word and their order, sometimes used as a learning or spelling aid. I hope this helps!
  
  Me: Please use that word I just asked you to smert in a sentence. It means "to not have an idea how a word is spelled".
  
  ChatGPT: Here is a sentence using the word "sdfkjhsndf", which means "to not have an idea how a word is spelled":
  "I sdfkjhsndfed the spelling of 'onomatopoeia' and had to look it up in the dictionary."
  • sebzim4500 3 years ago

    Interestingly it doesn't do it reliably when you give it an actual word.

        To smert "consequence," please see below:

    The first letter is "c".
    The second letter is "o".
    The third letter is "n".
    The fourth letter is "s".
    The fifth letter is "e".
    The sixth letter is "q".
    The seventh letter is "u".
    The eighth letter is "e".
    The ninth letter is "n".
    The tenth letter is "c".
    The eleventh letter is "e".
    Lastly, the twelfth letter is "s".
    • greatgib 3 years ago

      I don't know the inner working of chatgpt, but my best guess would be that they use something like Levenshtein distance on word to match to the nearest known word to reduce the space of known words and to accommodate to typos.

      When it replied to you, it took the exact extract from the input that is between quotes but internally it is mapped to "consequences" closest word to "consequence," if we consider the comma as any character.

  • benjismith 3 years ago

    Good question! I don't know :)

  • benjismith 3 years ago

    Okay, somebody posted a thread on Twitter explaining how this works...

    The language model is capable of generating python scripts to solve certain text-processing tasks, and then it re-prompts itself by reading the python outputs back into the language model. Very clever!

    https://twitter.com/goodside/status/1598253337400717313

    Other tricks include... prompting itself to lookup wikipedia entries, and then re-prompt itself with snippets from the resulting wikipedia page. Each user prompt is inserted into a template prompt with instructions to the model about the limitations of its capabilities.

    • yesenadam 3 years ago

      Thank you, that's a fascinating thread.

thegeomaster 3 years ago

Very interesting. I was not aware, for example, of the embedding vector lookup. The transformers I have worked with typically used a simple one-hot token representation, but they were domain-specific and not trained on natural language. How are these embeddings trained?

mastersummoner 3 years ago

God love you, your idea of "not very complicated" is absolutely fascinating.

  • benjismith 3 years ago

    Lol, good point!

    I just meant "this isn't related to Thinking Fast and Slow. It's just the tokenizer".

    But yeah, the inner workings of the language model are so complicated as to be almost completely incomprehensible, even after years of study. Touche!

    • mastersummoner 3 years ago

      I suspect it's "not very complicated" for anybody with a decent grounding in the tech. Which is very much not me!