Introducing talkie: a 13B vintage language model from 1930

<p><strong><a href="https://talkie-lm.com/introducing-talkie">Introducing talkie: a 13B vintage language model from 1930</a></strong></p> New project from <a href="https://nlevine.org">Nick Levine</a>, <a href="http://www.cs.toronto.edu/~duvenaud/">David Duvenaud</a>, and <a href="https://en.wikipedia.org/wiki/Alec_Radford">Alec Radford</a> (of GPT, GPT-2, Whisper fame).</p> <p><a href="https://huggingface.co/talkie-lm/talkie-1930-13b-base">talkie-1930-13b-base</a> (53.1 GB) is a "13B language model trained on 260B tokens of historical pre-1931 English text". </p> <p><a href="https://huggingface.co/talkie-lm/talkie-1930-13b-it">talkie-1930-13b-it</a> (26.6 GB) is a checkpoint "finetuned using a novel dataset of instruction-response pairs extracted from pre-1931 reference works", designed to power a chat interface. You can <a href="https://talkie-lm.com/chat">try that out here</a>.</p> <p>Both models are Apache 2.0 licensed. Since the training data for the base model is entirely out of copyright (the USA copyright cutoff date is currently January 1, 1931), I'm hoping they later decide to release the training data as well.</p> <p>Their report suggests some fascinating research objectives for this class of model, including:</p> <ul> <li>How good are these models at predicting the future? "we calculated the surprisingness of short descriptions of historical events to a 13B model trained on pre-1931 text"</li> <li>Can these models invent things that are past their knowledge cutoffs? "As Demis Hassabis has asked, could a model trained up to 1911 independently discover General Relativity, as Einstein did in 1915?"</li> <li>Can they be taught to program? "Figure 3 (left-hand side) shows an early example of such a test, measuring how well models trained on pre-1931 text can, when given a few demonstration examples of <a href="https://github.com/openai/human-eval">Python programs</a>, write new correct programs."</li> </ul> <p>I have a long-running interest in what I call "ve