Historical generative models
Language models that generate text and are pretrained on historical sources, with the documented cutoff of each release.
Daniel Göttlich, Dominik Loibner, Guohui Jiang and Hans-Joachim Voth
A research project announcing Ranke-4B, a family of 4B-parameter models trained on historical books and newspapers with knowledge cutoffs between 1913 and 1946. When reviewed, the weights, data, code and working paper were described as forthcoming.
- Method
- Historical pretraining from scratch (announced)
- Period
- Planned cutoffs: 1913, 1929, 1933, 1939 and 1946
- Size
- 4B
Resources: Project page
Technical details for History LLMs (Ranke-4B)
- Announced, not released: no public checkpoints were found for any cutoff.
- Pre-release notes describe a Qwen3-architecture model trained from scratch on about 80B tokens, with one checkpoint trained on pre-1900 text and then continued on later text for each cutoff.
- Licence: Not documented.
Xiaoxi Luo, Zachary Shinnick, Niclas Griesshaber, Yixuan Wang, Junchi Yu, Freda Shi, Philip Torr and Yao Lu
An academic study of whether a model pretrained only on English text from 1700 to 1913 can be both capable and grounded in its period. The paper also introduces historically grounded instruction data and a benchmark for checking the knowledge cutoff.
- Method
- Historical pretraining from scratch
- Period
- Text from 1700–1913; 1913 cutoff
- Language
- English
- Size
- 7B (7.24B per the paper)
Resources: Paper · Weights
Technical details for TypewriterLM
- Paper title: A Language Model from 1913: Pretraining on Historical Text (accepted at EMNLP 2026, per arXiv).
- Base model: typewriter-1913-7B-base, trained on TypewriterCorpus, about 54B tokens, mostly from Institutional Books.
- Instruction tuning: History-LIMA responses are grounded in pre-1913 text, with instructions written by Claude Opus and reviewed by hand. History-SelfInstruct (287,860 pairs) pairs historical passages with instructions generated by the base model. See History-Event and the datasets under Research perspectives.
- Further checkpoints with 1938 in their names are on Hugging Face but are not described in the paper.
- Model and dataset licences: Not documented. The corpus and datasets were not public on Hugging Face when reviewed.
Nick Levine, David Duvenaud and Alec Radford
A 13B model trained on about 260B tokens of mostly English text published before 1931, released as base and chat checkpoints. A twin with the same architecture trained on modern web text is released for comparison.
- Method
- Historical pretraining from scratch
- Period
- Text published before 1931
- Language
- Primarily English
- Size
- 13B
Resources: Blog post · Weights (base) · Weights (chat) · Code · Demo
Technical details for talkie
- talkie-1930-13b-base: base model trained on books, newspapers, periodicals, journals, patents and case law.
- talkie-1930-13b-it: chat model. Post-training uses instruction pairs built from historical texts, then online DPO with Claude Sonnet 4.6 as judge, and a final stage on synthetic chats with Claude Opus 4.6. The authors note this AI feedback can introduce anachronisms.
- talkie-web-13b-base: a modern-web comparison model, not a historical model.
- Licence stated: Apache 2.0.
Trip Venturella
A family of Llama-style models trained from scratch on British print from 1800 to 1899, from 345M to 2.7B parameters, with a 2.7B chat model. Most of the chat model's training replies are verbatim period text.
- Method
- Historical pretraining from scratch; chat model fine-tuned
- Period
- British texts published 1800–1899
- Language
- English
- Size
- 345M, 500M, 1.2B and 2.7B base; 2.7B chat
Resources: Blog post · Weights · Chat model · Dataset (chat tuning) · Code · Demo
Technical details for Mr. Chatterbox
- Base releases: Mr.Chatterbox-345M, -500M, -1.2B and -2.7B (the 2.7B on about 54B tokens). An earlier small model was released separately.
- Chat release: Mr.Chatterbox-2.7B-Chat. The project documents that some user prompts in the tuning data were written or paraphrased by modern LLMs, and that modern models were used for judging and tagging.
- Licence stated: weights CC BY-NC-SA 4.0 (changed from MIT in October 2026); code MIT.
zakarth on Hugging Face
GPT-NeoX models of about 1.4B and 160M parameters trained on texts from 1800 to 1899, each released as a base model and a chat model. The chat models are tuned to play a fictional persona, Miss Violet Hartwell of Kensington in 1899.
- Method
- Historical pretraining; chat models persona-tuned
- Period
- 1800–1899
- Language
- English
- Size
- About 1.4B and 160M
Resources: Weights · Demo
Technical details for Violet
- Base models: violet-1b4 and violet-160m. The cards do not explicitly say training started from scratch. The 160M card says its pretraining corpus includes some synthetic data.
- Chat models: violet-1b4-chat and violet-160m-chat, fine-tuned with synthetic data to build the chat format. The generator of the synthetic data is not documented.
- Licence stated: CC0 1.0 for weights and code.
Michael Hla
An experiment testing whether a model trained only on text from before 1900 can reach ideas from later physics. It includes general base and chat models and separate physics-focused experiments.
- Method
- Historical pretraining from scratch
- Period
- Text before 1900, with comparison models to 1905 and 1900–1964
- Language
- English
- Size
- 3.3B
Resources: Blog post · Code · Weights
Technical details for GPT-1900 (Machina Mirabilis)
- General models: a nanochat-based base model trained on about 22B tokens, and a chat model tuned on synthetic conversations generated from pretraining excerpts by a modern LLM (not named).
- Physics experiments: further training on pre-1900 physics texts and reinforcement learning with Claude Sonnet 4 as judge.
- Licence stated: MIT for the repository and most model cards.
Cristi Constantin (croqaz)
A hobby project training small Llama-architecture base models on texts written before 1900. Later models add synthetic rewrites of period text to the training mix.
- Method
- Historical pretraining from scratch
- Period
- Texts written before 1900
- Language
- English
- Size
- 75M to 340M
Resources: Code · Weights (340M) · Weights (Piston-and-Prose) · Blog post · Demo
Technical details for Vintage-LLM
- Base releases: vintage-LLM-340m-v1-base, Sprocket-and-Say (77M), Piston-and-Prose-sm (75M) and -lg (141M).
- Piston-and-Prose-lg's pretraining mix includes period texts rewritten by Gemma and completions generated by TypewriterLM and talkie.
- A chat model is planned but not released.
- Licences stated: MIT for the repository and 340M model; Apache 2.0 for Sprocket and Piston models.
Unbounded Labs
A 2.8B nanochat-based model trained on English public-domain books published before 1931, with additional training on pre-1930 scientific text, and a chat variant.
- Method
- Historical pretraining from scratch; chat variant fine-tuned
- Period
- 1930 cutoff
- Language
- English
- Size
- 2.8B
Resources: Blog post · Weights (base) · Weights (chat) · Code
Technical details for Bartholomew
- Base: trained on about 20B tokens from Institutional Books, with about 600M tokens of pre-1930 STEM text added in a later stage.
- Chat (bartholomew-sft): tuned on about 416,000 synthetic rows generated from corpus excerpts (the blog says DeepSeek was used), plus about 12,000 rows of authentic period question-and-answer text.
- Evaluated with Vintage CORE (see Research perspectives).
- Licence stated: MIT.
Historical adaptation and style transfer
Projects that start from modern pretrained models and adapt them to historical text or style. They retain knowledge from their modern pretraining.
Pierre-Carl Langlais
A chat model fine-tuned to answer in the style of seventeenth-century texts. It is a fine-tune of the modern OpenHermes-2-Mistral-7B model, so its knowledge comes from modern pretrained weights.
- Method
- Fine-tuning of a modern pretrained model
- Period
- Seventeenth-century texts, mostly from EEBO and Gallica
- Language
- English, French, Latin
- Size
- 7B
Resources: Weights · Dataset · Demo
Technical details for MonadGPT
- Base weights: teknium/OpenHermes-2-Mistral-7B, itself built on Mistral-7B-v0.1.
- Fine-tuned on about 11,000 early modern texts in ChatML format.
- Licence stated on the model card: Apache 2.0.
Tianyi Qiu, Yang Zhang, Xuchuan Huang, Jasmine Xinze Li, Jiaming Ji and Yaodong Yang (PKU-Alignment)
A framework for studying how moral values change over time, which its authors call progress alignment. Its HistLlama3 models are Meta Llama 3 models given continued pretraining on historical text, one model per century from the thirteenth to the twenty-first.
- Method
- Continued pretraining of a modern model
- Period
- Thirteenth to twenty-first centuries, one model per century
- Size
- 8B and 70B
Resources: Paper · Code · Weights · Dataset · Leaderboard · Project page
Technical details for ProgressGym and HistLlama3
- Base weights: Meta-Llama-3-8B and Meta-Llama-3-70B.
- Corpus of about 38 GB from the Internet Archive and Library of Congress, Project Gutenberg, EEBO and Pile of Law.
- Pretrain and instruct variants are released for each century. The instruction data (ProgressGym-TimelessQA, about 3,000 pairs drawn from LIMA, Dolly-15k and Alpaca) was selected with GPT-4-based filtering.
- Licence stated in the README: CC BY 4.0 for code, data and models.
zakarth on Hugging Face
A style-transfer adapter that rewrites modern English into Victorian register while keeping its content, intended to help build period-consistent text. It is an adapter on the modern Qwen3-8B model, not a historically trained base model.
- Method
- Style transfer (adapter on a modern model)
- Period
- Target style from PG19 books published 1837–1901
- Language
- English
- Size
- Adapter on Qwen3-8B
Resources: Weights
Technical details for Patina
- QLoRA adapter (rank 32), about 87M trainable parameters on Qwen/Qwen3-8B.
- Trained on about 59,000 pairs matching real Victorian paragraphs with modernised versions. The modern versions were generated by DeepSeek, and another LLM was used to check facts when selecting outputs.
- Licence stated on the model card: Apache 2.0.
Time-bounded and temporally conditioned models
Models trained only on text up to a given date, or conditioned on time, mostly to study forecasting, finance or language change without look-ahead from later text.
Suproteem Sarkar
A family of encoder models with yearly knowledge cutoffs, for research that needs language models limited to the text available at a given time. Each yearly model continues training from the previous year's model.
- Method
- Time-bounded pretraining (encoder, one model per year)
- Period
- v1: 1900–1963, American newspapers. v2: text from 1750–2024, with one model per year from 1900
- Language
- English
Resources: Paper (SSRN) · Weights (v1) · Project page · Dataset (American Stories)
Technical details for StoriesLM
- v1: BERT fill-mask models for each year from 1900 to 1963, trained on the American Stories newspaper dataset. Licence stated: MIT.
- v2: ModernBERT-base models. The first is trained from scratch on 1750–1900 text, then one model per year to 2024, using newspapers, public-domain books, the Congressional Record, newswire and arXiv text. Licence: Not documented.
Songrun He, Linying Lv, Asaf Manela and Jimmy Wu
Series of encoder and decoder models trained from scratch on text available up to the end of each year, for finance and economics research that must avoid look-ahead bias.
- Method
- Time-bounded pretraining
- Period
- Yearly cutoffs, 1999–2024
- Language
- English
- Size
- ChronoBERT 149M; ChronoGPT about 1.5B
Resources: Paper · Weights · Instruction dataset
Technical details for ChronoBERT and ChronoGPT
- Paper title: Chronologically Consistent Large Language Models.
- Base models are released for each year. Instruction-tuned ChronoGPT versions use modern instruction datasets (including Tulu 3 and Self-Instruct) filtered to pre-2000 content with a GPT-4.1 classifier, described in a separate working paper.
Yutong Yan, Raphael Tang, Zhenyu Gao, Wenxi Jiang and Yao Lu
Twelve models trained from scratch, each on data before an annual cutoff from 2013 to 2024, with instruction-tuned versions. The paper evaluates them on financial forecasting where later information would bias results.
- Method
- Time-bounded pretraining
- Period
- Annual cutoffs, 2013–2024
- Language
- English
- Size
- 1.3B (per the paper)
Resources: Paper · Weights · Demo
Technical details for DatedGPT
- Paper title: DatedGPT: Preventing Lookahead Bias in Large Language Models with Time-Aware Pretraining.
- Base and instruct checkpoints are on Hugging Face. Weights licence: Not documented. How the instruction data was built: Not documented on the model cards.
Bryan T. Kelly, Semyon Malamud, Johannes Schwab and Teng Andrea Xu
Decoder models up to 4B parameters trained on chronologically filtered web text, with point-in-time checkpoints for backtests in finance and social science. The paper also evaluates portfolios built from the models' embeddings.
- Method
- Time-bounded pretraining
- Period
- Checkpoints from 2013 to 2024
- Language
- English
- Size
- Up to 4B
Resources: Paper (arXiv) · Paper (NBER)
Technical details for Scaling Point-in-Time Language Models
- arXiv title: Scaling Point-in-Time Language Models: Enabling Economic Evaluation of Embeddings.
- Trained on about 1 trillion tokens of FineWeb filtered by date.
Brandon Duderstadt and Hayden Helm (Calcifer Computing)
An encoder that predicts both a document's text and its year, used to study trends in American English from 1990 to 2019. It is initialised from the modern Etin400M encoder, so it is conditioned on time rather than limited to historical text.
- Method
- Temporal conditioning (fine-tuned modern encoder)
- Period
- 1990–2019
- Language
- American English
Resources: Report
Technical details for TLM-1: A Model of the Language Process
- Trained on about 750 million words of the Corpus of Contemporary American English, with a year token added to each sequence.
- Weights, code and licence: Not documented.
Junjie Dong, Zhuoqi Lyu and Qing Ke
A series of encoder models for tracking how scientific writing on arXiv changes over time. A base model trained from scratch on papers up to 2008 is followed by models trained further year by year.
- Method
- Time-bounded pretraining (encoder, yearly continued training)
- Period
- Base on arXiv papers up to 2008, then yearly
- Language
- English
Resources: Paper · Weights
Technical details for AnnualBERT
- Paper title: Towards understanding evolution of science through language model series.
- RoBERTa architecture with whole-word tokens, trained on about 1.7 million arXiv papers.
- Model sizes and licence: Not documented.
Historical encoders and multilingual research
Encoder models and related work for analysing historical text in many languages. They are tools for classification, tagging and retrieval, not generative models with strict knowledge cutoffs.
Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza and Mariona Coll Ardanuy (Living with Machines)
Word2vec, fastText, BERT and Flair models trained on about 5.1B tokens of English books published from 1760 to 1900, released for historical research. The BERT models start from modern BERT base and are trained further on historical time slices.
- Method
- Mixed: static embeddings trained on historical text; BERT continued from a modern model
- Period
- Books published 1760–1900, with time-slice variants
- Language
- English
Resources: Code · Paper · Weights (Zenodo) · Weights (Hugging Face)
Technical details for histLM (Living with Machines)
- BERT time slices: 1760–1900, 1760–1850, 1850–1875, 1875–1890 and 1890–1900.
- Licences stated: code MIT; models CC BY 4.0.
Enrique Manjavacas Arevalo and Lauren Fonteyn
BERT-base models pretrained on historical English (MacBERTh, 1450–1950) and historical Dutch (GysBERT, 1500–1950) for humanities research.
- Method
- Encoder pretraining on historical text
- Period
- 1450–1950 (English); 1500–1950 (Dutch)
- Language
- English, Dutch
Resources: Project page · Paper (MacBERTh) · Paper (GysBERT) · Weights (MacBERTh) · Weights (GysBERT)
Technical details for MacBERTh and GysBERT
- MacBERTh: trained from scratch on about 3.9B tokens from EEBO, ECCO, COHA, CLMET3.1, Evans and Hansard.
- GysBERT: trained on DBNL and Delpher newspapers; whether it was trained from scratch is not stated on its model card.
- Licence stated on both model cards: MIT.
Wenjun Qiu and Yang Xu
A BERT model for studying how word meanings change over time, built by continuing bert-base-uncased on the Corpus of Historical American English.
- Method
- Continued pretraining of a modern encoder
- Period
- Twentieth century (1910s–2000s for the main variant)
- Language
- American English
Resources: Paper · Code
Technical details for HistBERT
- Model and code licence: Not documented.
Stefan Schweter, Luisa März, Katharina Schmid and Erion Çano
Multilingual BERT models pretrained on historical newspapers and books in five European languages, developed for named entity recognition in historical text.
- Method
- Encoder pretraining on historical text
- Language
- German, French, English, Finnish, Swedish
- Size
- About 110M and 135M, plus smaller variants
Resources: Paper · Weights
Technical details for hmBERT
- Pretrained on 130 GB of OCR text with a new historical vocabulary; no starting checkpoint is named.
- English data: British Library books, 1800–1900.
- Licence stated: MIT.
Christin Beck and Marisa Köllner
A BERT model trained from scratch on German texts from about 750 to 1999 CE, from Old High German onward, for research on lexical semantic change.
- Method
- Encoder pretraining from scratch on historical text
- Period
- About 750–1999 CE
- Language
- Historical German
Resources: Paper · Weights · Code
Technical details for GHisBERT
- Licences stated: model MIT; code GPL-3.0.
FreEM project (led by Simon Gabay)
Corpora and tools for Early Modern French, including D'AlemBERT, a RoBERTa-base model pretrained on the FreEMmax corpus for tasks such as part-of-speech tagging.
- Method
- Encoder pretraining on historical text
- Period
- Early Modern French
- Language
- French
Resources: Project page · Weights (D'AlemBERT)
Technical details for FreEM and D'AlemBERT
- Whether D'AlemBERT was trained from scratch or continued from an existing model: Not documented on the pages reviewed.
- Licence stated on the model card: Apache 2.0.
Tomás Freitas Osório and Henrique Lopes Cardoso
An encoder for historical Portuguese, built by continuing the pretraining of the modern Albertina model on eighteen corpora from the twelfth to the twentieth centuries, with variants for individual periods.
- Method
- Continued pretraining of a modern encoder
- Period
- Twelfth to twentieth centuries
- Language
- Portuguese
- Size
- About 100M
Resources: Paper · Weights · Dataset
Technical details for PortOldBERT
- Base weights: Albertina PT-PT 100M.
- The paper states a CC BY-NC 4.0 licence; the model card lists none.
David Bamman and Patrick J. Burns
A BERT model for Latin trained on about 643M tokens from classical to modern sources, evaluated on part-of-speech tagging, word sense disambiguation and semantic search.
- Method
- Encoder pretraining on Latin text
- Period
- Classical era to the twenty-first century
- Language
- Latin
Resources: Code and weights · Paper
Technical details for Latin BERT
- Uses a Latin-specific tokenizer; no starting checkpoint is named.
- Licence stated: MIT.
Wang Dongbo, Liu Chang, Zhu Zihe and colleagues
BERT and RoBERTa models trained further on the full text of the Siku Quanshu, the Qing-dynasty collection, for processing classical Chinese in digital humanities.
- Method
- Continued pretraining of modern encoders
- Language
- Classical Chinese
Resources: Code · Weights (SikuBERT) · Weights (SikuRoBERTa)
Technical details for SikuBERT and SikuRoBERTa
- Base weights: Chinese BERT and RoBERTa-wwm-ext.
- Licence stated: Apache 2.0.
Ivan P. Yamshchikov, Alexey Tikhonov, Yorgos Pantis, Charlotte Schubert and Jürgen Jost
An Ancient Greek BERT, adapted from a Modern Greek BERT, used to study the authorship of texts attributed to Plutarch. Published at EMNLP 2022.
- Method
- Continued pretraining of a modern encoder
- Language
- Ancient Greek
Resources: Paper · Weights
Technical details for BERT in Plutarch's Shadows
- Base weights: nlpaueb/bert-base-greek-uncased-v1.
- Licence stated on the model card: MIT.
David Mezzetti (NeuML)
A small BERT model trained from scratch on English books published from 1700 to 1899, released with a sentence-embeddings variant.
- Method
- Encoder pretraining from scratch on historical text
- Period
- Books from 1700–1899
- Language
- English
- Size
- 22.7M
Resources: Blog post · Weights · Dataset
Technical details for H.G. BERT Small
- Licence stated: Apache 2.0.
Stefan Schweter
Byte-level ByT5 models pretrained on historical newspapers and books in several European languages. Some training runs continue from the original ByT5 checkpoint.
- Method
- Mixed: continued pretraining of ByT5 for some runs
- Language
- English, German, French, Finnish, Swedish, Dutch, Norwegian
Resources: Code · Weights
Technical details for hmByT5
- The English run starts from ByT5 Small; German and French runs continue from the English model.
- Licence stated: MIT.
Emanuela Boros
An experiment adding lightweight period information to a French Europeana BERT, trained on public-domain French newspapers and tested on masked-language modelling and named entity recognition.
- Method
- Adaptation of a modern encoder with temporal adapters
- Period
- Five periods from before 1850 to after 1945
- Language
- French
Resources: Code · Weights
Technical details for Long-Horizon Historical BERT
- Base weights: dbmdz/bert-base-french-europeana-cased.
- Licence stated: MIT for the repository and v1 model; v2 model card lists none.
Research perspectives and evaluation
Talks, papers and benchmarks about what historical and time-bounded models can show and how to evaluate them.
Owain Evans
A talk on language models trained only on data up to a chosen date, and their uses for forecasting, testing whether ideas can be rediscovered, and humanistic study. It discusses obstacles including limited data, contamination from later text and cost.
- Method
- Research perspective (talk)
Resources: Transcript · Video
Ted Underwood, Laura K. Nelson and Matthew Wilkens
Tests whether modern models can simulate historical writing. Prompting with period prose did not produce period-consistent style, and fine-tuning fooled an automated judge but not human readers. The authors tentatively suggest pretraining on period text may be needed.
- Method
- Research and evaluation
Resources: Paper
Ted Underwood, Ziliang Qiu, Sarah Griebel, Laura K. Nelson, Edwin Roland, Wenyi Shang and Matthew Wilkens
A benchmark of 866 questions drawn from 187 period sources, testing whether language models can respond like English-language writers in specific contexts from 1831 to 1930.
- Method
- Benchmark
- Period
- 1831–1930
- Language
- English
Resources: Paper · Code and sample · Hugging Face
Technical details for Chronologic
- Paper title: Chronologic: Measuring Language Models' Ability to Represent the Past. Benchmark release: Chronologic-EN-1.0.
- The full question set is on Hugging Face under controlled access. The GitHub repository includes a 100-question public sample and MIT-licensed code.
From the TypewriterLM paper
Resources described in the TypewriterLM paper (see the TypewriterLM entry above). History-Event is an evaluation benchmark of historical events used to test knowledge and cutoff adherence. History-LIMA and History-SelfInstruct are instruction-tuning datasets.
- Method
- Benchmark and instruction datasets
- Language
- English
Resources: Paper · TypewriterLM entry
Technical details for History-Event, History-LIMA and History-SelfInstruct
- History-Event: 2,344 events drawn from Wikipedia century timelines, 1700–2025. History-LIMA: 1,030 examples. History-SelfInstruct: 287,860 instruction–response pairs.
- The paper says datasets are at hf.co/typewriter-ai; when reviewed, that organisation listed models but no public datasets.
Jonathan Duran-Ortiz (Unbounded Labs)
A version of the DataComp-LM CORE benchmark adapted for models with a 1930 knowledge cutoff. It was introduced with the Bartholomew model (see above) and has two 20-task bundles: one with post-1930 knowledge filtered out and one rewritten in period style.
- Method
- Benchmark
- Period
- 1930 cutoff
- Language
- English
Resources: Code · Dataset · Blog post
Technical details for Vintage CORE
- The Bartholomew post also reports a History-Event test using an independent reconstruction (jbduran/history-event-reconstruction), which its card says is not the TypewriterLM authors' dataset.
- Code licence: MIT. Benchmark rows keep their original third-party terms (Hugging Face licence listed as "other").
Michael E. W. Varnum, Nicolas Baumard, Mohammad Atari and Kurt Gray
Proposes historical large language models, trained on texts from past societies, as a way to study psychology across time, and discusses limits such as small corpora, missing benchmarks and over-representation of elite writers. Published in PNAS, 2024.
- Method
- Research perspective
Resources: Paper (PNAS) · Full text (PMC)
Guy D. Rosin, Ido Guy and Kira Radinsky
Adds time to text as extra context and masks it during training, then evaluates the resulting model on detecting semantic change and predicting when a sentence was written. Published at WSDM 2022.
- Method
- Temporal conditioning (research method)
Resources: Paper
Kai Nylund, Suchin Gururangan and Noah A. Smith
Introduces time vectors: the change in a model's weights after fine-tuning on text from one period. Vectors for neighbouring periods lie close together, and interpolating between them adapts a model to intervening periods.
- Method
- Research method (temporal adaptation)
Resources: Paper