Historical AI training data · est. 1980
Thirty-three years of authentic human conversation, captured before the web, before search rankings, before machine-written text. The Usenet Corpus is one of the largest privately held pre-LLM datasets in existence — cleaned, threaded, and verified.
405.8M posts · 18,335 newsgroups · 0 AI-generated tokens
* pre-omission figure — distinct-post count not yet recomputed for the as-distributed corpus; see Provenance
The holdings
Every major Usenet hierarchy, processed and validated — from the technical discourse that built the internet to the cultural argument that defined it.
Row-level figures reflect the corpus as processed, before the 12-newsgroup omission disclosed under Provenance (affects ALT, COMP, REC, and NEWS by ~0.55% of posts corpus-wide). The distributed total below is what's actually licensed.
| Hierarchy | Posts | Tokens | Character |
|---|---|---|---|
| alt.* | 228.6M | 58.8B | The largest, wildest hierarchy — culture, philosophy, hobbies, everything. |
| rec.* | 77.2M | 16.5B | Recreation, arts, sport. 99.7% English, exceptionally clean signal. |
| comp.* | 50.0M | 10.3B | Technical computing from the people who built the net. Coverage from 1980. |
| soc.* | 22.4M | 8.2B | Society and culture. The most multilingual hierarchy — 100+ languages. |
| sci.* | 12.0M | 3.3B | Dense scientific debate — physics, biology, mathematics, and more. |
| misc.* | 10.9M | 2.7B | Jobs, health, everyday life. High relevance for social-science research. |
| talk.* | 4.3M | 1.3B | Debate and argument — politics, religion, philosophy. Adversarial discourse. |
| news.* | 2.1M | 1.7B | Meta-discussion and FAQs. Highest average post length in the corpus. |
| humanities.* | 0.6M | 0.2B | Literature, philosophy, classics. Small but 99.6% English, high signal. |
| Processed total | 408M | 103.1B | pre-omission |
| Distributed total | 405.8M | 102.5B | as licensed — 18,335 newsgroups |
Record schema
Every post is one line of gzipped JSON Lines. The Threaded edition adds three fields; the cleaned corpus carries the first six. Nothing else differs between them.
{
"id": "msg-4c9f1a\u2026e07b",
"group": "comp.lang.c",
"date": "14 Mar 1991 09:22:41 GMT",
"author": "ptrwizard",
"subject": "Re: pointers vs. arrays",
"text": "Not quite. An array name decays\u2026",
"thread_id": "8f3c\u2026d2",
"thread_position": 4,
"thread_depth": 2
}| id | string | Stable per-post id — msg- + SHA-256 of the original Message-ID. |
| group | string | Newsgroup the post was archived under. |
| date | string | Original post date (RFC 2822). |
| author | string | Display name as posted. Email addresses removed; where the header held only an address (about 8% of posts) the value is [redacted]. |
| subject | string | Subject line. |
| text | string | Post body. |
| thread_id | string | Conversation identifier. Threaded edition. |
| thread_position | int | 0-based order within the thread. Threaded edition. |
| thread_depth | int | Reply depth from the thread root (root = 0). Threaded edition. |
Delivered as gzipped JSON Lines (.jsonl.gz), one file per newsgroup, per hierarchy directory.
Licensing
License the complete corpus, the thread-reconstructed edition, or a targeted extract. Academic and commercial terms available; every license is delivered through a private HuggingFace repository.
The complete cleaned archive as distributed — every hierarchy, every year, every newsgroup that passes current malware scanning. Maximum coverage for pretraining, fine-tuning, and chronological modeling.
The full corpus rebuilt into conversation trees from original References headers. Every post carries its place in the discussion — built for dialogue, reasoning, and multi-turn training.
The eight managed hierarchies without alt.* for a higher signal-to-noise ratio — or any custom slice: specific groups, temporal windows, topic areas. Token-count verified before delivery.
Browsable showcase excerpts of both corpora on HuggingFace — complete reconstructed conversations you can read and evaluate today, before you ever send an inquiry.
Applications
Active inquiries span frontier AI labs, computational linguistics, economics, and the digital humanities.
Pre-web, human-only discourse with zero AI contamination — language as it evolved across three decades before engagement optimization existed.
Post-level timestamps across 33 years, ideal for models that must reflect only what was knowable at a given point in time.
The threaded edition provides full conversation trees — 190.8M multi-turn arguments and debates at scale, from authentic human exchanges.
Large-scale analysis of sentiment and discourse evolution — already in use for generational economic-sentiment work at leading universities.
The comp.* and sci.* hierarchies offer dense technical discourse and early-internet computing culture underrepresented in modern datasets.
A continuous record of public discourse from networked academia through mass internet adoption — an irreplaceable pre-social-media window.
Provenance & data quality
A deterministic, fully logged pipeline from raw MBOX archives to delivered corpus. Every transformation is documented; every claim is verifiable.
alt.binaries.* and adult groups removed entirely before record-level work — negligible NLP value, high noise.
MIME inspection for residual UUencoded and base64 payloads; records carrying binary content dropped.
Email addresses replaced with an [email] token in post bodies and removed from author names (a bare-address sender reads [redacted]); Message-IDs replaced with SHA-256 hashes.
Every record tagged with fastText LID-176 — 96.6% English across 100+ languages, so subsets can be filtered by language.
Conversation trees rebuilt from References headers into 190.8M threads with per-post position and depth.
Duplication measured, not hidden: ~15.8% of posts are cross-posted, retained and labeled by shared id rather than stripped.
A two-stage ClamAV pass — per-post signature scanning plus whole-file span localization for multi-part payloads — removed every signature match, concentrated in the security and warez groups. The delivered corpus passes a whole-file ClamAV scan with zero detections, and every removal is logged. No raw Usenet dump can say the same.
Twelve newsgroups (2.24M posts / 0.58B tokens corpus-wide, ~0.55% of the processed total) are withheld from both releases. Their content — security and hacking discussion, net-abuse reports, and a JavaScript programming group — matches an outdated antivirus signature; the identical files scan clean under current ClamAV definitions. They're withheld so the distributed dataset passes automated scanning without exception, not because the content itself failed review. Affected groups: alt.games.tombraider, alt.games.mame, alt.hacker, alt.support.depression.medication, alt.quotations, comp.lang.javascript, comp.security.misc, rec.skydiving, rec.radio.amateur.equipment, rec.arts.comics.dc.universe, rec.music.rock-pop-r+b.1970s, news.admin.net-abuse.sightings. Full detail is in each data card.
On rights. The underlying posts are public Usenet communications, originally published in openly accessible newsgroups. The compiled corpus is a derivative work produced through substantial original effort — format conversion, deduplication measurement, sanitization, binary and malware removal, PII redaction, language detection, thread reconstruction, and curatorial decisions about scope.
This basis is consistent with the database and compilation-rights frameworks under which data vendors and research-dataset curators routinely operate. OwnedByDanes LLC is a registered Florida limited liability company; licensing agreements are governed by U.S. law.
Get in touch
Custom subsets, temporal slices, and hierarchy-specific extracts on request. Typical response within two business days.
◆ In active use for generational economic-sentiment research at a leading U.S. university.
Pretraining, fine-tuning, evaluation, and retrieval for AI products. Full corpus or targeted subsets.
Commercial inquiryGrant-friendly terms for universities and non-commercial research. Reproducible, per-hierarchy access.
Academic inquiry