33 years of authentic human discourse — before SEO, before algorithms, before AI. The Usenet Corpus 1980–2013 is one of the largest privately held pre-LLM text datasets in existence.
Every major Usenet newsgroup hierarchy, processed and validated. From technical discourse that built the internet to cultural conversations that defined an era.
License the full corpus or a targeted subset. Academic and commercial terms available. All licenses include delivery via private HuggingFace repository.
The complete corpus. Every hierarchy, every year, every newsgroup. Maximum coverage for pretraining, fine-tuning, and chronological modeling.
The eight major hierarchies without alt.*. Higher signal-to-noise ratio for research requiring cleaner, more focused training data.
Specific hierarchies, date ranges, or topic areas. Previous subsets include pre-2005 temporal slices and sentiment-focused social group extracts.
Thread-reconstructed version with conversation chain GUIDs, thread position, and depth fields. Rebuilt from original MBOX source files with References headers intact.
Active research and licensing inquiries span AI labs, economics research, computational linguistics, and digital humanities.
Pre-web, human-only discourse with zero AI contamination. Covers language evolution across three decades before engagement optimization existed.
Timestamped at post level across 33 years. Ideal for chronologically consistent models that need text data available only at specific points in time.
Large-scale text analysis of sentiment, attitude, and discourse evolution. Currently in use for generational economic sentiment research at leading universities.
The comp.* and sci.* hierarchies offer dense technical discourse for domain-specific fine-tuning. Early internet computing culture is underrepresented in modern datasets.
A continuous record of public discourse from networked academic communities through mass internet adoption. An irreplaceable window into pre-social-media culture.
Threaded corpus version in development will provide full conversation trees — multi-turn argumentation and debate at scale, from authentic human exchanges.
A two-stage cleaning pipeline applied to raw MBOX archives. 99% of files validated clean with zero email or PII violations.
alt.binaries.* removed entirely — UUencoded binary content with negligible NLP value. Adult content newsgroups excluded. Most binary data removed before record-level processing began.
Posts with identical Message-IDs removed. Cross-file deduplication applied. Usenet cross-posting artifacts eliminated.
MIME header inspection for residual UUencoded and base64 binary content. Records with binary payloads dropped entirely.
Email addresses replaced with [email] token throughout post bodies. Author email addresses stripped. Message-IDs SHA-256 hashed to msg-<hex32>.
MBOX → structured JSONL, gzip-compressed. Language detection on every record using Meta's fasttext LID-176. 96.6% English, 100+ languages total.
Full-corpus validation scan with zero-tolerance checks for email addresses, binary content, and duplicate IDs. 99% of 18,347 files validated clean.
The underlying posts are public Usenet communications originally published in publicly accessible newsgroups. The compiled, processed corpus represents a derivative work produced through substantial original effort — format conversion, deduplication, sanitization, binary removal, PII redaction, language detection, and curatorial decisions about scope and inclusion.
This basis is consistent with standard database and compilation rights frameworks under which publishers, data vendors, and research dataset curators routinely operate.
OwnedByDanes LLC is a registered Florida limited liability company. All licensing agreements are governed by US law.
Academic and commercial licenses available. Custom subsets, temporal slices, and hierarchy-specific extracts on request. Typical response within 2 business days.
todd@ownedbydanes.com