Appearance
Curation is the product: what this wave of dataset releases gets right
The data bottleneck moved
Five years ago, the question was "where do we get enough tokens?" The answer was always: scrape more. This wave of releases says something different. The bottleneck is now the decisions you make about the data, and the datasets that shipped this quarter are collections of decisions, encoded as schemas, QC flags, license fields, and revision history.
I went through seven new Hugging Face releases, spanning pretraining corpora, synthetic agentic traces, egocentric video, voice evaluation, and a refreshed benchmark. What they share is curation made explicit. Provenance isn't a README paragraph anymore; it's a column. Quality isn't a vibe; it's a qc_status field with a list of machine-written reasons.
The download numbers back this up. Wikipedia sits at 276,000 monthly downloads because it solved the access problem cleanly, not because it contains more text than anyone else. ZGCM-1-Data pulled over 31,000 downloads in its first month, and that dataset's headline feature is bookkeeping.
Complete corpora: arxiv sources and wikipedia snapshots
secemp9/arxiv-complete is the most literal version of "give me all the data." It indexes arXiv papers with every source artifact: the main .tex file, each included .eps figure, the compiled PDF per version, even the original PostScript. Every row carries a sha256, a path_encoding, and for compressed files a stored_as pointer so you can un-gzip deterministically. That makes it usable for training on LaTeX, on figures, or for OCR-style work on the PDF stream, not just for text extraction.
The wikimedia/wikipedia dataset makes the opposite move. Instead of raw dumps, it ships cleaned article text, parsed with mwparserfromhell, references and boilerplate sections stripped. You can load any language and date combo by name, so load_dataset("wikimedia/wikipedia", "20231101.en") gives you the English snapshot with 6.4 million articles in a single split.
| arxiv-complete | wikimedia/wikipedia | |
|---|---|---|
| Content | Paper source files, PDFs, PostScript | Cleaned encyclopedia articles |
| Key fields | sha256, path_encoding, stored_as | id, url, title, text |
| Granularity | File-level, per paper version | Article-level, per dump date |
| Languages | Mostly English STEM | 300+ language subsets |
| Typical use | LaTeX and figure training, OCR | Language modeling, retrieval |
Neither is flashy. Both solve a pain that only shows up when you write the parser yourself. arXiv's source tarballs are messy to walk, and Wikipedia dumps are messier. Packaging them into a typed schema is real work, and it's the kind people underrate until they've done it.
Quick Take: the hard problem in this wave of releases is encoding quality decisions into the data itself, so the next team doesn't have to redo them.
ZGCM-1-Data: provenance as a design principle
zgcagi/ZGCM-1-Data is the training collection behind ZGCM-1, a 7.39B-parameter dense model for math and agentic search. 7.39B parameters means the model runs on a single high-end GPU, no cluster required. The dataset runs much larger: 4,191.4B pretraining tokens, roughly 600.9B in long-context midtraining, and an SFT stage of 4.57M samples packed into 19.46B tokens.
The design that stands out is the index_only record pattern. Some sources can't be redistributed. Nemotron-CC needs manual approval. The Stack v2 has per-file licenses. GHArchive-derived code can't be re-hosted. So ZGCM ships locators instead of text. You get the upstream repo, revision, and a SHA-256 of the record, but no model-visible content. If your training run needs that data, you request access upstream, resolve the locator, and pull it yourself. It's a legal and practical middle ground, and it's the first time I've seen it done at this scale with a consistent schema.
Copy the stage split even if you ignore the model. Pretrain stage 1 is a 990.1B-token curriculum; stage 2 continues on the full 3,201.3B-token mixture. Midtrain ramps context from 16K to 64K to 256K, mixing short-prefix data with target-length examples so the model doesn't lose general ability. 256K context means the final model can take an entire codebase or a long research paper in one pass. Every sample keeps a stable source identifier, a normalized source_group, upstream repo and revision, and item-level licenses. You can trace any training example back to its origin and its legal status.
That bookkeeping matters because the collection mixes licenses. The repo-level label is other, and the database license odc-by-1.0 explicitly does not replace item-level terms. Some records fall under CC-BY-NC-4.0, which forbids commercial use. If you train on this, you're the one who has to respect that per record, and the schema gives you the fields to do it. That's the whole point.
| Stage | Config | Size | Purpose |
|---|---|---|---|
| Pretrain | stage 1 | 990.1B tokens | Curriculum learning |
| Pretrain | stage 2 | 3,201.3B tokens | Full mixture, continued |
| Midtrain | 16K | ~180B tokens | Capability training |
| Midtrain | 64K | ~240B tokens | 64K long context |
| Midtrain | 256K | ~180.9B tokens | 256K long context |
| SFT | sft | 4.57M samples, 19.46B tokens | Instruction, reasoning, code, tool use |
Synthetic agentic traces: the value is in the filter
MoreThought/Fable-5.1-Max-Reasoning-Filtered-10000x is a different beast. It's 10,000 multi-turn agentic coding traces generated by Fable 5.1 at max reasoning effort, then deduplicated and heavily filtered. The result: about 500 million tokens of step-by-step chain-of-thought programming, averaging 50K tokens per trace. That's roughly one full working session per example, a scale most SFT sets only reach in aggregate.
The interesting number isn't the total. Generating traces is cheap; deciding which ones teach a smaller model anything is the job. The card lists what survived: algorithm design and optimization, advanced debugging, multi-step logic under complex constraints. If you're distilling agentic behavior into a smaller model, this is the format you want, but you should re-filter for your own target. A trace that's high-quality for one student model may be too verbose for another. My take after browsing the schema: treat the 10K as candidates; filter to what your student model actually needs.
Key numbers10,000 traces in Fable-5.1, about 50K tokens each, 1.97 GB after dedup 4,191.4B pretraining tokens in ZGCM-1-Data, plus ~600.9B midtraining 4.57M SFT samples packed into 19.46B tokens 6.4M articles in the English Wikipedia 20231101 snapshot
Benchmarks need maintenance: TMMLU+ v1.1
ikala/tmmluplus is the reminder that evaluation sets age. TMMLU+ covers 66 subjects in Traditional Chinese, built around Taiwan's educational and professional contexts, from junior high exams to tax law and veterinary pharmacology. It's about six times larger than the original TMMLU.
Version 1.1 is where it gets instructive. Four rule-based scans flagged 6,116 of the 22,742 questions as potentially problematic. Humans reviewed the 691 where verification suggested a change. The verdict: 197 answers were wrong and got corrected, and 539 questions were removed entirely because their stems, options, or answer keys couldn't be defended. That's a 2.4% defect rate in a published benchmark, found only because the team went looking.
| Category | v1.0 test | v1.1 test | Removed |
|---|---|---|---|
| STEM | 3,458 | 3,369 | 89 |
| Social Sciences | 5,958 | 5,864 | 94 |
| Humanities | 1,763 | 1,723 | 40 |
| Other (Business, Health, Misc.) | 8,939 | 8,724 | 215 |
| Total | 20,118 | 19,680 | 438 |
The leaderboard against v1.1 shows how far the top has moved. claude-opus-5 sits at 96.16 average, with every category above 94. At that level the benchmark is close to saturated for frontier models, which makes the remaining variance mostly about question quality. You can still load revision="v1.0", but you shouldn't, because you'd be measuring against questions the maintainers have publicly flagged as broken.
QC as a dataset feature: tracker-pov
eidon-ai/tracker-pov looks like a niche release: egocentric chest-camera video of people doing household tasks. Cooking, cleaning, making the bed, folding laundry, doing dishes, watering plants. The video is genuinely useful for robotics and activity recognition, but the standout is the QC layer.
Every recording carries a qc_status of valid, flagged, or invalid, and the rows that aren't valid include machine-written reasons. "Expected task is cooking but frames show object inspection and unpacking." "Hand presence below 0.22 threshold in the majority of frames." "Majority of frames too dark to discern scene content." The same pipeline computes hand_presence_ratio, stability_score, lighting_score, and average_optical_flow.
| Row | task | qc_status | duration (s) | hand presence | why it failed (if any) |
|---|---|---|---|---|---|
| 1 | cooking | flagged | 11.5 | 0.33 | Unpacking, not cooking |
| 7 | cleaning | invalid | 7.14 | 0.0 | No hands, sorting items |
| 8 | folding laundry | valid | 32.09 | 0.86 | |
| 12 | cooking | valid | 127.99 | 0.81 | |
| 16 | cooking | invalid | 7.52 | 0.0 | Too dark |
| 18 | doing the dishes | valid | 43.16 | 0.89 | |
| 35 | cooking | flagged | 98.43 | 0.95 | Eating, not cooking |
Look at row 35: hand presence at 0.95, good lighting, and the model still flags it because the activity is eating a meal, not preparing one. Recording quality and task match are separate axes, and this pipeline keeps them apart. Most video datasets blur those two things. When I trained on this kind of data before, the failure cases I hit were always activity mismatches, and they're the hardest to catch downstream. Shipping the reasons alongside the video turns debugging from guesswork into a query.
The voice side of multimodal evaluation gets the same treatment in Harland/OmniVChat. Each multi-turn conversation is scored against tiered key points: tier 0 checks the response language matches the user's, tier 1 checks the model directly confirms it can hear the user, tier 2 checks depth and specificity. "The response explicitly confirms the connection is normal" is the kind of criterion that sounds trivial until a model optimizes around it. Structured rubrics as dataset fields make those failures visible.
What trips people up
Treating index_only records as usable training data. They contain locators, not tokens. Your dataloader will silently produce empty examples unless you resolve them against approved upstream copies first. Check release_mode before you build the pipeline.
Ignoring item-level licenses. The repo-level other label on ZGCM-1-Data hides a mix of permissive, CC-BY-NC, and access-gated sources. If you assume "it's on Hugging Face, it's fine," you'll find out during a compliance review, not during training.
Comparing benchmark scores across revisions. TMMLU+ v1.0 and v1.1 are different question sets, and the maintainers showed that about 2.4% of the original had real defects. Record which revision a score came from, and don't mix revisions in a leaderboard.
Using filtered synthetic traces as ground truth. Fable-5.1's filtering is tuned to Fable's own notion of quality. When I distilled traces like these into a smaller model, re-filtering by task type and by the student's failure modes worked far better than taking the top-N by a single quality score.
Trusting QC flags blindly in video data. The tracker-pov pipeline is good, but it's a model. Row 12 is 128 seconds at 0.81 hand presence with a near-gigabyte video file; you still want human audits on the valid set before you train a policy on it.
One thing to remember: every dataset in this release wave ships with quality judgments baked in, as columns, statuses, rubrics, or revision history. Keep those judgments when you build on the data. Stripping them is how datasets turn back into scrapes.
What to do with this wave of data
If you're pretraining or mid-training a model, copy the ZGCM staging pattern: curriculum first, full mixture second, progressive long-context third, with provenance columns on every record. The index_only pattern is the defensible way to use access-gated sources like Nemotron-CC and The Stack v2 without redistributing them; standardize on it early.
If you're distilling agentic coding ability, Fable-5.1's filtered traces are a credible SFT source at a useful scale, 10K sessions and 500M tokens. But re-filter them for your target model and your eval. The upstream filter removed low-quality traces; it didn't remove traces that teach your specific student badly.
If you're evaluating multilingual or agentic models, move to TMMLU+ v1.1 as your Traditional Chinese benchmark and record the revision in every result. For voice and video, adopt the tiered-rubric and QC-reason patterns from OmniVChat and tracker-pov. The next wave of evaluation will be judged on whether failures are explainable, not just on whether scores move.