AI EVALUATION · DATASETS
The test data that makes evaluation mean something.
Import any public benchmark, upload the data you already own, or manufacture a labelled adversarial corpus for a system that has never been attacked before. One registry, every kind of test data.
Most teams can assemble a few hundred question-and-answer pairs from their own logs. Almost nobody has a labelled corpus of jailbreak attempts tailored to their own assistant, their own domain and their own rules. That last capability is the point: Kosmoy generates one.
Four routes bring data in — a HuggingFace import, a file upload, synthetic red-team generation, and seed prompt sets that capture what normal usage of your system looks like. Everything routes through your own gateway, so quota, cost and controls apply exactly as they do everywhere else in Kosmoy.
What it does.
HuggingFace import
The world's largest open dataset repository, one field away. Preview configurations, splits, row count and columns before you commit; large repositories import in the background.
Bring your own files
JSONL, NDJSON, JSON, CSV or Parquet. Validated before storage, with the input and expected-answer columns suggested and split grouping preserved.
Synthetic adversarial generation
Describe what to test; Kosmoy writes the attacks — parallel generation, a refinement pass, rotating techniques, deduplicated and validated row by row.
Grounded in your domain
Seed prompt sets dress every generated attack as your traffic, in your domain — the difference between a useful red-team dataset and generic jailbreak boilerplate.
Labelled for readable reports
Every row tagged with attack surface, technique, harm severity and OWASP category, so results break down by risk and by technique, not one undifferentiated score.
Deterministic, portable, capped
Coverage selection is reproducible and runs before generation, so a budget genuinely reduces spend. Red-team sets export for standard open-source tooling too.
Module questions, answered straight.
Why do datasets come first in AI evaluation?
Every claim you make about an AI system — that it is accurate, stays on topic, cannot be talked into something harmful — is only as credible as the data you tested it against. A demonstration proves a system can work; a dataset proves how it behaves across hundreds of cases, repeatably, after every change you ship. Kosmoy's Datasets section is where that test data comes from.
How do I bring in existing evaluation data?
Two routes. Import any public dataset or benchmark from HuggingFace — type a repository name and Kosmoy previews configurations, splits, row count and detected columns before you commit. Or upload your own files in JSONL, NDJSON, JSON, CSV or Parquet; Kosmoy validates before anything is stored, suggests which column is the input and which is the expected answer, and preserves any split grouping. Nothing is stored until the file passes.
How does Kosmoy generate adversarial red-team data?
You describe what to test; Kosmoy generates the attacks — concrete, ready-to-send adversarial prompts, each labelled with the risk it probes, the technique it uses, its severity and its OWASP category. A guided wizard grounds generation in seed prompts from your own domain, lets you pick from 23 attack surfaces across four families, sets the volume and cost cap, and optionally expands every prompt into disguised variants. This is the capability most teams lack: almost nobody has a labelled corpus of jailbreak attempts tailored to their own assistant.
What attack surfaces and techniques are covered?
23 attack surfaces across four families: ten harmful-content categories, six behavioural rules (language fidelity, prompt injection, system-prompt leak, off-topic adherence, PII disclosure, output-format violation), four indirect/agentic injection surfaces aligned to the OWASP LLM Top 10 (LLM01, LLM02, LLM06), and three over-refusal controls. Attacks are framed with eleven techniques — from roleplay and false authority to payload splitting and many-shot — and can be disguised with around thirty obfuscation techniques across encoding, transformation and model-driven rewriting.
What data types can a dataset hold?
One registry holds them all, each typed so the platform knows how to evaluate it: question-and-answer, chat and conversation, tool use, retrieval, instruction, summarization, translation, classification, preference, guardrail, red teaming and seed prompts. A public benchmark, a spreadsheet of agent traces exported from production and a generated adversarial corpus sit side by side — each previewable, exportable and reusable across as many evaluations as you like.
Is dataset generation reproducible for audits?
Yes. Coverage selection is deterministic — the same configuration always produces the same dataset, which matters when a regulator asks you to reproduce a result — and it runs before generation, so a cost cap genuinely reduces spend rather than discarding work already paid for. Red-team datasets also export in the format expected by standard open-source red-teaming tooling, so the corpus you generate in Kosmoy is not locked inside it.
Generate an adversarial dataset for your assistant.
Describe what it must never do; leave with a labelled corpus you can run today and re-run after every change.