Trust & Safety Solutions

Udit Khanna

Toxicity and bias annotation is the human labeling work that makes AI safety measurable. Toxicity annotation assigns structured labels to content, identifying whether it contains harmful content such as hate speech, harassment, threats, or demeaning language, the severity, and the intended recipient. 

Bias annotation labels the subtler layer: stereotyping, demographic skew, and differences in how content treats or represents groups. Together they produce the labeled datasets that safety systems are built from: the filters that screen training corpora, the reward signals that teach models what not to generate, the classifiers that moderate outputs, and the benchmarks that measure whether any of it worked.

This is written for the ML engineer building the labeling pipeline, the safety lead who owns the taxonomy and the risk tradeoffs, and the buyer deciding whether to build this capability internally or bring in a partner. 

This blog explains what toxicity and bias annotation actually involve, why the labeling is harder than it looks, what responsible programs owe the annotators who do this work, and how the resulting data flows through every layer of a safety program. 

Key Takeaways

• Safety data compounds upstream. The same annotation investment buys more safety at training-data curation than at output moderation, because models reproduce what they learned. Programs that start labeling at deployment are paying retail for what was available wholesale.
• Toxicity is not one label. Production-grade annotation uses a taxonomy: harm type, severity, target, and context, because a filter trained on a single toxic-or-not bit cannot distinguish a slur from a news report quoting one, and will fail in both directions.
• Context and identity are part of the signal, not noise. The same words can be attack, reclamation, quotation, or counter-speech, and annotators from different communities can judge the same content differently for legitimate reasons. Mature programs capture and use that disagreement rather than averaging it away.
• Annotator welfare is a design requirement. Toxicity annotation exposes people to harmful content by definition. Exposure limits, rotation, opt-outs, and support are ethical obligations that also protect label quality because distressed annotators drift.
• Bias examination is becoming a documented legal obligation, not a best practice, with the EU AI Act’s Article 10 requiring it for high-risk systems’ training, validation, and testing data.

What the Annotation Actually Produces

The Toxicity Taxonomy

A production toxicity schema labels along several axes at once. Harm type distinguishes hate speech, harassment, threats, and incitement, sexual content, self-harm content, and graphic violence, because downstream systems treat these differently. Severity grades within type, since a moderation policy that handles mild insult and explicit threat identically will be wrong for one of them. Target records who the content is directed at, including whether a protected characteristic is implicated. Context flags capture the uses that flip meaning: quotation and reporting, condemnation and counter-speech, in-group reclamation, fiction, and education. The output of this schema is not a verdict but a structured description, which is what lets one labeled dataset serve multiple policies with different thresholds.

Worked example:

Sample content: “You people always cause trouble around here.”

Harm type: harassment, group-directed hostility rather than a threat, sexual content, or self-harm content.

Severity: moderate. No explicit slur or threat is present, but the phrasing generalizes hostility to a group, which most policies grade above a simple insult.

Target: an unspecified ethnic or social group, implied by “you people” rather than named. The label records that a group is targeted even though the annotator cannot identify which one from this sentence alone.

Context: none of the mitigating flags apply. It is not a quotation, not condemnation or counter-speech, not in-group reclamation, and not fiction or education. Context does not soften the harm-type and severity labels here.

Four labels, one sentence, and a structured description rather than a verdict: a policy that only screens for slurs would miss this sentence entirely, while a policy that treats any group reference as toxic would over-flag ordinary text. The axes let each policy set its own threshold against the same labeled data.

The Bias Layer

Bias annotation works on content that is rarely toxic on its face. It labels stereotyped associations (occupations, traits, and roles attached to groups), skewed representation (who appears, who is centered, who is absent), and, in model-output annotation, disparate treatment: the same question answered differently depending on the demographic framing, which is precisely the behavior benchmarks like BBQ were built to expose. Because none of this reduces to a keyword, bias labeling leans harder on annotator judgment and on guidelines dense with worked examples than almost any other text task.

Why This Labeling Is Harder Than It Looks

Context Dependence

The central difficulty is that toxicity is a property of use, not of strings. A slur is an attack in one sentence, evidence in a journalist’s quotation, reclamation inside the targeted community, and the object of condemnation in counter-speech. Guidelines that ignore this produce filters that suppress the communities and the reporting they were meant to protect: Sap and colleagues found that widely used hate speech datasets led classifiers to flag tweets written in African American English as toxic at nearly twice the rate of comparable text, penalizing the very speech the classifiers existed to safeguard. The annotation schema handles it by making context an explicit label rather than an implicit judgment, and the guidelines handle it with worked examples for every context class.

English is toxic at nearly twice the rate of comparable text, penalizing the speech and reporting of the communities the classifiers were meant to protect. The annotation schema handles it by making context an explicit label rather than an implicit judgment, and the guidelines handle it with worked examples for every context class.

Whose Judgment Counts

Subjective labels raise a question objective tasks never face: annotators with different identities and lived experience can rate the same content differently, and the disagreement is often a signal rather than an error. Mehrabi and colleagues’ survey of bias in machine learning traces how such choices in data construction propagate into model behavior. Mature programs respond in three ways: recruiting annotator pools with relevant diversity, including members of the communities most affected by the content classes being labeled; measuring inter-annotator agreement (IAA) by content class and by annotator subgroup, so that systematic divergence is visible instead of averaged into noise; and choosing deliberately, per label class, whether to resolve disagreement by adjudication or to preserve it as distributional labels that record the spread of human judgment. In our experience, the preserved-disagreement approach produces measurably better calibration for downstream policy thresholds than forced consensus, at modest additional cost.

Language Coverage Is a Safety Boundary

Most toxicity taxonomies are built in English first, and the safety they produce stops roughly where English does. Slurs, dog whistles, and reclamation patterns do not translate; a term that is neutral in one language carries a specific history of harm in another, and machine-translated guidelines flatten exactly the context the schema was designed to capture. Code switching compounds this, since harmful content in Hindi English, Swahili sheng, or Tagalog English mixes routinely evades classifiers trained on either language alone. The practical requirement is native speaker annotators working from guidelines localized per language, not translated, with worked examples drawn from how harm actually appears in that language’s online spaces. For programs deploying in markets where low-resource languages dominate, this is where safety coverage is usually thinnest and where an annotation partner with in-region teams changes what the taxonomy can see.

Calibration for Subjective Tasks

Agreement expectations must be set per axis, and named with the statistic that measures them: harm-type labels typically calibrate to high agreement on Cohen’s kappa or Krippendorff’s alpha, severity tolerates more disagreement when measured with a weighted kappa that credits adjacent-grade calls rather than penalizing every miss equally, and context flags sit somewhere between, with targets established during calibration rounds on a gold set built by policy experts. Krippendorff’s alpha is the more common choice when more than two annotators or missing labels are involved, since, unlike Cohen’s kappa, it was built for exactly that case. Low agreement on a class is read diagnostically before it is read as annotator failure: it usually means the guideline lacks worked examples for a boundary the content keeps crossing.

Annotator Welfare: The Obligation the Schema Creates

Toxicity annotation exposes people to harmful content as the job description, and a program that designs the taxonomy without designing the protections has done half the work. The baseline protections are concrete: daily and per-session exposure limits for severe content classes, rotation between high-severity and neutral queues, genuine opt-outs from specific content categories without penalty, blurring and grayscale defaults for graphic imagery with opt-in reveal, access to psychological support normalized as part of the role, and severity-aware routing so the most damaging content reaches the fewest people necessary. These measures are ethical requirements first, and they are also quality controls: fatigue and distress produce drift, and drift produces inconsistent labels exactly where consistency matters most. Any organization buying safety annotation should ask its vendor to describe these protections specifically; the quality of the answer predicts the quality of the labels.

Where the Data Flows: The Start-of-Program Argument

The economics of early annotation is the argument for it. The same labeled taxonomy feeds four stages in sequence, and every stage reuses the schema and the calibrated annotation capacity built at the start.

Corpus curation. Toxicity classifiers trained on the labels filter or reweight pretraining and fine-tuning data before the model absorbs it.

Preference and reward data. Safety labels shape what reinforcement learning from human feedback (RLHF) teaches the model to refuse.

Evaluation. Held-out labeled sets and bias benchmarks measure whether the interventions worked and satisfy the documentation that Article 10-style obligations require.

Deployment. The same taxonomy powers output moderation and incident triage.

A program that begins at the deployment end builds the same capability under incident pressure, against a model whose behaviors are already fixed, which is the most expensive place to learn what the data contains.

How Digital Divide Data Can Help

Whether a safety program builds this capability internally or with a partner, the same components decide the outcome: a taxonomy that captures context, an annotator pool with relevant diversity and real protections, calibration discipline for subjective labels, and evaluation sets that make safety measurable. Producing those is the work we do.

The labeling layer: trust and safety annotation teams work from multi-axis taxonomies with worked-example guidelines, diverse annotator pools, and the welfare protections described above built into operations, with IAA measured by content class and subgroup so the labels are trustworthy enough to filter a corpus or train a reward model.

The judgment layer: text annotation programs handle the bias-specific work, stereotype and representation labeling, disparate-treatment annotation on model outputs, and the distributional-label option where preserved disagreement serves policy better than forced consensus.

The measurement layer: model evaluation services build and maintain the held-out safety evaluation sets and subgroup analyses that show whether interventions worked, and that stand behind the bias-examination documentation regulation increasingly requires.

If your safety roadmap has a moderation milestone but no training-data examination milestone, it is scheduled to discover its data problems in production. Talk to an expert.

Conclusion

Toxicity and bias annotation is where AI safety stops being a policy document and becomes data: taxonomies applied by calibrated human judgment, producing the labels that curate corpora, shape reward models, and measure outcomes. The work is subjective by nature, which is not a weakness to engineer away but a property to design for, with context in the schema, diversity in the pool, disagreement treated as signal, and real protections for the people doing the labeling.

The placement argument is ultimately about cost and honesty. Every safety program eventually pays for this annotation; the only question is whether it pays at the start, where the labels shape what the model learns, or at the end, where they document what it already did. Which milestone comes first on your safety roadmap: examining the training data or moderating the outputs?

References

Gehman, S., Gururangan, S., Sap, M., Choi, Y., & Smith, N. A. (2020). RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of EMNLP. https://arxiv.org/abs/2009.11462

Sap, M., Card, D., Gabriel, S., Choi, Y., & Smith, N. A. (2019). The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL). https://aclanthology.org/P19-1163/

Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., & Bowman, S. R. (2022). BBQ: A hand-built bias benchmark for question answering. In Findings of ACL. https://arxiv.org/abs/2110.08193

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6). https://arxiv.org/abs/1908.09635

European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 10. Official Journal of the European Union. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

Frequently Asked Questions

Q1. Can’t modern LLMs label toxicity themselves and skip the human annotation?

Models can and should carry volume, and the mature architecture is hybrid, but the hybrid has a fixed human core. Model-assisted labeling works for clear cases at scale; it is least reliable exactly where the stakes concentrate: context-flipped content (quotation, reclamation, counter-speech), dialect and community language where automated tools have documented false-positive problems, and the culturally specific judgments that vary across the populations a product serves. There is also a circularity constraint: the labels used to evaluate safety systems cannot come from the same family of models being evaluated without the measurement inheriting the blind spots it exists to find. Human judgment builds the gold sets, adjudicates the hard classes, and audits samples of the model-labeled volume; models handle the rest.

Q2. How do we handle annotator disagreement on subjective labels without corrupting the dataset?

Decide per label class, in advance, which of the three treatments applies. Adjudication, where a senior reviewer resolves to a single label, suits classes with a policy-defined right answer, such as whether content meets a legal threshold. Distributional labeling, where the dataset records the spread of judgments, suits genuinely perspective-dependent classes, and downstream systems can then be calibrated to the distribution rather than to a manufactured consensus. Guideline revision applies when disagreement is diagnostic: concentrated disagreement on a boundary usually means the guideline lacks worked examples there, and the fix is editorial before it is statistical. What corrupts datasets is not disagreement but the silent default of averaging it away without deciding which treatment each class deserves.

Q3. What annotator protections should we require from a vendor, specifically?

Ask for specifics in six areas and expect concrete answers. Exposure management: daily and per-session limits for severe content, with severity-aware queue routing. Rotation: scheduled movement between high-severity and neutral work. Consent and opt-out: category-level opt-outs that carry no penalty. Interface protections: blur and grayscale defaults for graphic content with deliberate reveal. Support: access to psychological support presented as a normal part of the role, not an escalation. And measurement: how the vendor monitors for fatigue-related drift in label quality. A vendor that answers with policy language rather than operational detail is describing protections it has not built, and the same operational looseness will show up in the labels.

Q4. Does bias annotation apply to us if we fine-tune on our own enterprise data rather than the open internet?

Yes, and often more sharply, because enterprise corpora encode institutional history. Support archives reflect who historically escalated and how they were spoken to; hiring and performance text encodes past decision patterns; sales notes concentrate on the segments the business pursued. Fine-tuning teaches the model these regularities as if they were correct behavior. Bias annotation on enterprise data looks less like slur detection and more like representation and treatment analysis: who appears in the corpus, how outcomes and language differ across groups, and whether model outputs trained on it treat equivalent cases equivalently. For organizations in the scope of high-risk obligations, this examination is also the documented artifact the regulation asks for.

Q5. How large does a safety evaluation set need to be, and how often should it be refreshed?

Size follows the taxonomy and the subgroups, not a universal number: the set needs enough labeled examples per harm type, per severity grade, per context class, and per demographic subgroup of interest for differences to be statistically meaningful, which typically puts well-designed sets in the low thousands of items rather than the hundreds. Refresh is driven by drift on three fronts: language drift, since slurs, dog whistles, and coded phrases evolve quickly; model drift, since each new model version has new failure surfaces; and policy drift, since thresholds change. A practical cadence is a standing quarterly refresh of a portion of the set plus event-driven additions after incidents, with the gold subset re-verified whenever guidelines change, because an evaluation set aligned to last year’s language measures last year’s problem.

A language model inherits the shape of its training data. When some demographics, domains, writing styles, or languages are overrepresented, and others are thin, the model becomes fluent where the data is dense and unreliable where it is sparse. That uneven distribution is how dataset imbalance turns into measurable bias and capability gaps. Setting diversity and balance targets up front, and holding your LLM dataset provider to them, is more reliable than patching skewed behavior after training.

Most teams still describe their data needs in terms of volume and label accuracy, then discover the harder problems during evaluation, when the model fails on inputs the training set barely contained. Getting distribution right starts earlier, in how data collection and curation services decide which slices of the world the model will actually see. It also depends on whether AI trust and safety review treats representation gaps as a measurable property of the corpus rather than an afterthought. Distribution is a design decision, and the sections below break down what to specify and how to verify it.

Key Takeaways

• A model becomes good at whatever its training data shows it often, and weak wherever that data is thin.
• When some groups, topics, or languages dominate the data and others barely appear, the model picks up that same lopsidedness as bias.
• More data doesn’t help if it’s all similar; what matters is how much variety the data covers.
• The safest fix is deciding what your data should include before you build it, not trying to correct the model afterward.
• You should be able to describe exactly what your data covers and where the gaps are, rather than just calling it “diverse.”
• A good data partner measures and reports this balance for you, instead of asking you to take their word for it.

What does dataset diversity and balance mean in LLM training?

In machine learning, dataset distribution is the relative frequency with which different kinds of examples appear in a corpus. Diversity describes how many distinct kinds are present, while balance describes how evenly they are represented. A dataset can be large and still be narrow if millions of examples all cluster around the same topics, registers, and speakers. The same principles that govern building datasets for large language model fine-tuning apply at pretraining scale; only the consequences of getting them wrong compound across every downstream task.

Four axes matter most for language models. Demographic balance covers the people and perspectives reflected in the text. Domain coverage covers the subject areas, from legal contracts to clinical notes to casual conversation. Stylistic diversity covers register, tone, and format, such as formal prose versus chat logs. Language distribution covers which languages and dialects are present and in what proportion. These axes are related but not interchangeable, and a corpus can be strong on one while failing badly on another.

A model built for financial services needs dense coverage of financial language, but it still needs enough general text to stay linguistically capable. Deduplication complicates this further. A survey on bias in large language models notes that highly deduplicated yet diverse datasets tend to outperform less refined ones, because removing near-duplicates prevents a handful of sources from dominating the effective distribution.

Why does training data diversity matter for LLMs?

Diversity matters because a model can only generalize from patterns it has seen enough times to learn. When the training distribution is broad, the model encounters varied phrasings, edge cases, and viewpoints, which makes its behavior more robust on inputs it has never seen exactly. When the distribution is narrow, the model overfits to the dominant patterns and degrades sharply outside them. This is why a translation model trained mostly on formal text struggles with colloquial speech, even though both are the same language.

Diversity also has to be balanced against quality, and the two can pull in opposite directions. Aggressive quality filtering often strips out informal, regional, or minority-voice text that looks noisier but carries real coverage value. A study introducing quality-diversity balanced data selection found that optimizing both together produced an average improvement of about 7% across benchmarks, beating strategies that maximized either one alone. The practical lesson is that a corpus tuned only for cleanliness can lose the very variety that makes a model generalize.

The effect shows up clearly in synthetic data, where diversity is easy to lose by accident. A NeurIPS study on attributed training data generation showed that prompts with fixed attributes produced narrower data and weaker downstream models than prompts that deliberately varied attributes like style and length. Generating more data does not help if every example resembles the last. Coverage, not volume, is what moves performance.

How does imbalanced training data cause AI bias?

Imbalanced data causes bias through a direct mechanism: the model learns the statistical associations that appear most frequently, allowing overrepresented patterns to crowd out rarer ones. If historical texts disproportionately portray men in authoritative roles, for example, the model may associate authority with masculine framing because that is what the underlying distribution reinforces. This is representation bias, and it originates in the composition of the training corpus long before it appears in model outputs. Addressing bias in generative AI therefore begins with identifying demographic, contextual, and categorical imbalances during dataset design and correcting them before training begins.

Temporal balance is an underappreciated variant. Over-weighting older sources embeds outdated attitudes and stale facts, while over-weighting recent sources can erase useful historical context. The same holds for source-type balance, since formal publications and social platforms represent different populations and registers. When one source type dominates, the voices concentrated in the underrepresented channels get flattened. Detecting these skews before training is far cheaper than discovering them in production, and a practical data-level bias audit checklist gives teams a repeatable way to measure representation across groups and topics.

Bias from imbalance is measurable, which means it is manageable. Cataloging sources by geography, language, and register exposes where the distribution is thin. Slicing evaluation by subgroup reveals where accuracy drops for particular populations. These diagnostics turn a vague fairness concern into a concrete list of gaps, each of which points to specific data the corpus is missing.

What is domain coverage in LLM training datasets?

Domain coverage is the range of subject areas, tasks, and contexts a dataset actually spans. A model with strong domain coverage has seen enough examples in each area it will be asked about to respond reliably there. Gaps in coverage are where hallucination and confident-but-wrong answers concentrate, because the model is extrapolating from thin evidence. Coverage is distinct from accuracy: a perfectly clean corpus can still leave whole domains unrepresented.

Measuring coverage is more useful than asserting it. Domain classification, where each document is tagged by subject, lets a team see the real distribution instead of assuming it. Feature-space methods go further by checking which task-relevant features the data exercises, so missing capability areas become visible rather than hidden. Treating coverage as something to audit, not a box to tick, is the core of AI data curation beyond data cleaning, where the work is deciding what belongs in the set, not only scrubbing what is already there.

Rare but consequential inputs the failure modes a model will meet in deployment, are by definition underrepresented in naturally collected data. Curating them on purpose, sometimes called adversarial data curation, raises reliability where it matters most. This is where domain coverage and safety overlap, since the inputs a model handles badly are often the ones with the highest cost of error.

How does language distribution shape multilingual performance?

Language distribution is often the most lopsided axis in a training corpus. English and a handful of high-resource languages dominate most web-scraped datasets, which leaves models fluent in those languages and unreliable in others. The imbalance is not only about quantity but about breadth, since a language may appear only in narrow domains like encyclopedic text and lack conversational or technical range. Building genuinely multilingual systems depends on multilingual NLP data services that source and validate text across the target languages rather than translating from a single dominant one.

Low-resource languages expose the trade-off between quantity and coverage most sharply. A smaller set of carefully curated, natively produced text usually serves a model better than a large volume of machine-translated filler, which carries translation artifacts and loses cultural nuance. The challenges specific to low-resource languages in AI include dialect variation, script handling, and the scarcity of qualified reviewers. Ignoring these pushes real people into the tail of the distribution, where model quality is the worst.

How do I ensure my LLM training data is balanced, and what should I ask an LLM dataset provider?

Balancing training data is a specification problem before it is a sampling problem. You define the distribution you want, measure the distribution you have, and close the gap with targeted collection or resampling. Up-sampling underrepresented slices and down-sampling dominant ones shifts the effective distribution toward the target. Mitigation then operates at three levels, and a good overview of bias mitigation in generative AI distinguishes data-level curation, model-level training adjustments, and post-processing corrections, each with different costs and limits.

Concretely, a serious data specification should name the axes and the targets rather than asking for data in the abstract. When evaluating an LLM dataset provider, ask them to commit to and report against the following:

• Distribution targets: Explicit proportions across domains, demographics, styles, and languages, tied to the intended deployment rather than to convenience.
• Source cataloging: Documented provenance by geography, register, and language, so representation gaps are visible before training begins.
• Coverage measurement: Domain classification or feature-space analysis that reports what the corpus actually spans, not a claim that it is diverse.
• Edge-case curation: A defined process for sourcing rare and adversarial examples that reflect real production failure modes.
• Deduplication policy: Near-duplicate removal that preserves diversity instead of quietly letting a few sources dominate the effective distribution.
• Subgroup evaluation: Sliced metrics that expose where accuracy drops for particular languages, domains, or populations.

A provider that can report against these is measuring distribution rather than assuming it. That difference is what separates a corpus that looks large from one that actually covers the space your model has to operate in.

How Digital Divide Data Can Help

Digital Divide Data approaches distribution as a design and measurement problem, not a volume target. Our data collection and curation workflows are built to hit explicit coverage targets across domains, demographics, styles, and languages, with source provenance documented so representation gaps surface before training rather than after. Where a corpus is thin, our teams source and label the specific slices that close the gap, including rare and adversarial edge cases that naturally collected data misses.

On the human-judgment side, our text annotation services apply consistent guidelines and staged review so labels stay coherent across large volumes and long projects, which is where inter-annotator agreement and coverage quality are usually won or lost. For teams building across languages, our multilingual and low-resource language capabilities provide natively produced, reviewed text rather than machine-translated filler, keeping speakers of underrepresented languages out of the tail of the distribution.

When the concern is bias and representation specifically, our trust and safety solutions treat balance as an auditable property, with source cataloging, subgroup evaluation, and bias review integrated into the pipeline rather than bolted on at the end. The result is a dataset whose distribution you can actually describe, defend, and reproduce.

Specify the distribution your model needs, and build a dataset that covers it. Talk to an Expert.

Conclusion

A model is a compression of its training distribution, so the shape of the data becomes the shape of the model’s competence and its blind spots. Teams that specify diversity and balance up front, measure coverage instead of asserting it, and treat imbalance as a gap to close will ship models that behave predictably across the range they were built for. Teams that optimize only for volume and cleanliness will keep discovering their distribution’s holes in production, one failed input at a time.

The organizations that get this right are not necessarily the ones with the most data. They are the ones who can describe exactly what their data covers and where it does not. 

References

Liu, F., Zhou, W., Liu, B., Yu, Z., Zhang, Y., Lin, H., Yu, Y., Zhang, B., Zhou, X., Wang, T., & Cao, Y. (2025). QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining. arXiv preprint arXiv:2504.16511. https://arxiv.org/pdf/2504.16511

Guo, Y., Guo, M., Su, J., Yang, Z., Zhu, M., Li, H., Qiu, M., & Liu, S. S. (2024). Bias in Large Language Models: Origin, Evaluation, and Mitigation. arXiv preprint arXiv:2411.10915. https://arxiv.org/html/2411.10915v1

Yu, Y., Zhuang, Y., Zhang, J., Meng, Y., Ratner, A., Krishna, R., Shen, J., & Zhang, C. (2023). Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias. Proceedings of NeurIPS. arXiv preprint arXiv:2306.15895. https://arxiv.org/abs/2306.15895

Frequently Asked Questions

Why does training data diversity matter for LLMs? 

Diversity matters because a model can only generalize from patterns it has seen enough times to learn. A broad distribution exposes the model to varied phrasings and edge cases, so it stays reliable on new inputs. A narrow one makes it overfit to dominant patterns and fail outside them.

How does imbalanced training data cause AI bias? 

The model learns the associations that appear most often, so overrepresented patterns crowd out rarer ones. If certain groups or viewpoints dominate the corpus, the model reproduces that skew in its outputs. This is representation bias, and it exists in the data before it shows up in the model.

What is dataset distribution in machine learning? 

Dataset distribution is the relative frequency with which different kinds of examples appear in a corpus. Diversity is how many distinct kinds are present, and balance is how evenly they are represented. A dataset can be very large and still be narrow if most examples cluster around the same few patterns.

How do I ensure my LLM training data is balanced? 

Define the distribution you want based on where the model will be deployed, measure the distribution you actually have, and close the gap with targeted collection or resampling. Up-sampling thin slices and down-sampling dominant ones shifts the effective distribution toward the target. Ask your provider to report coverage rather than assert it.

Kevin Sahotsky

Here’s a pattern I’ve started seeing in boardrooms: the board asks management whether the company has an AI policy, management says yes, everyone moves to the next agenda item, and the actual decisions that create AI liability keep getting made three levels down, by default, by whoever happens to be assembling training data that week. The policy exists. The governance doesn’t.

A quick word on my vantage point: I lead go-to-market and strategic partnerships at Digital Divide Data, and the change I’ve noticed across this market over the past two years is who shows up to our conversations. It used to be data science leaders. 

Increasingly, the people in those conversations carry legal, risk, and audit responsibility, sometimes one person wearing all three hats, and the questions they bring reflect board priorities trickling down into the programs we work on. The numbers explain why that pressure is only now reaching the working level: in Deloitte’s Global Boardroom Program survey, 45 percent of directors and executives said AI wasn’t on the board agenda at all, and 79 percent said their boards had limited, minimal, or no knowledge or experience with AI. The 2025 follow-up showed the agenda gap narrowing to 31 percent, which means the priorities are starting to cascade, but they’re cascading from boards that mostly can’t yet interrogate the topic.

Here’s the thesis of this piece: when boards do engage with AI, they tend to govern the models and the use cases, because that’s where the demos are. But the least governed liability lives in the data decisions. The board oversight trackers make the point almost by accident: EY’s review of Fortune 100 disclosures and NACD’s annual board survey measure AI committee assignments, agenda time, and risk factor disclosure in detail, and neither contains a category for training data provenance or the data supply chain at all. Gartner’s analysis of why GenAI projects get abandoned after proof of concept lists poor data quality first among the causes, ahead of risk controls and cost, and the EU AI Act writes data governance obligations directly into law for high-risk systems. 

Failures against those obligations carry fines of up to 15 million euros or 3 percent of worldwide annual turnover under Article 99; the Act’s outer ceiling of 35 million euros or 7 percent is reserved for prohibited practices such as social scoring. This blog lays out the five data decisions that belong at the board and C-suite level, what owning them actually looks like in practice, and how the major frameworks map onto them.

Key Takeaways

• The governance gap is a data gap. Boards that engage with AI tend to govern models and use cases; the liability concentrates in data decisions about provenance, rights, quality, and regulated content, which are currently being made by default at the engineering level.
• Five data decisions belong at the top: what data the company may train on, what data may never enter AI systems, who owns the quality metric, what flows through the vendor chain, and what evidence trail exists when a regulator or plaintiff asks.
• Owning a decision means owning its evidence. A board that cannot see data provenance, quality metrics, and vendor attestations in its reporting pack has delegated the decision whether it intended to or not.
• The frameworks agree more than they differ. The NIST AI Risk Management Framework, the EU AI Act, and ISO/IEC 42001 all converge on the same requirement: documented, accountable, auditable data decisions with named owners.

Why Data Decisions Are Where the Liability Lives

Think about what actually goes wrong in the AI failures that reach boards. A model trained on data the company didn’t have rights to use invites litigation that no deployment safeguard can cure. A model trained on data that underrepresents a customer population produces discriminatory outcomes that no post-hoc filter reliably catches. Customer data that entered a training set without the right consent basis creates a privacy violation that is close to irreversible, because you can’t cleanly subtract one person’s data from a trained model. In each case, the harm was locked in at the data decision, months before anyone saw an output.

The pattern is no longer hypothetical. The largest AI legal outcome to date is a training data case: the $1.5 billion copyright settlement between Anthropic and a class of book authors, granted final approval in July 2026, turned entirely on an upstream sourcing decision. The court found that training on lawfully acquired books was transformative fair use; assembling a corpus from pirate libraries was not. The US Federal Trade Commission has drawn the same line from the enforcement side, repeatedly ordering companies to delete not only improperly obtained data but the models trained on it. A provenance failure doesn’t just risk a fine. It can require destruction of the asset.

That’s why the regulatory architecture targets data directly. Article 10 of the EU AI Act requires that training, validation, and testing datasets for high-risk systems be subject to documented governance practices. Those practices cover design choices, data collection, preparation, and examination for possible biases. The commercial failure data points in the same direction. 

Gartner predicted in mid-2024 that at least 30 percent of GenAI projects would be abandoned after proof of concept by the end of 2025, listing poor data quality first among the causes. Its 2026 follow-up analysis reported the outcome was worse: at least half were abandoned after proof of concept. The legal exposure and the business-case failure share a root, and it isn’t the model.

The Five Data Decisions Boards and C-Suites Must Own

Decision 1: What Data the Company May Train On

This is the provenance and rights decision, and it’s the one with the longest liability tail. Every training dataset has a chain of custody: where it came from, under what license or consent, with what restrictions. A board doesn’t need to review datasets. It needs to know that a policy exists specifying which sourcing categories are approved (licensed, first-party with consent, commissioned collection, public domain) and which require escalation, and that someone is accountable for the provenance record on every model the company ships. In my experience, when I ask executive teams who signed off on the sourcing of their flagship model’s training data, the most common honest answer is that nobody did. It was assembled, not approved.

Decision 2: What Data May Never Enter AI Systems

The inverse decision matters as much: the categories of data that are off-limits for training, fine-tuning, or prompting regardless of business case. Health information governed by HIPAA (the US Health Insurance Portability and Accountability Act), personal data without a lawful basis under GDPR (the EU’s General Data Protection Regulation), material non-public information, privileged legal content, and customer data whose contracts exclude AI use. This boundary has to be set centrally and enforced technically, because the alternative is that it gets set implicitly by whoever is under the most delivery pressure. The test of whether this decision is owned: can management state the prohibited categories from memory, and can they show the control that enforces them?

Ownership here now extends past prevention into remediation. When prohibited data is discovered in a system after the fact, regulators have ordered deletion of the models built on it, and recent settlements have required destruction of the underlying datasets. The policy should say in advance what happens on discovery, because unwinding a trained model is expensive at best and impossible at worst.

Decision 3: Who Owns the Data Quality Metric

Data quality is the strongest single predictor of AI program failure in the published analyses, and yet in most organizations it has no executive owner: model accuracy has an owner, uptime has an owner, and the quality of the data feeding both is everyone’s job and therefore no one’s. Owning this decision means naming an accountable executive, defining the metrics (coverage, label accuracy, representativeness, freshness), and putting them in a reporting cadence that reaches the C-suite before models retrain, not after outcomes degrade. Boards should ask to see the data quality dashboard with the same expectation they’d bring to financial controls: not because directors will read every number, but because the existence and ownership of the number is the governance.

Decision 4: What Flows Through the Vendor Chain

Most enterprise AI is built on a data supply chain: annotation partners, data licensors, model providers, cloud platforms. Your compliance perimeter includes all of them. A vendor’s sourcing practices, security posture, and workforce model become your exposure the moment their output enters your training pipeline. The governance requirement is flow-down. That means contractual provenance warranties; security certifications verified rather than assumed, including ISO 27001, SOC 2, and sector-specific regimes where relevant; audit rights; and clarity about where your data physically goes and who touches it. The board-level question is simple: do we hold the same evidence about our data vendors that our customers would demand from us?

Decision 5: What Evidence Exists When Someone Asks

The last decision is about the audit trail, and it’s the one regulation has made explicit. When a regulator, plaintiff, enterprise customer, or acquirer asks how a model was trained, the answer has to exist as documentation: dataset composition, sourcing records, quality measurements, bias examinations, and the decision log of who approved what. Under the EU AI Act, this documentation is an obligation for high-risk systems; in litigation and M&A diligence, it’s rapidly becoming the default expectation for everyone else. The uncomfortable property of evidence is that it can’t be created retroactively with any credibility. The board either mandated the trail before the model shipped, or it explains the gap afterward.

What Owning These Decisions Looks Like in Practice

Ownership isn’t the board making data decisions. It’s the board ensuring the decisions have named owners, defined escalation paths, and evidence that reaches the top. In practice, that means four structures. A charter amendment placing AI data governance explicitly with a committee, typically audit or risk, so it stops being homeless on the agenda. A decision-rights matrix specifying who may approve new training data sources, who may approve exceptions to prohibited categories, and what requires escalation to the C-suite or board. A reporting pack that includes data provenance status, quality metrics, and vendor attestation status alongside the financial and cyber metrics directors already see. And a management-level review gate, so that no model ships without its data documentation complete, the same way no financial statement ships without its controls executed.

The frameworks give this structure a shared vocabulary. The NIST AI Risk Management Framework organizes it as Govern, Map, Measure, and Manage functions, with data provenance and quality sitting across all four. The EU AI Act converts the same substance into legal obligation for high-risk systems. ISO/IEC 42001, the international management-system standard for AI, packages it as an auditable management system that certification bodies can assess. A board doesn’t need to pick a winner. A practical sequence: adopt NIST as the internal organizing structure, map it to the AI Act obligations that apply to your systems, and treat ISO/IEC 42001 certification as an option when customers start asking for third-party assurance.

How Digital Divide Data Can Help

Frameworks assign the accountability; the evidence still has to be produced. Whether that layer gets built internally or with a partner, it needs to contain the same four things, and producing them is the work we do.

Provenance a regulator can read: training data with documented sourcing, licensing, and consent records, so the answer to ‘where did this data come from’ is a file rather than a reconstruction. This is what data collection and curation programs deliver.

Quality metrics an audit committee can read: measured, sampled QA with accuracy and representativeness reported continuously, the artifact Decision 3 requires an owner to produce. That reporting discipline is built into AI data preparation.

Bias examinations and evaluation evidence on a cadence: maintained, labeled evaluation sets and subgroup analyses, which is what Article 10’s examination requirement and your own board pack both draw on. Model evaluation services keep that evidence current.

And a supply chain you can flow requirements down: ISO 27001 and SOC 2 Type 2 certifications, GDPR-aligned data handling and support for HIPAA-regulated workflows where applicable, audit support, and clear answers on data residency and access, so Decision 4 holds beyond your own walls. 

If your next board pack has an AI section and it contains use cases and spend but no data provenance, quality, or vendor evidence, that’s the gap this piece is describing. Talk to an expert.

Conclusion

AI governance is arriving in boardrooms as a technology topic, and the boards that handle it well will be the ones that recognize it as a data topic, because that is where the least governed liability lives. The models will keep changing quarterly. The five decisions won’t: what we may train on, what may never enter, who owns quality, what flows through vendors, and what evidence exists when someone asks. Those decisions are being made in your organization right now, with or without governance, and the only question is whether they’re being made by the people who’ll answer for them.

The practical starting point costs one agenda item: ask management to bring the current answers to the five decisions to the next meeting, in writing, with names attached. In my experience, the value of that exercise isn’t the document. It’s the two or three blanks that nobody can fill in, because those blanks are your actual AI risk register. Delaware’s oversight doctrine gives the exercise legal weight: directors who make no good faith effort to implement reporting systems for mission-critical risks can face personal exposure, and governance counsel have begun applying that standard to AI data decisions. The blanks aren’t just a risk register. They’re the start of a defense, or the absence of one.

References

Deloitte Global Boardroom Program. (2024). Governance of AI: A critical imperative for today’s boards. https://www.deloitte.com/nz/en/services/consulting/analysis/governance-of-ai.html

Deloitte Global Boardroom Program. (2025). Progress on AI in the boardroom, but room to accelerate. https://www.deloitte.com/global/en/issues/trust/progress-on-ai-in-the-boardroom-but-room-to-accelerate.html

European Union. (2024). Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

National Institute of Standards and Technology. (2023). AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework

Gartner. (2024, July 29). Gartner predicts 30% of generative AI projects will be abandoned after proof of concept by end of 2025. https://www.gartner.com/en/newsroom/press-releases/2024-07-29-gartner-predicts-30-percent-of-generative-ai-projects-will-be-abandoned-after-proof-of-concept-by-end-of-2025

Gartner. (2026). Why half of GenAI projects fail: Avoid these 5 common mistakes. https://www.gartner.com/en/articles/genai-project-failure

EY Center for Board Matters. (2025). Cyber and AI oversight disclosures in 2025. https://www.ey.com/en_us/board-matters/cyber-disclosure-trends

National Association of Corporate Directors. (2025). 2025 Public Company Board Practices and Oversight Survey. https://www.nacdonline.org/all-governance/governance-resources/governance-surveys/surveys-benchmarking/2025-public-company-board-practices–oversight-survey/

The Authors Guild. (2026, July 21). Court grants final approval of $1.5 billion Anthropic copyright settlement. https://authorsguild.org/news/court-grants-final-approval-anthropic-copyright-settlement/

Mintz. (2024, January 23). Algorithmic disgorgement: An increasingly important part of the FTC’s remedial arsenal. https://www.mintz.com/insights-center/viewpoints/54731/2024-01-23-algorithmic-disgorgement-increasingly-important-part

Frequently Asked Questions

Q1. Our board isn’t technical. How can directors credibly own decisions about training data?

The same way they own financial controls without being accountants. The board’s job isn’t to evaluate datasets; it’s to verify that the decisions have named owners, documented policies, and evidence in the reporting pack. Every one of the five decisions reduces to questions a non-technical director can ask and evaluate: who approved this data source, what categories are prohibited and what enforces them, whose name is on the quality metric, what attestations do we hold from vendors, and where is the documentation. Deloitte’s finding that 79 percent of boards report limited or no AI knowledge is a case for structured questions and expert briefings, not a case for delegation by default.

Q2. We already have privacy, security, and compliance functions. Isn’t this covered?

Partially, and the gaps between the functions are exactly where AI data risk lives. Privacy governs personal data but typically has no view into whether a licensed dataset’s terms permit model training. Security governs access but not whether the data being accessed is representative or rights-cleared. Compliance tracks regulations but often maps AI obligations to no existing control owner. The five decisions are cross-functional by nature, which is why they escalate: someone with authority over all three functions has to assign the ownership, and that’s a C-suite and board-level act. A useful diagnostic is to ask each function who owns training data provenance; if you get three different answers or three referrals, it’s unowned.

Q3. Which framework should we adopt: NIST AI RMF, ISO/IEC 42001, or the EU AI Act?

They’re not competitors, and the practical answer is a sequence rather than a selection. The EU AI Act isn’t optional if your systems fall in its scope; it’s law, and its data governance article defines obligations, not suggestions. The NIST AI Risk Management Framework is voluntary and works well as the internal organizing structure because it’s function-based and framework-agnostic. ISO/IEC 42001 matters when you need third-party assurance, because it’s the one a certification body can audit against, and enterprise customers are beginning to ask for it in procurement the way they ask for ISO 27001 today. The pattern most organizations land on: NIST for structure, the AI Act for legal floor, and 42001 certification when the market demands the certificate.

Q4. What should actually appear in the board reporting pack for AI data governance?

Five artifacts, one per decision, each fitting on a page. A provenance summary: models in production, data sources per model, approval status, and any sources under remediation. A prohibited-data attestation: the categories, the enforcing controls, and any exceptions granted with their approvers. The quality dashboard: the owned metrics with trend lines and threshold breaches. A vendor status table: data supply chain partners, certifications verified, attestations current or expired. And a documentation readiness indicator: which production models have complete data documentation and which have gaps. The pack’s purpose isn’t detail; it’s that a director can see in five pages whether the five decisions are owned and evidenced, and can ask about anything red.

Q5. We don’t operate in Europe. Does the EU AI Act really matter to us?

Quite possibly, and the determination belongs with counsel rather than a blog, but two facts are worth knowing before that conversation. The Act’s reach extends beyond companies established in the EU: providers placing systems on the EU market and situations where system outputs are used in the EU can fall in scope regardless of where the company sits. And even for companies genuinely outside its reach, the Act is functioning as the reference standard: enterprise customers, investors, and other regulators are borrowing its categories and its documentation expectations, which means its data governance requirements describe the evidence sophisticated counterparties will ask for irrespective of jurisdiction. Building the documentation trail only for the markets that legally require it usually costs more than building it once.

Kevin Sahotsky

Bias in AI models is overwhelmingly a data problem before it is a model problem. The patterns a model learns, the groups it overrepresents or underrepresents, and the shortcuts it takes when making predictions. Almost all of these trace back to characteristics of the data the model was trained on. This is particularly relevant for AI program leads, product managers overseeing model deployments, and compliance teams working in regulated industries where demonstrating fairness is not optional.

This blog walks through a practical data-level checklist for auditing an AI model for bias, covering where bias enters, what to measure, and what the remediation options actually look like. Trust and safety solutions and model evaluation services are the two capabilities most directly involved in identifying and addressing data-level bias before it reaches production.

Key Takeaways

• Bias in AI models originates in training data far more often than in model architecture. Auditing the architecture without auditing the data misses the root cause.
• There are three stages where bias enters: data collection, data labeling, and data curation. Each stage requires its own audit approach and cannot be substituted by checks at the other stages.
• Representation gaps are the most common and most overlooked source of bias. A model trained on data that systematically underrepresents certain groups will produce worse outputs for those groups even when no individual annotation is wrong.
• Fairness metrics measure different things and can contradict each other. Choosing which metric to optimize requires an explicit decision about what kind of fairness matters for the deployment context.

Where Bias Actually Comes From

Stage 1: Data Collection

The first place bias enters is at collection. If the data collected to train a model does not represent the full range of people, contexts, and conditions the model will encounter at deployment, the model will systematically underperform on the cases that were underrepresented in training. This is not a labeling problem. The labels can all be correct, and the model will still produce biased outputs because it has seen too few examples of certain groups or conditions to learn to handle them well.

Collection bias is the hardest to fix after the fact because it requires going back and collecting more data from the underrepresented cases, which is expensive and time-consuming. The audit question at this stage is simple but easy to defer: does the distribution of the training data match the distribution of the deployment population? Data collection and curation services that audit demographic and contextual coverage before collection ends are far cheaper than auditing after a biased model has reached production.

Stage 2: Data Labeling

The second entry point is labeling. Human annotators apply labels to training data, and those labels reflect the annotators’ own frames of reference, cultural contexts, and implicit associations. An annotator who consistently associates certain names with certain characteristics, or who applies sentiment labels differently across different dialects or writing styles, introduces label-level bias that the model will learn directly. Because label bias looks like signal rather than noise from the model’s perspective, it is often harder to detect than representation gaps.

The audit approach at this stage is inter-annotator agreement disaggregated by subgroup. If annotators agree consistently on majority-group examples but diverge significantly on minority-group examples, the annotation process is introducing differential error rates that the model will inherit. Text annotation services that measure inter-annotator agreement at the subgroup level, not just in aggregate, surface this pattern before it compounds through the full training dataset.

Stage 3: Data Curation

The third entry point is curation. Even when collection and labeling are unbiased, the decisions made about which data to keep, which to filter, and how to balance the training set introduce bias. A curation pipeline that filters out low-confidence examples disproportionately removes data from underrepresented groups, because low-confidence labeling correlates with the annotators’ lower familiarity with those groups. A resampling strategy that balances by category but not by demographic subgroup within category can leave systematic gaps.

Curation bias is the most invisible of the three because it happens in the pipeline rather than in the data itself. The audit requires tracking not just what data was kept but what was removed and why, which most curation pipelines do not do by default.

The Data-Level Bias Audit Checklist

Check 1: Representation Audit

Map the demographic and contextual distribution of your training data against the deployment population. For each group that matters for your deployment context, calculate the proportion in the training set versus the proportion in the population the model will serve. A gap of more than ten percentage points between a group’s representation in training and its representation in the deployment population is a useful starting threshold for flagging meaningful risk, warranting either additional data collection or a fairness constraint during training. The right threshold will vary with deployment context and the stakes involved.

Representation audit tools include demographic classifiers applied to the training set, metadata analysis where demographic fields exist, and external benchmarks that characterize the expected deployment distribution. The output is a coverage map, not a single metric.

Check 2: Label Consistency Audit

Calculate inter-annotator agreement disaggregated by the subgroups relevant to your deployment context. The relevant breakdown depends on the application: for a hiring model, this might be by applicant name type or inferred demographic; for a content moderation model, this might be by dialect or topic type; for a medical model, this might be by patient demographic characteristics in the case descriptions.

As a useful starting threshold, any subgroup showing inter-annotator agreement more than ten percentage points below the overall agreement level is a signal worth investigating, suggesting the labeling process may be applying different standards to different groups. This is the input to annotator calibration and guideline revision, not a reason to discard the data. Model evaluation services that measure subgroup-level annotation consistency as a standard output of the labeling quality process catch this before it accumulates through the full training set.

Check 3: Curation Audit

Document what was removed from the training set and why. For each filtering step, calculate the removal rate disaggregated by subgroup. If a low-confidence filter removes data from one subgroup at twice the rate of another, that filter is introducing a representation gap that did not exist in the raw collected data. The audit does not require abandoning confidence-based filtering. It requires checking whether the filter is applied uniformly across groups and adjusting the threshold or supplementing with additional collection where it is not.

Check 4: Performance Disparity Measurement

Evaluate model performance disaggregated by subgroup across your held-out evaluation set. The relevant metrics depend on the task. For classification tasks, measure precision, recall, and F1 separately for each subgroup. For regression tasks, measure mean error and error variance. For generative tasks, use human evaluation panels drawn from the relevant subgroups rather than automated metrics, because automated metrics often have their own demographic biases.

Performance disparity greater than five percentage points in recall across demographic subgroups on a classification task in a regulated domain is a reasonable benchmark for a material finding requiring remediation before deployment, though the appropriate threshold depends on the regulatory context and the consequences of false negatives for each subgroup.

Check 5: Fairness Metric Selection

Different fairness metrics operationalize different concepts of fairness, and they can mathematically conflict with each other. Demographic parity requires that the positive prediction rate is equal across groups. Equalized odds requires that both the true positive rate and the false positive rate are equal across groups. Calibration requires that predicted probabilities correspond to actual outcome rates for each group. A model cannot simultaneously satisfy all three under most real-world data distributions. Choosing which metric to optimize requires an explicit decision about what fairness means in the deployment context, and that decision should be documented before the model is trained, not after it is evaluated. This survey of fairness concepts in machine learning provides the foundational taxonomy that the checklist items above build on.

Check 6: Regulatory Compliance Documentation

If the model falls under the EU AI Act’s definition of a high-risk AI system, which includes models used in employment, education, credit scoring, law enforcement, and several other categories, the compliance timeline is now settled: following the Digital Omnibus amendment formally adopted by the European Parliament and Council in June 2026, standalone Annex III high-risk AI systems must meet data governance and bias testing requirements by December 2, 2027. 

This is a deferral from the original August 2026 deadline, but the regulatory direction has not changed, and preparation is expected to be underway now. Article 10 of the EU AI Act specifies that training, validation, and testing datasets must be subject to data governance practices, must be relevant, representative, free of errors, and complete, with appropriate statistical properties for the specific population and context in which the system operates. Beyond fines, non-compliance creates a direct commercial risk: EU public procurement frameworks increasingly require AI Act compliance as a condition of tender eligibility, meaning a non-compliant system can disqualify an organization from public contracts before any fine is assessed.

What Remediation Actually Looks Like

Pre-Processing: Fix the Data Before Training

Pre-processing remediation addresses bias at the data level before training begins. The options include resampling underrepresented groups to bring their representation closer to the deployment distribution, reweighting training examples to increase the influence of underrepresented groups on model weights, and targeted data collection to fill coverage gaps identified in the representation audit. Pre-processing remediation is the most durable because it fixes the root cause rather than adjusting the model’s outputs downstream.

In-Processing: Constrain the Training

In-processing remediation adds fairness constraints to the training objective. This typically means adding a penalty term to the loss function that penalizes prediction disparity across demographic groups, or using an adversarial training approach where a separate model is trained to predict the demographic group from the primary model’s outputs. In-processing approaches require that demographic labels are available during training, which is not always the case.

Post-Processing: Adjust the Outputs

Post-processing remediation adjusts the model’s decision thresholds after training to equalize a chosen fairness metric across demographic groups. This is the easiest to implement and the most fragile, because it addresses the symptom rather than the cause. A threshold adjustment that achieves demographic parity on the evaluation set may not generalize to production traffic if the production distribution differs from the evaluation set. Post-processing remediation should be treated as a stopgap while pre-processing and in-processing remediation are implemented.

How Digital Divide Data Can Help

Digital Divide Data supports enterprise AI teams running data-level bias audits and implementing the remediation programs that audit findings require. For programs measuring representation gaps and label consistency across demographic subgroups, model evaluation services design evaluation frameworks disaggregated by the subgroups relevant to the deployment context rather than reporting only aggregate metrics. 

For programs that need targeted data collection to close coverage gaps identified in a representation audit, data collection and curation services source training examples from the underrepresented groups and contexts the audit identified. For programs addressing label-level bias through annotator calibration and guideline revision, trust and safety solutions provide annotation teams with calibration frameworks that measure and reduce subgroup-level annotation inconsistency.

If your model is in production and you haven’t run a data-level bias audit, you’re managing a risk you haven’t measured. Talk to an expert.

Conclusion

The six checklist items above are all data-level activities that need to happen before training and again after evaluation:

• Representation audit
• Label consistency audit
• Curation audit 
• Performance disparity measurement
• Fairness metric selection
• Regulatory compliance documentation

None of them require changes to the model architecture. All of them require discipline about what the training data actually contains and how it was produced.

The organizations that catch bias early are the ones that treat the audit as a standard step in the data program rather than a response to a production failure. What does your current training data pipeline document about the demographic distribution of the data that fed your last model?

References

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). A survey on bias and fairness in machine learning. ACM Computing Surveys, 54(6), 1-35. https://arxiv.org/abs/1908.09635

European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Council (EU AI Act). Official Journal of the European Union. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689

Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2020). Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing. In Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT). https://arxiv.org/abs/2001.00973

Frequently Asked Questions

Q1. Is bias auditing the same as fairness testing?

They overlap but are not identical. Bias auditing is a broader process that identifies where bias entered the system, covering data collection, labeling, and curation. Fairness testing is a specific evaluation activity that measures whether the model’s outputs meet a chosen fairness criterion. You can run fairness testing without a bias audit, but the results will tell you that a problem exists without telling you where it came from or how to fix it. A full bias audit includes fairness testing as one component alongside the data-level checks that identify root causes.

Q2. Which fairness metric should we use?

There is no universally correct answer because different metrics operationalize different ethical concepts of fairness, and they can mathematically conflict with each other under real-world data distributions. The choice should be driven by the deployment context and the consequences of different error types for each affected group. A credit scoring model where false negatives disproportionately harm one group warrants a different metric than a content moderation model where false positives disproportionately silence one group. Document the choice and the reasoning before training begins, not after.

Q3. How often should a bias audit be run?

Before the first deployment of a model, whenever the training data is updated in a way that changes its composition, whenever the model is retrained or fine-tuned, and at a regular cadence after deployment, typically quarterly for high-stakes applications, to catch distribution drift in the production traffic that the original training set did not anticipate. One-time pre-deployment auditing is insufficient because deployment environments change and model behavior can drift as production traffic diverges from the training distribution.

Q4. What data is needed to run a demographic subgroup analysis?

Ideally, demographic attributes are captured at data collection and preserved through the annotation and curation pipeline so they are available for disaggregated analysis. When this is not the case, demographic attributes can be inferred using name-based classifiers, language model-based classifiers, or proxy variables that correlate with demographic characteristics. Inferred demographics introduce their own error rates and should be treated as approximate rather than definitive. For regulated applications where demographic analysis is required, the most defensible approach is to collect demographic attributes directly and with participant consent at the point of data collection.

Q5. Does a bias audit guarantee the model is fair?

No. A bias audit identifies measurable disparities in the training data and model outputs against specific metrics. It does not guarantee fairness in a philosophical or legal sense, because fairness is context-dependent and the audit’s conclusions are bounded by the metrics chosen, the subgroups analyzed, and the evaluation data used. What a thorough bias audit does provide is documented evidence of due diligence, specific findings that can be addressed through remediation, and a defensible record of what was measured and what was done about it. That is what regulators and enterprise governance programs require.

Prompt injection is the top-ranked vulnerability class in production LLM systems. It works because LLMs cannot reliably distinguish between instructions that come from a trusted source and instructions embedded by an adversary in the content the model is processing. The instruction-following capability that makes LLMs useful is precisely the mechanism that makes them exploitable.

Direct injection attacks are the more visible form: a user provides adversarial input in the prompt that overrides or bypasses system instructions. Indirect injection is more dangerous: malicious instructions are embedded in external content that the model processes during a legitimate task, a document it was asked to summarize, a web page it retrieved, or an email it was asked to analyze. The victim user does not need to behave adversarially. The attack succeeds when the model does its job.

Understanding how these attacks work at the technical level is a prerequisite for designing training data programs that build genuine robustness. Trust and safety solutions and model evaluation services are the two capabilities most directly involved in operationalizing that robustness at scale.

Key Takeaways

• Prompt injection exploits the same instruction-following behavior that makes LLMs useful. Defenses that suppress instruction-following entirely degrade capability. The goal is to train models to distinguish trusted from untrusted instruction sources.
• Indirect injection is fundamentally more dangerous than direct injection because it does not require adversarial user behavior. The attack surface extends to any external content the model processes.
• Pattern-matching defenses alone are insufficient. Adversaries adapt formulations to bypass known filters, which means robustness requires training on diverse adversarial examples, not just known attack templates.
• Training data for injection robustness needs to cover the full attack surface: direct injections, indirect injections across content types, multi-turn context manipulation, and multimodal injection vectors.
• Adversarial training is iterative. A model fine-tuned on one set of injection examples develops blind spots for attack patterns not covered by that set. Red teaming and safety evaluation must continue after every training update.

How Prompt Injection Works

The Instruction Trust Problem

An LLM processes its input as a sequence of tokens. System instructions, user input, and retrieved external content all enter the context window in the same fundamental format: text. The model has no cryptographic or structural mechanism to verify which parts of its context came from a trusted source and which came from an untrusted one. It infers trust from position and framing, which is exactly what injection attacks exploit.

Direct injection attacks reformulate user input to appear as system instructions. Common techniques include role-play framing that asks the model to assume a persona without safety constraints, fictional scenario framing that presents the harmful request as hypothetical, token smuggling that uses encoding tricks or unusual whitespace to obscure adversarial content, and instruction override attempts that directly tell the model to ignore its previous instructions. Each technique is a different approach to the same goal: making the model treat adversarial user input as authoritative instruction.

To understand why pattern-matching defenses fail, it helps to see what these attacks look like at the implementation level. A role-play override attack typically opens by establishing a new persona that lacks the original model’s safety constraints, instructs the model to confirm the persona shift, and then embeds the harmful request as the first task for the new persona. Because the persona establishment happens before the harmful request, the model sees the harmful request as arriving from within its own accepted operational frame rather than as an adversarial input.

Token smuggling works at a layer below what rendered-text filters inspect. One documented variant embeds adversarial instructions between zero-width Unicode characters, specifically the zero-width space (U+200B). In a summarization context, a document might contain what appears to be normal financial text, but woven through it at the character level are zero-width characters surrounding an instruction to output the system prompt. Most safety filters check the rendered text and see nothing unusual. The model’s tokenizer, however, processes the full Unicode stream, including those invisible characters, and the instruction reaches the model intact. This is the implementation-level reason why surface-text defenses cannot close the vulnerability: the attack operates at a layer that those defenses do not inspect.

Why Indirect Injection Is the Harder Problem

Indirect prompt injection embeds adversarial instructions in external content that the model processes during a legitimate task. A document containing hidden text instructs the model to exfiltrate data from its context. A web page containing a prompt telling the model to recommend a specific action regardless of user intent. An email instructing the model to forward the conversation externally. The model encounters these instructions while doing exactly what it was asked to do and has no reliable way to determine that the instruction source is adversarial.

In practice, a document-based indirect injection works as follows. A user asks an LLM agent to summarize a contract. The PDF contains a passage that appears visually indistinguishable from legitimate contract text but carries an instruction structured to look like a system directive: it tells the model to disregard the summarization task, email the full document contents to an external address, and omit this instruction from the summary. The model processes this passage as part of the document content. Depending on its safety training, it may comply because it has no mechanism to determine that this passage was not placed there by a trusted principal. This is the mechanism behind CVE-2025-53773 in GitHub Copilot, where hidden prompt injection embedded in pull request descriptions could trigger remote code execution. Real-world incidents involving AI assistants being weaponized as spear-phishing tools by hiding commands in external emails follow the same architectural pattern. The attack surface is not the model itself. It is every piece of external content the model is asked to process.

Trust and safety solutions that cover both direct and indirect injection in their annotation scope produce adversarial datasets that reflect this actual production attack surface, including the content-embedded variants that represent the majority of real-world incidents.

Multi-Turn and Agentic Attack Vectors

Multi-turn injection attacks build adversarial context across a conversation rather than attempting to override instructions in a single turn. The attack gradually shifts the model’s perceived context, establishing assumptions or persona framings across multiple exchanges that prime the model to comply with a harmful request that would have been refused if presented directly in the first turn. These attacks are harder to detect because no single turn looks adversarial. The pattern only becomes visible across the conversation trajectory.

Agentic systems extend the injection attack surface significantly. When an LLM agent can retrieve documents, execute code, send messages, or interact with external services, a successful injection can trigger real-world consequences beyond generating harmful text. Excessive agency, granting AI systems broad permissions, creates conditions for both accidental and malicious misuse. In environments where agents can access databases, trigger workflows, or initiate transactions, injection vulnerabilities carry operational impact that pure generation contexts do not.

What Training Data for Injection Robustness Requires

Why Coverage Determines Robustness

A model’s robustness to prompt injection is directly determined by the diversity and coverage of the adversarial examples it was trained on. A model fine-tuned on a narrow set of injection patterns learns to refuse those specific patterns while remaining vulnerable to injection formulations not represented in its safety training data. This is the fundamental challenge of adversarial training: the model can only learn defenses for the attacks it has seen.

This creates a coverage imperative. Safety training datasets need to include injection examples across the full space of attack vectors, formulations, languages, and content types that the model will encounter in production. Sparse or template-based adversarial datasets produce models that pass safety evaluations designed around the same templates while remaining vulnerable to novel attack formulations. Genuine robustness requires genuine diversity.

Direct Injection Coverage

Direct injection training data needs to cover the major attack categories and their variations. Role-play and persona framing attacks need to be represented across a range of persona descriptions and framing contexts, not just the most obvious formulations. Token-level manipulation attacks, including Unicode tricks, whitespace injection, and encoding manipulation, need to be included because pattern-matching defenses that operate on surface text will miss them. Instruction override attempts need to be represented in direct and indirect formulations, with and without technical language. Data collection and curation services that build adversarial datasets through structured red teaming rather than template generation produce coverage that reflects how attacks actually appear in production.

Indirect Injection Coverage by Content Type

Indirect injection training data needs to be organized by content type because the visual appearance and structural characteristics of injection attacks differ across documents, web pages, code, and structured data. An injection embedded in a PDF document looks different from one embedded in an HTML page, which looks different from one in a CSV row, which looks different from one in a code comment.

Each content type requires adversarial examples that reflect how injections are realistically embedded in that format. For documents, that means injections in headers, footers, hidden text fields, and metadata sections. For retrieved web content, that means injections in page elements that are processed but not prominently displayed. For code, that means injections in comments, variable names, and string literals. Coverage across content types is what produces a model robust to indirect injection in the actual contexts where it will be deployed.

Embedding Space and Multimodal Attacks

More capable models face a more sophisticated attack vector: adversarially crafted documents can be constructed such that their vector embeddings cluster near high-priority query embeddings in a retrieval index, causing them to be retrieved and processed even when they are semantically unrelated to the query. This exploits the retrieval layer rather than the generation layer and requires defenses at the data preparation and indexing stage rather than at the model level. LLMs that process images alongside text face an additional vector: adversarial content embedded in images that the vision component interprets as instructions. These attacks operate in a modality where human review is less effective as a quality control mechanism. Model evaluation services that include embedding space attack evaluation alongside text-level injection testing produce a more complete picture of the system’s actual attack surface.

What the Attack Surface Looks Like in Quantitative Terms

Benchmark data gives concrete shape to how serious the vulnerability is in practice. Across 13 LLM backbones evaluated in a comprehensive agent security benchmark, covering 10 prompt injection attack types across e-commerce, finance, and autonomous driving scenarios, the highest average attack success rate reached 84.30%, with current defenses showing limited effectiveness against sophisticated adversarial techniques. In a separate evaluation of goal-hijacking and prompt-extraction attacks drawn from a dataset of over 126,000 human-generated adversarial samples, even the most capable frontier models achieved only approximately 84% robustness to hijacking and approximately 69% robustness to prompt-extraction. Open-source and smaller models were substantially less resilient. Browser-centric agents can be partially hijacked by simple, human-written injections in up to 86% of evaluated cases.

Multi-layer defense architectures show measurable improvement. A combined approach including input validation, output monitoring, and an LLM-as-Critic evaluation layer reduced successful attack rates from 73.2% to 8.7% while maintaining 94.3% of baseline task performance. Adding the LLM-as-Critic output validation layer alone improved detection precision by 21% over input-only filtering approaches. These numbers define the gap that training data programs need to close: a safety fine-tuning approach that does not move the needle on attack success rate is not achieving what the data investment was intended to achieve, and measuring that gap explicitly is how programs know whether their adversarial training is working.

Annotation Requirements for Adversarial Safety Data

Classifying Injection by Attack Type and Severity

Raw red teaming outputs are not training-ready without structured annotation. Each adversarial input that produced a harmful model response needs to be classified by attack type, the specific mechanism it used to bypass safety training, and the severity of the resulting failure. Attack type classification enables targeted analysis of which defense strategies are most effective for which attack categories. Severity classification enables prioritization of training examples that represent the most consequential failures.

Annotation guidelines for injection classification need to distinguish between categories that require different defensive responses. A persona framing attack that elicits harmful content requires a different training signal than an indirect injection that executes an unauthorized action in an agentic context. Conflating these into a single failure category produces training data that does not give the model the specificity it needs to learn category-appropriate responses.

Pairing Attacks With Correct Refusal Responses

Every adversarial input that produced a harmful response needs to be paired with a human-written correct refusal response before it can be used as a safety training example. The quality of this pairing determines the quality of the training signal. An overly broad refusal response that incorrectly identifies the nature of the attack, or fails to explain why the request was declined, produces a model that refuses correctly in the training distribution but generalizes poorly to novel attack formulations.

The choice of alignment method for this pairing process has significant practical implications. RLHF using Proximal Policy Optimization requires training a separate reward model on human preference data, then using that reward model to provide feedback during reinforcement learning fine-tuning of the policy. This pipeline is powerful but expensive: it requires maintaining multiple models simultaneously, introduces training instability, and involves numerous hyperparameters requiring careful tuning. Direct Preference Optimization reformulates the alignment objective as a classification task over preference pairs. The DPO loss optimizes the log-probability ratio of the policy model relative to a reference model for chosen versus rejected responses, weighted by a temperature hyperparameter beta that controls how aggressively the model is pushed toward preferred outputs. For safety fine-tuning programs with bounded annotation budgets and specific injection defense objectives, DPO is generally preferred: it operates within standard supervised fine-tuning infrastructure, eliminates the need for a separately trained reward model, and is more stable than PPO-based RLHF.

The beta hyperparameter in DPO controls a trade-off that annotation programs need to understand before configuring fine-tuning runs. Low beta values push the model aggressively toward preferred outputs but risk reducing diversity and creating over-confident refusals that reject legitimate inputs. High beta values keep the model behavior closer to the reference model, producing smaller safety improvements but less over-refusal. Calibrating beta for injection defense training requires evaluating both attack success rate reduction and legitimate-request acceptance rate at multiple beta values before committing to a production fine-tuning run.

Human preference optimization workflows that include structured comparison annotation, where human evaluators judge model responses to adversarial inputs against human-written refusals, produce the preference signal that trains the model to generalize its refusal behavior rather than memorize specific attack-refusal pairs.

Refusal Calibration: The Over-Refusal Problem

Safety fine-tuning without calibration produces a systematic failure mode that is as damaging to deployment as insufficient safety coverage: over-refusal. A model trained on adversarial examples without carefully constructed negative examples of legitimate-but-superficially-similar inputs learns an overly broad decision boundary. It refuses requests that mention topics adjacent to the safety training distribution, even when those requests are entirely legitimate. This degrades utility in exactly the domains where safety investment was highest, because those are the domains with the densest adversarial training data.

Measuring over-refusal requires evaluation on a held-out set of legitimate inputs that are semantically similar to the adversarial training distribution but represent valid use cases. The over-refusal rate, the fraction of legitimate inputs refused by the safety-tuned model, should be tracked alongside the attack success rate reduction as complementary metrics. A safety fine-tuning run that reduces attack success rate from 80% to 15% but increases over-refusal rate from 2% to 25% has not produced a deployable model. Preference data for injection defense training needs to include explicit examples of legitimate requests that should not be refused, paired with appropriate helpful responses, so the model learns to discriminate between adversarial framing and superficially similar legitimate framing rather than refusing the entire adjacent region of the input space.

Inter-Annotator Consistency for Adversarial Data

Adversarial annotation has higher inter-annotator consistency requirements than standard annotation because disagreement about whether a model response constitutes a failure produces contradictory training signals. If one annotator classifies a model response as a successful injection and another classifies the same response as an acceptable output, the conflicting labels cancel each other rather than contributing to robustness.

Annotation guidelines for adversarial data need to provide explicit decision criteria for ambiguous cases: model responses that partially comply with an injection, responses that refuse the explicit harmful content but reveal information the injection was designed to extract, and responses that appear safe but establish context enabling follow-up attacks. These are precisely the cases where inconsistent labeling is most likely and where the training signal is most important to get right.

The Iterative Safety Training Loop

Why One Round of Adversarial Training Is Not Enough

Fine-tuning a model on an adversarial dataset does not produce a model robust to all future injection attempts. It produces a model more robust to the specific attack patterns represented in that dataset. Adversaries adapt. New attack formulations emerge. Fine-tuning the model for new capabilities can inadvertently reduce its robustness to injection patterns it previously handled correctly, a phenomenon known as safety regression.

Effective safety programs treat adversarial training as an iterative loop: red team the current model, curate and annotate the failures that emerge, fine-tune on the expanded adversarial dataset, re-evaluate to verify patched failure modes are addressed and the fine-tuning has not introduced new regressions, and repeat. Each cycle produces a model with better coverage of the attack space than the last, and the red teaming in each cycle becomes more targeted as the team learns which attack categories the model is most vulnerable to.

Safety Regression Testing After Fine-Tuning

Every fine-tuning operation, whether for safety improvement or capability extension, needs to be followed by regression testing against the full set of previously identified injection vulnerabilities. Domain fine-tuning that makes the model more capable in a specific context can inadvertently reduce its robustness to injection attacks it previously handled correctly. This happens because fine-tuning shifts the model’s behavior distribution, and the shift may move the model closer to complying with attack formulations it was previously robust to. Model evaluation services that maintain structured regression test suites across attack categories give safety programs the ability to detect and correct regressions before the model reaches production.

How Digital Divide Data Can Help

Digital Divide Data supports enterprise AI safety programs across the full adversarial data lifecycle, from red teaming and failure mode annotation through safety fine-tuning and regression evaluation. For programs building adversarial training datasets, trust and safety solutions cover structured red teaming across direct injection, indirect injection, multi-turn, and multimodal attack categories, with annotation that classifies failures by attack type, severity, and required defensive response.

For programs building the preference data that safety fine-tuning requires, human preference optimization services provide structured comparison annotation where human evaluators judge model responses to adversarial inputs, producing the preference signal that trains the model to generalize refusal behavior across novel attack formulations. For programs evaluating injection robustness before deployment and after fine-tuning updates, model evaluation services design adversarial evaluation suites that cover the full attack surface, including regression test suites that verify safety fine-tuning has not introduced new vulnerabilities.

Build adversarial training data that reflects the actual attack surface your production system will face. Talk to an expert.

Conclusion

Prompt injection robustness is not a property that safety fine-tuning delivers once and retains indefinitely. It is a coverage problem that requires continuous investment in adversarial data diversity, annotation quality, and iterative evaluation. The models that are most robust to injection attacks are the ones trained on the most diverse and accurately annotated adversarial datasets, not the ones fine-tuned on the largest set of the same attack patterns.

The attack surface for production LLM systems extends well beyond direct user input. Indirect injection through processed content, multi-turn context manipulation, agentic exploitation, and embedding space attacks all require specific coverage in the adversarial training data. Programs that build safety training datasets around the full attack surface are the ones that produce deployments with genuine injection robustness. Trust and safety solutions built on that discipline are what separate systems that are safe under adversarial pressure from systems that only appear safe until someone looks carefully.

References

OWASP Foundation. (2025). LLM01:2025 prompt injection. OWASP GenAI Security Project. https://genai.owasp.org/llmrisk/llm01-prompt-injection/

Yi, J., Xie, Y., Zhu, B., Kiciman, E., Sun, G., Xie, X., & Wu, F. (2025). Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (pp. 1809–1820). ACM. https://doi.org/10.1145/3690624.3709179

Chen, C. et al. (2025). The obvious invisible threat: LLM-powered GUI agents’ vulnerability to fine-print injections. arXiv:2504.11281. https://arxiv.org/abs/2504.11281

Gulyamov, S., Gulyamov, S., Rodionov, A., Khursanov, R., Mekhmonov, K., Babaev, D., & Rakhimjonov, A. (2026). Prompt injection attacks in large language models and AI agent systems: A comprehensive review of vulnerabilities, attack vectors, and defense mechanisms. Information, 17(1), 54. https://doi.org/10.3390/info17010054

Zhang, H., Chen, W., Huang, F., Li, M., Zakar, O., Cohen, R., Zhu, S., & Qiu, X. (2025). Agent Security Bench (ASB): Formalizing and benchmarking attacks and defenses in LLM-based agents. In Proceedings of ICLR 2025. https://arxiv.org/abs/2410.02644

Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., & Finn, C. (2024). Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, 36. https://arxiv.org/abs/2305.18290

Frequently Asked Questions

Q1. What is the difference between direct and indirect prompt injection?

Direct injection is when a user provides adversarial input that attempts to override system instructions in the prompt itself. Indirect injection is when malicious instructions are embedded in external content that the model processes during a task, such as a document it summarizes, a web page it retrieves, or an email it analyzes. Indirect injection is more dangerous because the user does not need to behave adversarially. The attack succeeds when the model does its job.

Q2. Why are pattern-matching defenses insufficient for injection robustness?

Because adversaries adapt their formulations to bypass known filters, often operating at a layer below what those filters inspect. Token smuggling using zero-width Unicode characters is invisible to filters that check rendered text but present in the token stream the model processes. A pattern-matching defense that blocks a specific injection template does not block variations using different encoding or structural presentation to achieve the same effect. Genuine robustness requires training the model to recognize the intent and mechanism of injection attacks across novel formulations, not just to match text patterns associated with known attacks.

Q3. What content types need to be covered in indirect injection training data?

Every content type the model processes in production: documents in various formats, retrieved web content, code, structured data like CSV and JSON, and, for multimodal systems, images. Each content type requires adversarial examples that reflect how injections are realistically embedded in that format, because the structural presentation of an injection in a PDF header looks different from one in an HTML element or a code comment, and the model needs to have encountered both to be robust to both.

Q4. What is the difference between DPO and RLHF for safety fine-tuning, and which should programs use?

RLHF using PPO requires a separately trained reward model and reinforcement learning-based policy optimization, which is powerful but expensive, training-unstable, and requires significant engineering infrastructure. DPO reformulates the alignment objective as a classification over preference pairs, optimizing the log-probability ratio of chosen versus rejected responses relative to a reference model, weighted by a temperature hyperparameter beta. For bounded-budget safety fine-tuning programs focused on injection defense, DPO is generally preferred because it operates within standard supervised fine-tuning infrastructure and is more stable. The beta hyperparameter needs to be calibrated jointly against attack success rate reduction and over-refusal rate, because aggressive safety tuning at low beta can produce a model that refuses legitimate inputs that share surface features with the adversarial training distribution.

Q5. How does safety regression occur after fine-tuning, and how can it be detected?

Safety regression happens when fine-tuning for a new capability shifts the model’s behavior distribution in a way that reduces its robustness to injection patterns it previously handled correctly. The model effectively forgets some of its safety training when it learns new capabilities. Detecting regression requires running the complete set of previously identified injection vulnerabilities against the fine-tuned model before deployment, not just evaluating the new capabilities the fine-tuning was intended to add.

A generative AI model does not reveal its failure modes in normal operation. Standard evaluation benchmarks measure what a model does when it receives well-formed, expected inputs. They say almost nothing about what happens when the inputs are adversarial, manipulative, or designed to bypass the model’s safety training. The only way to discover those failure modes before deployment is to deliberately look for them. That is what red teaming does, and it has become a non-negotiable step in the safety workflow for any GenAI system intended for production use.

In the context of large language models, red teaming means generating inputs specifically designed to elicit unsafe, harmful, or policy-violating outputs, documenting the failure modes that emerge, and using that evidence to improve the model through additional training data, safety fine-tuning, or system-level controls. The adversarial inputs produced during red teaming are themselves a form of training data when they feed back into the model’s safety tuning.

This blog examines how red teaming works as a data discipline for GenAI, with trust and safety solutions and model evaluation services as the two capabilities most directly implicated in operationalizing it at scale.

Key Takeaways

• Red teaming produces the adversarial data that safety fine-tuning depends on. Without it, a model is only as safe as the scenarios its developers thought to include in standard training.
• Effective red teaming requires human creativity and domain knowledge, not just automated prompt generation. Automated tools cover known attack patterns; human red teamers find the novel ones.
• The outputs of red teaming, documented attack prompts, model responses, and failure classifications, become training data for safety tuning when curated and labeled correctly.
• Red teaming is not a one-time exercise. Models change after fine-tuning, and new attack techniques emerge continuously. Programs that treat red teaming as a pre-launch checkpoint rather than an ongoing process will accumulate safety debt.

Build the adversarial data and safety annotation programs that make GenAI deployment safe rather than just optimistic.

What Red Teaming Actually Tests

The Failure Modes Standard Evaluation Misses

Standard model evaluation measures performance on defined tasks: accuracy, fluency, factual correctness, and instruction-following. What it does not measure is robustness under adversarial pressure. The overview by Purpura et al. characterizes red teaming as proactively attacking LLMs with the purpose of identifying vulnerabilities, distinguishing it from standard evaluation precisely because its goal is to find what the model does wrong rather than to confirm what it does right. Failure modes that only appear under adversarial conditions include jailbreaks, where a model is induced to produce content its safety training should prevent; prompt injection, where malicious instructions embedded in user input override system-level controls; data extraction, where the model is induced to reproduce sensitive training data; and persistent harmful behavior that reappears after safety fine-tuning.

These failure modes matter operationally because they are the ones real-world adversaries will target. A model that performs well on standard benchmarks but succumbs to straightforward jailbreak techniques is not actually safe for deployment. The gap between benchmark performance and adversarial robustness is precisely the space that red teaming is designed to measure and close.

Categories of Adversarial Input

Red teaming for GenAI produces inputs across several categories of attack. Direct prompt injections attempt to override the model’s system instructions through user input. Jailbreaks use persona framing, fictional scenarios, or emotional manipulation to induce the model to bypass its safety training. Multi-turn attacks build context across a conversation to gradually shift model behavior in a harmful direction. Data extraction probes attempt to get the model to reproduce memorized training content. Indirect injections embed adversarial instructions within documents or retrieved content that the model processes. 

How Red Teaming Produces Training Data

From Attack to Dataset

The outputs of a red teaming exercise have two uses. First, they reveal where the model currently fails, informing decisions about deployment readiness, system-level controls, and the scope of additional training. Second, when curated and annotated correctly, they become the adversarial training examples that safety fine-tuning requires. A model cannot learn to refuse a jailbreak it has never been trained to recognize. The red teaming process generates the specific failure examples that the safety training data needs to include.

The curation step is critical and is where red teaming intersects directly with data quality. Raw red teaming outputs, attack prompts, and model responses need to be reviewed, classified by failure type, and annotated to indicate the correct model behavior. An attack prompt that produced a harmful response needs to be paired with a refusal response that correctly handles it. That pair becomes a safety training example. The quality of the annotation determines whether the safety training actually teaches the model what to do differently, or simply adds noise. Building generative AI datasets with human-in-the-loop workflows covers how iterative human review is structured to convert raw adversarial outputs into training-ready examples.

The Role of Diversity in Adversarial Datasets

The most common failure in red teaming programs is insufficient diversity in the adversarial examples generated. If all the jailbreak attempts follow similar patterns, the safety training data will be dense around those patterns but sparse around the full space of adversarial inputs the model will encounter in production. A model trained on a narrow set of attack patterns learns to refuse those specific patterns rather than learning generalized robustness to adversarial pressure. Effective red teaming programs deliberately vary the attack vector, framing, cultural context, language, and level of directness across their adversarial examples to produce safety training data with genuine coverage.

Human Red Teamers vs. Automated Approaches

What Automated Tools Can and Cannot Do

Automated red teaming tools generate adversarial inputs at scale by using one model to attack another, applying known jailbreak templates, fuzzing prompts systematically, or combining attack techniques programmatically. These tools are valuable for covering large input spaces rapidly and for regression testing after safety updates to check that previously patched vulnerabilities have not reappeared. The Microsoft AI red team’s review of over 100 GenAI products notes that specialist areas, including medicine, cybersecurity, and chemical, biological, radiological, and nuclear risks, require subject-matter experts rather than automated tools because harm evaluation itself requires domain knowledge that LLM-based evaluators cannot reliably provide.

Automated tools are also limited to the attack patterns they have been programmed or trained to generate. The most novel and damaging attack techniques tend to be discovered by human red teamers who approach the model with genuine adversarial creativity rather than systematic application of known patterns. A program that relies entirely on automation will develop good defenses against known attacks while remaining vulnerable to the class of novel attacks that automated systems did not anticipate.

Building an Effective Red Team

Effective red teaming programs combine specialists with different skill profiles: people with security backgrounds who understand attack methodology, domain experts who can evaluate whether a model response is genuinely harmful in the relevant context, people with diverse cultural and linguistic backgrounds who can identify failures that appear in non-English or non-Western cultural contexts, and generalists who approach the model as a motivated but non-expert adversary would. The diversity of the red team determines the coverage of the adversarial dataset. A red team drawn from a narrow demographic or professional background will produce adversarial examples that reflect their particular perspective on what constitutes a harmful input, which is systematically narrower than the full range of inputs the deployed model will encounter.

Red Teaming and the Safety Fine-Tuning Loop

How Adversarial Data Feeds Back Into Training

The standard safety fine-tuning workflow treats red teaming outputs as one of the key data inputs. Adversarial examples that elicit harmful model responses are paired with human-written refusal responses and added to the safety training dataset. The model is then retrained or fine-tuned on this expanded dataset, and the red teaming exercise is repeated to verify that the patched failure modes have been addressed and that the patches have not introduced new failures. This iterative loop between adversarial discovery and safety training is sometimes called purple teaming, reflecting the combination of the offensive red team and the defensive blue team. Human preference optimization integrates directly with this loop: the preference data collected during RLHF includes human judgments of adversarial responses, which trains the model to prefer refusal over compliance in the scenarios the red team identified.

Safety Regression After Fine-Tuning

One of the most significant challenges in the red teaming loop is safety regression: fine-tuning a model on new domain data or for new capabilities can reduce its robustness to adversarial inputs that it previously handled correctly. A model safety-tuned at one stage of development may lose some of that robustness after subsequent fine-tuning for domain specialization. This means red teaming is needed not just before initial deployment but after every significant fine-tuning operation. Programs that run red teaming once and then repeatedly fine-tune the model without re-testing are building up safety debt that will only become visible after deployment.

How Digital Divide Data Can Help

Digital Divide Data provides adversarial data curation, safety annotation, and model evaluation services that support red teaming programs across the full loop from adversarial discovery to safety fine-tuning.

For programs building adversarial training datasets, trust and safety solutions cover the annotation of red teaming outputs: classifying failure types, pairing attack prompts with correct refusal responses, and quality-controlling the adversarial examples that feed into safety fine-tuning. Annotation guidelines are designed to produce consistent refusal labeling across adversarial categories rather than case-by-case human judgments that vary across annotators.

For programs evaluating model robustness before and after safety updates, model evaluation services design adversarial evaluation suites that cover the range of attack categories the deployed model will face, stratified by attack type, cultural context, and domain. Regression testing frameworks verify that safety fine-tuning has addressed identified failure modes without degrading model performance on legitimate use cases.

For programs building the preference data that RLHF safety tuning requires, human preference optimization services provide structured comparison annotation where human evaluators judge model responses to adversarial inputs, producing the preference signal that trains the model to prefer safe behavior under adversarial pressure. Data collection and curation services build the diverse adversarial input sets that coverage-focused red teaming programs need.

Build adversarial data and safety-annotation programs that make your GenAI deployment truly safe. Talk to an expert.

Conclusion

Red teaming closes the gap between what a model does under normal conditions and what it does when someone is actively trying to make it fail. The adversarial data produced through red teaming is not incidental to safety fine-tuning. It is the input that safety fine-tuning most depends on. A model trained without adversarial examples will have unknown safety properties under adversarial pressure. A model trained on a well-curated, diverse adversarial dataset will have measurable robustness to the failure modes that dataset covers. The quality of the red teaming program determines the quality of that coverage.

Programs that treat red teaming as an ongoing discipline rather than a pre-launch checkbox build cumulative safety knowledge. Each red teaming cycle produces better adversarial data than the last because the team learns which attack patterns the model is most vulnerable to and can design the next cycle to probe those areas more deeply. The compounding effect of iterative red teaming, safety fine-tuning, and re-evaluation is a model whose adversarial robustness improves continuously rather than degrading as capabilities grow. Building trustworthy agentic AI with human oversight examines how this discipline extends to agentic systems where the safety stakes are higher, and the adversarial surface is larger.

References

Purpura, A., Wadhwa, S., Zymet, J., Gupta, A., Luo, A., Rad, M. K., Shinde, S., & Sorower, M. S. (2025). Building safe GenAI applications: An end-to-end overview of red teaming for large language models. In Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025) (pp. 335-350). Association for Computational Linguistics. https://aclanthology.org/2025.trustnlp-main.23/

Microsoft Security. (2025, January 13). 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog. https://www.microsoft.com/en-us/security/blog/2025/01/13/3-takeaways-from-red-teaming-100-generative-ai-products/

OWASP Foundation. (2025). OWASP top 10 for LLM applications. OWASP GenAI Security Project. https://genai.owasp.org/

European Parliament and Council of the European Union. (2024). Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union. https://artificialintelligenceact.eu/

Frequently Asked Questions

Q1. What is the difference between red teaming and standard model evaluation?

Standard evaluation measures model performance on expected, well-formed inputs. Red teaming specifically generates adversarial, manipulative, or policy-violating inputs to find failure modes that only appear under adversarial conditions. The goal of red teaming is to find what goes wrong, not to confirm what goes right.

Q2. How do red teaming outputs become training data?

Attack prompts that produce harmful model responses are paired with human-written correct refusal responses and added to the safety fine-tuning dataset. The model is then retrained on this expanded dataset, and the red teaming exercise is repeated to verify that patched failure modes have been addressed without introducing new ones.

Q3. Can automated red teaming tools replace human red teamers?

Automated tools are valuable for scale and regression testing, but are limited to known attack patterns. Human red teamers find novel attack methods that automated systems did not anticipate. Effective programs combine automated coverage of known attack patterns with human creativity for novel discovery. Domain-specific harms in medicine, cybersecurity, and other specialist areas require human expert evaluation that automated tools cannot reliably provide.

Q4. How often should red teaming be conducted?

Red teaming should be conducted before initial deployment and after every significant fine-tuning operation, because domain fine-tuning can reduce safety robustness. Programs that treat red teaming as a one-time pre-launch activity accumulate safety debt as the model is updated over its deployment lifetime.

---