How Researchers Deliver OECD-Aligned, QMRF-Ready QSAR Model Validation

A QSAR model is not validated until it passes internal robustness checks, external validation on an independent test set, and per-prediction applicability-domain checks, following the OECD's five model-validation principles. Run cross-validation and y‑scrambling first. Then hold out a genuinely independent test set you never touched during feature selection or tuning. Finally, define your applicability domain and check every new prediction against it before you report a number, and prepare your QMRF fields as you go rather than after the fact.

TL;DR:

• Internal validation methods like k-fold cross-validation, leave-one-out, and y‑scrambling are essential for assessing model robustness, but do not replace external validation with independent data.
• External test sets must be separated before descriptor filtering or feature selection to avoid data leakage, with splitting strategies tailored to dataset size and structure, such as Kennard–Stone or cluster-based splits.
• The applicability domain should be checked for each prediction using methods like distance or leverage, and flagged if a compound falls outside the defined boundary, ensuring prediction reliability.
• Validation metrics should include multiple measures such as external R² or Q²F2, RMSE, MAE, CCC, and y‑scrambling distribution, with no single statistic sufficient for model assessment.
• Proper data curation, detailed documentation, and reproducible workflows are critical for credible validation, enabling effective review and regulatory acceptance.

Table of Contents

• What Does QSAR Model Validation Actually Require?
• Internal Validation Methods: Which To Run and When
• External Validation and Independent Test Sets: Getting It Right
• Applicability Domain: Methods and Per-Prediction Checks
• Validation Metrics: What They Mean and Where They Mislead
• How Data Curation Problems Quietly Undermine Validation
• A Reproducible Validation Workflow You Can Follow Step by Step
• Practitioner Notes on Reproducible Validation
• Answering Reviewers: What Regulators and Referees Actually Ask
• Sources
• FAQ

What Does QSAR Model Validation Actually Require?

Validated does not mean "fits the training data well." It means the model survives three separate, sequential tests, and skipping any one of them is why so many published QSAR models fall apart on the first external dataset someone else throws at them.

The OECD guidance on (Q)SAR model validation sets five principles that every defensible model needs to satisfy:

• A defined endpoint. The biological or physicochemical property must be unambiguous, with the assay, units, and conditions specified, not a vague label like "activity."
• An unambiguous algorithm. The modeling method, descriptor set, and any preprocessing steps must be fully specified so someone else could rebuild the model from your description.
• A defined applicability domain. You need an explicit boundary describing the chemical space where the model's predictions are trustworthy, not an implicit assumption that it works everywhere.
• Appropriate goodness-of-fit, robustness, and predictivity measures. These are three distinct properties, not one number, and conflating them is one of the most common reporting errors in the field.
• A mechanistic interpretation, where possible. Not mandatory in every case, but expected when the model informs a regulatory decision.

The Quantitative Model Reporting Format (QMRF) operationalizes these principles into a structured template covering endpoint definition, algorithm details, training/test set composition, internal and external performance statistics, and applicability domain description. Reviewers and regulators use it as a checklist, so populating it as you build the model, not after submission, saves considerable rework.

Layered on top of the 2007 principles, the (Q)SAR Assessment Framework adds prediction-assessment criteria for evaluating a specific prediction rather than the model overall: is the input structure correctly represented, does the compound fall inside the AD, is the underlying model reliable for this chemical class, and is the prediction fit for the intended regulatory or screening purpose. The QAF supplies Model, Prediction, and Result Checklists that map directly onto these four questions, and they're worth adopting even outside a regulatory submission because they force you to document reasoning you'd otherwise skip.

Internal Validation Methods: Which To Run and When

Internal validation answers one question: how much does performance degrade when the model sees data it wasn't trained on, within the same dataset? It's a robustness check, not proof of predictivity, and treating it as the latter is where a lot of otherwise careful modeling goes wrong.

1. k‑fold cross-validation for datasets above roughly 50 to 100 compounds. Ten folds is the conventional default; five folds work fine for smaller sets where ten would leave too few compounds per fold to be stable.
2. Leave-one-out (LOO) for very small datasets, under about 30 to 40 compounds, where k‑fold would waste too much data per fold. LOO tends to overestimate performance for larger sets, so don't default to it out of habit once your dataset grows.
3. Leave-many-out (LMO), removing 20 to 30 percent of compounds per iteration across many repetitions, as a middle ground that's less optimistic than LOO but more stable than a single k‑fold split on modest datasets.
4. Bootstrap resampling, drawing samples with replacement and repeating hundreds of times, when you need a confidence interval around a metric rather than a single cross-validated estimate.
5. Y‑scrambling (also called response permutation testing), randomly shuffling the target property while keeping descriptors fixed, then refitting the model. A model with genuine signal should collapse toward random performance; if a scrambled model still fits reasonably well, you've built a model that's fitting descriptor noise, not chemistry.

Run y‑scrambling across many permutations as a practical compromise between statistical confidence and compute cost, and report the distribution of scrambled R² or Q² values next to your real model's score rather than a single scrambled run. A wide gap between the real score and the scrambled distribution is the signal you're looking for.

Pro Tip: Don't run y‑scrambling once and call it done. A single permutation can land anywhere in the distribution by chance. Run the full set, plot the histogram, and report where your real model's metric falls relative to it.

Method choice interacts with model type in a way that trips up a lot of researchers. Goodness-of-fit and robustness track closely together for linear methods like multiple linear regression, but they decouple for nonlinear methods such as artificial neural networks and support vector regression. A random forest or SVR model can show excellent fit statistics while its cross-validated robustness is mediocre, because these methods have enough flexibility to fit noise convincingly. That's exactly why internal validation for flexible, nonlinear models needs to lean harder on resampling and chance-correlation checks, and lighter on raw fit metrics.

None of this replaces external testing. Cross-validation and bootstrapping are foundational for assessing robustness, but they operate on data drawn from the same pool the model was built on, which means they can't tell you how the model behaves on chemistry it has genuinely never encountered. Internal validation exists to guide hyperparameter choices and catch obvious chance correlations before you spend an external test set. It is a gate, not a finish line.

External Validation and Independent Test Sets: Getting It Right

An external test set is chemistry the model has never seen in any form, not during training, not during feature selection, and not during hyperparameter tuning. That last part is where most leakage actually happens, and it's subtler than reusing training compounds by mistake.

The distinction from internal cross-validation folds matters more than it sounds. A CV fold is held out for one iteration and then folded back into the ensemble of training runs; the overall modeling pipeline, including descriptor selection, still saw the full dataset before any fold was created. A true external set is separated before descriptor filtering, before feature selection, before you even look at correlations. If you calculate variable importance or drop low-variance descriptors using the full dataset and only then split into train and test, you've already leaked information across the boundary.

Splitting strategies compared

Random splitting is the fastest option and works reasonably well on large, chemically diverse datasets. It fails on small or clustered datasets because near-duplicate compounds can land on both sides of the split, inflating apparent predictivity without testing anything meaningful.

Kennard–Stone splitting selects training points to maximize coverage of descriptor space, then assigns the remainder to test. It's the better default when your dataset has structural clusters, because it forces the test set to include chemistry genuinely different from the training set rather than near-neighbors of it.

Cluster-based or stratified splitting groups compounds by scaffold or activity class first, then samples proportionally from each group into train and test. This is the right call when you need the test set to represent every activity class or chemotype at roughly the same ratio as the full dataset, particularly for imbalanced datasets where one class dominates.

Time-split or prospective validation holds out compounds synthesized or tested after a cutoff date, evaluating the model against genuinely future data. It's the closest thing to a real-world deployment test and the gold standard when historical data allows it, though it's rarely available for smaller academic datasets.

Common leakage points to watch for:

• Selecting descriptors or filtering low-variance features using the entire dataset before splitting.
• Reusing the same external test set repeatedly across multiple model iterations, which turns it into a de facto validation set and inflates reported performance over time.
• Standardizing or scaling descriptors using statistics (mean, standard deviation) calculated across the full dataset instead of the training set alone.
• Allowing near-duplicate compounds, salts, or stereoisomers of the same parent structure to appear in both train and test.
• Treating repeated experimental measurements of the same compound as independent data points during splitting.

Applicability Domain: Methods and Per-Prediction Checks

The applicability domain is the region of chemical space where a model's predictions carry the reliability its validation statistics claim. Outside that region, a prediction is not scientifically defensible for regulatory or discovery decisions, regardless of how good the model's headline metrics look, because you've extrapolated past the chemistry the model actually learned from.

OECD guidance explicitly ties reported performance to the AD used to generate it, and notes that narrowing the AD tends to raise measured performance simply because you've excluded the harder-to-predict edge cases. That's a real trap: a model can look better on paper purely because its AD shrank, not because the underlying chemistry understanding improved.

Common AD methods, in rough order of computational cost:

• Distance-based or Tanimoto-similarity methods compare a new compound's fingerprint or descriptor vector against the training set and flag it if it falls outside a similarity threshold. Cheap to compute, intuitive to explain, but sensitive to the descriptor or fingerprint choice.
• Leverage-based approaches (the Williams plot is the classic implementation) use the training descriptor matrix to compute a leverage value per compound; high leverage combined with a large residual flags an unreliable prediction. Works well for linear models, less cleanly for highly nonlinear ones.
• Density-based methods estimate how densely populated the local descriptor space is around a query compound, flagging low-density regions as outside the domain. More robust to descriptor choice than simple distance cutoffs, at higher compute cost.
• Conformal prediction and ensemble-variance approaches use the spread of predictions across an ensemble, or a formal conformal-prediction framework, to generate a per-compound confidence interval rather than a binary in/out flag. This is increasingly the preferred approach because it quantifies uncertainty directly rather than reducing it to yes/no.

Pro Tip: Run AD checks per prediction, not just once at the model level. A model can have a well-defined overall AD and still hand you an individual compound sitting right at its edge. Report the AD status alongside every prediction you deliver, not as a footnote in the methods section.

When a compound lands outside the AD, say so explicitly and report the prediction with a flag rather than silently delivering a number indistinguishable from a confident one. If confirmatory testing is the next step, sourcing well-characterized research peptides for benchtop follow-up is often faster than trying to force a prediction the model was never built to make.

Validation Metrics: What They Mean and Where They Mislead

No single metric tells the whole story, and treating R² as sufficient on its own is probably the single most common validation error in published QSAR work. Comparative analysis of five widely used external-validation criteria shows each has blind spots, which is why the recommendation across the literature has shifted from "report R²" to "report several metrics that fail in different ways."

• R² and Q²(LOO) measure how well the model explains variance internally; Q²(LOO) is the leave-one-out cross-validated version and is more conservative than raw R², but both are computed on training data and say nothing about external predictivity.
• RMSE and MAE give error magnitude in the original units of the endpoint, which is more interpretable to a bench chemist than a unitless correlation coefficient. MAE is less sensitive to outliers than RMSE, so reporting both flags whether a few bad predictions are dragging your RMSE up.
• Q²F2 (external Q²) applies the cross-validation logic to the external test set specifically, using the training set mean as the reference rather than the test set's own mean. It's a stricter, more honest measure of predictivity than a simple external R².
• CCC (concordance correlation coefficient) penalizes systematic bias, a model that's consistently high or low, in a way that a correlation coefficient alone doesn't catch. Two models can have identical R² with very different CCC if one has a systematic offset.

Sample size drives this harder than most researchers assume: validation statistics computed on fewer than 20 to 30 compounds are unstable enough that a single removed or added data point can swing Q² noticeably, and that instability is worse for flexible nonlinear models than for simple linear regression.

A practical minimum reporting set: internal Q²(LOO or CV), external R² or Q²F2, RMSE and MAE in original units, CCC, and the y‑scrambling comparison. Phrase conclusions against all five together, not against whichever one happened to look best. A model with strong Q²F2 but weak CCC is telling you something specific: it's ranking compounds correctly but shifted systematically, which matters enormously if you're using it to set an absolute potency cutoff rather than to rank candidates.

How Data Curation Problems Quietly Undermine Validation

Bad input data produces validation statistics that look fine and predictions that fail in the field, and the fix has to happen before modeling starts, not after.

1. Deduplicate aggressively. Remove exact structural duplicates and near-duplicates arising from different salt forms, tautomers, or stereochemistry annotations that describe the same underlying compound.
2. Harmonize units and assay conditions. IC50 values pulled from different assay formats, cell lines, or incubation times are not directly comparable, and merging them without normalization injects noise the model will try to explain with descriptors instead.
3. Resolve repeated measurements explicitly. When a compound has multiple experimental values, decide on a rule, median, geometric mean, or exclusion of outlier replicates, and apply it consistently rather than letting duplicates inflate the effective dataset size.
4. Capture assay metadata. Keep the assay type, species, and conditions attached to every data point so you can stratify or filter later if you spot an inconsistency.

Practitioner experience across QSAR groups consistently points to naive random splitting combined with unresolved duplicate measurements as the leading cause of validation statistics that look excellent on paper and collapse on the next real dataset. If the same compound (or a near-identical analog) appears in both train and test because of unrecognized duplication, your external validation isn't external anymore.

Simple diagnostics catch a lot of this before it becomes a published error: plot residuals against predicted value and look for structure (a fan shape usually means variance isn't constant across the range); check the error distribution for a suspiciously tight cluster near zero, which often means near-duplicates are inflating apparent accuracy; and compare performance on structurally novel test compounds specifically against performance on close analogs of training compounds. A large gap between those two groups is a leakage flag, not a fluke.

A Reproducible Validation Workflow You Can Follow Step by Step

Treat this as the sequence to run, not a menu to pick from selectively.

1. Define the endpoint precisely. Specify the assay, units, and any activity cutoffs before touching descriptors.
2. Curate the dataset. Deduplicate, harmonize assay conditions, and resolve repeated measurements as described above.
3. Split the data with a documented rationale. Choose random, Kennard–Stone, cluster-based, or time-split according to dataset structure, and write down why.
4. Run internal validation. Cross-validation or bootstrap for robustness, y‑scrambling across 50 to 100 permutations for chance-correlation checks.
5. Run external validation. Evaluate the untouched test set only after the model and its hyperparameters are finalized.
6. Define and apply the applicability domain. Choose a method appropriate to your model type and check every prediction, training and external, against it.
7. Report the full metric set and populate QMRF fields. Endpoint definition, algorithm description, AD boundaries, internal and external statistics, all in one document.

Reproducibility depends on details that are easy to skip under deadline pressure: fixed random seeds for every split and resampling step, recorded software versions for your modeling library and descriptor calculator, the exact descriptor filtering pipeline, and stored artifacts from the actual training run rather than a rerun summary written from memory.

For supplementary information aimed at reviewers or regulators, include the raw split assignments (which compound went to train versus test), the full metric table across all five recommended statistics, the y‑scrambling distribution plot, and the completed QMRF, not just a summary paragraph referencing it.

Practitioner Notes on Reproducible Validation

Reproducible-validation practice comes down to habits, not tools: fixing random seeds at every stage, keeping QMRF fields populated as you build rather than backfilling them before submission, and treating the applicability domain as a per-prediction question rather than a one-time model property. Innovabiotech's own modeling notes on reproducible QSAR validation workflows walk through seed control and QMRF-ready reporting in more depth than fits here.

A few recurring judgment calls worth flagging from practice:

• Kennard–Stone splitting earns its extra computational cost specifically when the dataset has visible structural clusters. On a chemically diverse, evenly distributed dataset, it barely outperforms random splitting.
• Leave-one-out cross-validation misleads more often than researchers expect once a dataset grows past 40 or 50 compounds. It tends to report overly optimistic Q² values precisely when a k‑fold approach would be more honest.
• Applicability domain checks matter more for virtual screening validation than for retrospective model benchmarking, because screening applies the model to compound libraries the training set never anticipated.

Answering Reviewers: What Regulators and Referees Actually Ask

Reviewers rarely challenge your model's architecture. They challenge your validation choices, and the questions are predictable enough to prepare for in advance.

Document your split rationale before anyone asks for it: why random versus Kennard–Stone, and what the descriptor-space coverage looked like on each side of the split. Have the y‑scrambling distribution ready as a figure, not a sentence, because "we ran y‑scrambling and it looked fine" invites a follow-up question every time. Show applicability domain checks per prediction, not as an aggregate model statistic, since that's the level regulators actually evaluate at under the QAF.

On scope: a model validated for ranking compounds in a screening campaign is not automatically acceptable for a regulatory read-across decision, and claiming otherwise is the fastest way to lose a reviewer's confidence. If your endpoint or chemical space touches anything with regulatory weight, prospective validation on genuinely new data is worth the wait before you commit the model to a decision it can't yet support.

Share code, descriptor pipelines, and a completed QMRF whenever the venue allows it. It costs little at submission time and it's the difference between a model someone can audit and one they simply have to trust.

— Hooman

Sources

• Principles of QSAR models validation: internal and external
• The relevance of goodness‑of‑fit, robustness and prediction validation categories of OECD‑QSAR principles with respect to sample size and model type
• Comparison of various methods for validity evaluation of QSAR models

FAQ

What Is the Difference Between Internal and External QSAR Validation?

Internal validation (cross-validation, bootstrapping) measures robustness using only training data, while external validation measures true predictivity on a genuinely independent test set the model never saw during training or tuning.

How Many Y-Scrambling Permutations Should I Run?

Run 50 to 100 permutations as a practical balance between statistical confidence and compute time, then report the distribution of scrambled scores alongside your real model's metric.

What Counts as Being Outside the Applicability Domain?

A compound falls outside the applicability domain when it sits beyond the chemical space boundary defined by methods like Tanimoto similarity, leverage, density, or conformal prediction, meaning the model's validated performance doesn't reliably extend to it.

Which Splitting Method Should I Use for a Small, Clustered Dataset?

Kennard–Stone or cluster-based splitting works better than random splitting on small or structurally clustered datasets, because it forces the test set to represent chemistry distinct from the training set rather than near-duplicates of it.

Is a Model Validated for Screening Acceptable for Regulatory Use?

Not automatically. A model validated for screening purposes may need additional assessment, including a more rigorous applicability domain check and possibly prospective validation, before it's acceptable for a regulatory decision.

What Should a QMRF Include Beyond Basic Model Description?

A complete QMRF documents the defined endpoint, algorithm and descriptor pipeline, training and test set composition, internal and external validation statistics, and the applicability domain, mapped against the OECD principles and QAF checklists.

Innovabiotech supports biotech and pharma teams that need computational modeling paired with confirmatory experimental follow-up, from protein engineering and chimeric protein design through custom peptide design and optimization once a validated model points toward a lead worth synthesizing.

Recommended

• Deep QSAR Modeling with Reproducible Validation for Medicinal Chemists
• Validating Virtual Screening Results: 2026 Best Practices