A Practical Guide to QSAR Model Validation: Internal, Cross, and External Checks

A QSAR model that only fits the training table has not been validated. Internal fit, cross-validation, and an external set answer three different questions, and a paper needs all three. The thresholds below are the ones this note reports, from Golbraikh and Tropsha, Roy and coauthors, Chirico and Gramatica, and the QSARINS-style reporting built on them [1]–[5]. They are not a new cutoff set.

Fig. 1 · Three checks, all required

The OECD guidance and the papers at the end are the source of the three-tier frame [6]. A strong training $R^2$ with a weak external $R^2$ is a failed model. The assay-first note covers what has to be true before any of these numbers mean anything: a curated column and a test set that is scored once.

The three-tier validation framework

The table consolidates the standard parameters across internal, cross-validation, and external validation. Threshold values reflect widely used criteria in the literature [1]–[5].

Tier	Parameter	Threshold
Internal	$R^2_{tr}$	≥ 0.600
Internal	$R^2_{adj}$	close to $R^2_{tr}$
Internal	LOF (Friedman lack of fit)	lowest possible
Internal	$\mathrm{CCC}_{tr}$	> 0.800
Internal	$\mathrm{RMSE}_{tr}$, $\mathrm{MAE}_{tr}$	lowest possible
Internal	$\Delta K$ (QUIK rule)	≥ 0.05
Internal	$s$ (standard error of estimate)	lowest possible
Internal	$F$	highest possible
Cross-validation	$R^2_{cv}$ ($Q^2_{LOO}$)	> 0.500
Cross-validation	$\mathrm{RMSE}_{cv}$, $\mathrm{MAE}_{cv}$	lowest possible
Cross-validation	$\mathrm{CCC}_{cv}$	> 0.800
Cross-validation	$Q^2_{LMO}$	> 0.500
Cross-validation	$R^2_{Yscr}$, $Q^2_{Yscr}$	lowest possible
External	$\mathrm{RMSE}_{ex}$, $\mathrm{MAE}_{ex}$	lowest possible
External	$R^2_{ex}$	> 0.600
External	$Q^2_{F1}$, $Q^2_{F2}$, $Q^2_{F3}$	> 0.600
External	$\mathrm{CCC}_{ex}$	> 0.800 (0.85 as first proposed)
External	$(R^2 - R^2_0)/R^2$ or $(R^2 - R'^2_0)/R^2$	< 0.1
External	$k$ or $k'$ (slopes through the origin)	0.85 to 1.15

Key inequality to remember: $R^2 > Q^2$ and $\mathrm{RMSE}_{tr} < \mathrm{RMSE}_{cv}$, together with a low $\mathrm{RMSE}_{ex}$ and $\mathrm{MAE}_{ex}$ and a high $R^2_{ex}$, are all required at once for a reliable model.

Internal validation: necessary but not sufficient

Internal statistics describe how well your model fits the training data. $R^2_{tr}$ must be at least 0.600, but the more diagnostic check is the gap between $R^2_{tr}$ and $R^2_{adj}$. A large gap signals that some descriptors are contributing noise rather than signal and should be removed.

The lack-of-fit (LOF) statistic penalises overfitting more aggressively than $R^2$ alone. The concordance correlation coefficient $\mathrm{CCC}_{tr}$ above 0.800 is particularly informative because it jointly measures precision and accuracy, punishing models that are systematically biased even if they show high correlation.

$\Delta K$ is a lesser-known but useful check on the descriptors themselves. It is the QUIK rule of Todeschini and coworkers: the multivariate correlation of the descriptors plus the response, $K_{XY}$, should exceed the correlation among the descriptors alone, $K_{XX}$ [7]. QSARINS-style reports accept a model when $\Delta K = K_{XY} - K_{XX}$ is at least 0.05. A smaller gap means the descriptors are mostly explaining each other, a collinearity that $R^2$ hides.

Cross-validation: testing generalisability on training data

Cross-validation estimates predictive ability without touching the external set. $Q^2_{LOO}$ above 0.500 is the minimum bar; many reviewers now expect 0.600 or above for publication. Leave-many-out ($Q^2_{LMO}$) is more conservative and more reliable for small datasets, where removing a single compound can give optimistic results.

Y-scrambling is often overlooked but should be reported in every paper. If $R^2_{Yscr}$ and $Q^2_{Yscr}$ are not close to zero, your model is capturing artefacts in the data structure rather than a true structure–activity relationship.

External validation: the real test

Compounds in the external set were never seen during training. This makes external metrics the gold standard. $R^2_{ex}$ above 0.600 and $\mathrm{CCC}_{ex}$ above 0.800 are the headline numbers. Chirico and Gramatica proposed 0.85 for CCC when they introduced it for QSAR [3]; 0.800 is the looser cut many QSARINS papers report. The Golbraikh–Tropsha criteria add geometric checks on the predicted-versus-observed line: a slope through the origin, $k$ or $k'$, between 0.85 and 1.15, and an $R^2$ that barely changes when the line is forced through the origin [1].

The three external $Q^2$ variants differ in their reference. $Q^2_{F1}$ measures test-set errors against the training-set mean, $Q^2_{F2}$ against the test-set mean (Schüürmann and coworkers), and $Q^2_{F3}$ against the training-set mean but scaled by the training-set variance (Consonni, Ballabio and Todeschini) [4], [8]. Reporting all three gives reviewers and regulators a more complete picture than any single metric alone.

Practical checklist before submission

Before submitting a QSAR paper, run through each tier. A model that passes internal checks but fails cross-validation almost certainly overfits. A model that passes cross-validation but fails external validation likely has a biased or unrepresentative training set, or the applicability domain has not been properly defined. Only models that clear all three tiers at once warrant regulatory or prospective use.

Finally, always report the full set of metrics rather than cherry-picking the ones that look best, and show the plot of predicted against observed values: Chirico and Gramatica’s follow-up found that high statistics can hide unacceptable predictions that the scatter plot shows at once [5]. Reviewers familiar with this framework will notice missing statistics, and missing Y-scrambling results in particular.

References

1. Golbraikh A, Tropsha A. Beware of q2! J Mol Graph Model 2002;20:269–276. doi:10.1016/S1093-3263(01)00123-1.
2. Roy K, Das RN, Ambure P, Aher RB. Be aware of error measures. Further studies on validation of predictive QSAR models. Chemom Intell Lab Syst 2016;152:18–33. doi:10.1016/j.chemolab.2016.01.008.
3. Chirico N, Gramatica P. Real external predictivity of QSAR models: how to evaluate it? Comparison of different validation criteria and proposal of using the concordance correlation coefficient. J Chem Inf Model 2011;51:2320–2335. doi:10.1021/ci200211n.
4. Consonni V, Ballabio D, Todeschini R. Comments on the definition of the Q2 parameter for QSAR validation. J Chem Inf Model 2009;49:1669–1678. doi:10.1021/ci900115y.
5. Chirico N, Gramatica P. Real external predictivity of QSAR models. Part 2. New intercomparable thresholds for different validation criteria and the need for scatter plot inspection. J Chem Inf Model 2012;52:2044–2058. doi:10.1021/ci300084j.
6. OECD. Guidance Document on the Validation of (Quantitative) Structure-Activity Relationship [(Q)SAR] Models. ENV/JM/MONO(2007)2.
7. Todeschini R, Consonni V, Mauri A, Pavan M. Detecting “bad” regression models: multicriteria fitness functions in regression analysis. Anal Chim Acta 2004;515:199–208. doi:10.1016/j.aca.2003.12.010.
8. Schüürmann G, Ebert RU, Chen J, Wang B, Kühne R. External validation and prediction employing the predictive squared correlation coefficient: test set activity mean vs training set activity mean. J Chem Inf Model 2008;48:2140–2145. doi:10.1021/ci800253u.