AP30 — AI Model Evaluation, Data, and Statistical Interpretation

Learning index and sources

• IPA scope A: Major Category 1 Basic Theory → Middle Category 1 Basic Theory → 3 Information Theory → (9) AI Technologies → ② Machine Learning, PDF page 10 / printed page 5.
• IPA scope B: Major Category 9 Corporate and Legal Affairs → Middle Category 22 Corporate Activities → 2 Business Analysis and Data Utilization → (2) Data Utilization → ② Data Processing and Analysis and ③ Statistical Methods in Data Analysis, PDF page 130 / printed page 126.
• Scope sources (checked 2026-10-21): IPA syllabus PDF · overview · revision information.
• Revision history: 2026-10-28 corrected the next-installment note so AP31 is described by its actual scope, law and standardization.
• Supplementary official technical references: scikit-learn: data leakage · classification metrics. These explain implementation practice; the IPA syllabus above defines the exam scope.
• Prerequisite: AP3 probability and statistics. Use that installment for probability and descriptive-statistics foundations; they are not retaught here.
• Series context: the previous installment, AP29 accounting, break-even analysis, and investment appraisal, and every other installment are listed in the public series index.

This study guide uses only fictional aggregate data, claims no IPA affiliation or endorsement, and does not guarantee a pass.

1. Split data by purpose

• Training data fits model parameters, such as regression coefficients or tree splits.
• Validation data selects hyperparameters, models, features, and decision thresholds.
• An untouched test set estimates final generalization performance after development choices are fixed.

Cross-validation belongs inside development: split the development portion into folds, fit on the training folds, validate on the held-out fold, and aggregate the scores. Keep the final test set outside that loop. Repeatedly checking test results and changing the model turns the test set into validation data; it can no longer provide an unbiased final estimate.

Random splitting is not always safe. Put records from the same person, device, or organization in one side by using a grouped split. For forecasting or other temporal tasks, use a time-aware split in which training precedes evaluation, rather than letting future records inform the past.

2. Prevent overfitting and leakage

Overfitting occurs when a model learns training-specific noise: training error keeps falling while validation or test error stops improving or worsens. Regularization, simpler models, more representative data, and development-only cross-validation can help, but final performance still comes from the untouched test set.

Data leakage lets information unavailable at prediction time influence training. Examples include computing standardization values or selecting features on the entire dataset, including the target in an input feature, or using future information to predict an earlier event.

Use a safe workflow for every learned preprocessing step:

1. Split first.
2. Fit imputation, scaling, encoding, and feature selection on training data only.
3. Transform validation and test data with those fitted training values; never refit on them.
4. Fit the model on the transformed training data, choose settings with validation data, then evaluate once on test data.

A pipeline helps bind preprocessing and modeling so that each cross-validation fold learns its transformations only from its training fold.

3. Read a confusion matrix

Define the positive class explicitly: here, “requires inspection” is positive. For 100 cases:

Actual / predicted	Positive	Negative
Positive	TP = 24	FN = 8
Negative	FP = 6	TN = 62

Using unrounded counts and rounding only final percentages to two decimal places:

• Accuracy = (TP + TN) / N = (24 + 62) / 100 = 86.00%.
• Precision = TP / (TP + FP) = 24 / 30 = 80.00%. Its denominator is predicted positives.
• Recall (sensitivity) = TP / (TP + FN) = 24 / 32 = 75.00%. Its denominator is actual positives.
• Specificity = TN / (TN + FP) = 62 / 68 = 91.18%. Its denominator is actual negatives.
• F1 = 2 × precision × recall / (precision + recall) = 48 / 62 = 77.42%.

Changing the decision threshold usually trades precision against recall. Lowering it often catches more positives but creates more false positives. Choose the metric and threshold from the costs of false negatives and false positives, not from one universal rule. ROC curves compare true-positive and false-positive rates across thresholds; PR curves emphasize precision and recall and are often especially informative when the positive class is rare. AUC summarizes a curve but does not select the operational threshold or encode business costs.

4. Class imbalance can hide failure

Suppose 990 cases are negative and 10 are positive. A classifier that always predicts negative has 99.00% accuracy, but positive-class recall is 0.00% and F1 is 0.00%. Because it predicts no positives, TP + FP = 0 and precision is mathematically undefined. A report must state its convention—for example, “undefined (software configured to display 0 with a warning)”—rather than silently claiming that mathematical precision equals zero.

Therefore inspect class counts, the confusion matrix, per-class metrics, and threshold-dependent curves. Accuracy may still be useful, but not by itself when the minority error matters.

5. Interpret statistical evidence within its limits

A population is the group about which a claim is intended; a sample is the observed subset. Sampling uncertainty means another sample can give another estimate. More digits do not remove that uncertainty.

• Selection bias: inclusion in the data is related to the outcome, so the sample does not represent the target population.
• Information bias: measurements, labels, or records are systematically inaccurate.
• Confounding bias: a third factor affects both the apparent cause and outcome, distorting their association.

Correlation alone does not establish causation. A predictive model may exploit an association without showing that changing a feature will change the outcome. Conclusions cannot be generalized beyond the sampling and data-generation process without justification. Check who and what was measured, how labels were produced, what time period and environment apply, missingness, and whether deployment conditions differ.

Term	Canonical distinction
train / validation / test	fit parameters / choose settings and thresholds / final generalization estimate
overfitting / leakage	learning training-specific noise / using unavailable or evaluation information
precision / recall	correct among predicted positives / found among actual positives
ROC / PR	TPR-versus-FPR view / precision-versus-recall view
population / sample	intended claim domain / observed subset

Practice questions — all original

Question 1

Which workflow best protects the final test estimate from leakage?

1. Split first; fit preprocessing and the model on training data, tune with validation data, and evaluate once on untouched test data.
2. Standardize the full dataset, then split it into training and test data.
3. Check test accuracy after every hyperparameter change and retain the best result.
4. Select features on the full labeled dataset, then perform cross-validation.

Answer: 1

• 1: Correct. Evaluation data neither teaches preprocessing nor guides development.
• 2: Incorrect. Full-data standardization leaks evaluation-set statistics.
• 3: Incorrect. Reuse makes the test set part of model selection.
• 4: Incorrect. Feature selection must occur inside each training fold.

Question 2

For TP=24, FP=6, FN=8, and TN=62, with “requires inspection” positive, which pair is correct?

1. Precision 75.00%; recall 80.00%
2. Precision 80.00%; recall 75.00%
3. Accuracy 91.18%; specificity 86.00%
4. F1 80.00%; accuracy 77.42%

Answer: 2

• 1: Incorrect. It swaps precision and recall.
• 2: Correct. 24/(24+6)=80.00%, and 24/(24+8)=75.00%.
• 3: Incorrect. Accuracy is 86.00%; specificity is 91.18%.
• 4: Incorrect. F1 is 77.42%; accuracy is 86.00%.

Question 3

Among 1,000 cases, 10 are positive. What correctly describes an always-negative classifier?

1. Accuracy 99%, positive recall 0%, F1 0; precision is undefined because no positive is predicted.
2. Accuracy 1%, positive recall 99%, and precision 100%.
3. Accuracy 99% proves the model is suitable regardless of error costs.
4. Positive recall is undefined because there are ten actual positives.

Answer: 1

• 1: Correct. It classifies 990 negatives correctly but finds no positive; the reporting convention for undefined precision must be stated.
• 2: Incorrect. It reverses the result and invents positive predictions.
• 3: Incorrect. Accuracy conceals total minority-class failure.
• 4: Incorrect. Recall has denominator ten and equals zero.

Question 4

Which statement is statistically sound?

1. Correlation proves that changing one variable causes the other to change.
2. A large convenience sample automatically represents every future user.
3. A confounder can influence both an apparent cause and an outcome, distorting their association.
4. More decimal places eliminate sampling uncertainty.

Answer: 3

• 1: Incorrect. Association alone does not establish causation.
• 2: Incorrect. Size does not repair selection bias or domain shift.
• 3: Correct. This is the central confounding mechanism.
• 4: Incorrect. Display precision does not remove sampling uncertainty.

Provenance and next installment

All four questions are original. Past-question year, season, AP label, original time band, question number, translation/adaptation, and official-answer comparison are all N/A; no IPA past question is used.

Next: AP31 — Law and standardization.