Build statistical judgment for actuarial risk modeling through 20 original selected-response questions and 60 substantive lessons. Practice choosing learning methods, calculating predictions, interpreting coefficients, diagnosing misspecification, and comparing models without confusing training fit with predictive performance. Worked examples connect regression, generalized linear models, time series, trees, principal components, and clustering to insurance applications. Explanations distinguish related quantities such as odds and probabilities, counts and rates, and confidence and prediction intervals. The domain allocation is editorial. Current official syllabus alignment and examination logistics have not been verified as of October 17, 2026; this representative preview is not a full-length mock examination.
What this guide covers
Build SRM modeling skills with 20 original questions and 60 lessons on prediction, regression, forecasting, and clustering.
Statistical learning foundations and predictive assessment
1. Responses determine supervised learning tasks
Supervised learning estimates a relationship between predictors and an observed response. A numerical response, such as claim severity, defines regression; a categorical response, such as claim occurrence, defines classification. Unsupervised learning instead investigates predictor structure without response labels, using methods such as clustering or principal components. The distinction depends on the objective and available response, not whether predictors are numerical or categorical. A segmentation can support later supervised modeling, but discovering groups alone does not demonstrate differences in future claims or establish predictive performance.
Worked example: An insurer has mileage, braking frequency, and nighttime driving share for each policyholder. Grouping similar driving profiles is unsupervised clustering. Predicting next year's claim indicator from those measurements becomes supervised classification once historical claim labels are available.
Common mistake: Numerical predictors do not make a task regression; the response type and intended prediction determine the supervised task.
2. Prediction accuracy and coefficient inference
Prediction asks how accurately a fitted procedure estimates outcomes for new observations. Inference asks what the data support about relationships, such as a coefficient's direction or magnitude under a specified model. A flexible model may predict well while providing no simple coefficient interpretation. Conversely, a clear regression coefficient may contribute little incremental predictive accuracy. Conditional association also differs from causation: controlling for measured predictors does not automatically remove confounding or selection effects. Match evaluation to the objective, using out-of-sample loss for prediction and suitable uncertainty calculations for inference.
Worked example: A renewal model uses hundreds of correlated transaction features and predicts cancellations accurately. A separate analysis asks how cancellation varies with tenure after controlling for premium. That question requires an interpretable model and uncertainty assessment, even if its predictive accuracy is slightly lower.
Common mistake: A statistically significant predictor is not automatically a useful predictor, and its coefficient does not automatically measure a causal effect.
3. Flexibility, structure, and interpretability
Model flexibility describes the range of relationships a method can represent. Additional polynomial terms, interactions, tree depth, or nonlinear basis functions can capture patterns a simple additive line misses. Flexibility can reduce approximation bias, but estimating more structure may increase sensitivity to the development sample. Interpretability depends on the representation: a short additive model offers direct conditional comparisons, while an ensemble often requires secondary summaries. Neither flexibility nor interpretability guarantees predictive quality. Choose structural complexity using the intended loss and validation design while checking whether explanations remain adequate for the application.
Worked example: Claim frequency increases sharply only when both annual mileage and nighttime driving are high. An additive model may miss that joint pattern. Adding an interaction or using trees can capture it, but the improvement should appear on appropriate validation observations.
Common mistake: A more flexible method does not automatically dominate a simpler model; sample size, noise, and the true relationship affect generalization.
4. Bias–variance decomposition of squared prediction error
At a fixed predictor value, expected squared prediction error decomposes into squared bias, variance of the fitted prediction across development samples, and irreducible response variance. Bias measures systematic departure from the true conditional mean. Estimation variance measures instability caused by changing the training data. Greater flexibility often lowers bias while raising variance, although the exact relationship depends on the method. Irreducible error remains even if the conditional mean is known. The decomposition averages over hypothetical training samples and new response noise; it does not separate one observed residual into three directly observable pieces.
Worked example: Suppose a prediction procedure has squared bias 4, estimation variance 3, and irreducible variance 9 at a particular risk profile. Its expected squared prediction error is 16. Reducing squared bias to 1 helps only if the associated variance increase stays below 3.
Common mistake: Irreducible error is not training failure; it represents outcome variation remaining even when the conditional mean is correctly known.
5. Training error, test error, and learning curves
Training error evaluates the observations used to estimate the model, so it generally understates prediction error on independent observations. For nested least-squares models fitted without penalties, adding predictors cannot increase minimized training RSS. Test error can nevertheless rise when added flexibility fits sample noise. High training and validation errors suggest underfitting or weak available information; low training error with substantially higher validation error suggests overfitting. Learning curves can plot error against complexity or sample size. Interpret their axes carefully, and compare models using the same loss and appropriately separated observations.
Worked example: A shallow tree has training MSE 18 and validation MSE 19. A very deep tree has training MSE 2 and validation MSE 27. The deeper tree fits the development observations better, but its larger validation loss indicates poorer generalization for this comparison.
Common mistake: A falling training error does not establish improved prediction, because the same observations helped determine the increasingly flexible fit.
6. Data leakage includes learned preprocessing
A final test set estimates performance after the modeling procedure has been chosen. Validation observations guide choices such as tuning parameters, transformations, and feature selection. Any preprocessing estimated from data belongs inside that procedure: imputation values, scaling constants, and selected variables must be learned without assessment observations. Reusing test results to revise the model converts the test set into development information. Splitting must also respect the prediction unit, so repeated policies or related observations may require grouped separation. The goal is to reproduce what information would actually be available for a new prediction.
Worked example: Before cross-validation, an analyst selects the ten variables most correlated with claim severity using the entire dataset. Every validation fold has influenced that selection. Repeating variable selection separately within each training fold removes this particular leakage source and evaluates the actual selection procedure.
Common mistake: Preprocessing is not automatically harmless; even outcome-free transformations can use information that should remain unavailable during validation or testing.
7. Validation-set, k-fold, and leave-one-out estimates
A validation-set estimate trains once on one subset and evaluates another, making it simple but sensitive to the split and reduced training size. K-fold cross-validation trains k times, holding out each fold in turn and aggregating validation losses. Leave-one-out is the case k = n, with training sets of size n − 1. Its errors are strongly dependent because training sets overlap substantially; it is not guaranteed to have lower variance than moderate-fold cross-validation. When tuning is extensive, an outer assessment loop or untouched test set is needed to evaluate the selected procedure.
Worked example: With 100 independent observations, five-fold cross-validation fits five models on 80 observations each and predicts 20 held-out observations per fit. Leave-one-out fits on 99 observations each time. Both estimate prediction loss, but neither permits fitting preprocessing using its held-out responses.
Common mistake: Leave-one-out does not create independent error estimates, and using more folds does not guarantee a less variable performance estimate.
8. MSE, RMSE, and MAE target different losses
Mean squared error averages squared residuals, root mean squared error takes its square root, and mean absolute error averages absolute residuals. Squaring gives large errors disproportionate influence, while MAE penalizes each additional error unit linearly. RMSE has the response's original units; MSE has squared units. Under the corresponding population losses, the conditional mean minimizes expected squared error and a conditional median minimizes expected absolute error. Thus, choosing a loss can change the optimal prediction, especially for skewed outcomes. Compare measures on consistent observations and response scales, with any operational weights explicitly defined.
Worked example: For prediction errors of −1, 2, and 3 thousand currency units, MSE is (1 + 4 + 9)/3 = 14/3 squared-thousand units. RMSE is approximately 2.16 thousand units, while MAE is 2 thousand units. These values summarize different penalties.
Common mistake: RMSE and MAE are not interchangeable merely because both use response units; their treatment of large errors differs materially.
9. Classification thresholds, prevalence, and ranking
Sensitivity is TP/(TP + FN), specificity is TN/(TN + FP), and precision is TP/(TP + FP). Lowering a probability threshold generally identifies more positives but also more false positives. Precision depends on event prevalence as well as classifier behavior, so it may change across portfolios. ROC curves compare sensitivity with false-positive rate across thresholds; precision–recall curves emphasize positive predictions and detected events. A ranking measure does not establish probability calibration or choose an operational threshold. Select thresholds using the consequences of errors, capacity constraints, and appropriately evaluated probabilities.
Worked example: A referral model identifies 40 of 50 actual fraud cases and refers 100 claims overall. Sensitivity is 80%, but precision is only 40%. Increasing review capacity may justify a lower threshold, provided the additional detected cases warrant the additional false referrals.
Common mistake: High overall accuracy can coexist with poor event detection when the positive outcome is uncommon, so accuracy alone can mislead.
Linear regression estimation and interpretation
10. Least squares from centered sums
For simple linear regression with an intercept, the least-squares slope is b1 = Sxy/Sxx, where Sxy sums centered predictor–response products and Sxx sums squared centered predictor values. The intercept is b0 = ȳ − b1x̄, ensuring the fitted line passes through the sample means. Least squares minimizes the sum of squared vertical response residuals, rather than perpendicular distances to the line. A slope is identifiable only when the predictor varies, so Sxx must be positive. Its units are response units per predictor unit, which should carry through predictions and interpretation.
Worked example: Suppose average tenure is 5 years, average annual loss is 12 hundred currency units, Sxy is 24, and Sxx is 8. The slope is 3 per year and the intercept is −3. At tenure 6, predicted loss is 15 hundred units.
Common mistake: Using uncentered cross-products in the intercept-model slope formula generally gives a different answer and can silently fit the wrong relationship.
11. Conditional coefficients in multiple regression
In a multiple linear regression, a coefficient describes the change in fitted conditional mean for a one-unit predictor increase while the other included predictors remain fixed. This differs from a simple-regression association, which can combine the predictor's own relationship with associations involving omitted variables. Correlation among predictors can therefore change coefficient magnitudes or signs when controls are added. The interpretation also depends on transformations and interaction terms. A conditional coefficient is meaningful within supported predictor combinations; holding other variables fixed may otherwise describe an unrealistic extrapolation rather than a useful comparison.
Worked example: A model predicts annual loss as 100 + 8m + 20u, where m is mileage in thousands and u indicates urban residence. The mileage coefficient compares policies with the same residence indicator. A two-thousand-mile increase corresponds to 16 additional fitted currency units.
Common mistake: A multiple-regression coefficient is not the overall difference between all high- and low-predictor observations, because other included predictors are held fixed.
12. Standard errors, t tests, and coefficient intervals
A coefficient standard error estimates sampling variability under the fitted model and its error assumptions. To test H0: βj = β0, form t = (bj − β0)/SE(bj), using the appropriate residual degrees of freedom under normal linear-model inference. A confidence interval is bj plus or minus the selected t critical value times its standard error. Statistical significance depends on uncertainty as well as effect magnitude; a small effect can be precisely estimated. These calculations do not account automatically for data-driven variable selection, misspecified variance, or repeated testing across many candidate models.
Worked example: A coefficient estimate is 1.8 with standard error 0.6. Testing a zero coefficient gives t = 3. If the supplied critical value is 2.05, the corresponding interval is 1.8 ± 1.23, or 0.57 to 3.03 in coefficient units.
Common mistake: A 95% confidence interval does not assign a 95% probability to a fixed parameter after the observed interval has been calculated.
13. Partial F tests for nested regressions
A partial F test evaluates whether a block of additional linear-regression coefficients improves fit beyond a reduced model. For q added coefficients, F = [(RSSreduced − RSSfull)/q]/[RSSfull/dfFull]. The models must be nested and fitted to the same response observations, and the classical reference assumes the usual normal, independent, constant-variance errors. The overall regression F test compares all nonintercept terms against an intercept-only model. A significant block test does not imply that every coefficient in the block is individually significant or that the improvement is operationally important.
Worked example: A reduced model has RSS 180. Adding two terms lowers RSS to 150, and the full model has 30 residual degrees of freedom. The partial F statistic is [(180 − 150)/2]/(150/30) = 3, to be compared with a supplied reference distribution.
Common mistake: Comparing nonnested models with the partial F formula does not create a valid nested-model test, even when both use least squares.
14. Mean uncertainty versus individual prediction uncertainty
At predictor vector x0, estimated mean-response variance is s²h0, where h0 = x0′(X′X)−1x0. An individual future response adds its own error variance, giving s²(1 + h0). With the same confidence level and model assumptions, the prediction interval is therefore wider than the confidence interval for the conditional mean. Both usually widen for predictor combinations far from the observed design. Increasing sample size can reduce mean-estimation uncertainty, but it does not eliminate individual response noise. Extrapolation also introduces possible model misspecification that these formulas do not automatically capture.
Worked example: Suppose s = 4, h0 = 0.25, and the supplied interval multiplier is 2. The mean-response half-width is 2 × 4 × √0.25 = 4. The individual prediction half-width is 2 × 4 × √1.25, approximately 8.94 response units.
Common mistake: A very narrow interval for expected loss does not imply that individual claims will fall close to that expected value.
15. Reference coding for categorical predictors
With an intercept, a categorical predictor having g levels can be represented by g − 1 indicator variables and a reference level. Each indicator coefficient compares its level's fitted mean with the reference, conditional on other included terms. A contrast between two nonreference levels subtracts their coefficients, with uncertainty depending on their covariance as well as their standard errors. Changing the reference changes coefficient labels and values but not fitted predictions for an equivalent model. Including an intercept and a full set of level indicators creates exact linear dependence unless another identification constraint is imposed.
Worked example: A region model uses North as reference, with intercept 200, South coefficient 30, and West coefficient −10. Holding other predictors fixed, South exceeds North by 30 and West by 40 currency units. Releveling to West changes coefficients but preserves these fitted differences.
Common mistake: A nonreference category coefficient compares with the designated reference, not with the average of every other category in the dataset.
16. Interactions make effects depend on other predictors
An interaction term allows one predictor's conditional effect to depend on another. In E[Y | x, I] = β0 + β1x + β2I + β3xI, the slope in the reference group is β1 and the slope when I = 1 is β1 + β3. The group contrast at x is β2 + β3x. Consequently, β2 alone is the group contrast specifically at x = 0. Centering x can make that main effect correspond to a more meaningful predictor value without changing fitted predictions. Interpret interactions through explicit contrasts or slopes rather than isolated coefficients.
Worked example: A fitted model is 50 + 4x + 12I − xI, with x measured in years. The reference slope is 4, while the other group's slope is 3. At x = 5, the fitted group difference is 12 − 5 = 7.
Common mistake: When an interaction is included, the main-effect coefficient does not generally represent the same group difference at every predictor value.
17. Polynomial regression remains linear in coefficients
Polynomial regression adds powers of predictors while remaining linear in the unknown coefficients. For a quadratic mean β0 + β1x + β2x², the local slope is β1 + 2β2x, so β1 alone is the slope at zero rather than a universal effect. A turning point occurs at −β1/(2β2) when β2 is nonzero, but its practical interpretation requires support within the observed predictor range. Centering predictors can reduce numerical dependence among polynomial terms. High-degree polynomials may fit training patterns closely while behaving unstably near boundaries or beyond the data.
Worked example: Expected cost is modeled as 100 + 6x − 0.5x² for an observed range of x from 0 to 8. The local slope at x = 4 is 6 − 4 = 2. The fitted maximum occurs at x = 6.
Common mistake: A curved fitted relationship does not make least-squares estimation nonlinear when the model is still linear in its unknown coefficients.
18. Log-response models require retransformation
A regression for log(Y) models the conditional mean on the log scale. Exponentiating that fitted value does not generally recover E[Y | x], because the exponential is nonlinear. With normal log errors of constant variance σ², the conditional median is exp(η) and the conditional arithmetic mean is exp(η + σ²/2). A smearing estimate can instead average exponentiated residuals under suitable assumptions. Log-scale coefficients describe multiplicative changes after exponentiation, while percentage changes for finite differences are exp(βΔx) − 1. Heteroscedastic log errors can make the retransformation adjustment depend on predictors.
Worked example: A fitted log severity is 4, with normal log-error variance 0.36. The fitted median is exp(4), approximately 54.6 currency units. The fitted arithmetic mean is exp(4.18), approximately 65.4. A coefficient of 0.10 implies an approximately 10.5% multiplicative increase.
Common mistake: Exponentiating a fitted log response does not automatically estimate the original-scale arithmetic mean, even when the log-scale regression is correctly specified.
Regression diagnostics, model selection, and regularization
19. Residual curvature identifies missing mean structure
A residual-versus-fitted plot examines whether systematic response structure remains after fitting the mean model. A smooth curved pattern suggests that the conditional mean may need nonlinear terms, transformations, or interactions. Random scatter around zero is more consistent with the specified mean, though it does not prove all assumptions. Residual-versus-predictor plots can identify which relationships remain inadequately modeled, especially when several predictors are involved. Diagnose mean structure separately from changing residual spread. A better-looking residual plot should accompany validation evidence rather than justify unlimited feature additions using the same data.
Worked example: A straight-line severity model leaves positive average residuals at low and high ages but negative residuals in the middle. Adding a suitable quadratic age term addresses this curvature hypothesis. A funnel-shaped spread without a curved average would instead raise a variance question.
Common mistake: Residuals can average to zero in an intercept model while still showing substantial systematic curvature across fitted values or predictors.
20. Heteroscedasticity, robust uncertainty, and weighted fitting
Heteroscedasticity means conditional error variance changes with predictors or fitted values. Under exogeneity, ordinary least-squares coefficients can remain unbiased, but conventional constant-variance standard errors may be wrong and estimation may be inefficient. Heteroscedasticity-robust standard errors adjust uncertainty without changing the fitted coefficients. Weighted least squares instead changes estimation, using weights proportional to inverse error variance when that variance structure is credible. These weights are not automatically exposure weights or squared inverse standard deviations chosen without justification. Response transformations may stabilize variance, but they also change the modeled mean and interpretation.
Worked example: Two groups have error variances 4 and 16 under a proposed variance model. Inverse-variance weights are proportional to 1/4 and 1/16, so the first group's observations receive four times the weight. Robust standard errors would leave the original least-squares coefficients unchanged.
Common mistake: A funnel-shaped residual plot does not by itself establish biased coefficients; it primarily challenges constant-variance uncertainty calculations and efficiency.
21. Residual dependence changes information content
Residual dependence means errors from different observations are correlated, as can occur across time or within policyholder groups. Even when the conditional mean is adequate, conventional independent-error standard errors can misstate uncertainty because observations contribute overlapping information. Plotting residuals in their natural order or examining lag correlations helps reveal serial patterns. Remedies may involve modeling error correlation, adding omitted time structure, or using an appropriate covariance estimator. A random residual-versus-fitted cloud does not rule out dependence, because that plot discards ordering. Validation splitting should respect the same dependence structure.
Worked example: Monthly aggregate-loss residuals alternate between long runs of positive and negative values. Their scatter against fitted values looks unremarkable, but the time plot suggests persistence. A model with suitable time effects or correlated errors may better represent the remaining information.
Common mistake: Uncorrelated-looking residuals against fitted values do not establish independence, because serial or group dependence may only appear in the appropriate ordering.
22. Outliers, leverage, and Cook's distance
A response outlier has an unusual response relative to the fitted conditional relationship. Leverage measures unusualness of predictor values through the design matrix. Influence describes how much fitted coefficients or predictions change when an observation is removed or perturbed. Cook's distance combines residual size and leverage to summarize an observation's effect on a least-squares fit. A high-leverage point close to the fitted surface may have little observed influence, while a substantial residual can become especially influential at high leverage. Investigate data quality and model structure before deciding whether an unusual observation should be excluded.
Worked example: One policy has an extreme mileage value but a loss close to the fitted line. Another has ordinary mileage and a much larger residual. The first has greater leverage, yet the second may alter fitted predictions more; deletion diagnostics distinguish their actual influence.
Common mistake: High leverage does not prove a data error or justify automatic deletion; it describes predictor geometry rather than validity.
23. Multicollinearity and variance inflation
Multicollinearity occurs when predictors are strongly explained by combinations of other predictors. The data then contain limited information for separating their individual conditional effects, increasing coefficient uncertainty and sensitivity to small changes. For predictor j, VIFj = 1/(1 − Rj²), where Rj² comes from regressing that predictor on the others with an intercept. Large VIF signals inflated variance relative to a corresponding orthogonal design, without creating a universal decision threshold. Predictions within the observed predictor relationships may remain stable even when individual coefficients are unstable. Exact collinearity prevents unique ordinary least-squares coefficient identification.
Worked example: Annual mileage and annual fuel use are highly correlated. Regressing mileage on the other predictors gives R² = 0.90, so its VIF is 10. The model may predict loss adequately while estimating mileage's separate conditional coefficient imprecisely.
Common mistake: Multicollinearity does not automatically produce poor predictions or biased least-squares coefficients; its central problem is separating effects with limited independent information.
24. Best subset and stepwise search paths
Best-subset selection compares all predictor subsets, usually retaining the best fit at each model size before applying a selection criterion. Forward stepwise starts with a small model and adds the predictor producing the greatest permitted improvement at each step. Backward stepwise starts with the full candidate model and removes terms. Stepwise procedures follow restricted paths, so they can miss the best subset of a given size. Backward selection requires an estimable starting model. Any search can overfit the development sample, making independent validation of the complete selection procedure important.
Worked example: With predictors A, B, and C, forward selection might choose A first and then B, never evaluating the two-variable model containing only B and C. Best-subset selection evaluates all three two-variable combinations, but it still needs a criterion beyond raw training fit.
Common mistake: Forward and backward selection need not return the same model, because their restricted search paths depend on different earlier decisions.
25. Adjusted R-squared, AIC, and BIC penalize complexity
Ordinary R-squared cannot decrease when predictors are added to an intercept-containing least-squares model. Adjusted R-squared penalizes added terms through residual degrees of freedom and can decrease. Likelihood criteria instead combine fit and complexity: AIC = −2 log L + 2k and BIC = −2 log L + k log n. Lower values are preferred within valid comparable candidates. BIC imposes the larger per-parameter penalty when log n exceeds two. These criteria serve different theoretical objectives and need not agree with cross-validated predictive loss. Comparisons require compatible likelihoods, observations, and parameter-count conventions.
Worked example: Two models on 100 observations improve maximized log likelihood by 2 when one parameter is added. AIC changes by −4 + 2 = −2 and favors expansion. BIC changes by −4 + log(100), approximately 0.61, and favors the smaller model.
Common mistake: The model with the highest ordinary R-squared is not automatically preferred once complexity or out-of-sample predictive performance matters.
26. Ridge regression shrinks correlated coefficients
Ridge regression minimizes a squared-error objective plus λ times the sum of squared penalized coefficients. The usual intercept is unpenalized. Increasing λ shrinks slopes toward zero, trading additional bias for potentially lower estimation variance, especially with correlated predictors. Ridge generally retains nonzero coefficients rather than selecting a sparse subset. Predictor scaling matters because the penalty acts on coefficient magnitudes; standardization makes penalties more comparable across measurement units. Select λ using appropriate validation, with scaling estimated inside each training fold. Objective normalization conventions differ, so numerical λ values are not transferable without checking the formula.
Worked example: Mileage in miles and vehicle age in years enter a ridge model. Without scaling, their coefficient magnitudes reflect different units, distorting the penalty comparison. Standardizing both within each training fold allows validation to assess a more comparable shrinkage choice.
Common mistake: Ridge's smaller coefficients do not mean weaker predictive value; coordinated shrinkage can stabilize predictions when predictors contain overlapping information.
27. The lasso combines shrinkage with sparse selection
The lasso adds λ times the sum of absolute penalized coefficients to the fitting objective. Its penalty geometry permits some estimated slopes to equal zero, producing sparse models while shrinking retained coefficients. As with ridge, scaling and tuning are essential, and the intercept is ordinarily unpenalized. With strongly correlated predictors, the lasso can select one variable while excluding a close substitute, making selected sets unstable across samples. A zero coefficient under a chosen penalty is a modeling decision, not proof of no underlying association. Post-selection conventional inference requires additional care.
Worked example: Two nearly interchangeable mileage measures compete in a standardized lasso fit. One receives a nonzero coefficient and the other is zero at the selected penalty. Another development sample may reverse their roles while giving similar predictions, showing why sparse selection is not unique scientific attribution.
Common mistake: A lasso-selected variable set is not guaranteed to identify the true generating predictors, especially when several predictors are strongly correlated.
Generalized linear models
28. The three components of a GLM
A generalized linear model specifies a conditional response distribution, a linear predictor η = x′β, and a link relating the conditional mean to that predictor: g(μ) = η. The link transforms the mean, not the observed response. Distribution choice determines the mean–variance relationship, while link choice determines how predictors affect the mean and whether fitted values respect its range. Classical linear regression is a normal identity-link example. A response distribution and link should be chosen together with the data-generating question; fitting a linear predictor alone does not specify a complete probabilistic model.
Worked example: For claim occurrence, a Bernoulli response supplies variance p(1 − p), the logit link maps p to unrestricted real values, and η combines policy predictors. Applying the inverse link returns a fitted probability between zero and one for each policy.
Common mistake: A GLM does not generally regress a transformed observed response; its link is applied to the conditional response mean.
29. Matching Bernoulli, Poisson, and gamma responses
Bernoulli models describe binary outcomes, Poisson models describe nonnegative counts, and gamma models can describe strictly positive continuous outcomes. Common choices include a logit link for Bernoulli probabilities and a log link for count or gamma means. Poisson assumes conditional variance equals the conditional mean, whereas a gamma GLM commonly uses variance proportional to the squared mean. A gamma response cannot directly represent zero claim amounts, so frequency and positive severity may require separate treatment. These are candidate assumptions, not rules guaranteeing fit; inspect support, conditional variability, and residual behavior.
Worked example: An insurer separately models whether a claim occurs, how many claims arise over exposure, and positive payment size given a claim. Bernoulli, Poisson, and gamma distributions respectively match these response supports, while the count and severity assumptions still require diagnostic assessment.
Common mistake: A log link does not make different response distributions equivalent; their conditional variances and likelihoods remain distinct modeling assumptions.
30. Logistic coefficients describe odds ratios
Logistic regression uses log[p/(1 − p)] = x′β. Increasing predictor xj by one unit while other terms remain fixed multiplies fitted odds by exp(βj). It does not multiply probability by that amount. Convert any linear predictor using p = exp(η)/(1 + exp(η)). The probability change associated with a fixed odds ratio depends on the starting probability, and interactions can make the odds ratio depend on other predictors. Logistic coefficients are conditional associations within the specified model; neither an odds ratio nor a probability contrast automatically has a causal interpretation.
Worked example: A coefficient of log(2) doubles fitted odds. Starting from probability 0.20 gives odds 0.25, new odds 0.50, and new probability 1/3. Starting from probability 0.50 instead gives new probability 2/3, demonstrating that the probability increase is not fixed.
Common mistake: An odds ratio of two does not generally double probability; converting between odds and probabilities is essential for correct interpretation.
31. Log-link effects multiply conditional means
For a log-link GLM, log μ = x′β and therefore μ = exp(x′β). A predictor increase Δx multiplies the conditional mean by exp(βΔx), holding other terms fixed. The exact percentage change is 100[exp(βΔx) − 1], with 100βΔx only an approximation for small changes. Unlike a regression on log(Y), this model directly specifies the original-scale conditional mean and needs no lognormal retransformation correction. Interactions alter the relevant coefficient contrast, and offsets can contribute fixed multiplicative factors such as exposure. Interpret the response units before describing the resulting ratio.
Worked example: A gamma log-link severity model has coefficient 0.15 for a binary coverage indicator. Otherwise comparable policies have fitted mean severities in ratio exp(0.15), approximately 1.162. If the reference fitted mean is 500 currency units, the other fitted mean is approximately 581.
Common mistake: A log-link GLM and ordinary regression of log payments are different models, even when both display a linear equation involving logarithms.
32. Exposure offsets separate counts from rates
A count model with log exposure offset has log μi = log ei + xi′β, so μi = ei exp(xi′β). The offset's coefficient is fixed at one, imposing proportionality between expected count and exposure for otherwise identical risks. The modeled rate is exp(xi′β), while the expected count multiplies that rate by exposure in consistent units. Estimating a free coefficient on log exposure instead changes the model and no longer imposes simple proportionality. Zero exposure requires separate handling because log zero is undefined; positive count exposure must also represent the relevant opportunity for events.
Worked example: A fitted frequency is 0.12 claims per policy-year. A policy observed for six months has exposure 0.5 policy-years and expected count 0.06. A comparable policy observed for two years has expected count 0.24, without changing its annual risk rate.
Common mistake: An exposure offset is not an estimated ordinary predictor coefficient, and an expected count should not be reported as an annual rate.
33. Likelihood fitting, IRLS, and separation
GLM coefficients are commonly estimated by maximizing the specified conditional likelihood. Iteratively reweighted least squares updates a working response and weights based on the current fitted means, converting each step into a weighted linear-model calculation. These computational weights arise from the link and variance function rather than being arbitrary importance scores. Convergence does not establish correct distributional assumptions. In logistic regression, complete separation can cause ordinary maximum-likelihood coefficients to diverge because predictors perfectly distinguish outcomes. Extremely large estimates and unstable standard errors can therefore signal an estimation boundary rather than overwhelming precision.
Worked example: Every policy with a particular indicator has a claim, and every policy without it has no claim. A logistic likelihood can improve as that indicator's coefficient grows indefinitely. Reporting a huge coefficient with a conventional Wald interval does not resolve the separation problem.
Common mistake: A numerical optimizer's completion message does not guarantee finite, stable estimates or validate the fitted response distribution and link.
34. Deviance and nested likelihood-ratio comparisons
Deviance compares a fitted model's likelihood with that of a saturated model that fits each response as fully as the framework permits. Lower deviance indicates better unpenalized fit on comparable observations. For suitable nested models, a deviance reduction is a likelihood-ratio statistic, often compared asymptotically with a chi-square distribution having degrees of freedom equal to the added parameter count. Regularity conditions and the specified dispersion matter. A reduction does not itself prove overall adequacy, predictive improvement, or correct variance assumptions. Quasi-likelihood models do not supply an ordinary full likelihood for every likelihood-based comparison.
Worked example: A Poisson model with three added terms reduces deviance from 145 to 136. The comparison statistic is 9 on three added degrees of freedom under the stated likelihood assumptions. This tests the block of terms, while residual diagnostics address remaining lack of fit.
Common mistake: A lower training deviance is not sufficient to prefer a larger model; added terms ordinarily improve fit even when they generalize poorly.
35. Overdispersion and unmodeled heterogeneity
A Poisson GLM assumes Var(Y | x) = E[Y | x]. Conditional variability exceeding that mean is overdispersion relative to the model. A Pearson statistic divided by residual degrees of freedom can provide a diagnostic indication, though omitted mean structure, dependence, outliers, and small-sample behavior also affect interpretation. Quasi-Poisson methods estimate a dispersion multiplier for uncertainty, while negative-binomial models provide a different count distribution with extra variability. These approaches do not repair every source of misspecification. Distinguish conditional overdispersion from the unconditional variation created merely by combining risks with different fitted means.
Worked example: Policies with the same fitted Poisson mean of two claims show substantially more variability than two in repeated comparable groups. A latent risk mixture could explain this pattern. A negative-binomial model may capture that heterogeneity, but omitted covariates should also be considered.
Common mistake: A portfolio-wide variance exceeding its mean does not alone prove conditional overdispersion, because different policy means can create that difference.
36. Pearson and deviance residual diagnostics
A Pearson residual scales the observed-minus-fitted difference by the model-implied standard deviation, with dispersion handled according to the chosen convention. A deviance residual is the signed square root of an observation's deviance contribution. Both help examine remaining structure, unusual observations, and variance mismatch, but they emphasize discrepancies differently. Discrete or strongly skewed responses need not produce residuals resembling normal linear-regression errors. Inspect residuals against fitted means, important predictors, exposure, and relevant ordering. Leverage and influence can also matter in GLMs, so a large raw response difference is not automatically the most consequential observation.
Worked example: A count residual of three claims is more surprising when the fitted mean is one than when it is one hundred. Pearson scaling reflects the model-implied variability. A residual pattern increasing with exposure may indicate a rate specification or exposure assumption that needs investigation.
Common mistake: GLM residuals need not form a normal cloud simply because the model is appropriate; response discreteness and variance structure affect their appearance.
Time series models and forecasting
37. Trend, seasonality, and stochastic dependence
A trend is a sustained change in a series' level, while seasonality is a pattern recurring at a known calendar period. Serial dependence describes relationships between observations separated in time and can remain after trend or seasonality is modeled. Additive decompositions suit roughly constant seasonal amplitudes; multiplicative behavior suggests amplitudes that scale with level, sometimes motivating a log transformation for positive observations. A visually rising series does not by itself distinguish deterministic trend from a stochastic trend. Plot observations in time order and use the application calendar to identify meaningful candidate components.
Worked example: Monthly claims rise over several years and peak every winter. If winter excesses remain about 200 claims, an additive seasonal component may be plausible. If winter excesses instead stay near 20% of the current level, a multiplicative representation may be more suitable.
Common mistake: Seasonality is a recurring calendar pattern, not simply any sequence of positive lag correlations or an unexplained upward trend.
38. Weak stationarity and appropriate differencing
Weak stationarity requires a constant finite mean, constant finite variance, and autocovariance depending on lag rather than calendar time. Detrending removes a modeled deterministic trend; first differencing forms Yt − Yt−1 and can remove a stochastic level trend. Seasonal differencing uses Yt − Yt−s for period s. A variance-stabilizing transformation addresses changing scale rather than necessarily removing dependence. These operations are not interchangeable, and unnecessary differencing can introduce avoidable dependence or increase forecast uncertainty. Choose the operation based on the suspected structure, then reassess the transformed series rather than assuming stationarity has been achieved.
Worked example: A monthly index has a stable annual pattern but a wandering level. First differences may address the level behavior, while a lag-12 difference targets seasonal persistence. Applying both without diagnosing the separate features can unnecessarily complicate the series and its forecast reconstruction.
Common mistake: Differencing is not a universal cleaning step; a trend-stationary series and a random walk can require different modeling treatments.
39. Autocovariance and autocorrelation quantify lag dependence
For a weakly stationary series with mean μ, lag-k autocovariance is γ(k) = Cov(Yt, Yt−k), and autocorrelation is ρ(k) = γ(k)/γ(0). Autocovariance carries squared response units, while autocorrelation is unitless and lies between minus one and one. Positive lag correlation indicates a tendency for deviations with matching signs, rather than a guarantee that successive observations increase. Sample estimators differ in denominator and mean treatment, so use the stated convention in calculations. Stationary autocovariance is symmetric in lag, and estimated correlations contain sampling noise rather than revealing model order exactly.
Worked example: A stationary index has variance 25 index-units squared and lag-two autocovariance 10 index-units squared. Its lag-two autocorrelation is 10/25 = 0.4. This describes dependence between centered observations two periods apart, not a 40% expected increase in the index.
Common mistake: Autocorrelation is not a growth rate, and a positive value does not imply that the series must rise from one period to the next.
40. Partial autocorrelation separates direct lag relationships
The partial autocorrelation at lag k measures the linear association between observations k periods apart after accounting for intervening lags. It can be obtained as the final lag coefficient in an autoregression containing lags one through k. For an ideal stationary AR(p) process, the population PACF cuts off after p while the ACF generally tails off. For an invertible MA(q), the population ACF cuts off after q while the PACF generally tails off. Sample plots blur these patterns, so model identification also needs estimation, diagnostics, and forecasting assessment rather than a rigid visual rule.
Worked example: An AR(1) series has lag-one correlation 0.7 and lag-two correlation 0.49. The lag-two relationship is transmitted through the intermediate observation. Its population lag-two partial autocorrelation is zero, even though its ordinary lag-two autocorrelation remains positive.
Common mistake: A nonzero ordinary lag-two correlation does not by itself establish an AR(2) term, because shorter-lag dependence can transmit correlation.
41. AR(1) persistence, mean reversion, and forecasts
A stationary AR(1) can be written Yt − μ = φ(Yt−1 − μ) + εt with |φ| < 1. Its autocorrelation is φ raised to the lag, and its variance is σ²ε/(1 − φ²). The h-step conditional forecast is μ + φ^h(Yt − μ), showing mean reversion. With known parameters, forecast-error variance is σ²ε times the sum of φ^(2j) for j from zero to h − 1. Negative φ creates alternating dependence. If the model instead uses intercept c, the stationary mean is c/(1 − φ), not c.
Worked example: For Yt = 4 + 0.8Yt−1 + εt, the stationary mean is 20. Starting from 30, the two-step forecast is 20 + 0.8²(10) = 26.4. The process remains persistent while its forecast gradually approaches the long-run mean.
Common mistake: The intercept in an autoregressive equation is generally not its long-run mean; the lag coefficient changes that relationship.
42. Moving averages are finite innovation filters
An MA(1) model written Yt = μ + εt + θεt−1 combines current and previous white-noise innovations. Its variance is σ²ε(1 + θ²), lag-one autocovariance is θσ²ε, and autocovariances vanish beyond lag one. Finite moving-average processes are stationary with finite-variance innovations; invertibility is a separate condition that supports a unique stable innovation representation. Under this sign convention, positive θ gives positive lag-one correlation. White noise requires zero autocovariance at nonzero lags, but independence is stronger unless additional distributional assumptions are made. Always check the stated MA sign convention before calculating.
Worked example: Let θ = 0.5 and innovation variance be 4. The MA(1) variance is 4(1.25) = 5, lag-one covariance is 2, and lag-one correlation is 0.4. Lag-two covariance is zero because the observations share no innovation terms.
Common mistake: A moving-average time-series model is not merely a rolling average of observed responses; its defining terms are unobserved innovations.
43. Random-walk forecasts accumulate innovation uncertainty
A random walk with drift satisfies Yt = Yt−1 + d + εt. With known drift and independent mean-zero innovations, the h-step forecast is Yt + hd, and its forecast-error variance is hσ²ε. The level process does not revert to a fixed long-run mean, and its uncertainty grows without bound as the horizon increases. First differences equal d + εt, giving a stationary process under the stated innovation assumptions. Estimating drift adds uncertainty beyond the known-parameter formula. Forecast intervals additionally require a distributional approximation or another justified method for the sum of future innovations.
Worked example: An index currently equals 100, estimated drift is treated as known at 2 units per month, and innovation variance is 9. The four-month point forecast is 108. Under the known-drift assumption, forecast-error variance is 36 and standard deviation is 6 units.
Common mistake: A random-walk forecast does not approach a fixed historical average, and its forecast variance does not stabilize at longer horizons.
44. Rolling-origin assessment and residual whiteness
Rolling-origin evaluation fits a forecasting procedure using information available at each historical origin and tests predictions at the intended future horizon. Expanding windows retain all prior data; sliding windows emphasize a fixed recent span. Transformations and tuning must obey the same time boundary. Compare procedures using losses aligned with the application, and recognize that overlapping multi-step forecast errors can be dependent. Residual autocorrelation suggests unused linear forecasting information or misspecification, while uncorrelated residuals do not prove optimality. Percentage errors can be unstable near zero, so their suitability depends on the response scale.
Worked example: A three-month-ahead procedure is fitted through month 60 and tested at month 63, then fitted through month 61 and tested at month 64. Averaging those errors evaluates the intended horizon while preserving chronology, unlike fitting once to the complete series.
Common mistake: Small in-sample residuals do not establish good forecasts, and shuffled validation can grant a forecasting procedure information from its future.
Decision trees and ensemble methods
45. Regression splits minimize within-node RSS
A regression tree partitions predictor space into regions and predicts the mean development response within each terminal region under squared-error loss. For a proposed split, compute the RSS in both children about their own means and add them. The preferred split gives the largest reduction from parent RSS, equivalently the smallest combined child RSS. Child sample sizes are already reflected in their RSS values; averaging unweighted child mean squared errors would use a different objective. Candidate split quality is local to the current node and does not guarantee the globally optimal final tree.
Worked example: A node contains responses 2, 4, 8, and 10. Its mean is 6 and RSS is 40. A split placing 2 and 4 together and 8 and 10 together gives child means 3 and 9, combined RSS 4, and reduction 36.
Common mistake: The best regression split minimizes response variation within its children, rather than maximizing the numerical distance between predictor cut points.
46. Gini, entropy, and classification error differ
For class proportions pk, Gini impurity is 1 − Σpk², entropy is −Σpk log pk, and classification error is 1 − max pk. All equal zero for a pure node, but Gini and entropy respond more sensitively to changes in class composition than classification error. Evaluate a split using child impurities weighted by child observation counts or explicitly defined observation weights. Classification error is often useful for interpreting terminal predictions but can overlook improvements in probability purity. Specify the logarithm base when a numerical entropy value is requested; consistent bases preserve candidate rankings.
Worked example: A binary node with proportions 0.8 and 0.2 has Gini impurity 1 − 0.64 − 0.04 = 0.32 and classification error 0.20. Two equally sized children with that composition retain weighted Gini 0.32, even though their majority labels may differ.
Common mistake: A split's child impurities must be size-weighted; a tiny pure child should not count equally with a large mixed child.
47. Recursive binary splitting creates rectangular regions
Standard tree construction repeatedly divides one current region into two using a predictor and split rule. For numerical predictors, axis-aligned thresholds create rectangular regions in predictor space. Predictions are constant within terminal nodes, making trees naturally represent interactions: a predictor's effect can depend on earlier branch decisions. Greedy splitting chooses the best immediate improvement rather than searching every possible tree. Deep trees can capture fine structure but may have unstable boundaries and high variance. Minimum leaf size, maximum depth, and pruning constrain complexity; ordinary regression-tree predictions also do not extrapolate linear trends beyond terminal-region means.
Worked example: A tree first splits annual mileage at 15,000 and then splits vehicle age only in the high-mileage branch. Age therefore affects predictions differently by mileage region. A policy far beyond the observed mileage range still receives a terminal-node mean rather than an extrapolated slope.
Common mistake: A tree's simple local rules do not guarantee stable predictions; small development-sample changes can alter early splits and many downstream assignments.
48. Cost-complexity pruning selects subtrees
Cost-complexity pruning evaluates a subtree using a criterion such as training RSS plus α times its number of terminal nodes. Increasing α places greater cost on complexity and favors smaller subtrees. A large initial tree generates a nested sequence of candidate pruned trees, after which validation or cross-validation selects an appropriate complexity level. Pruning can remove branches whose apparent local training improvements do not generalize. The selected subtree need not be the best tree among every imaginable structure; it is chosen from the available pruning sequence. A one-standard-error rule can favor simplicity when validation results are close.
Worked example: One subtree has RSS 90 and six leaves; another has RSS 100 and three leaves. At α = 2, their costs are 102 and 106, favoring six leaves. At α = 5, costs are 120 and 115, favoring three leaves.
Common mistake: The pruning penalty is not selected merely to minimize training RSS; validation assesses how the complexity tradeoff affects new observations.
49. Bootstrap aggregation reduces unstable-model variance
Bagging fits models to bootstrap samples drawn with replacement from the development observations, then aggregates their predictions. Regression commonly averages predicted values; classification may aggregate class votes or estimated probabilities. It is especially effective for unstable learners such as deep trees because averaging smooths their sample-specific variation. Correlation limits the benefit: for equally variable components with common pairwise correlation ρ, ensemble variance is proportional to ρ + (1 − ρ)/B. More trees reduce the uncorrelated part but cannot remove the shared component. Bagging does not automatically correct systematic bias or invalid data separation.
Worked example: Suppose individual tree predictions have variance 25 and pairwise correlation 0.2. Averaging 100 trees gives variance 25[0.2 + 0.8/100] = 5.2 under this simplified model. Even infinitely many such trees retain variance 5 because their errors share a correlated component.
Common mistake: Bootstrap samples are not disjoint partitions; each contains repeated observations and omits others, creating different fitted trees from the same development data.
50. Random forests trade tree strength for decorrelation
Random forests extend bagged trees by considering only a random subset of predictors at each split. This prevents a dominant predictor from forcing nearly identical top-level structures throughout the ensemble, often lowering tree correlation. Fewer available predictors can weaken individual trees, so predictive performance reflects a tradeoff between component strength and diversity. The number of candidate predictors, leaf-size controls, and sampling choices can be tuned using suitable assessment. Predictor importance also requires care: correlated features can share or obscure importance, and some impurity-based measures favor predictors with more possible splits.
Worked example: A portfolio has one strong mileage measure and several additional useful predictors. Bagged trees repeatedly split on mileage first. Restricting candidate predictors sometimes forces alternative early splits, making tree errors less similar and potentially improving the average prediction despite weaker individual trees.
Common mistake: Random forests do not simply choose one permanent predictor subset for the entire ensemble; candidate predictor subsets are typically redrawn at splits.
51. Out-of-bag predictions use omitted bootstrap observations
For a bootstrap sample of n draws from n observations, a particular observation is omitted with probability (1 − 1/n)^n, approaching exp(−1). Out-of-bag prediction aggregates only trees whose bootstrap samples omitted that observation. Comparing these predictions with observed responses provides an internal assessment without a separate validation split. However, repeated tuning against out-of-bag error can still adapt to that assessment, and preprocessing learned from all observations can undermine separation. Ordinary bootstrap out-of-bag assessment also does not automatically respect temporal forecasting or grouped dependence. It evaluates the sampling design actually used, not every deployment setting.
Worked example: In a forest of 600 bootstrap trees, a development observation may be omitted from roughly 220 trees. Its out-of-bag prediction averages those trees' predictions, excluding trees trained on it. The resulting error can guide tuning, while an untouched test set remains useful for final assessment.
Common mistake: An observation's out-of-bag prediction must exclude trees that trained on it; using all trees produces an ordinary fitted ensemble prediction.
52. Boosting follows the loss gradient sequentially
Gradient boosting adds learners sequentially to improve the current prediction function. Under squared-error loss, fitting the next learner to current residuals follows the negative loss gradient. Other losses require different working targets, so classification boosting is not simply regression on raw probability residuals in every implementation. The learning rate shrinks each update, tree depth controls interaction complexity, and iteration count controls accumulated flexibility. Smaller learning rates often require more iterations and still need validation. Unlike bagging, boosting's learners depend on earlier fits, and excessive iterations can fit noise rather than useful structure.
Worked example: A current prediction is 100 and a newly fitted residual tree predicts a correction of 20. With learning rate 0.05, the updated prediction is 101. The following tree uses residuals from that updated model rather than repeatedly applying the original correction.
Common mistake: A small learning rate does not eliminate overfitting; enough sequential updates can still make the model excessively flexible.
Unsupervised learning: principal components and clustering
53. Principal components maximize variance under orthogonality
For centered predictors, the first principal component is a unit-length linear combination with the greatest sample variance. Later components maximize remaining variance subject to orthogonality to earlier loading directions. The loading vectors are eigenvectors of the chosen covariance or correlation matrix, and their eigenvalues give component variances. PCA uses predictor variation without a response, so it does not directly optimize claim prediction or identify causal factors. Orthogonal component directions yield uncorrelated scores in the fitted sample, although uncorrelatedness does not generally imply statistical independence. Component signs are arbitrary if loadings and scores are reversed together.
Worked example: Two strongly positively correlated standardized predictors may have first loading direction approximately (0.707, 0.707), capturing their common movement. A second direction approximately (0.707, −0.707) captures their difference. The first explains more variation, but it is not necessarily more useful for predicting claims.
Common mistake: The first principal component captures the most predictor variance, not necessarily the strongest relationship with a separate insurance outcome.
54. Loadings define directions; scores locate observations
A loading vector specifies the coefficients forming a component, while an observation's score is the inner product of its preprocessed predictor values with that vector. Large absolute loadings indicate strong contributions to the direction under that preprocessing convention. Loading signs show which variables move together or oppose one another within the component. Scores locate observations along the resulting axis. For new observations, apply the development sample's centering and scaling before projection. Some software uses alternative loading normalizations, so distinguish unit eigenvectors from correlation-style loadings before interpreting reported numerical magnitudes.
Worked example: A unit loading vector is (0.6, 0.8). An observation with centered predictor values (3, −1) has score 0.6(3) + 0.8(−1) = 1.0. Reversing the loading vector changes its score to −1.0 while leaving the represented axis unchanged.
Common mistake: A loading belongs to a predictor–component relationship, whereas a score belongs to an observation; exchanging them obscures the PCA interpretation.
55. Explained variance and low-rank reconstruction
A component's proportion of variance explained is its eigenvalue divided by the sum of all eigenvalues. Cumulative explained variance sums these proportions over retained components. Projecting centered observations onto the first r loading vectors and reconstructing them yields the rank-r approximation minimizing squared reconstruction error under the chosen preprocessing. A scree plot displays eigenvalues and can reveal diminishing returns, but no universal explained-variance cutoff suits every application. Low-variance directions may still contain predictive or anomaly information. If retained scores feed a supervised model, choose the number of components using the relevant predictive assessment.
Worked example: Eigenvalues are 4, 1.5, and 0.5, totaling 6. The first component explains two-thirds of the variance, and the first two explain 5.5/6, approximately 91.7%. Keeping two components reduces dimension, but the discarded direction could still matter for a particular response.
Common mistake: High cumulative explained variance establishes good predictor reconstruction under the chosen scale, not automatically good prediction of an external outcome.
56. Centering and scaling define geometric importance
Centering subtracts each predictor's mean, so PCA describes variation around the data center rather than distance from an arbitrary origin. Scaling by standard deviations gives each nonconstant predictor unit variance and corresponds to correlation-matrix PCA. Without scaling, high-variance measurement units can dominate covariance PCA and Euclidean clustering. Standardization changes the analytical question rather than universally improving it: large natural variation may be substantively meaningful. For new observations, reuse fitted preprocessing constants. Constant predictors cannot be standardized by a zero standard deviation, and outliers can distort ordinary means, standard deviations, and distance geometry.
Worked example: Mileage is measured in tens of thousands while a driving-risk proportion lies between zero and one. Raw Euclidean distances largely reflect mileage units. Standardization gives both variables comparable variance contributions, but the analyst should decide whether that balance matches the intended similarity definition.
Common mistake: Changing units can change unscaled PCA and distance-based clusters; the geometry is not automatically invariant to how predictors are measured.
57. Lloyd's k-means alternates assignment and mean updates
K-means seeks a partition minimizing the sum of squared Euclidean distances from observations to their assigned cluster centers. Lloyd's algorithm alternates nearest-center assignment with updating each center to its assigned observations' arithmetic mean. Each step does not increase the objective, but convergence can reach a local rather than global minimum. The arithmetic mean is appropriate because the objective uses squared Euclidean distance; changing the dissimilarity may require a different representative. Standard k-means assumes a chosen number of clusters and works most naturally for compact, roughly spherical groups under the selected predictor scale.
Worked example: For observations 1, 3, 8, and 10 with initial centers 1 and 9, nearest-center assignment forms groups {1, 3} and {8, 10}. Updating gives centers 2 and 9. The new squared-distance total is 1 + 1 + 1 + 1 = 4.
Common mistake: A converged k-means solution is not guaranteed to be globally optimal, so a single initialization can give a misleading partition.
58. Initialization, outliers, and selecting k
K-means results depend on starting centers because its objective can have multiple local minima. Multiple starts or structured initialization improve the search but do not guarantee discovery of a uniquely meaningful partition. Squared distances make extreme observations influential, potentially pulling centers away from dense groups. Increasing k cannot increase the globally minimized within-cluster sum of squares, so raw fit alone always rewards more clusters. An elbow plot, silhouette analysis, stability checks, and domain usefulness can help evaluate complexity. Irregular shapes, unequal densities, or overlapping populations may make k-means an unsuitable representation despite a numerically good objective.
Worked example: Most policyholders have standardized scores near zero, but one recording error has score 40. With two clusters, k-means may dedicate a center to that observation. Checking data validity and comparing multiple starts can reveal that the apparent second group lacks substantive meaning.
Common mistake: The smallest within-cluster sum of squares does not choose k by itself, because adding centers systematically improves that training objective.
59. Hierarchical linkage determines which groups merge
Agglomerative hierarchical clustering starts with singleton observations and repeatedly merges the closest clusters under a linkage rule. Single linkage uses the smallest cross-cluster dissimilarity and can chain groups through nearby intermediates. Complete linkage uses the largest cross-cluster dissimilarity and favors tighter diameter control. Average linkage averages cross-cluster dissimilarities. Ward linkage instead seeks a small increase in within-cluster squared variation under compatible Euclidean assumptions. The dissimilarity and linkage together define the hierarchy. Merges are not ordinarily undone, so early choices constrain later structure and can make results sensitive to scaling or unusual observations.
Worked example: For clusters {0, 2} and {5, 9} on a line, cross-cluster distances are 5, 9, 3, and 7. Single linkage gives distance 3, complete linkage gives 9, and average linkage gives 6. These different definitions can produce different subsequent merges.
Common mistake: A hierarchical clustering method is not fully specified by its distance metric alone; the linkage rule changes the cluster-to-cluster comparison.
60. Dendrogram cuts and silhouette comparisons
A dendrogram records the sequence of hierarchical merges and their linkage heights. A horizontal cut selects the groups still separate at that height; vertical separation between merges may suggest candidate partitions but does not establish a true population count. Silhouette analysis compares an observation's average within-cluster dissimilarity a with its smallest average dissimilarity b to another cluster, using s = (b − a)/max(a, b). Values near one indicate strong separation, near zero indicate ambiguity, and negative values suggest a closer alternative cluster. Results remain dependent on preprocessing, dissimilarity, and the clustering objective.
Worked example: An observation has average within-cluster distance 2 and average distance 5 to its nearest alternative cluster. Its silhouette is (5 − 2)/5 = 0.6. Comparing average silhouettes across candidate dendrogram cuts helps assess separation, alongside cluster stability and practical interpretability.
Common mistake: Dendrogram leaf spacing is usually a display choice; merge heights and membership structure carry the clustering information rather than horizontal gaps.