A More General Methodology to Spot Model Error

A More General Methodology to Spot Model Error

Model second-order effects and fragility: Assume we have the right model (a generous assumption) but are uncertain about the parameters. As a generalization, we use a simple function f(x|ᾱ), where ᾱ is the average expected input variable. Let φ be the distribution of α over its domain. The philosopher's stone bias (convexity bias) arises from stochasticizing the parameter α deemed fixed. This bias ω_A is measured as the difference between (a) the function f integrated across potential values of α, and (b) f estimated for a single value of α deemed its average:

$$ \omega_A = \int f(x|\alpha) \phi(\alpha) d\alpha - f(x|\bar{\alpha}) $$

The central equation: Fragility is a partial philosopher's stone below a threshold K. The missed fragility ω_B is assessed by comparing the two integrals below K to capture the effect on the left tail:

$$ \omega_B(K) = \left( \int_{-\infty}^{K} \int f(x|\alpha) \phi(\alpha) d\alpha dx \right) - \left( \int_{-\infty}^{K} f(x|\bar{\alpha}) dx \right) $$

A symmetric perturbation of α around its midpoint gives the following displayed approximation. Its right-hand side is pointwise in x; despite the left-hand label, obtaining the tail quantity requires integration below the threshold:

$$ \omega_B(K) \approx \frac{1}{2} \left( f(x|\bar{\alpha}+\Delta\alpha) + f(x|\bar{\alpha}-\Delta\alpha) \right) - f(x|\bar{\alpha}) $$

Note that antifragility ω_C integrates from K to infinity. We can probe ω_B by point estimates of f at level X ≤ K:

$$ \omega'_B(X) = \frac{1}{2} \left( f(X|\bar{\alpha}+\Delta\alpha) + f(X|\bar{\alpha}-\Delta\alpha) \right) - f(X|\bar{\alpha}) $$

Under that pointwise approximation, the tail quantity is obtained by integration:

$$ \omega_B(K) = \int_{-\infty}^{K} \omega'_B(X) dx $$

This leads to the fragility detection heuristic (Taleb, Canetti, et al., 2012). If ω'_B(X) has constant sign for X ≤ K, then ω_B(K) has the same sign. The detection heuristic perturbates the tails to probe fragility.

Portfolio fallacies: Taleb challenges the defense that portfolio optimization is necessarily better than no model because it encourages diversification. His concern is that apparent precision can encourage overallocation. It causes them to take more open positions due to perceived statistical offsets, making them vulnerable to model error and tail events. For example, an investor who thinks they know the statistical properties (σ_A, σ_B, ρ_AB) will allocate ω_A, ω_B to target risk. The lower the perceived correlation ρ_AB, the worse the exposure to model error. The illustration claims that when ρ_AB is thought to be 0, overallocation for extreme events is 1/3, and when it is thought to be -1, overallocation is maximal. The assumptions needed to derive these allocations are not supplied here. With leverage, this leads to disasters like Long-Term Capital Management. Correlations are unstable: 0.8 over one period becomes -0.2 over another. Taleb (1997) deals with stochastic correlations; one is only safe shorting a correlation at 1 and buying at -1, corresponding to the 1/n heuristic.

Kelly Criterion vs. Markowitz: Mean-variance optimization uses estimates of expected returns and covariances, together with a return or risk objective. Kelly allocation maximizes expected logarithmic growth and also depends on a model of returns; portfolio applications can require their joint distribution. Neither method eliminates estimation error or guarantees avoidance of ruin. Taleb favors bounded trial and error and discusses Kelly in that context, citing Thorp (1971, 1998) and Haigh (2000).

Corporate Finance: Corporate finance uses point projections, not distributional projections. Perturbating cash flow projections (e.g., in the Gordon valuation model) with continuously varying jumps (especially under fat-tailed distributions) can markedly increase expected value for companies deemed expensive, something the market prices heuristically but without explicit reason.

Parameter uncertainty can change a nonlinear model’s results even when the model’s form is correct. The direction and practical importance of that change depend on the function and the outcomes being assessed.