Imre Kondor (Complexity Science Hub Vienna) – Regularized portfolio selection

Institutional portfolios are large, the number of different assets or risk factors N can be of the order of hundreds or thousands, while the sample size T (the number of observations per asset) is limited. Therefore, the estimation of the various parameters of these portfolios will be prone to a high degree of sampling error. The optimization of such portfolios belongs to the realm of high-dimensional statistics, which offers mathematically consistent and disciplined methods for reining in the large sample fluctuations. These methods are known as regularization, and consists in adding various terms to the objective function to supress large fluctuations. Regularization introduces bias, but hopefully a reasonable trade-off can be reached between bias and sample fluctuations.

In most practical applications regularized optimization is carried out numerically on empirical data. In contrast, here we derive analytic results to be able to identify the instabilities arising in these problems. We discuss two important cases. The regularization of the variance by the l1 norm (that is imposing a constraint on the sum of absolute values of the portfolio weights, a special case of which is the ban on short positions), and the regularization of Expected Shortfall by an l2 norm (the sum of the squares of the portfolio weights).

In the variance + l1 case we discover a new instability which is easy to overlook in numerical work and has apparently been overlooked so far in the literature. We observe that when one has plenty of data (the ratio r=N/T<<1) the regularizer plays hardly any role. If r is larger (0.3 or 0.4 or higher), the regularizer takes over fast, and determines the optimum. The range where an actual tradeoff can be observed is rather limited, and the new instability sets in at r=2.

The talk will discuss the financial meaning of the observed instabilities and of the regularization procedure.

The talk will also point to an apparently little known fact that regulators must be aware of: a number of standard statistical program packages contain a „hidden” l2 regularizer whose effect is generally negligible, but which kicks in under special circumstances, for example when a singular matrix has to be inverted. The software can smoothly perform this impossible task, tantamount to dividing by zero, without explicitly alerting the user. It is quite plausible to assume that similar smoothing and stabilizing tricks abound in banks’ risk management packages.

Tuesday, May 22, 2018, 11.00 a.m.

Oesterreichische Nationalbank
Otto-Wagner-Platz 3
1090 Wien