Algorithmic Learning in a Random World

This book is about conformal prediction, an approach to prediction that originated in machine learning in the late 1990s. The main feature of conformal prediction is the principled treatment of the reliability of predictions. The prediction algorithms described — conformal predictors — are provably valid in the sense that they evaluate the reliability of their own predictions in a way that is neither over-pessimistic nor over-optimistic (the latter being especially dangerous). The approach is still flexible enough to incorporate most of the existing powerful methods of machine learning. The book covers both key conformal predictors and the mathematical analysis of their properties. Algorithmic Learning in a Random World contains, in addition to proofs of validity, results about the efficiency of conformal predictors. The only assumption required for validity is that of "randomness" (the prediction algorithm is presented with independent and identically distributed examples); in later chapters, even the assumption of randomness is significantly relaxed. Interesting results about efficiency are established both under randomness and under stronger assumptions. Since publication of the First Edition in 2005 conformal prediction has found numerous applications in medicine and industry, and is becoming a popular machine-learning technique. This Second Edition contains three new chapters. One is about conformal predictive distributions, which are more informative than the set predictions produced by standard conformal predictors. Another is about the efficiency of ways of testing the assumption of randomness based on conformal prediction. The third new chapter harnesses conformal testing procedures for protecting machine-learning algorithms against changes in the distribution of the data. In addition, the existing chapters have been revised, updated, and expanded.