Foundations Of Machine Learning Pdf Download

The book is dedicated to all students interested in machine learning who are not content with only running lines of (deep-learning) code but who are eager to learn about this discipline's assumptions, limitations, and perspectives. When I was a student, my dream was to become an AI researcher and save humankind with intelligent robots. For several reasons, I abandoned such ambitions (but you never know). In exchange, I discovered that machine learning is much more than a conventional research domain since it is intimately associated with the scientific process transforming observations into knowledge. The first version of this book was made publicly available in 2004 with two objectives and one ambition. The first objective was to provide a handbook to ULB students since I was (and still am) strongly convinced that a decent course should come with a decent handbook. The second objective was to group together all the material that I consider fundamental (or at least essential) for a Ph.D. student to undertake a thesis in my lab. At that time, there were already plenty of excellent machine learning reference books. However, most of the existing work did not sufficiently acknowledge what machine learning owes to statistics and concealed (or did not make explicit enough, notably because of incomplete or implicit notation) important assumptions underlying the process of inferring models from data. The ambition was to make a free academic reference on the foundations of machine learning available on the web. There are several reasons for providing free access to this work: I am a civil servant in an institution that already takes care of my salary; most of the material is not original (though its organisation, notation definition, exercises, code and structure represent the primary added value of the author); in many parts of the world access to expensive textbooks or reference material is still difficult for the majority of students; most of the knowledge underlying this book was obtained by the author thanks to free (or at least non charged) references and, last but not least, education seems to be the last societal domain where a communist approach may be as effective as rewarding. Personally, I would be delighted if this book could be used to facilitate the access of underfunded educational and research communities to state-of-the-art scientific notions.

Discover the world's research

20+ million members
135+ million publications
700k+ research projects

Join for free

Handbook

Statistical foundations of machine learning

Second edition

Gianluca Bontempi

Machine Learning Group

Computer Science Department

ULB, Universit´e Libre de Bruxelles,

Brussels, Belgium

mlg.ulb.ac.be

September 17, 2021

And indeed all things that are

known have number. For it is not

possible that anything whatsoever be

understood or known without this.

Philolaus, 400 BC

Not everything that can be counted

counts, and not everything that

counts can be counted.

W. B. Cameron, 1963

Preface to the 2021 edition

The book is dedicated to all students interested in machine learning who are not

content with only running lines of (deep-learning) code but who are eager to learn

about this discipline's assumptions, limitations, and perspectives. When I was a

student, my dream was to become an AI researcher and save humankind with in-

telligent robots. For several reasons, I abandoned such ambitions (but you never

know). In exchange, I discovered that machine learning is much more than a con-

ventional research domain since it is intimately associated with the scientiﬁc process

transforming observations into knowledge.

The ﬁrst version of this book was made publicly available in 2004 with two

objectives and one ambition. The ﬁrst objective was to provide a handbook to ULB

students since I was (and still am) strongly convinced that a decent course should

come with a decent handbook. The second objective was to group together all the

material that I consider fundamental (or at least essential) for a Ph.D. student to

undertake a thesis in my lab. At that time, there were already plenty of excellent

machine learning reference books. However, most of the existing work did not

suﬃciently acknowledge what machine learning owes to statistics and concealed (or

did not make explicit enough, notably because of incomplete or implicit notation)

important assumptions underlying the process of inferring models from data.

The ambition was to make a free academic reference on the foundations of ma-

chine learning available on the web. There are several reasons for providing free

access to this work: I am a civil servant in an institution that already takes care

of my salary; most of the material is not original (though its organisation, notation

deﬁnition, exercises, code and structure represent the primary added value of the

author); in many parts of the world access to expensive textbooks or reference ma-

terial is still diﬃcult for the majority of students; most of the knowledge underlying

this book was obtained by the author thanks to free (or at least non charged) refer-

ences and, last but not least, education seems to be the last societal domain where

a communist approach may be as eﬀective as rewarding. Personally, I would be de-

lighted if this book could be used to facilitate the access of underfunded educational

and research communities to state-of-the-art scientiﬁc notions.

Though machine learning was already a hot topic at the end of the 20th century,

nowadays, it is deﬁnitely surrounded by a lot of hype and excitement. The number

of publications describing or using a machine learning approach in the last decades

is countless, making it impossible to address the heterogeneity of the domain in

a single book. Therefore, it is interesting to check how much material from the

ﬁrst edition is still useful: reassuringly enough, the more the nature of the content

is fundamental, the less it is prone to obsolescence. Nevertheless, a lot of new

things (not only deep learning) happened in the domain, and, more speciﬁcally, I

realised the importance of some fundamental concepts that were neglected in the

ﬁrst edition.

In particular, during those years, I realised the importance of exposing young

researchers to notions of multivariate dependency and independence. These notions

are brilliantly summarised in the topic of graphical models whose knowledge is es-

sential to grasp aspects of dimensionality reduction and feature selection. Secondly,

I (re)discovered that the foundations of machine learning lie in epistemology, the

branch of philosophy aiming to explain the meaning of knowledge and the process of

discovering it. Third, I became convinced that a process of discovering knowledge

from data should not be limited to modelling associations but aimed at discovering

causal mechanisms. Finally, I added a number of exercises, R scripts, and Shiny

dashboards to visualise and illustrate (sometimes too abstract) probabilistic and

estimation notions. In this sense, I am convinced that the adoption of Monte Carlo

simulation to introduce probabilistic concepts should be a more common habit in

introductory statistics classes.

For sure, I am strongly indebted to a lot of authors and their publications. I

hope I acknowledged them adequately in the bibliography. If I did not give enough

credit to some of the existing works, please do not hesitate to contact me. Last

but not least, the book is dedicated to all my ULB students and MLG researchers

in whom I have tried for many years to inculcate complex concepts of statistical

learning. Their eyes staring at my hand-waving, while I was trying to elucidate

some abstruse notions, were the best indicators of how to adapt, select and improve

the book's content.

To all those who want to send a note or continue to follow my machine learning

journey, see you on my blog https://datascience741.wordpress.com.

Acknowledgements

Though the book is not peer-reviewed, the added value of writing a handbook for

students and researchers is that they are typically very careful readers and willing to

pinpoint mistakes, inconsistencies, bad English and (a lot of) typos. First, I would

like to thank (in random order) the MLG researchers who sent me very useful

comments: Abhilash Miranda, Yann-a¨el Le Borgne, Souhaib Ben Taieb, Jacopo

De Stefani, Patrick Meyer, Olivier Caelen, Liran Lerman. Thanks as well to the

following students and readers (in random order) for their comments and remarks:

Robin de Haes, Mourad Akandouch, Zheng Liangliang, Olga Ibanez Sol´e, Maud

Destree, Wolf De Wulf, Dieter Vandesande, Miro-Manuel Matagne, Henry Morgan,

Pascal Tribel. A big thank to all of you! And do not hesitate to drop me an email

if you have comments or remarks!

Contents

Index 4

1 Introduction 15

1.1 Notations ................................. 22

2 Setting the foundations 27

2.1 Deductivelogic .............................. 27

2.2 Formal and empirical science . . . . . . . . . . . . . . . . . . . . . . 28

2.3 Induction, projection, and abduction . . . . . . . . . . . . . . . . . . 29

2.4 Hume and the induction problem . . . . . . . . . . . . . . . . . . . . 30

2.5 Logical positivism and veriﬁcationism . . . . . . . . . . . . . . . . . 31

2.6 Popper and the problem of induction . . . . . . . . . . . . . . . . . . 32

2.7 Instrumentalism ............................. 33

2.8 Epistemology and machine learning: the cross-fertilisation . . . . . . 33

3 Foundations of probability 37

3.1 The random model of uncertainty . . . . . . . . . . . . . . . . . . . . 37

3.1.1 Axiomatic deﬁnition of probability . . . . . . . . . . . . . . . 39

3.1.2 Visualisation of probability measures . . . . . . . . . . . . . . 39

3.1.3 Symmetrical deﬁnition of probability . . . . . . . . . . . . . . 40

3.1.4 Frequentist deﬁnition of probability . . . . . . . . . . . . . . 41

3.1.5 The Law of Large Numbers . . . . . . . . . . . . . . . . . . . 42

3.1.6 Independence and conditional probability . . . . . . . . . . . 42

3.1.7 Thechainrule .......................... 45

3.1.8 The law of total probability and the Bayes' theorem . . . . . 45

3.1.9 Direct and inverse conditional probability . . . . . . . . . . . 47

3.1.10 Logics and probabilistic reasoning . . . . . . . . . . . . . . . 49

3.1.11 Combined experiments . . . . . . . . . . . . . . . . . . . . . . 50

3.1.12 Array of joint/marginal probabilities . . . . . . . . . . . . . . 52

3.2 Randomvariables............................. 54

3.3 Discrete random variables . . . . . . . . . . . . . . . . . . . . . . . . 55

3.3.1 Parametric probability function . . . . . . . . . . . . . . . . . 55

3.3.2 Expected value, variance and standard deviation of a discrete

r.v.................................. 55

3.3.3 Entropy and relative entropy . . . . . . . . . . . . . . . . . . 58

3.4 Continuous random variable . . . . . . . . . . . . . . . . . . . . . . . 59

3.4.1 Mean, variance, moments of a continuous r.v. . . . . . . . . . 59

3.4.2 Univariate Normal (or Gaussian) distribution . . . . . . . . . 60

3.5 Jointprobability ............................. 61

3.5.1 Marginal and conditional probability . . . . . . . . . . . . . . 62

3.5.2 Independence........................... 63

3.5.3 Chainrule............................. 64

6CONTENTS

3.5.4 Conditional independence . . . . . . . . . . . . . . . . . . . . 65

3.5.5 Entropy in the continuous case . . . . . . . . . . . . . . . . . 66

3.5.5.1 Joint and conditional entropy . . . . . . . . . . . . . 66

3.6 Bivariate continuous distribution . . . . . . . . . . . . . . . . . . . . 67

3.6.1 Correlation ............................ 68

3.7 Normal distribution: the multivariate case . . . . . . . . . . . . . . . 70

3.7.1 Bivariate normal distribution . . . . . . . . . . . . . . . . . . 71

3.7.2 Gaussian mixture distribution . . . . . . . . . . . . . . . . . . 72

3.7.3 Linear transformations of Gaussian variables . . . . . . . . . 72

3.8 Mutualinformation............................ 73

3.8.1 Conditional mutual information . . . . . . . . . . . . . . . . . 74

3.8.2 Joint mutual information . . . . . . . . . . . . . . . . . . . . 74

3.8.3 Partial correlation coeﬃcient . . . . . . . . . . . . . . . . . . 75

3.9 Functions of random variables and Monte Carlo simulation . . . . . 76

3.10 Linear combinations of r.v. . . . . . . . . . . . . . . . . . . . . . . . 77

3.10.1 The sum of i.i.d. random variables . . . . . . . . . . . . . . . 77

3.11Conclusion ................................ 77

3.12Exercises ................................. 78

4 Graphical models 85

4.1 Conditional independence and multivariate distributions . . . . . . . 85

4.2 Directed acyclic graphs . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.3 Bayesiannetworks ............................ 86

4.3.1 Bayesian network and d-separation . . . . . . . . . . . . . . . 90

4.3.2 D-separation and I-map . . . . . . . . . . . . . . . . . . . . . 91

4.3.2.1 D-separation and faithfulness . . . . . . . . . . . . . 91

4.3.3 Skeleton and I-equivalence . . . . . . . . . . . . . . . . . . . . 93

4.3.4 Stable distributions . . . . . . . . . . . . . . . . . . . . . . . 94

4.4 Markovnetworks............................. 94

4.4.1 Separating vertices, separated subsets and independence . . . 95

4.4.2 Directed and undirected representations . . . . . . . . . . . . 95

4.5 Conclusions................................ 96

5 Parametric estimation 97

5.1 Classicalapproach ............................ 97

5.1.1 Point estimation . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.2 Empirical distributions . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.3 Plug-in principle to deﬁne an estimator . . . . . . . . . . . . . . . . 100

5.3.1 Sampleaverage.......................... 101

5.3.2 Sample variance . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.4 Sampling distribution . . . . . . . . . . . . . . . . . . . . . . . . . . 101

5.4.1 Shiny dashboard . . . . . . . . . . . . . . . . . . . . . . . . . 102

5.5 The assessment of an estimator . . . . . . . . . . . . . . . . . . . . . 103

5.5.1 Bias and variance . . . . . . . . . . . . . . . . . . . . . . . . . 103

5.5.2 Estimation and the game of darts . . . . . . . . . . . . . . . 104

5.5.3 Bias and variance of ˆ

µ...................... 104

5.5.4 Bias of the estimator ˆ

σ2 ..................... 105

5.5.5 A tongue-twister exercise . . . . . . . . . . . . . . . . . . . . 106

5.5.6 Bias/variance decomposition of MSE . . . . . . . . . . . . . . 107

5.5.7 Consistency............................ 107

5.5.8 Eﬃciency ............................. 108

5.6 The Hoeﬀding's inequality . . . . . . . . . . . . . . . . . . . . . . . . 108

5.7 Sampling distributions for Gaussian r.v.s . . . . . . . . . . . . . . . . 109

5.8 The principle of maximum likelihood . . . . . . . . . . . . . . . . . . 109

CONTENTS 7

5.8.1 Maximum likelihood computation . . . . . . . . . . . . . . . 111

5.8.2 Maximum likelihood in the Gaussian case . . . . . . . . . . . 111

5.8.3 Cramer-Rao lower bound . . . . . . . . . . . . . . . . . . . . 113

5.8.4 Properties of m.l. estimators . . . . . . . . . . . . . . . . . . 114

5.9 Interval estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

5.9.1 Conﬁdence interval of µ ..................... 115

5.10 Combination of two estimators . . . . . . . . . . . . . . . . . . . . . 117

5.10.1 Combination of m estimators .................. 118

5.10.1.1 Linear constrained combination . . . . . . . . . . . 118

5.11 Testing hypothesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119

5.11.1 Types of hypothesis . . . . . . . . . . . . . . . . . . . . . . . 119

5.11.2 Types of statistical test . . . . . . . . . . . . . . . . . . . . . 119

5.11.3 Pure signiﬁcance test . . . . . . . . . . . . . . . . . . . . . . . 120

5.11.4 Tests of signiﬁcance . . . . . . . . . . . . . . . . . . . . . . . 120

5.11.5 Hypothesis testing . . . . . . . . . . . . . . . . . . . . . . . . 121

5.11.6 The hypothesis testing procedure . . . . . . . . . . . . . . . . 122

5.11.7 Choiceoftest........................... 123

5.11.8 UMP level-α test......................... 125

5.11.9 Likelihood ratio test . . . . . . . . . . . . . . . . . . . . . . . 125

5.12Parametrictests ............................. 125

5.12.1 z-test (single and one-sided) . . . . . . . . . . . . . . . . . . . 126

5.12.2 t-test: single sample and two-sided . . . . . . . . . . . . . . . 127

5.13 A posteriori assessment of a test . . . . . . . . . . . . . . . . . . . . 128

5.14Conclusion ................................ 129

5.15Exercises ................................. 130

6 Nonparametric estimation and testing 135

6.1 Nonparametric methods . . . . . . . . . . . . . . . . . . . . . . . . . 135

6.2 Estimation of arbitrary statistics . . . . . . . . . . . . . . . . . . . . 136

6.3 Jackknife ................................. 137

6.3.1 Jackknife estimation . . . . . . . . . . . . . . . . . . . . . . . 137

6.4 Bootstrap ................................. 139

6.4.1 Bootstrap sampling . . . . . . . . . . . . . . . . . . . . . . . 139

6.4.2 Bootstrap estimate of the variance . . . . . . . . . . . . . . . 139

6.4.3 Bootstrap estimate of bias . . . . . . . . . . . . . . . . . . . . 141

6.4.4 Bootstrap conﬁdence interval . . . . . . . . . . . . . . . . . . 141

6.4.5 The bootstrap principle . . . . . . . . . . . . . . . . . . . . . 142

6.5 Randomisation tests . . . . . . . . . . . . . . . . . . . . . . . . . . . 143

6.5.1 Randomisation and bootstrap . . . . . . . . . . . . . . . . . . 144

6.6 Permutationtest ............................. 144

6.7 Considerations on nonparametric tests . . . . . . . . . . . . . . . . . 145

6.8 Exercises ................................. 146

7 Statistical supervised learning 147

7.1 Introduction................................ 147

7.2 Estimating dependencies . . . . . . . . . . . . . . . . . . . . . . . . . 150

7.3 Dependency and classiﬁcation . . . . . . . . . . . . . . . . . . . . . . 152

7.3.1 The Bayes classiﬁer . . . . . . . . . . . . . . . . . . . . . . . 154

7.3.2 Inverse conditional distribution . . . . . . . . . . . . . . . . . 155

7.4 Dependency and regression . . . . . . . . . . . . . . . . . . . . . . . 157

7.5 Assessment of a learning machine . . . . . . . . . . . . . . . . . . . . 158

7.5.1 An illustrative example . . . . . . . . . . . . . . . . . . . . . 159

7.6 Functional and empirical risk . . . . . . . . . . . . . . . . . . . . . . 164

7.6.1 Consistency of the ERM principle . . . . . . . . . . . . . . . 165

8CONTENTS

7.6.2 Key theorem of learning . . . . . . . . . . . . . . . . . . . . . 166

7.6.2.1 Entropy of a set of functions . . . . . . . . . . . . . 167

7.6.2.2 Distribution independent consistency . . . . . . . . 168

7.6.3 The VC dimension . . . . . . . . . . . . . . . . . . . . . . . . 169

7.7 Generalisation error . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7.7.1 The decomposition of the generalisation error in regression . 170

7.7.2 The decomposition of the generalisation error in classiﬁcation 173

7.8 The hypothesis-based vs the algorithm-based approach . . . . . . . . 174

7.9 The supervised learning procedure . . . . . . . . . . . . . . . . . . . 175

7.10 Validation techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

7.10.1 The resampling methods . . . . . . . . . . . . . . . . . . . . . 177

7.11Concludingremarks ........................... 179

7.12Exercises ................................. 179

8 The machine learning procedure 181

8.1 Introduction................................ 181

8.2 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

8.3 Experimental design . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

8.4 Data pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . 183

8.5 Thedataset ................................ 184

8.6 Parametric identiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . 185

8.6.1 Error functions . . . . . . . . . . . . . . . . . . . . . . . . . . 185

8.6.2 Parameter estimation . . . . . . . . . . . . . . . . . . . . . . 186

8.6.2.1 The linear least-squares method . . . . . . . . . . . 186

8.6.2.2 Iterative search methods . . . . . . . . . . . . . . . 186

8.6.2.3 Gradient-based methods . . . . . . . . . . . . . . . 187

8.6.2.4 Gradient descent . . . . . . . . . . . . . . . . . . . . 187

8.6.2.5 The Newton method . . . . . . . . . . . . . . . . . . 189

8.6.2.6 The Levenberg-Marquardt algorithm . . . . . . . . 190

8.6.3 Online gradient-based algorithms . . . . . . . . . . . . . . . . 192

8.6.4 Alternatives to gradient-based methods . . . . . . . . . . . . 192

8.7 Regularisation............................... 193

8.8 Structural identiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . 194

8.8.1 Model generation . . . . . . . . . . . . . . . . . . . . . . . . . 194

8.8.2 Validation............................. 195

8.8.2.1 Testing ......................... 195

8.8.2.2 Holdout......................... 196

8.8.2.3 Cross-validation in practice . . . . . . . . . . . . . . 196

8.8.2.4 Bootstrap in practice . . . . . . . . . . . . . . . . . 196

8.8.2.5 Complexity based criteria . . . . . . . . . . . . . . . 197

8.8.2.6 A comparison of validation methods . . . . . . . . . 199

8.8.3 Model selection criteria . . . . . . . . . . . . . . . . . . . . . 199

8.8.3.1 The winner-takes-all approach . . . . . . . . . . . . 199

8.8.3.2 The combination of estimators approach . . . . . . 200

8.9 Partition of dataset in training, validation and test . . . . . . . . . . 201

8.10 Evaluation of a regression model . . . . . . . . . . . . . . . . . . . . 201

8.11 Evaluation of a binary classiﬁer . . . . . . . . . . . . . . . . . . . . . 202

8.11.1 Balanced Error Rate . . . . . . . . . . . . . . . . . . . . . . . 203

8.11.2 Speciﬁcity and sensitivity . . . . . . . . . . . . . . . . . . . . 203

8.11.3 Additional assessment quantities . . . . . . . . . . . . . . . . 203

8.11.4 Receiver Operating Characteristic curve . . . . . . . . . . . . 204

8.11.5 Precision-recall curves . . . . . . . . . . . . . . . . . . . . . . 204

8.12 Multi-class problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

8.13Concludingremarks ........................... 207

CONTENTS 9

8.14Exercises ................................. 207

9 Linear approaches 211

9.1 Linearregression ............................. 211

9.1.1 The univariate linear model . . . . . . . . . . . . . . . . . . . 211

9.1.2 Least-squares estimation . . . . . . . . . . . . . . . . . . . . . 212

9.1.3 Maximum likelihood estimation . . . . . . . . . . . . . . . . . 214

9.1.4 Partitioning the variability . . . . . . . . . . . . . . . . . . . 214

9.1.5 Test of hypotheses on the regression model . . . . . . . . . . 215

9.1.5.1 Thet-test........................ 215

9.1.6 Interval of conﬁdence . . . . . . . . . . . . . . . . . . . . . . 216

9.1.7 Variance of the response . . . . . . . . . . . . . . . . . . . . . 216

9.1.8 Coeﬃcient of determination . . . . . . . . . . . . . . . . . . . 217

9.1.9 Multiple linear dependence . . . . . . . . . . . . . . . . . . . 217

9.1.10 The multiple linear regression model . . . . . . . . . . . . . . 217

9.1.11 The least-squares solution . . . . . . . . . . . . . . . . . . . . 218

9.1.12 Least-squares and non full-rank conﬁgurations . . . . . . . . 219

9.1.13 Properties of least-squares estimators . . . . . . . . . . . . . 219

9.1.14 Variance of the prediction . . . . . . . . . . . . . . . . . . . . 220

9.1.15 The HAT matrix . . . . . . . . . . . . . . . . . . . . . . . . . 220

9.1.16 Generalisation error of the linear model . . . . . . . . . . . . 221

9.1.16.1 The expected empirical error . . . . . . . . . . . . . 221

9.1.16.2 The PSE and the FPE . . . . . . . . . . . . . . . . 223

9.1.17 The PRESS statistic . . . . . . . . . . . . . . . . . . . . . . 226

9.1.18 Dual linear formulation . . . . . . . . . . . . . . . . . . . . . 227

9.1.19 The weighted least-squares . . . . . . . . . . . . . . . . . . . 228

9.1.20 Recursive least-squares . . . . . . . . . . . . . . . . . . . . . . 228

9.1.20.1 1st Recursive formulation . . . . . . . . . . . . . . . 229

9.1.20.2 2nd Recursive formulation . . . . . . . . . . . . . . 230

9.1.20.3 RLS initialisation . . . . . . . . . . . . . . . . . . . 230

9.1.20.4 RLS with forgetting factor . . . . . . . . . . . . . . 230

9.2 Linear approaches to classiﬁcation . . . . . . . . . . . . . . . . . . . 231

9.2.1 Linear discriminant analysis . . . . . . . . . . . . . . . . . . . 232

9.2.1.1 Discriminant functions in the Gaussian case . . . . 233

9.2.1.2 Uniform prior case . . . . . . . . . . . . . . . . . . . 234

9.2.1.3 LDA parameter identiﬁcation . . . . . . . . . . . . . 236

9.2.2 Perceptrons............................ 236

9.2.3 Support vector machines . . . . . . . . . . . . . . . . . . . . . 238

9.3 Conclusion ................................ 242

9.4 Exercises ................................. 242

10 Nonlinear approaches 245

10.1 Nonlinear regression . . . . . . . . . . . . . . . . . . . . . . . . . . . 247

10.1.1 Artiﬁcial neural networks . . . . . . . . . . . . . . . . . . . . 248

10.1.1.1 Feed-forward architecture . . . . . . . . . . . . . . . 248

10.1.1.2 Back-propagation . . . . . . . . . . . . . . . . . . . 250

10.1.1.3 Approximation properties . . . . . . . . . . . . . . . 253

10.1.2 From shallow to deep learning architectures . . . . . . . . . . 254

10.1.3 From global modelling to divide-and-conquer . . . . . . . . . 257

10.1.4 Classiﬁcation and Regression Trees . . . . . . . . . . . . . . . 257

10.1.4.1 Learning in Regression Trees . . . . . . . . . . . . . 259

10.1.4.2 Parameter identiﬁcation . . . . . . . . . . . . . . . . 259

10.1.4.3 Structural identiﬁcation . . . . . . . . . . . . . . . . 259

10.1.5 Basis Function Networks . . . . . . . . . . . . . . . . . . . . . 262

10 CONTENTS

10.1.6 Radial Basis Functions . . . . . . . . . . . . . . . . . . . . . . 262

10.1.7 Local Model Networks . . . . . . . . . . . . . . . . . . . . . . 262

10.1.8 Neuro-Fuzzy Inference Systems . . . . . . . . . . . . . . . . . 263

10.1.9 Learning in Basis Function Networks . . . . . . . . . . . . . . 265

10.1.9.1 Parametric identiﬁcation: basis functions . . . . . . 266

10.1.9.2 Parametric identiﬁcation: local models . . . . . . . 266

10.1.9.3 Structural identiﬁcation . . . . . . . . . . . . . . . . 268

10.1.10 From modular techniques to local modelling . . . . . . . . . . 268

10.1.11 Local modelling . . . . . . . . . . . . . . . . . . . . . . . . . . 270

10.1.11.1 Nadaraya-Watson estimators . . . . . . . . . . . . . 270

10.1.11.2 Higher order local regression . . . . . . . . . . . . . 272

10.1.11.3 Parametric identiﬁcation in local regression . . . . . 272

10.1.11.4 Structural identiﬁcation in local regression . . . . . 275

10.1.11.5 The kernel function . . . . . . . . . . . . . . . . . . 275

10.1.11.6 The local polynomial order . . . . . . . . . . . . . . 275

10.1.11.7 The bandwidth . . . . . . . . . . . . . . . . . . . . . 276

10.1.11.8 The distance function . . . . . . . . . . . . . . . . . 277

10.1.11.9 The selection of local parameters . . . . . . . . . . . 278

10.1.11.10Bias/variance decomposition of the local constant

model.......................... 279

10.2 Nonlinear classiﬁcation . . . . . . . . . . . . . . . . . . . . . . . . . . 281

10.2.1 Direct estimation via regression techniques . . . . . . . . . . 281

10.2.1.1 The nearest-neighbour classiﬁer . . . . . . . . . . . 281

10.2.2 Direct estimation via cross-entropy . . . . . . . . . . . . . . . 284

10.2.3 Density estimation via the Bayes theorem . . . . . . . . . . . 285

10.2.3.1 Naive Bayes classiﬁer . . . . . . . . . . . . . . . . . 285

10.2.3.2 SVM for nonlinear classiﬁcation . . . . . . . . . . . 287

10.3 Is there a best learner? . . . . . . . . . . . . . . . . . . . . . . . . . . 288

10.4Conclusions ................................ 290

10.5Exercises ................................. 292

11 Model averaging approaches 309

11.1Stackedregression ............................ 309

11.2Bagging .................................. 310

11.3Boosting.................................. 312

11.3.1 The Ada Boost algorithm . . . . . . . . . . . . . . . . . . . . 312

11.3.2 The arcing algorithm . . . . . . . . . . . . . . . . . . . . . . . 314

11.3.3 Bagging and boosting . . . . . . . . . . . . . . . . . . . . . . 315

11.4RandomForests ............................. 315

11.4.1 Why are Random Forests successful? . . . . . . . . . . . . . . 316

11.5 Gradient boosting trees . . . . . . . . . . . . . . . . . . . . . . . . . 316

11.6Conclusion ................................ 317

11.7Exercises ................................. 317

12 Feature selection 319

12.1 Curse of dimensionality . . . . . . . . . . . . . . . . . . . . . . . . . 320

12.2 Approaches to feature selection . . . . . . . . . . . . . . . . . . . . . 325

12.3Filtermethods .............................. 325

12.3.1 Principal component analysis . . . . . . . . . . . . . . . . . . 326

12.3.1.1 PCA: the algorithm . . . . . . . . . . . . . . . . . . 327

12.3.2 Clustering............................. 329

12.3.3 Ranking methods . . . . . . . . . . . . . . . . . . . . . . . . . 329

12.4Wrappingmethods ............................ 330

12.4.1 Wrapping search strategies . . . . . . . . . . . . . . . . . . . 331

CONTENTS 11

12.4.2 The Cover and van Campenhout theorem . . . . . . . . . . . 332

12.5Embeddedmethods ........................... 332

12.5.1 Shrinkage methods . . . . . . . . . . . . . . . . . . . . . . . . 333

12.5.1.1 Ridge regression . . . . . . . . . . . . . . . . . . . . 333

12.5.1.2 Lasso .......................... 334

12.5.2 Kernelmethods.......................... 335

12.5.3 Dual ridge regression . . . . . . . . . . . . . . . . . . . . . . . 336

12.5.4 Kernel function . . . . . . . . . . . . . . . . . . . . . . . . . . 336

12.6 Similarity matrix and non numeric data . . . . . . . . . . . . . . . . 337

12.7 Averaging and feature selection . . . . . . . . . . . . . . . . . . . . . 338

12.8 Information-theoretic perspective . . . . . . . . . . . . . . . . . . . . 338

12.8.1 Relevance, redundancy and interaction . . . . . . . . . . . . . 338

12.8.2 Information-theoretic ﬁlters . . . . . . . . . . . . . . . . . . . 341

12.8.3 Information-theoretic notions and generalisation . . . . . . . 341

12.9 Assessment of feature selection . . . . . . . . . . . . . . . . . . . . . 342

12.10Conclusion ................................ 343

12.11Exercises ................................. 344

13 From prediction to causal knowledge 347

13.1 About the notion of cause . . . . . . . . . . . . . . . . . . . . . . . . 348

13.2 Causality and dependencies . . . . . . . . . . . . . . . . . . . . . . . 349

13.2.1 Simpson's paradox . . . . . . . . . . . . . . . . . . . . . . . . 351

13.3 Causal vs associational knowledge . . . . . . . . . . . . . . . . . . . 353

13.4 The two main problems in causality . . . . . . . . . . . . . . . . . . 355

13.5 Causality and potential outcomes . . . . . . . . . . . . . . . . . . . . 355

13.5.1 Causaleﬀect ........................... 356

13.5.2 Estimation of causal eﬀect . . . . . . . . . . . . . . . . . . . . 357

13.5.3 Assignment mechanisms assumptions . . . . . . . . . . . . . . 358

13.5.4 About unconfoundness . . . . . . . . . . . . . . . . . . . . . . 358

13.5.5 Randomised designs . . . . . . . . . . . . . . . . . . . . . . . 359

13.5.5.1 Estimation of the treatment eﬀect . . . . . . . . . . 360

13.5.5.2 Stratiﬁed (or conditionally) randomised experiments 361

13.5.6 Observational study . . . . . . . . . . . . . . . . . . . . . . . 361

13.5.7 Strategies for estimation in observational studies . . . . . . . 362

13.6 From potential outcomes to graphical models . . . . . . . . . . . . . 362

13.7 Causal Bayesian network . . . . . . . . . . . . . . . . . . . . . . . . . 363

13.7.1 Causal networks and Structural Causal Models . . . . . . . . 365

13.7.2 Pre and post-intervention distributions . . . . . . . . . . . . . 365

13.7.3 Causal eﬀect estimation and identiﬁcation . . . . . . . . . . . 366

13.7.3.1 Backdoor criterion . . . . . . . . . . . . . . . . . . . 368

13.7.3.2 Beyond suﬃcient set: do-calculus . . . . . . . . . . 370

13.7.4 Selectionbias ........................... 370

13.8Counterfactual .............................. 372

13.9 Causal structure identiﬁcation . . . . . . . . . . . . . . . . . . . . . . 374

13.9.1 Constraint-based approaches . . . . . . . . . . . . . . . . . . 375

13.9.1.1 Normal conditional independence test . . . . . . . . 375

13.9.1.2 Skeleton discovery . . . . . . . . . . . . . . . . . . . 376

13.9.1.3 Dealing with immoralities in the skeleton . . . . . . 376

13.9.1.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . 378

13.10Beyond conditional independence . . . . . . . . . . . . . . . . . . . . 378

13.10.1 Causality and feature selection . . . . . . . . . . . . . . . . . 379

13.10.2 Beyond observational equivalence . . . . . . . . . . . . . . . . 379

13.10.2.1 Learning directionality in bivariate associations . . . 380

13.11Concluding remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . 382

12 CONTENTS

14 Conclusions 383

14.1 About ML limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 383

14.2Abitofethics............................... 384

14.3 Take-homenotions ............................ 385

14.4Recommendations ............................ 385

A Unsupervised learning 387

A.1 Probability density estimation . . . . . . . . . . . . . . . . . . . . . . 387

A.1.1 Nonparametric density estimation . . . . . . . . . . . . . . . 387

A.1.1.1 Kernel-based methods . . . . . . . . . . . . . . . . . 388

A.1.1.2 k-Nearest Neighbors methods . . . . . . . . . . . . . 389

A.1.2 Semi-parametric density estimation . . . . . . . . . . . . . . . 389

A.1.2.1 Mixture models . . . . . . . . . . . . . . . . . . . . 389

A.1.2.2 The EM algorithm . . . . . . . . . . . . . . . . . . . 390

A.1.2.3 The EM algorithm for the mixture model . . . . . . 390

A.2 K-meansclustering............................ 392

B Linear algebra notions 393

B.1 Rankofamatrix ............................. 393

B.2 Innerproduct............................... 393

B.3 Diagonalisation .............................. 394

B.4 QRdecomposition ............................ 394

B.5 Singular Value Decomposition . . . . . . . . . . . . . . . . . . . . . . 394

B.6 Chain rules of diﬀerential calculus . . . . . . . . . . . . . . . . . . . 395

B.7 Quadraticnorm.............................. 396

B.8 Quadratic programming . . . . . . . . . . . . . . . . . . . . . . . . . 396

B.9 The matrix inversion formula . . . . . . . . . . . . . . . . . . . . . . 396

C Probabilistic notions 399

C.1 Common univariate discrete probability functions . . . . . . . . . . . 399

C.1.1 The Bernoulli trial . . . . . . . . . . . . . . . . . . . . . . . . 399

C.1.2 The Binomial probability function . . . . . . . . . . . . . . . 399

C.2 Common univariate continuous distributions . . . . . . . . . . . . . . 400

C.2.1 Uniform distribution . . . . . . . . . . . . . . . . . . . . . . . 400

C.2.2 The chi-squared distribution . . . . . . . . . . . . . . . . . . 400

C.2.3 Student's t -distribution ..................... 400

C.2.4 F-distribution........................... 402

C.3 Common statistical hypothesis tests . . . . . . . . . . . . . . . . . . 402

C.3.1 χ2 -test: single sample and two-sided . . . . . . . . . . . . . . 402

C.3.2 t-test: two samples, two sided . . . . . . . . . . . . . . . . . . 402

C.3.3 F-test: two samples, two sided . . . . . . . . . . . . . . . . . 403

C.4 Transformation of random variables and vectors . . . . . . . . . . . . 403

C.5 Correlation and covariance matrices . . . . . . . . . . . . . . . . . . 404

C.6 Convergence of random variables . . . . . . . . . . . . . . . . . . . . 404

C.6.1 Example.............................. 405

C.7 The central limit theorem . . . . . . . . . . . . . . . . . . . . . . . . 405

C.8 The Chebyshev's inequality . . . . . . . . . . . . . . . . . . . . . . . 405

C.9 Empirical distribution properties . . . . . . . . . . . . . . . . . . . . 405

C.10Usefulrelations .............................. 406

C.11 Minimum of expectation vs. expectation of minimum . . . . . . . . . 406

C.12 Taylor expansion of function . . . . . . . . . . . . . . . . . . . . . . 407

C.13 Proof of Eq. (7.5.28) . . . . . . . . . . . . . . . . . . . . . . . . . . . 407

C.14 Biasedness of the quadratic empirical risk . . . . . . . . . . . . . . . 407

CONTENTS 13

D Plug-in estimators 409

E Kernel functions 411

F Companion R package 413

G Companion R Shiny dashboards 415

G.1 List of Shiny dashboards . . . . . . . . . . . . . . . . . . . . . . . . . 415

Chapter 1

Introduction

Over the last decades, a growing number of organisations have been allocating a

vast amount of resources to construct and maintain databases and data warehouses.

In scientiﬁc endeavours, data refers to carefully collected observations about some

phenomenon under study. In business, data capture information about economic

trends, critical markets, competitors, and customers. In manufacturing, data record

machinery performances and production rates in diﬀerent conditions. There are

essentially two reasons why people gather increasing volumes of data. First, they

think some valuable assets are implicitly coded within them, and, second, computer

technology enables eﬀective data storage and processing at reduced costs.

The idea of extracting useful knowledge from volumes of data is common to many

disciplines, from statistics to physics, from econometrics to system identiﬁcation

and adaptive control. The procedure for ﬁnding useful patterns in data is known

by diﬀerent names in diﬀerent communities, viz., knowledge extraction, pattern

analysis, data processing. In the artiﬁcial intelligence community, the most common

name is machine learning [71]. More recently, the set of computational techniques

and tools to support the modelling of a large amount of data is grouped under the

more general label of data science.

The need for programs that can learn was stressed by Alan Turing, who argued

that it might be too ambitious to write from scratch programs for tasks that even

humans must learn to perform. This handbook aims to present the statistical

foundations of machine learning intended as the discipline which deals with the

automatic design of models from data. In particular, we focus on supervised learning

problems (Figure 1.1), where the goal is to model the relation between a set of input

variables and one or more output variables, which are considered to be dependent

on the inputs in some manner.

Since the handbook deals with artiﬁcial learning methods, we do not take into

consideration any argument of biological or cognitive plausibility of the learning

methods we present. Learning is postulated here as a problem of statistical estima-

tion of the dependencies between variables on the basis of empirical data.

The relevance of statistical analysis arises as soon as there is a need to extract

useful information from data records obtained by repeatedly measuring an observed

phenomenon. Suppose we are interested in learning about the relationship1 between

two observed variables x(e.g. the height of a child) and y (e.g. the weight of a

child), which are quantitative observations of some phenomenon of interest (e.g.

obesity during childhood). Sometimes, the a priori knowledge that describes the

relation between x and y is available. In other cases, no satisfactory theory exists,

and all that we can use are repeated measurements of x and y.

1Note that the term relation simply denotes the statistical association (due to a probabilistic

dependency) between the two variables and has no causal connotation.

16 CHAPTER 1. INTRODUCTION

Figure 1.1: The supervised learning setting. Machine learning aims to infer from

observed data the best model of the stochastic input/output dependency.

In this book, our focus is the second situation where we assume that only a set of

observed data is available. The reasons for addressing this problem are essentially

two. First, the more complex is the input/output relation, the less eﬀective will be

the contribution of a human expert in extracting a model of the relation. Second,

data-driven modelling may be a valuable support for the designer also in modelling

tasks where she can take advantage of existing knowledge.

Though machine learning is becoming a central component in many (so-called)

intelligent applications, we deem that simply considering it as a powerful computa-

tional technology would be utterly reductive. The process of extracting knowledge

from observations lies at the root of the modern scientiﬁc process, and the most

challenging issues in machine learning relate to well-established philosophical and

epistemological problems, notably induction or the notion of truth. This is the

reason why we added in this new version of the handbook a preliminary chapter to

situate the machine learning problem into the broader context of human knowledge

acquisition.

Modelling from data

Modelling from data is often viewed as an art, mixing an expert's insight with the

information contained in the observations. A typical modelling process cannot be

considered as a sequential process but is better represented as a loop with many

feedback paths and interactions with the model designer. Various steps are repeated

several times aiming to reach, through continuous reﬁnements, a good description

of the phenomenon underlying the data.

The modelling process consists of a preliminary phase that brings the data from

their original form to a structured conﬁguration and a learning phase that aims to

select the model, or hypothesis, that best approximates the data (Figure 1.2).

The preliminary phase can be decomposed in the following steps:

Problem formulation. Here the model designer chooses a particular application

domain, a phenomenon to be studied, a number of descriptive variables and

hypothesises the existence of a (stochastic) relation (or dependency) between

the measurable variables. The deﬁnition of the input variables (and where

necessary their transformations) is a very crucial step and is called feature

Figure 1.2: The modelling process and its decomposition in the preliminary phase

and learning phase.

18 CHAPTER 1. INTRODUCTION

Figure 1.3: A training set for a simple supervised learning problem with one input

variable x and one output variable y . The dots represent the observed samples.

engineering. It is important to stress here the proactive role played by the

human (in contrast to a tabula rasa approach), and that this role is a necessary

condition for any knowledge process.

Experimental design. This step aims to return a dataset which, ideally, should

be made of observations that are well-representative of the phenomenon in

order to maximise the performance of the modelling process [55].

Pre-processing. In this step, raw data are cleaned to make learning easier. Pre-

processing includes a large set of actions on the observed data, such as noise

ﬁltering, outlier removal, missing data treatment [124], feature selection, and

so on.

Once the preliminary phase has returned the dataset in a structured input/output

form (e.g. a two-column table), called training set , the learning phase begins. A

graphical representation of a training set for a simple learning problem with one

input variable xand one output variable y is given in Figure 1.3. This manuscript

will mostly focus on this second phase assuming that the preliminary steps have

already been performed by the model designer.

Suppose that, on the basis of the collected data, we wish to learn the unknown

dependency existing between the xvariable and the yvariable. The knowledge of

this dependency could shed light on the observed phenomenon and let us predict

the value of the output yfor a given input (e.g. what is the expected weight of a

child who is 120cm tall?). What is diﬃcult and tricky in this task is the ﬁniteness

and the random nature of data. For instance, a second set of observations of the

same pair of variables could produce a dataset (Figure 1.4) that is not identical to

the one in Figure 1.3 though both originate from the same measurable phenomenon.

This simple fact suggests that a simple interpolation of the observed data would

not produce an accurate model of the data.

The goal of machine learning is to formalise and optimise the procedure which

brings from data to model and consequently from data to predictions. A learning

procedure can be concisely deﬁned as a search, in a space of possible model conﬁg-

urations, of the model which best represents the phenomenon underlying the data.

As a consequence, a learning procedure requires both a search space where possible

solutions may be found and an assessment criterion that measures the quality of

the solutions in order to select the best one.

The search space is deﬁned by the designer using a set of nested classes with

increasing capacity (or representation power). For our introductory purposes, it is

Figure 1.4: A second realisation of the training set for the same phenomenon ob-

served in Figure 1.3. The dots represent the observed examples.

Figure 1.5: Training set and three parametric models which belong to the class of

ﬁrst order polynomials.

suﬃcient to consider here a class as a set of input/output models (e.g. the set of

polynomial models) with the same model structure (e.g. second-order degree) and

the capacity of the class as a measure of the set of input/output mappings which

can be approximated by the models belonging to the class.

Figure 1.5 shows the training set of Figure 1.3 together with three parametric

models which belong to the class of ﬁrst-order polynomials. Figure 1.6 shows the

same training set with three parametric models, which belong to the class of second-

order polynomials.

The reader could visually decide whether the class of second-order models is

more suitable or not than the ﬁrst-order class to model the dataset. At the same

time, she could guess which among the three plotted models is the one that produces

the best ﬁtting.

In real high-dimensional settings, however, a visual assessment of the quality

of a model is neither possible nor suﬃcient. Data-driven quantitative criteria are

therefore required. We will assume that the goal of learning is to achieve a good

statistical generalisation. This means that the learned model is expected to return

an accurate prediction of the dependent (output) variable for new (unseen) values

of the independent (input) variables. By new values we intend values which are not

part of the training set but are generated by the same stochastic process.

Once the classes of models and the assessment criteria are ﬁxed, the goal of

a learning algorithm is to search i) for the best class of models and ii) for the

best parametric model within such a class. Any supervised learning algorithm is

then made of two nested loops denoted as the structural identiﬁcation loop and the

parametric identiﬁcation loop.

20 CHAPTER 1. INTRODUCTION

Figure 1.6: Training set and three parametric models which belong to the class of

second-order polynomials.

Structural identiﬁcation is the outer loop that seeks the model structure which

is expected to have the best accuracy. It is composed of a validation phase, which

assesses each model structure on the basis of the chosen assessment criterion, and a

selection phase which returns the best model structure on the basis of the validation

output. Parametric identiﬁcation is the inner loop that returns the best model for a

ﬁxed model structure. We will show that the two procedures are intertwined since

the structural identiﬁcation requires the outcome of the parametric step in order to

assess the goodness of a class.

Statistical machine learning

On the basis of the previous section, we could argue that learning is nothing more

than a standard problem of optimisation. Unfortunately, the reality is far more

complex. In fact, because of the ﬁnite amount of data and their random nature,

there exists a strong correlation between parametric and structural identiﬁcation

steps, which makes non-trivial the problem of assessing and, ﬁnally, choosing the

prediction model. In fact, the random nature of the data demands a deﬁnition of

the problem in stochastic terms and the adoption of statistical procedures to choose

and assess the quality of a prediction model. In this context, a challenging issue is

how to determine the class of models more appropriate to our problem. Since the

results of a learning procedure are found to be sensitive to the class of models chosen

to ﬁt the data, statisticians and machine learning researchers have proposed over

the years a number of machine learning algorithms. Well-known examples are linear

models, neural networks, local modelling techniques, support vector machines, and

regression trees. The aim of such learning algorithms, many of which are presented

in this book, is to combine high generalisation with an eﬀective learning procedure.

However, the ambition of this handbook is to present machine learning as a sci-

entiﬁc domain that goes beyond the mere collection of computational procedures.

Since machine learning is deeply rooted in conventional statistics, any introduc-

tion to this topic must include some introductory chapters to the foundations of

probability, statistics and estimation theory. At the same time, we intend to show

that machine learning widens the scope of conventional statistics by focusing on a

number of topics often overlooked by statistical literature, like nonlinearity, large

dimensionality, adaptivity, optimisation and analysis of massive datasets.

It is important to remark, also, that the recent adoption of machine learning

models is showing the limitation of pure black-box approaches, targeting accuracy

at the cost of interpretability. This is made evident by the embedding of automatic

approaches in decision-making processes with impact on ethical, social, political, or

juridical aspects. While we are personally skeptical about gaining any interpretabil-

ity from a large number of parameters and hyperparameters underlying a supervised

learner, we are conﬁdent that human insight can be obtained by techniques able

to reduce or modularise large variate tasks. In this direction, feature selection and

causal inference techniques are promising approaches to master the complexity of

data-driven modelling and return human accessible descriptions (e.g. in the form

of mechanisms).

This manuscript aims to ﬁnd a good balance between theory and practice by

situating most of the theoretical notions in a real context with the help of practical

examples and real datasets. All the examples are implemented in the statistical

programming language R [160] made available by the companion package gbcode

(Appendix F). In this second edition, we provide as well a number of Shiny dash-

boards (Appendix G) to give the reader a more tangible idea of somewhat abstract

concepts. For an introduction to R we refer the reader to [53, 189]. This prac-

tical connotation is particularly important since machine learning techniques are

nowadays more and more embedded in plenty of technological domains, like bioin-

formatics, robotics, intelligent control, speech and image recognition, multimedia,

web and data mining, computational ﬁnance, business intelligence.

Outline

The outline of the book is as follows. Chapter 2 is one of the novelties of the second

edition. Its aim is to situate the process of modelling from data in a larger epistemo-

logical domain dealing with the problem of extracting knowledge from observations.

We deem it interesting to show how some of the formal problems addressed in the

book dates back to old philosophical disputes and works. Chapter 3 summarises the

relevant background material in probability. Chapter 4 has been added to introduce

graphical modelling, a ﬂexible and interpretable way of representing large variate

problems in probabilistic terms. In particular, this formalism puts into evidence the

importance of conditional independence as a key notion to illustrate the properties

of dependencies and simplify the modelling of large dimensional tasks. Chapter 5 in-

troduces the parametric approach to parametric estimation and hypothesis testing.

Chapter 6 presents some nonparametric alternatives to the parametric techniques

discussed in Chapter 5. Chapter 7 introduces supervised learning as the statistical

problem of assessing and selecting a hypothesis function on the basis of input/output

observations. Chapter 8 reviews the steps which lead from raw observations to a

ﬁnal model. This is a methodological chapter that introduces some algorithmic

procedures underlying most of the machine learning techniques. Chapter 9 presents

conventional linear approaches to regression and classiﬁcation. Chapter 10 intro-

duces some machine learning techniques which deal with nonlinear regression and

classiﬁcation tasks. Chapter 11 presents the model averaging approach, a recent and

powerful way for obtaining improved generalisation accuracy by combining several

learning machines. Chapter 12 deals with the problem of dimensionality reduction

and in particular with feature selection strategies. Chapter 13 has been added in the

2nd edition to make clear the limitations of associational approaches and to stress

the risk of wrong extrapolation and biases if pure statistical results are interpreted

in a causal manner. We believe that causal reasoning represents the ultimate step

in the data analytics process going from data to knowledge.

Although the book focuses on supervised learning, some related notions of un-

supervised learning and density estimation are presented in Appendix A.

22 CHAPTER 1. INTRODUCTION

1.1 Notations

Throughout this manuscript, boldface denotes random variables and normal font is

used for instances (realisations) of random variables. Strictly speaking, one should

always distinguish in notation between a random variable and its realisation. How-

ever, we will adopt this extra notational burden only when the meaning is not

clear from the context. Then we will use Prob {z } (or (p (z )) as a shorthand for

Prob {z =z} ((pz (z )) when the identity of the random variable is clear from the

context.

As far as variables are concerned, lowercase letters denote scalars or vectors of

observables, greek letters denote parameter vectors, and uppercase denotes matrices.

Uppercase in italics denotes generic sets while uppercase in greek letters denotes

sets of parameters.

Gender-neutral pronoun: computer sciences suﬀer from the gender issue and

probably much more than other sciences. Of course, you won't ﬁnd any solution in

this book but the author (a man) felt odd in referring to a generic reader by using

a masculine pronoun only. He then decided to use as much as possible a "(s)he"

notation or, alternatively, a (balanced) random gender choice.

Generic notation

-θ : Parameter vector.

-θ : Random parameter vector.

-M : Matrix.

- [N× n ] or [N, n]: Dimensionality of a matrix with N rows and ncolumns.

-MT : Transpose of the matrix M.

- diag[m1 , . . . , mN ]: Diagonal matrix with diagonal [m1 , . . . , mN ]

-M : Random matrix.

-ˆ

θ: Estimate of θ.

-ˆ

θ: Estimator of θ.

-τ : Index in an iterative algorithm.

Probability Theory notation

- Ω : Set of possible outcomes.

-ω : Outcome (or elementary event).

-{E} : Set of possible events.

-E : Event.

- Prob {E} : Probability of the event E.

- (Ω, {E}, Prob {·} ): Probabilistic model of an experiment.

-Z : Domain of the random variable z.

1.1. NOTATIONS 23

-P (z ): Probability distribution of a discrete random variable z . Also Pz (z).

-F (z ) = Prob {z ≤z } : Distribution function of a continuous random variable

z. Also Fz (z ).

-p (z ): Probability density of a continuous r.v.. Also pz (z).

-E [z ]: Expected value of the random variable z.

-Ex [z ] = R X z (x, y)p(x ) dx : Expected value of the random variable z averaged

over x.

- Var [z ]: Variance of the random variable z.

-LN (θ ): Likelihood of a parameter θgiven the dataset DN .

-lN (θ ): Log-Likelihood of a parameter θgiven the dataset DN .

-U (a, b ): univariate uniform probability density between a and b≥ a .

-N (µ, σ2 ): univariate Normal probability density with mean µand variance

σ2 (Section 3.4.2).

-z∼pz (z ): random variable z with probability density pz (z).

-z∼ N (µ, σ 2 ): random variable z with Normal density with mean µand

variance σ2 .

Learning Theory notation

-x : Multidimensional random input variable.

-xj :j th component of the multidimensional input variable.

-X ⊂ Rn : Input space.

-y : Multidimensional output variable.

-Y ⊂ R : Output space.

-xi :i th observation of the random vector x.

-xij :i th observation of the jth component of the random vector x.

-f (x ): Target regression function.

-w : Random noise variable.

-zi =hxi , yi i : Input-output example (also observation or data point): ith case

in training set.

-N : Number of observed examples in the training set.

-DN ={z1 , z2, . . . , zN } : Training set.

- Λ: Class of hypothesis.

-α : Hypothesis parameter vector.

-h (x, α ): Hypothesis function.

24 CHAPTER 1. INTRODUCTION

- Λs : Hypothesis class of capacity (or complexity) s.

-L (y, f (x, α )): Loss function.

-R (α ): Functional risk.

-α0 : arg minα∈Λ R (α ).

-Remp (α ): Empirical functional risk.

-αN : Parameter which minimises the empirical risk of DN

-GN : Mean integrated squared error (MISE).

-l : Number of folds in cross-validation.

-ˆ

Gcv : Cross-validation estimate of GN .

-ˆ

Gloo : Leave-one-out estimate of GN .

-Ntr : Number of examples used for training in cross-validation.

-Nts : Number of examples used for test in cross-validation.

-D(i) : Training set with the ith example set aside.

-αN(i ) : Parameter which minimises the empirical risk of D(i) .

-ˆ

Gbs : Bootstrap estimate of GN .

-D(b) : Bootstrap training set of size N generated by DN with replacement.

-α(b) : Parameter which minimises the empirical risk of the bootstrap set D(b) .

-B : Number of bootstrap examples.

Data analysis notation

-xi :i th row of matrix X.

-x·j :j th column of matrix X.

-xij :j th element of vector xi .

-Xij :ij th element of matrix X.

-q : Query point (point in the input space where a prediction is required).

- ˆ yq : Prediction in the query point.

- ˆ y−j

i: Leave-one-out prediction in x i with the j th s example set aside.

-eloo

j=y j −ˆ y−j

j: Leave-one-out error with the j th example set aside.

-K (· ): Kernel function.

-B : Bandwidth.

1.1. NOTATIONS 25

-β : Linear coeﬃcients vector.

-ˆ

β: Least-squares parameters vector.

-ˆ

β−j : Least-squares parameters vector with the jth example set aside.

-hj (x, α ): jth ,j = 1, . . . , m , local model in a modular architecture.

-ρj : Activation or basis function.

-ηj : Set of parameters of the activation function.

26 CHAPTER 1. INTRODUCTION

Chapter 2

Setting the foundations:

machine learning and

epistemology

Machine learning is a relatively new discipline, but its foundations rest on much

older notions like modelling, reasoning, information, truth, knowledge, uncertainty,

induction. Nowadays, much of those notions have a mathematical and/or com-

putational interpretation, also thanks to machine learning. Nevertheless, before

reaching a mathematical formalisation, they have been the object of an extensive

philosophical inquiry and discussion. The aim of this chapter (primarily inspired by

the book [81]) is to provide a rapid historical journey over the most important con-

tributions of philosophy to epistemology, the branch of philosophy of science that

investigates how humans extract and attain knowledge in the scientiﬁc process.

The two main phases of human reasoning are the acquisition of true knowledge

and its manipulation in a truth-preserving manner. Induction is concerned with the

ﬁrst part, while deductive logic addresses the second one. In ancient times, logic

was the only aspect of knowledge that deserved the attention of philosophers and

epistemologists. A possible reason was that, until the scientiﬁc revolution, it was a

common belief that either truth was inaccessible (e.g. the allegory of Plato's cave)

or could be attained only by an initiatory process of inspiration, made possible by

the benevolence of God.

2.1 Deductive logic

The most ancient discipline formalising the notions of truth, reasoning, and knowl-

edge is logic, whose origin dates back to Aristotle. Logic is concerned with deﬁning

the properties that reasoning mechanisms should have in order to transform con-

sistently true statements into other true statements. The objects of reasoning are

arguments, i.e. groups of propositions where a proposition is a statement that can

be either true or false. According to [106] an argument (or inference) is made of

two groups of statements, one of which (premises) is claimed to provide support for

the other (conclusions). For instance

If A, then C

C∴

28 CHAPTER 2. SETTING THE FOUNDATIONS

is an argument where the groups of premises is made of the two propositions ("If

A, then C" and "A") and the conclusion is the proposition "C". Premises are the

statements that deﬁne the evidence while the conclusion is the statement that the

evidence is supposed to imply. An argument consisting of exactly two premises and

one conclusion, like the one above, is called a syllogism. If one of the two premises is

in the conditional form (as the example above), it is called a hypothetical syllogism.

Logic cannot, in general, tell whether premises are true or false (factual claim). It

is instead concerned with the quality of the reasoning process, which links premises

to conclusions (inferential claim). Its purpose is to develop methods and techniques

that allow us to distinguish good arguments (where the premises do support the

conclusion) from bad ones. In particular, logic distinguishes between validity and

sound arguments. An argument is valid if

•it is logically impossible for the conclusion to be false when the premises are

true,

•conclusion is a logical consequence of (it follows from) the premises,

•it is truth-preserving, i.e. the conclusion is implicitly contained in the premises.

Two examples of valid arguments are

1. Premises: "If A, then C." and "A is true". Conclusion:"C is true".

2. Premises: "Every F is G." and "b is F". Conclusion: "b is G".

Validity is something that is determined by the relationship between premises and

conclusion ("does the premises support the conclusion?") and not by the actual

truth of premises and/or conclusions1. It follows that valid arguments are risk-free

arguments. Note also that the validity of an argument depends only on its form

(or pattern) and not on the content (i.e. no matter what are the substitutes for A

and C in the ﬁrst argument). A valid argument is also called a deductive argument.

Examples of deductive arguments are arguments in which the conclusion depends

on some arithmetic or geometric computations or mathematical demonstrations.

All arguments in pure mathematics are deductive.

An argument is sound if it is valid and its premises are true. Soundness for

deductive logic has to do with both the validity and truth of the premises. Every

sound argument, by deﬁnition, will have a true conclusion as well. For instance, the

argument

All Italians play pretty good football

Gianluca is Italian.

Gianluca plays pretty good football ∴

is valid since the conclusion follows necessarily from the premises but not sound

(otherwise, Gianluca would have been playing for Fiorentina AC).

2.2 Formal and empirical science

Epistemologists are used to distinguishing between formal and empirical sciences.

Deductive arguments are the workhorse of formal sciences like geometry and math-

ematics. Those disciplines are built on a number of axioms, taken for true, and

1with the exception that a deductive argument with true premises and a false conclusion is

necessarily invalid

2.3. INDUCTION, PROJECTION, AND ABDUCTION 29

on an eﬀective truth-preserving mechanism. As such, they reason about a concep-

tual world, not necessarily in relation to the material world, where it is possible

to deﬁne notions of truth, correctness, and soundness. On the empirical side, we

ﬁnd disciplines like physics, biology, and economics, whose statements are supposed

to have a strong relationship with (some aspects of) sensible human experiences.

Though empirical sciences often rely on formal sciences to deﬁne notions, concepts,

and models, the validity of an empirical science proposition does not derive exclu-

sively from its formal truth but essentially from the fact that its predictions are

in accordance with experimental observations. Empirical sciences make then use of

inductive arguments where the content of the conclusion is in some way intended

to go beyond the content of the premises: a typical example is a prediction about

a future event based on the observation of some events, i.e. the supervised learning

scenario illustrated in Figure 1.1.

Modern empirical science, and the critical analysis of its inductive basis, be-

gan around the 16th and 17th centuries when the demand for new technologies

(e.g. for military or exploration reasons) stimulated the inquiry into the origins of

knowledge. In 1620 Francis Bacon, an English philosopher (1561-1626), published

the Novum Organum , which presented an inductivist view of science. According to

Bacon, scientiﬁc reasoning consists of making generalisations, or inductions , from

observations to general laws of nature (e.g. moving to the conclusion that all swans

are white after a number of historical observations). In other terms, the observa-

tions are supposed to induce the formulation of natural laws in the mind of the

scientist.

2.3 Induction, projection, and abduction

Induction is deﬁned as an inference in which one takes the past as grounds for

beliefs about the future or the observed as grounds for beliefs about the unob-

served. In other words, an inference is ampliative , i.e. it has more content in the

conclusion than in the premises, unlike logical reasoning, which is deductive and

non-ampliative. In inductive inference, the premises or departure points are called

data or observations, and the conclusions are referred to as hypotheses2. A proba-

bilistic language is usually adapted to express a hypothesis derived from induction.

Induction has the following properties that contrast with the deductive pattern of

inference [19]

1. The conclusion (e.g. hypothesis h (D )) follows non-monotonically from the

premises (e.g. the dataset D). The addition of an extra premise (i.e. more

data) might change the conclusion even when the extra premise does not

contradict any of the other premises. In other terms, D1 ⊂D2 6⇒ h ( D1 ) ⊂

h( D2 ), where h( D ) is the inductive consequence of the set of observations D.

2. The truth of the premises is not enough to guarantee the truth of the conclu-

sion as there is no correspondence to the notion of deductive validity.

3. There is an information gain in induction since a hypothesis asserts more than

data alone.

Another substantial diﬀerence is that, while logical arguments derive their va-

lidity from their form, this does not apply to inductive arguments: two inductive

arguments may have the same form, but one may be good and the other not. So

inductive inference is both useful and unsafe: no conclusion is a guaranteed truth,

and it can dissolve even if no premise is removed.

2Note that this should not be confused with what mathematicians call mathematical induction

which is a kind of deduction

30 CHAPTER 2. SETTING THE FOUNDATIONS

There are several forms of inductive arguments:

1. Statement about a sample drawn from a population ⇒Statement about the

population as a whole

2. Statement about a population ⇒Statement about a sample

3. Statement about a sample ⇒ Statement about a new sample

4. Observation of facts ⇒ Hypothesis

The third form of inference is also called projection [81] and is implemented in statis-

tical learning by memory-based (e.g. lazy learning in Section 10.1.11) or transduc-

tion algorithms. The fourth form is also known as abduction , explanatory inference,

deduction in reverse or inference to the best explanation. Abduction is a less ambi-

tious form of induction since it does not infer to a generalisation but to a hypothesis

that explains the data. In abduction, given h→ D and the observation of D we

infer the condition h. The rationale is that explanatory considerations are a guide

to inference: in other words, the hypothesis that would (if correct) best explains

the evidence is the hypothesis that is most likely to be correct. Note that this is

the mechanism typically used in statistical hypothesis testing (Section 5.11).

An example of abduction is the Darwin theory. At his time, Darwin inferred the

hypothesis of natural selection because, though not entailed by biological evidence,

natural selection would provide the best explanation of that evidence. Darwin

did not witness speciﬁc cases of evolution but formulated his hypothesis as an

explanation of the available observations.

2.4 Hume and the induction problem

The downside of the induction success is its problematic and unsafe aspect, i.e. the

projection of regularity onto unseen cases. The main problem of induction is how

to justify the inference from the observed (data) to the unobserved (laws of nature),

from the past (historical time series) to the future (e.g. prediction).

David Hume (1711-1776) was a Scottish philosopher who studied the problem of

induction from a philosophic perspective. In 1739 he published A treatise of human

nature, one of the most inﬂuential books of Western philosophy. According to Hume,

all reasonings concerning nature are founded on experience, and all reasonings from

experience are founded on the supposition that the course of nature will continue

uniformly the same or in other terms that the future will be like the past. Any

attempt to show, based on experience, that a regularity that has held in the past

will hold in the future too will be circular (since based on the principle of regularity

itself).

So empirical sciences rely on a supposition that, as shown by Hume, has no

logical necessity. In other words, there is no contradiction in supposing that the

future could be totally unlike the past (Figure 2.1) since we have no logical reason

to expect that the past resembles the future.

So why do humans expect the future to be like the past? According to Hume,

this is part of human nature: we have inductive habits, but we cannot justify them.

The principle of uniformity of nature is not a priori true, nor it can be proved

empirically. There is no reason beyond induction to justify inductive reasoning.

Thus, Hume oﬀers a naturalistic explanation of the psychological mechanism by

which empirical predictions are made but not any rational justiﬁcation for this

practice. Our inductive practices rest on habit and custom and cannot be justiﬁed

by rational argument. Induction is psychologically natural to us [81].

2.5. LOGICAL POSITIVISM AND VERIFICATIONISM 31

Figure 2.1: Falsiﬁcation of the inductive hypothesis "Are all swans white?"

2.5 Logical positivism and veriﬁcationism

Logical positivism is a philosophical movement belonging to the wider family of

empiricism, which developed in Europe after World War I and was established by a

group of people (including Schlick, Neurath, and Carnap), also known as the Vienna

Circle. They were inspired by the developments in sciences at the beginning of the

XXth century, notably the work of Einstein. Two are the central ideas (or dogmas)

of logical positivism: the distinction between analytic and synthetic sentences and

the veriﬁability theory of meaning [81].

Analytic sentences are true or false, whatever is the world state. Analytical

truths (e.g. in mathematics and logic) are necessary but somewhat empty. Math-

ematics does not describe the world and is independent of experience: they are a

convention to use symbols in a particular way.

A synthetic sentence is true or false according to the actual state of the world.

The value of synthetic sentences resides then in their method of veriﬁcation. In

other words, knowing the meaning of a sentence boils down to know how to verify

it through observation. Veriﬁcationism is a strong empiricist principle: the only

source of knowledge and the only source of meaning is observation. There are two

categories of veriﬁable statements: i) observation statements (e.g. the temperature

is below zero) which are directly veriﬁable, and ii) theoretical statements (indirectly

veriﬁable) from which we can deduce observation statements.

Veriﬁcationists reject as "meaningless" statements speciﬁc to entire ﬁelds such

as metaphysics, theology, ethics since they do not imply veriﬁable observations.

Such statements may be meaningful in inﬂuencing emotions or human behaviour

but provide no truth value, information, or factual content.

Science consists then of veriﬁable and then meaningful claims. According to

the philosophy of logical positivism, a general statement or theory can be arrived

at by inductive reasoning. Moreover, if such a theory is veriﬁed by observation or

experiment, it can be promoted to a law. It follows that veriﬁability is the criterion

of what is and what is not science (demarcation criterion).

Logical positivists stress that almost none of the evidence in everyday life and

science may have the same degree of necessity as deductive logic. No evidence for a

scientiﬁc theory is ultimately decisive since there is always the possibility of error,

but this does not prevent science from being supported by evidence. The great aim

of science is to discover and establish generalisations since there is no alternative to

knowledge besides experience.

32 CHAPTER 2. SETTING THE FOUNDATIONS

However, the veriﬁcationist ambition of grounding scientiﬁc truth in experience

encountered some major problems related to the real possibility of verifying hy-

pothesis in practice:

1. pure observations do not exist: observations are always theory-laden , i.e. they

are inevitably aﬀected by the theoretical beliefs (or expectations) of the inves-

tigator. Observations are neither neutral nor exhaustive, even in a big data

world. To observe means to select what seems to be pertinent for the analysis,

and this demands a speciﬁc and voluntary action from the experimenter (e.g.

selection of the instrumentation or the language to communicate the results).

Unfortunately, the analyst is often unaware of such selection, inducing then

dangerous bias in the possible conclusions (Section 13.7.4).

2. no scientiﬁc assumption is testable in complete isolation (also known as the

problem of holism about testing): the dogma of veriﬁcationism is naive since,

in practice, only whole complex structured hypotheses may be submitted to

empirical tests. Our ideas and hypothesis have contact with the experience

only as a whole. Whenever we assess a theory by comparing it with observa-

tions, we need many additional assumptions to put a theoretical statement at

the same level of observations.

3. unobservable entities escape from veriﬁcation: one of the basic claims of logical

positivists is that all aspects of science can be reduced to observational state-

ments and submitted to veriﬁcation (in science, there are no depths, there is

surface everywhere ). However, many successful and universally accepted sci-

entiﬁc formulations rely on hidden structures and notions that are not directly

observable (or mapped to observations in a univocal manner). Consider, for

instance, the notions of gene or electron and the signiﬁcant impact they have

on the human understanding of reality.

Such criticisms contributed to the decline of the positivist program and opened the

way to alternative interpretations of the knowledge discovery process.

2.6 Popper and the problem of induction

Karl Popper (1902-1994) is generally regarded as one of the greatest philosophers

of science of the 20th century. His ﬁrst achievement was an original deﬁnition of

science based on the distinction between scientiﬁc and pseudo-scientiﬁc statements

(also known as the demarcation problem). The solution he proposes is called falsiﬁ-

cationism in opposition to the veriﬁcationism of positivists. Falsiﬁcationism claims

that a hypothesis is scientiﬁc if and only if it has the potential to be refuted by

some possible observation. To be scientiﬁc, a hypothesis has to entail testable pre-

diction; in other words, it has to be bold, to take a risk. For instance, All F is G is

a scientiﬁc statement while Some F is G is not. All scientiﬁc theories are univer-

sal in nature, and no ﬁnite collection of observation statements, however great, is

logically equivalent to or can justify an unrestricted universal proposition. At the

same time, we are never completely sure that a theory is true (aka fallibilism ). A

well-known example is Newton's physics which was considered for a long time as a

gold standard of scientiﬁc theory until it was shown to be false in several respects.

Popper was sceptical about all forms of conﬁrmation and notably about the

theory of conﬁrmation proposed by empiricists. According to him, the only good

reasoning is deductively valid reasoning. According to Popper, humans or scientists

do not make inductions; they make conjectures (or hypotheses) and test them (or

their logical consequences obtained by deduction). If the test is successful, the

conjecture is corroborated but never veriﬁed or proven. Conﬁrmation is thus a

2.7. INSTRUMENTALISM 33

myth : no theory or belief about the world can be proven. Though no number of

positive experimental outcomes can demonstrate the truth of a scientiﬁc theory, a

single genuine counter instance can refute it (modus tollens ). It follows that we learn

something by deduction and not by induction. If the empirical test of the conjecture

is not successful, the conjecture is refuted. The refutation of a hypothesis leads us

(or the scientiﬁc community) to revise it or devise a more robust one. The ﬁnal

result is that scientiﬁc laws are falsiﬁable yet strictly unveriﬁable.

Scientiﬁc knowledge evolves via a two-step cycle that repeats endlessly: the ﬁrst

stage is made of conjecture making. The second stage is attempted refutation, when

the hypothesis is submitted to critical testing. The most important qualities of a

scientist are then imaginative (almost artistic) creativity, and rigorous testing.

Also, according to Popper, there are no "pure" or theory-free observations. Ob-

servation is always selective: it needs a chosen object, a deﬁnite task, an interest, a

point of view, a problem. Observation is theory-laden and involves applying theoret-

ical terms, descriptive language, and a conceptual scheme to particular experimental

situations.

2.7 The hypothetico-deductive method and instru-

mentalism

Nowadays, the most commonly agreed vision of science (hypothetico-deductivism )

merges the main ideas of logic and induction, realism and empiricism, of veriﬁca-

tionism and falsiﬁcationism. According to this vision, science is a process where

scientists formulate hypotheses (e.g. inductive step after a preliminary stage where

observations were collected) and then deduce observational predictions from them.

If predictions are accurate, then the theory is supported (in agreement with logical

positivists) or (e.g. in Bayesian terms) its degree of truth increases. If predictions

are not accurate, the theory is disconﬁrmed (this is coherent with Popper). The

more tests a theory passes, the more conﬁdence we can have in its truth3.

If the value of scientiﬁc models is intimately related to the quality of prediction,

they should be seen more as useful tools (or instruments) than a faithful representa-

tion of reality. The notion of instrumentalism was introduced by Van Fraassen [184].

An instrumentalist does not worry about whether a theory is a true description of

the world (e.g. if electrons really exist). The role of a theory is to establish a good

prediction. The question of whether our theory has some deeper match in the real

world will never have an answer so we should stop asking it.

Van Fraassen thinks that the only aim of theories is to accurately describe the

observable parts of the world. If this happens, they are empirically adequate. Trying

to address the hidden nature of reality is of no interest to science.

2.8 Epistemology and machine learning: the cross-

fertilisation

This chapter sketched some contributions of epistemology to the understanding of

how humans extract and attain knowledge from observations, in particular during

the scientiﬁc endeavour.

Machine learning, the topic of this book, is a computationally based approach

aiming to produce knowledge from observed data. If we make the basic assumption

3Note also that the predominating use of probabilistic hypotheses to take into account noisy

observation is in contradiction with the restrictive vision of Popper on logical deduction.

34 CHAPTER 2. SETTING THE FOUNDATIONS

that both epistemology and machine learning refers to the same notion of knowl-

edge (i.e. knowledge useful for human beings), an epistemological approach can be

useful to understand both limits and potential of machine learning. The author

is convinced that a fruitful cross-fertilisation can derive from a stronger synergy

between epistemology and machine learning. In particular, he expects the following

contributions from a machine learning approach to the study of knowledge discov-

ery:

•machine learning deals intimately with induction or how observations can

induce and/or conﬁrm a theory, one of the most fundamental problems of

philosophy of science. Also, it implements in a reproducible and testable way

the mechanism of learning, generation of hypothesis, and testing.

•machine learning is today unavoidable in supporting discovery in scientiﬁc

domains where human experts would be overwhelmed by complexity and di-

mensionality.

•machine learning is a key factor of the revolution transforming all empirical

sciences into data sciences, i.e. inductive disciplines where the quality and the

accuracy of the discoveries are strictly dependent on the capacity of extracting

accurate information, predictions, or models from large amounts of observed

data.

•machine learning generalises and democratises the notion of observed evidence

by making it converging with the notion of data. Every instrument (or tool or

simulator) producing data can be taken as the starting point of a knowledge

discovery process. This extends the common notion of experimental evidence

adopted in conventional sciences, like physics. A ﬁnancial transaction, a tweet,

or a GPS trace may be for some domains as informative as a CERN multi-

million experience in physics.

•machine learning is the ultimate step in the scientiﬁc process moving from the

optimistic objective of ﬁnding true descriptions of reality to the more realistic

goal of attaining accurate models of observations.

At the same time, there a number of lessons that young data scientists could

learn from ancient and recent philosophers of science:

•A critical analysis of the role of observations and data: all empirical sciences

derive their justiﬁcation from the fact of being ﬁrmly founded on experiments.

The distinctive nature of machine learning, and a reason for its success, is the

automatic process of extracting knowledge from data. Observations and data

are then necessary conditions for triggering any knowledge discovery proce-

dure. There is, however, the risk to sanctify the role of data (or facts) as

an unquestionable and objective foundation of truth. This excess has been

several times discussed and criticised by epistemologists (notably the crit-

ics of logical positivism). Pure facts and theory-neutral observations do not

exist, not even in a big-data world. Observations (and more speciﬁcally ex-

periments) are never passive or beyond any suspicion: they are the results of

a speciﬁc human initiative (or intervention) that can be dictated by speciﬁc

objectives, constraints, and motivation. The presupposition that the truth

of empirical statements can be securely established only by observation is a

naive attitude that could lead to disastrous consequences (e.g. sexist or racist

AI applications due to sampling bias) [41].

•Skepticism about induction: the Hume analysis, conﬁrmed by theoretical anal-

ysis in machine learning (notable the no-free-lunch theorem) reminds us that

2.8. EPISTEMOLOGY AND MACHINE LEARNING: THE CROSS-FERTILISATION35

atabula rasa approach going from data to knowledge is not possible. There is

no univocal (or optimal) way of proceeding from observations to models since

every learning process relies on (explicit or more often implicit) assumptions.

This is also related to the notion of undetermination of theory by evidence,

which means that there will always be a range of alternative theories compat-

ible with observations.

•Importance of hypothesis generation and validation: this important lesson

comes straight from Popper and associates the scientiﬁc character of a knowl-

edge discovery process to the possibility of falsiﬁcation. In that sense, machine

learning complies with the Popper interpretation of science and goes further

by proposing a set of strategies for automatically generating hypotheses and

validating them by empirical evidence. In more actual terms, the best way

to ensure falsiﬁability to computation sciences is reproducibility and inter-

pretability. These two aspects are essential to guarantee the respect of high

standards of quality and rigour in computational approaches to knowledge

discovery. Forgetting the assumptions underlying any data-driven eﬀort may

lead to accepting biased conclusions and misinterpretations (e.g. from a causal

perspective), which are dangerously endorsed by the size of the dataset or the

complexity of the algorithmic approach.

•Model as tools: the adoption of complex representation of reality (though

characterised by high-level notions and principles) makes diﬃcult, if not un-

realistic, the validation of all the components of a model. As a consequence,

a model should not be considered as a faithful copy of reality but as a conve-

nient abstraction, which, if conﬁrmed by experimental validation, becomes a

useful instrument for prediction and decision making.

•The conﬁrmation of a hypothesis requires taking into account the procedures

involved in generating data: conﬁrmation of a hypothesis with observations

is not a go-nogo process. Since new evidence changes degrees of validity

(or degree of belief), a probabilistic approach is necessary. This is why any

introduction to machine learning needs ﬁrst an introduction to probability,

probabilistic reasoning, and then statistics.

36 CHAPTER 2. SETTING THE FOUNDATIONS

Chapter 3

Foundations of probability

Uncertainty is inescapable in the real world. Even without resort to indeterminism,

its pervasiveness is due to the complexity of reality and the limitations of human

observational skills and modelling capabilities. According to [119] uncertainty arises

because of limitations in our ability to observe the world, limitations in our ability

to model it, and possibly even because of innate nondeterminism. Probability theory

is one of many disciplines [143] concerned with the study of uncertain (or random)

phenomena. It is also, according to the author, one of the most successful ones

in terms of formalisation, theoretical and algorithmic developments and practical

applications. For this reason, in this book, we will adopt probability as the math-

ematical language to describe and quantify uncertainty. Uncertain phenomena,

although not predictable in a deterministic fashion, may present some regularities

and consequently be described mathematically by idealised probabilistic models.

These models consist of a list of all possible outcomes together with the respective

probabilities. The theory of probability makes it possible to infer from these models

the patterns of future behaviour.

This chapter presents the basic notions of probability which serve as a necessary

background to understand the statistical aspects of machine learning. We ask the

reader to become acquainted with two aspects: the notion of a random variable

as a compact representation of uncertain knowledge and the use of probability as

an eﬀective formal tool to manipulate and process such uncertain information. In

particular, we suggest the reader give special attention to the notions of conditional

and joint probability. As we will see in the following, these two related notions

are extensively used by statistical modelling and machine learning to deﬁne the

dependence and the relationships between random variables.

3.1 The random model of uncertainty

We deﬁne a random experiment as any action or process which generates results

or observations which cannot be predicted with certainty. Uncertainty stems from

the existence of alternatives. In other words, each uncertain phenomenon is charac-

terised by a multiplicity of possible conﬁgurations or outcomes. Weather is uncer-

tain since it can take multiple forms (e.g. sunny, rainy, cloudy,...). Other examples

of random experiments are tossing a coin, rolling dice, passing an exam or measuring

the time to reach home.

A random experiment is then characterised by a sample space Ω that is a (ﬁnite

or inﬁnite) set of all the possible outcomes (or conﬁgurations) ω of the experiment.

The elements of the set Ω are called experimental outcomes or realisations. For

example, in the die experiment, Ω = {ω1 , ω2, . . . , ω6 } and ωi stands for the outcome

38 CHAPTER 3. FOUNDATIONS OF PROBABILITY

corresponding to getting the face with the number i . If ω is the outcome of a

measurement of some physical quantity, e.g. pressure, then we could have Ω = R+ .

The representation of an uncertain phenomenon is the result of a modelling

activity and, as such, it is not necessarily unique. In other terms diﬀerent repre-

sentations of a random experiment are possible. In the die experiment, we could

deﬁne an alternative sample space made of two sole outcomes: numbers equal to

and diﬀerent from 1. Also, we could be interested in representing the uncertainty

of two consecutive tosses. In that case, the outcome would be the pair (ω (t) , ω(t+1) )

where ω(t) is the outcome at time t.

Uncertainty stems from variability. Each time we observe a random phenomenon,

we may observe diﬀerent outcomes. In probabilistic jargon, observing a random

phenomenon is interpreted as the realisation of a random experiment. A single

performance of a random experiment is called a trial. This means that after each

trial, we observe one outcome ωi ∈ Ω.

A subset of experimental outcomes is called an event. Consider a trial that

generated the outcome ωi : we say that an event Eoccurred during the trial if the

set E contains the element ωi . For example, in the die experiment, an event (denoted

odd number ) is the set of odd values E = {ω1 , ω3, ω5 } . This means that when we

observe the outcome ω5 the event odd number takes place.

An event composed of a single outcome, e.g. E ={ ω1 } is called an elementary

event.

Note that since events Eare subsets, we can apply to them the terminology of

the set theory:

•Ω refers to the certain event i.e. the event that occurs in every trial.

•the notation

Ec = {ω: ω / ∈ E}

denotes the complement of E.

•the notation

E1 ∪ E2 = {ω ∈ Ω : ω ∈ E1 OR ω ∈ E2}

refers to the event that occurs when E1 or E2 or both occur.

•the notation

E1 ∩ E2 = {ω ∈ Ω : ω ∈ E1 AND ω ∈ E2}

refers to the event that occurs when both E1 and E2 occur.

•two events E1 and E2 are mutually exclusive or disjoint if

E1 ∩ E2 = ∅ (3.1.1)

that is each time that E1 occurs, E2 does not occur as well.

•a partition of Ω is a set of disjoint sets Ej ,j = 1, . . . , J (i.e. Ej 1 ∩ Ej 2 =∅

∀ji , j2 ∈J ) such that

∪J

j=1E j = Ω

•given an event Ewe deﬁne the indicator function of E by

IE ( ω) = ( 1 if ω∈ E

0 if ω / ∈ E (3.1.2)

3.1. THE RANDOM MODEL OF UNCERTAINTY 39

Let us consider now the notion of class of events. An arbitrary collection of

subsets of Ω is not a class of events. We require that if E1 and E2 are events, the

same also holds for the intersection E1∩E2 and the union E1 ∪E2 . A set of events that

satisﬁes these conditions is called, in mathematical terms, a Borel ﬁeld [142]. We

will consider only Borel ﬁelds since we want to deal not only with the probabilities

of single events but also with the probabilities of their unions and intersections.

3.1.1 Axiomatic deﬁnition of probability

Probability is a measure of uncertainty. Once a random experiment is deﬁned, this

measure associates to each possible outcome ω a number between 0 and 1. It follows

that we can assign to each event Ea real number Prob {E} ∈ [0, 1] which denotes

the probability of the event E . The measure associated with the event including all

possibilities is 1. The function Prob {·} : 2Ω → [0, 1] is called probability measure or

probability distribution and must satisfy the following three axioms:

1. Prob {E} ≥ 0 for any E.

2. Prob { Ω} = 1

3. Prob {E1 ∪ E2 } = Prob {E1 } + Prob {E2 } if E1 and E2 are mutually exclusive

(Equation (3.1.1)).

These conditions are known as the axioms of the theory of probability [120]. The

ﬁrst axiom states that all the probabilities are nonnegative real numbers. The sec-

ond axiom attributes a probability of unity to the universal event Ω, thus providing

a normalisation of the probability measure. The third axiom states that the prob-

ability function must be additive for disjoint events, consistently with the intuitive

idea of how probabilities behave.

So from a mathematician perspective, probability is easy to deﬁne: it is a

countably additive set function deﬁned on a Borel ﬁeld, with a total mass of one.

Every probabilistic property, for instance E1 ⊂ E2 ⇒ Prob {E1 } ≤ Prob {E2 } or

Prob {E c } = 1 − Prob {E} , can be derived directly or indirectly from the axioms

(and only the axioms).

There are many interpretations and justiﬁcations of these axioms, and we dis-

cuss the frequentist and the Bayesian interpretation in Section 3.1.4 brieﬂy. What

is relevant here is that the probability function is a formalisation of uncertainty

and that most of its properties and results appear to be coherent with the human

perception of uncertainty [110].

3.1.2 Visualisation of probability measures

Since probabilistic events are sets of outcomes, Venn diagrams are a convenient

manner to illustrate the relations between events and the notion of probability

measure. Suppose that you are a biker and you are interested in representing

the variability of weather and traﬃc conditions in your town in the morning. In

particular, you are interested in the probability that the morning will be sunny (or

not) and the road busy (or not). In order to formalise your practical issue, you

could deﬁne the uncertainty about the morning state by deﬁning a sample space

which is the set of all possible morning conditions. Two events are of interest here:

sunny mornings and traﬃc conditions. What is the relationship and probability of

such two events? Figure 3.1 illustrates the sample space, the two events, and the

(hypothetical) probability measures by means of a Venn diagram and two diﬀerent

tabular representations. The three representations in Figure 3.1 convey the same

information in diﬀerent manners. Notwithstanding, they do not necessarily scale-

up in the same manner if we take into consideration a larger number of events.

40 CHAPTER 3. FOUNDATIONS OF PROBABILITY

0.2 0.25

0.1 0.45

Not

Sunny 0.25

0.2

Traﬃc

Sunny 0.1

Traﬃc

Sunny

Traﬃc

11 0.1

0.21 0

0.250 1

0.45

Traﬃc

Sunny

Figure 3.1: Visualisation of two events and probability measures: Venn diagram

(left), two-way table (center), probability distribution table (right)

0.15 0.15

0.05

1 1 0.051

0.051 01

0.0511 0

1 00 0.15

110 0.1

0.15

0.1

0.35

0 01

10 0

Traﬃc

Sunny

Polluted

Sunny

Traﬃc

Polluted

0.1

0.05

0.1

Figure 3.2: Visualisation of three events and related probability measures: Venn

diagram (left), probability distribution table (right)

For instance, for n events the Venn diagram should contain all 2n hypothetically

possible zones1.

Suppose that you are also interested in another type of event, i.e. the air quality.

Adding such an event to your probability representation would make your Venn

representation more complicated and the two-way table inadequate (Figure 3.2).

The visualisation will still be more diﬃcult to handle and interpret if we deal with

more than three events.

Given their diﬃculty of encoding information in realistic probabilistic settings,

Venn diagrams are a pedagogical yet very limited tool for representing uncertainty.

Once introduced the notion of probability, a major question remains still open:

how to compute the probability value Prob {E} for a generic event E? The assign-

ment of probabilities is perhaps the most diﬃcult aspect of constructing probabilistic

models. Although the theory of probability is neutral, that is it can make infer-

ences regardless of the actual probability values, its results will be strongly aﬀected

by the choice of a particular assignment. This means that if the assignments are

inaccurate, the predictions of the model will be misleading and will not reﬂect the

real behaviour of the modelled phenomenon. In the following sections, we are going

to present some procedures which are typically adopted in practice.

3.1.3 Symmetrical deﬁnition of probability

Consider a random experiment where the sample space is made of a ﬁnite number

Mof symmetric outcomes (i.e., they are equally likely to occur). Let the number

of outcomes that are favourable to the event E (i.e. the event E takes place if one

of them occurs) be ME .

An intuitive deﬁnition of probability (also known as the classical deﬁnition) of

the event E, that adheres to the axioms, is

Prob {E} =M E

M(3.1.3)

1see Wikipedia https://en.wikipedia.org/wiki/Venn_diagram

3.1. THE RANDOM MODEL OF UNCERTAINTY 41

In other words, according to the principle of indiﬀerence (a term popularised by

J.M. Keynes in 1921), we have that the probability of an event equals the ratio of its

favourable outcomes to the total number of outcomes provided that all outcomes are

equally likely [142]. The computation of this quantity requires combinatorial meth-

ods for counting the favourable outcomes. This is typically the approach adopted

for a fair die. Also, in most cases, the symmetric hypothesis is accepted as self-

evident: if a ball is selected at random from a bowl containing W white balls and B

black balls, the probability that we select a white one is W/( W+ B ).

Note that this number is determined without any experimentation and is based

on symmetrical and ﬁnite space assumptions. But how to be sure that the symmet-

rical hypothesis holds? and that is invariant? Think, for instance, to the probability

that a newborn be a boy. Is this a symmetric case? More generally, how would one

deﬁne the probability of an event if the symmetrical hypothesis does not necessarily

hold or the space is not ﬁnite?

3.1.4 Frequentist deﬁnition of probability

Let us consider a random experiment and an event E . Suppose we repeat the

experiment N times and that we record the number of times NE that the event E

occurs. The quantity N E

N(3.1.4)

comprised between 0 and 1 is known as the relative frequency of E . It can be

observed that if the experiment is carried out a large number of times under exactly

the same conditions, the frequency converges to a ﬁxed value for increasing N . This

observation led von Mises to use the notion of frequency as a foundation for the

notion of probability.

Deﬁnition 1.1 (von Mises) . The probability Prob {E} of an event Eis the limit

Prob {E} = lim

N→∞

N(3.1.5)

where N is the number of observations and NE is the number of times that E

occurred.

This deﬁnition appears reasonable, and it is compatible with the axioms in

Section 3.1.1. However, in practice, in any physical experience, the number Nis

ﬁnite2 , and the limit has to be accepted as a hypothesis, not as a number that can

be determined experimentally [142].

Moreover, the assumption under exactly the same conditions is not as innocuous

as it seems. How could you ensure that two experiments occur under exactly the

same conditions? And what do those conditions refer to? Temperature, humidity,

obsolescence of the equipment? Are humans really able to control exactly all of

them? Would you be able to reproduce the exact same conditions of an experiment?

Notwithstanding, the frequentist interpretation is very important to show the

links between theory and application. At the same time, it appears inadequate to

represent probability when it is used to model a subjective degree of belief. Think,

for instance, to the probability that your professor wins a Nobel Prize: how to deﬁne

in such case a number Nof repetitions?

An important alternative interpretation of the probability measure comes then

from the Bayesian approach. This approach proposes a degree-of-belief interpreta-

tion of probability according to which Prob {E} measures an observer's strength of

belief that E is or will be true [192]. This manuscript will not cover the Bayesian

2As Keynes said "In the long run we are all dead".

42 CHAPTER 3. FOUNDATIONS OF PROBABILITY

approach to statistics and data analysis for the sake of compactness, though the

author is well aware that Bayesian machine learning approaches are more and more

common and successful. Readers interested in the foundations of the Bayesian in-

terpretation of probability are referred to [110]. Readers interested in introductions

to Bayesian machine learning are referred to [78, 13].

3.1.5 The Law of Large Numbers

A well-known justiﬁcation of the frequentist approach is provided by the Weak Law

of Large Numbers, proposed by Bernoulli.

Theorem 1.2. Let Prob {E} =p and suppose that the event E occurs NE times in

Ntrials. Then, N E

Nconverges to pin probability, that is, for any  > 0 ,

Prob     

N− p    ≤ →1as N→ ∞

According to this theorem, the ratio NE /N is close to pin the sense that, for any

 > 0, the probability that | NE /N − p| ≤  tends to 1 as N → ∞. This result justiﬁes

the widespread use of the frequentist approach (e.g. in Monte Carlo simulation) to

illustrate or numerically solve probability problems. The relation between frequency

and probability is illustrated by the Shiny dashboard lawlarge.R (package gbcode).

Note that such a result does not imply that the number NE will be close to Np

as one could naively infer from (3.1.5). In fact,

Prob {NE = Np} ≈ 1

p2πN p(1 − p)→ 0, as N → ∞ (3.1.6)

For instance, in a fair coin-tossing game, this law does not imply that the ab-

solute diﬀerence between the number of heads and tails should oscillate close to

zero [180] (Figure 3.3). On the contrary, it could happen that the absolute diﬀer-

ence keeps growing (though at a slower rate than the number of tosses) as shown

in the R script freq.R and the Shiny dashboard lawlarge.R.

3.1.6 Independence and conditional probability

Let us consider two diﬀerent events. We have already introduced the notions of

complementary and disjoint events. Another important deﬁnition is the deﬁnition

of independent events and the related notion of conditional probability. This notion

is essential in machine learning since supervised learning aims to detect and model

(in)dependencies by estimating conditional probabilities.

Deﬁnition 1.3 (Independent events) . Two events E1 and E2 are independent if and

only if

Prob {E1 ∩ E2 } = Prob {E1 } Prob {E2 } (3.1.7)

and we write E1 ⊥⊥ E2 .

The probability Prob {E1 ∩ E2 } of seeing two events occurring together is also

known as joint probability and often noted as Prob {E1 ,E2 } . If two events are inde-

pendent the joint probability depends only on the two individual probabilities. As

an example of two independent events, think of two outcomes of a roulette wheel

or of two coins tossed simultaneously.

From an uncertain reasoning perspective, independence is a very simplistic as-

sumption since the occurrence (or the observation) of one event has no inﬂuence

on the occurrence of the other, or similarly that the second event has no memory

3.1. THE RANDOM MODEL OF UNCERTAINTY 43

Figure 3.3: Fair coin-tossing random experiment: evolution of the relative frequency

(left) and of the absolute diﬀerence (right) between the number of heads and tails

(R script freq.R in gbcode).

of the ﬁrst. In other words, independence considers the uncertainty of a complex

joint event as a function of the uncertainties of its components3. This makes the

reasoning much simpler but, at the same time, too rough.

Exercise

Suppose that a fair die is rolled and that the number ω appears. Let E1 be the

event that the number ωis even, E2 be the event that the number ωis greater than

or equal to 3, E3 be the event that the number ωis a 4,5 or 6.

Are the events E1 and E2 independent? Are the events E1 and E3 independent?

•

Let E1 be an event such that Prob {E1 }> 0 and E2 a second event. We deﬁne

the conditional probability of E2 , given that E1 has occurred, the revised probability

of E2 after we learn about E1 occurrence:

Deﬁnition 1.4 (Conditional probability) . If Prob {E1 }> 0 then the conditional

probability of E2 given E1 is

Prob {E2|E1 } = Prob {E1 ∩ E2}

Prob {E1 } (3.1.8)

The following result derives from the deﬁnition of conditional probability.

Lemma 1. If E1 and E2 are independent events, then

Prob {E1|E2 } = Prob {E1 } (3.1.9)

In qualitative terms, the independence of two events means that the fact of

observing (or knowing) that one of these events (e.g. E1 ) occurred does not change

the probability that the other (e.g. E2 ) will occur.

3We refer the interested reader to the distinction between extensional and intensional reasoning

in [147]. Extensional reasoning (e.g. logics) always makes an assumption of independence, while

intensional reasoning (e.g. probability) consider independence as an exception.

44 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Example

Let E1 and E2 two disjoint events with positive probability. Can they be indepen-

dent? The answer is no since

Prob {E1 ∩ E2 } = Prob {∅} = 0 6 = Prob {E1 } Prob {E2 }> 0

or equivalently Prob {E1|E2 } = 0. We can interpret this result by noting that if

two events are disjoint, the realisation of one of them is highly informative about

the realisation of the other. For instance, though it is very probable that Italy

will win the next football World Cup (Prob {E1 } >> 0) , this probability goes

to zero if the (rare yet possible) event E2 ("World cup won by Belgium") occurs

(Prob {E1|E2 } = 0). The two events are then dependent.

•

Exercise

Let E1 and E2 be two independent events, and E c

1the complement of E 1 . Are E c

and E2 independent?

•

Exercise

Consider the sample space Ω and the two events E1 and E2 in Figure 3.4. Suppose

that the probability of the two events is proportional to the surface of the regions.

From the Figure we compute

Prob {E1 } = 9

100 = 0. 09 (3.1.10)

Prob {E2 } = 20

100 = 0. 2 (3.1.11)

Prob {E1 ∩ E2 } = 1

100 = 0. 01 6 = Prob {E1 } Prob {E2 } (3.1.12)

Prob {E1 ∪ E2 } = 0.28 = Prob {E1 } + Prob {E2 } − Prob {E1 ∩ E2 } (3.1.13)

Prob {E1|E2 } = 1

20 = 0.05 6 = Prob {E1 } (3.1.14)

Prob {E2|E1 } = 1

96 = Prob {E2 } (3.1.15)

and then derive the following conclusions: the events E1 and E2 are neither disjoint

nor independent. Also, it is more probable that E2 occurs given that E1 occurred

rather than the opposite.

•

From (3.1.8) we derive

Prob {E1 ,E2 } = Prob {E1 } Prob {E2|E1 } (3.1.16)

If we replace the event E2 with the intersection of two events E2 and E3 , from (3.1.16)

we obtain

Prob {E1 ,E2 ,E3 } = Prob {E1 } Prob {E2 ,E3|E1 } =

Prob {E1 } Prob {E2|E3 , E1 } Prob {E3|E1 } = Prob {E1 ,E3 } Prob {E2|E3 , E1 }

If we divide both terms by Prob {E3 } we obtain

Prob {E1 ,E2|E3 } = Prob {E1|E3 } Prob {E2|E1 ,E3 } (3.1.17)

which is the conditioned version of (3.1.16).

3.1. THE RANDOM MODEL OF UNCERTAINTY 45

Figure 3.4: Events in a sample space.

3.1.7 The chain rule

The equation (3.1.16) shows that a joint probability can be factorised as the prod-

uct of a conditional and an unconditional probability. In more general terms, the

following rule holds.

Deﬁnition 1.5 (Chain rule) . For any sequence of events E1 ,E2 ,...,En ,

Prob {E1 ,E2 ,...,En } =

Prob {E1 } Prob {E2|E1 } Prob {E3|E1 ,E2 } . . . Prob {En|E1 ,E2 ,...,En−1 }

We will see in Chapter 4 that the chain rule factorisation and the notion of

conditional independence play a major role in the adoption of graphical models to

represent probability distributions.

3.1.8 The law of total probability and the Bayes' theorem

Let us consider an indeterminate practical situation where a set of events E1 , E2 ,...,

Ek may occur. Suppose that no two such events may occur simultaneously, but at

least one of them must occur. This means that E1 , E2 ,..., Ek are mutually exclusive

and exhaustive or, in other terms, that they form a partition of Ω. The following

two theorems can be proven.

Theorem 1.6 (Law of total probability) . Let Prob {Ei} ,i = 1, . . . , k denote the

probability of the ith event Ei and Prob {E|Ei } ,i = 1, . . . , k the conditional probability

of a generic event E given that Ei has occurred. It can be shown that

Prob {E} =

i=1

Prob {E|Ei } Prob {Ei } =

i=1

Prob {E ∩ Ei } (3.1.18)

The quantity Prob {E} is referred to as marginal probability and denotes the

probability of the event Eirrespective of the occurrence of other events. A common-

sense interpretation of this theorem is that if an event E (e.g. an eﬀect) depends

on the realisation of kdisjoint events (e.g. causes), the probability of observing

E, is a weighted average of each single conditional probability Prob {E|Ei } where

the weights are given by the marginal probabilities of each event Ei , i = 1, . . . , k.

For instance, we can compute the probability that the highway is busy once we

46 CHAPTER 3. FOUNDATIONS OF PROBABILITY

know the probability that an accident occurred or not (two disjoint events) and the

conditional probabilities of traﬃc given the occurrence (or not) of an accident.

Theorem 1.7 (Bayes' theorem) . The conditional ("inverse") probability of any Ei ,

i= 1 , . . . , k given that E has occurred is given by

Prob {Ei |E} = Prob {E|Ei } Prob {Ei}

j=1 Prob {E|E j }Prob {E j }=Prob {E,E i }

Prob {E} i = 1, . . . , k

(3.1.19)

It follows that the Bayes theorem is the only sound way to derive from a condi-

tional probability Prob {E2|E1 } its inverse

Prob {E1|E2 } = Prob {E2|E1 } Prob {E1}

Prob {E2 } (3.1.20)

Any alternative derivation (or shortcut) will lead inevitably to fallacious reasoning

and inconsistent results (see the Prosecutor fallacy discussion in Section 3.1.9 ).

It may be useful also to write a conditioning version of the total probability.

Given an event E0 and the set E1 , E2 ,..., Ek of mutually exclusive events:

Prob {E|E 0 } =

i=1

Prob {E|Ei , E0 } Prob {Ei|E 0 } (3.1.21)

From (3.1.20) and by conditioning on a third event E3 , we obtain a conditioning

version of the Bayes theorem

Prob {E1|E2 ,E3 } = Prob {E2|E1 ,E3 } Prob {E1|E3}

Prob {E2|E3 } (3.1.22)

as long as Prob {E2|E3 }> 0

Example

Suppose that k = 2 and

• E1 is the event: "Tomorrow is going to rain".

• E2 is the event: "Tomorrow is not going to rain".

• E is the event: "Tonight is chilly and windy".

The knowledge of Prob {E1} , Prob {E2 } and Prob {E|Ek} ,k = 1, 2 makes possible

the computation of Prob {Ek|E}.

•

Exercise

Verify the validity of the law of total probability and of the Bayes theorem for the

problem in Figure 3.5.

•

3.1. THE RANDOM MODEL OF UNCERTAINTY 47

Figure 3.5: Events in a sample space

3.1.9 Direct and inverse conditional probability

The notion of conditional probability is central in probability and machine learning,

but it is often prone to dangerous misunderstanding, for instance, when inappropri-

ately used in domains like medical sciences or law. The most common error consists

of taking a conditional probability Prob {E1|E2 } for its inverse Prob {E2|E1 } . This

is also known as the prosecutor fallacy, as discussed in an example later.

The ﬁrst important element to keep in mind is that for any ﬁxed E1 , the quantity

Prob {·|E1 } still satisﬁes the axioms of probability, i.e. the function Prob {·|E1 } is

itself a probability measure. Conditional probabilities are probabilities [27]. How-

ever, this does not generally hold for Prob {E1|·}, which corresponds to ﬁx the term

E1 on the left of the conditional bar. For instance if E2 ,E3 and E4 are disjoint events

we have

Prob {E2 ∪ E3∪ E4 |E1 } = Prob {E2|E1 } + Prob {E3|E1 } + Prob {E4|E1}

in agreement with the third axiom (Section 3.1.1) but

Prob {E1|E2 ∪ E3∪ E4 } 6 = Prob {E1|E2 } + Prob {E1|E3 } + Prob {E1|E4}

Also it is generally not the case that Prob {E2|E1 } = Prob {E1|E2} . As a conse-

quence if E1 and E2 are not independent then

Prob {E c

1|E 2 }= 1 − Prob {E 1 |E 2 }

but

Prob {E1|E c

2} 6= 1 − Prob {E 1 |E 2 }(3.1.23)

where Ec denotes the complement of E.

Another remarkable property of conditional probability, which is also a distinc-

tive aspect of probabilistic reasoning, is its non-monotonic property. Given a non

conditional probability Prob {E1 }> 0 a priori, we cannot say anything about the

conditional term Prob {E1|E2 } . This term can be larger, equal or smaller than

Prob {E1} . For instance if observing the event E2 makes the event more (less) prob-

able then Prob {E1|E2 }> Prob {E1 } (Prob {E1|E2 }< Prob {E1 } ). If the two events

are independent, then the probability of E1 does not change by conditioning. It

follows that the degree of belief of an event (or statement) depends on the context.

Note that this does not apply to conventional logical reasoning where the validity

of a statement is context-independent.

48 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Italians

football supporters

World population

Figure 3.6: Italians and football fans.

In more general terms, it is possible to say that any probability statement is con-

ditional since it has been formulated on the basis of an often implicit background

knowledge K . For instance, if we say that the probability of the event E="rain

tomorrow" is Prob {E} = 0. 9, we are implicitly taking into consideration the sea-

son, our location and probably the weather today. So we should better note it as

Prob {E|K} = 0. 9. As succinctly stated in [27] all probabilities are conditional, and

conditional probabilities are probabilities.

Exercise

Consider as sample space Ω the set of all human beings. Let us deﬁne two events:

the set E1 of Italians and the set E2 of football supporters. Suppose that the prob-

ability of the two events is proportional to the surface of the regions in Figure 3.6.

Are these events disjoint? Are they independent? What about Prob {E1|E2 } and

Prob {E2|E1 } ? Are they equal? If not, which one is the largest?

•

The prosecutor fallacy

Consider the following story: A crime occurs in a big city (1M of inhabitants), and

a deteriorated DNA trace of the murderer is collected. The DNA proﬁle matches

the proﬁle of a person in a police database. A geneticist is contacted, and she states

that the probability of ﬁnding a person with the same DNA proﬁle is one out of

100 thousand (i.e. 1e− 5 ). The prosecution lawyer asks for condemnation with

the following argument: "since the chance of ﬁnding an innocent man with such

characteristics is so tiny, then the probability that he is innocent will be tiny as well".

The jury is impressed and ready to proceed with a life sentence. Then the defendant

replies: "Do you know that the population of the city is 1M? So the average number

of persons matching such DNA proﬁle is 10. His chance of being innocent is not so

tiny since it is 9/10 and not one in 100000" Lacking any additional evidence, the

suspect is acquitted.

This short story is inspired by a number of real cases in court that were con-

fronted with the serious error of confounding direct and inverse conditional probabil-

ity [169]. The impact of such false reasoning is so relevant in law that it is known as

the Prosecutor's fallacy, a common default in reasoning when the collected evidence

is tiny if the accused was innocent.

3.1. THE RANDOM MODEL OF UNCERTAINTY 49

Let us analyse in probabilistic terms the fallacious reasoning that occurred in

the example above. Let consider a criminal case for which we have 10 suspects, i.e.

the responsible and 9 innocent persons (out of a 1 million population) matching

the DNA proﬁle. The probability of matching evidence (M) given that someone is

innocent (I) is very low

Prob {M |I } = 9

999999 ≈ 1e− 5

However, what is relevant here is not the probability of the evidence given that he

is innocent (Prob {M |I } ) but the probability that is innocent given the evidence

Prob {I |M } = Prob {M|I } Prob {I}

Prob {M } = 9/ 999999 × 999999/1000000

10/ 1000000 = 9/ 10.

We can rephrase the issue in the following frequentist terms. Given Ninhabi-

tants, m persons with DNA matching proﬁles and a single murderer, the following

table shows the distribution of persons

Match No match

Innocent m− 1N− m

Guilty 1 0

From the table above, it is easy to derive the inconsistency of the prosecutor fallacy

reasoning since

Prob {M|I } =m− 1

N−1 ≈Prob {M } = m

Prob {I |M } =m− 1

m>> Prob {M|I}

•

3.1.10 Logics and probabilistic reasoning

This section aims to present some interesting relationships between logic deduction

and probabilistic reasoning.

First, we show that we can write down a probabilistic version of the deductive

modus ponens rule of propositional logic (Section 2.1):

If E1 ⇒ E2 and E1 is true, then E2 is true as well.

Since E1 ⇒ E2 is equivalent in set terms to E1 ⊂ E2 we obtain

Prob {E2|E1 } = Prob {E1 ,E2}

Prob {E1 } = Prob {E1}

Prob {E1 } = 1

i.e. a translation of the modus ponens argument in the probabilistic language.

Interestingly enough, the probability theory provides us with a result also in the

case of true E2 . It is well-known that in propositional logic if E1 ⇒ E2 and E2 is

true, then nothing can be inferred about E1 . Probability theory is more informative

since in this case we may derive from E2 ⊂ E1 that

Prob {E1|E2 } = Prob {E1}

Prob {E2 }≥ Prob {E1}

Note that this is a probabilistic formulation of the abduction principle (Section 2.3).

In other words, probability supports the following common-sense reasoning: if both

E1 ⇒ E2 and E2 apply, then the conditional probability of E1 (i.e. the probability

of E1 once we know that E2 occurred) cannot be smaller than the unconditional

probability (i.e. the probability of E1 if we knew nothing about E2 ).

50 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Also the properties of transitivity and inverse modus ponens hold in probability.

Let us consider three events E1 ,E2 ,E3 . The transitivity principle in logics states

that

If E1 ⇒ E2 and E2 ⇒ E3 then E1 ⇒ E3

In probabilistic terms we can rewrite E1 ⇒ E2 as

Prob {E2|E1 } = 1

and E2 ⇒ E3 as

Prob {E3|E2 } = 1

respectively. From the law of total probability (Equation (3.1.18)) we obtain

Prob {E3|E1 } = Prob {E3|E2 ,E1 } Prob {E c

2|E 1 }

| {z }

+ Prob {E3|E2 ,E1}

| {z }

Prob {E2|E1}

| {z }

= 1

Inverse modus ponens in logics states that

If E1 ⇒ E2 then ¬E2 ⇒ ¬E1

In probabilistic terms from Prob {E2|E1 } = 1 it follows

Prob {E c

1|E c

2}= 1 − Prob {E 1 |E c

2}= 1 −

Prob {E c

2|E 1 }

| {z }

Prob {E1}

Prob {E c

2}= 1

Those results show that deductive logic rules can be seen as limiting cases of proba-

bilistic reasoning and conﬁrm the compatibility of probability reasoning with human

common sense.

3.1.11 Combined experiments

So far we assumed that all the events belong to the same sample space. However,

the most interesting use of probability concerns combined (or multivariate) random

experiments whose sample space

Ω=Ω 1×Ω 2×. . . Ω n

is the Cartesian product of the spaces Ωi ,i = 1, . . . , n . For instance, if we want to

study the probabilistic dependence between the height and the weight of a child we

deﬁne a joint sample space

Ω = { (w, h ) : w∈ Ωw , h ∈ Ωh }

made of all pairs (w, h) where Ωwis the sample space of the random experiment de-

scribing the weight and Ωh is the sample space of the random experiment describing

the height.

Note that all the properties studied so far also holds for events that do not belong

to the same univariate sample space. For instance, given a combined experiment

Ω = Ω1 × Ω2 two events E1 ∈ Ω1 and E2 ∈ Ω2 are independent iﬀ Prob {E1|E2 } =

Prob {E1}.

Some examples of real problems modelled by random combined experiments are

presented in the following.

3.1. THE RANDOM MODEL OF UNCERTAINTY 51

Gambler's fallacy

Consider a fair coin-tossing game. The outcome of two consecutive tosses can be

considered independent. Now, suppose that we observe a sequence of 10 consecutive

tails. We could be tempted to think that the chances that the next toss will be head

are now very large. This is known as the gambler's fallacy [180]. In fact, to witness

a very rare event (like 10 consecutive tails) does not imply that the probability of

the next event will change or rather that it will become suddenly dependent on the

past.

•

Example [192]

Let us consider a medical study about the relationship between the outcome of a

medical test and the presence of a disease. We model this study as a combination

of two random experiments:

1. the random experiment which models the state of the patient. Its sample

space is Ωs ={ H, S } where H and S stand for a healthy and a sick patient,

respectively.

2. the random experiment which models the outcome of the medical test. Its

sample space is Ωo = { +, −} where + and −stand for a positive and a

negative outcome of the test, respectively.

The dependency between the state of the patient and the outcome of the test

can be studied in terms of conditional probability.

Suppose that out of 1000 patients, 108 respond positively to the test and that

among them, 9 result to be aﬀected by the disease. Also, among the 892 patients

who responded negatively to the test, only 1 is sick. According to the frequentist in-

terpretation, the probabilities of the joint events Prob {E s , Eo } can be approximated

according to expression (3.1.5) by

Es =S Es =H

Eo = + 9

1000 =.009 108−9

1000 =.099

Eo = −1

1000 =.001 892−1

1000 =.891

Doctors are interested in answering the following questions. What is the proba-

bility of having a positive (negative) test outcome when the patient is sick (healthy)?

What is the probability of being in front of a sick (healthy) patient when a positive

(negative) outcome is obtained? From the deﬁnition of conditional probability we

derive

Prob {E o = +|E s =S} = Prob {E o = +, Es =S }

Prob {E s =S} = . 009

.009 + .001 = .9

Prob {E o = −|E s =H} = Prob {E o = −, Es =H }

Prob {E s =H} = . 891

.891 + .099 = . 9

According to these ﬁgures, the test appears to be accurate. Does this mean that

we should be scared if we test positive? Though the test is accurate, the answer is

negative, as shown by the quantity

Prob {E s =S |E o = +} = Prob {E o = +, Es =S }

Prob {E o = +} = . 009

.009 + .099 ≈ .08

This example conﬁrms that sometimes humans tend to confound Prob {E s|E o } with

Prob {E o|E s } and that the most intuitive response is not always the right one (see

example in Section 3.1.9).

•

52 CHAPTER 3. FOUNDATIONS OF PROBABILITY

3.1.12 Array of joint/marginal probabilities

Let us consider the combination of two random experiments whose sample spaces are

ΩA ={A1 , ··· , An } and ΩB ={B1 , ··· , Bm } , respectively. Assume that for each

pair of events (Ai , Bj ), i = 1, . . . , n ,j = 1, . . . , m we know the joint probability value

Prob {Ai , Bj } . The joint probability array contains all the necessary information

for computing all marginal and conditional probabilities by means of (3.1.18) and

(3.1.8).

B1 B2 ·· · Bm Marginal

A1 Prob { A1 , B1 } Prob { A1 , B2 } ·· · Prob { A1 , Bm } Prob { A1 }

A2 Prob { A2 , B1 } Prob { A2 , B2 } ·· · Prob { A1 , Bm } Prob { A2 }

An Prob { An , B1 } Prob { An , B2 } ·· · Prob { An , Bm } Prob { An }

Marginal Prob {B1 } Prob { B2 } ·· · Prob {Bm } Sum=1

where Prob {Ai } = P j=1,...,m Prob {Ai , Bj } and Prob { Bj } = P i=1,...,n Prob {Ai , Bj } .

Using an entry of the joint probability matrix and the sum of the corresponding

row/column, we may use (3.1.8) to compute the conditional probability as shown

in the following example.

Example: dependent/independent scenarios

Let us model the commute time to go back home for a ULB student living in

St. Gilles as a random experiment. Suppose that its sample space is Ωt = { LOW,

MEDIUM, HIGH}. Consider also an (extremely:-) random experiment represent-

ing the weather in Brussels, whose sample space is Ωw = {G=GOOD, B=BAD}.

Suppose that the array of joint probabilities is

G (in Bxl) B (in Bxl) Marginal

LOW 0.15 0.05 Prob { LOW} = 0.2

MEDIUM 0.1 0.4 Prob { MEDIUM} = 0.5

HIGH 0.05 0.25 Prob { HIGH} = 0.3

Prob { G} = 0. 3 Prob { B} = 0. 7 Sum=1

According to the above probability function, is the commute time dependent on

the weather in Bxl? Note that if weather is good

LOW MEDIUM HIGH

Prob {·| G} 0.15/0.3=0.5 0.1/0.3=0.33 0.05/0.3=0.16

Else if weather is bad

LOW MEDIUM HIGH

Prob {·| B} 0.05/0.7=0.07 0.4/0.7=0.57 0.25/0.7=0.35

Since Prob {·| G } 6 = Prob {·| B } , i.e. the probability of having a certain commute time

changes according to the value of the weather, the relation (3.1.9) is not satisﬁed.

Consider now the dependency between an event representing the commute time

and an event describing the weather in Rome.

G (in Rome) B (in Rome) Marginal

LOW 0.18 0.02 Prob { LOW} = 0.2

MEDIUM 0.45 0.05 Prob { MEDIUM} = 0.5

HIGH 0.27 0.03 Prob { HIGH} = 0.3

Prob { G} = 0. 9 Prob { B} = 0. 1 Sum=1

Our question now is: is the commute time dependent on the weather in Rome?

If the weather in Rome is good we obtain

LOW MEDIUM HIGH

Prob {·| G} 0.18/0.9=0.2 0.45/0.9=0.5 0.27/0.9=0.3

3.1. THE RANDOM MODEL OF UNCERTAINTY 53

E1E2E3 P(E1 ,E2 ,E3 )

CLEAR RISING DRY 0.4

CLEAR RISING WET 0.07

CLEAR FALLING DRY 0.08

CLEAR FALLING WET 0.10

CLOUDY RISING DRY 0.09

CLOUDY RISING WET 0.11

CLOUDY FALLING DRY 0.03

CLOUDY FALLING WET 0.12

Table 3.1: Joint probability distribution of the three-variable probabilistic model of

the weather

while if the weather in Rome is bad

LOW MEDIUM HIGH

Prob {·| B} 0.02/0.1=0.2 0.05/0.1=0.5 0.03/0.1=0.3

Note that the probability of a commute time event does NOT change according to

the value of the weather in Rome, e.g. Prob { LOW| B} = Prob { LOW} . Try to

answer now the following question. If you would like to predict the commute time

in Brussels, which event would return more information on it: the weather in Rome

or in Brussels?

•

Example: three sample spaces

Consider a probabilistic model of the day's weather based on the combination of

the following random descriptors where

1. the ﬁrst represents the sky condition and its sample space is Ωs = {CLEAR,

CLOUDY}.

2. the second represents the barometer trend and its sample space is Ωb = {RISING,

FALLING}.

3. the third represents the humidity in the afternoon and its sample space is

Ωh ={ DRY,WET}.

Let the joint probability values be given by Table 3.1. From the joint values we

can calculate the probabilities P ( CLE AR, RIS IN G )=0 .47 and P(C LOUDY ) =

0. 35 and the conditional probability value

P( DRY | C LEAR, RI SIN G) = P(DRY , CLEAR, RI SI NG)

P( CLE AR, RIS IN G)= 0 .40

0. 47 ≈ 0.85

Take the time now to compute yourself other probabilities: for instance what is

the probability of having a cloudy sky in wet conditions? Does a rising barometer

increase or not this probability? Is the event "clear sky and falling barometer"

independent from the event "dry weather"?

•

54 CHAPTER 3. FOUNDATIONS OF PROBABILITY

3.2 Random variables

Machine learning and statistics are concerned with numeric data and measurements

while so far we have mainly been dealing with categories. What is then the link

between the notion of random experiment and data? The answer is provided by the

concept of random variable.

Consider a random experiment and the associated triple (Ω, {E}, Prob {·} ). Sup-

pose that we have a mapping rule z : Ω → Z ⊂ R such that we can associate with

each experimental outcome ωa real value z = z (ω ) in the domain Z. We say that

zis the value taken by the random variable z when the outcome of the random

experiment is ω . Henceforth, in order to clarify the distinction between a random

variable and its value, we will use the boldface notation for denoting a random

variable (as in z) and the normal face notation for the eventually observed value

(as in z = 11).

Since there is a probability associated with each event Eand we have a mapping

from events to real values, a probability distribution can be associated with z.

Deﬁnition 2.1 (Random variable) . Given a random experiment (Ω, {E}, Prob {·}),

a random variable zis the result of a mapping z : Ω → Z that assigns a number z

to every outcome ω . This mapping must satisfy the following two conditions:

•the set {z ≤z }is an event for every z.

•the probabilities

Prob {z = ∞} = 0 Prob {z = −∞} = 0

Given a random variable z ∈ Z and a subset I ⊂ Z we deﬁne the inverse

mapping

z−1 (I ) = {ω ∈ Ω | z(ω )∈I } (3.2.24)

where z −1 (I ) ∈ {E} is an event. On the basis of the above relation we can associate

a probability measure to zaccording to

Prob {z ∈I } = Prob  z−1 (I ) = Prob {ω ∈ Ω |z (ω )∈I } (3.2.25)

Prob {z =z} = Prob  z−1 (z ) = Prob {ω ∈ Ω |z (ω ) = z} (3.2.26)

In other words, a random variable is a numerical quantity, linked to some exper-

iment involving some degree of randomness, which takes its value from some set Z

of possible real values. The notion of r.v. formalizes the notion of numeric measure-

ments, which is indeed a mapping between an event (e.g. your body temperature)

and a number (e.g. in the range Z = { 35,...,41} returned by the thermometer).

Another experiment might be the rolling of two six-sided dice and the r.v. z might

be the sum (or the maximum) of the two numbers showing in the dice. In this case,

the set of possible values is Z ={ 2,...,12} (or Z = { 1,..., 6} ).

Example

Suppose that we have to decide when to go home and watch Fiorentina AC playing

the Champion's League ﬁnal match against Anderlecht. In order to make such a

decision, a quantity of interest is the (random) commute time z for getting from

ULB to home. Our personal experience is that this time is a positive number that

is not constant: for example, z1 = 10 minutes, z2 = 23 minutes, z3 = 17 minutes,

where zi is the time taken on the ith day of the week. The variability of this quantity

is related to a complex random process with a large sample space Ω (depending, for

example, on the weather condition, the weekday, the sports events in town, and so

on). The probabilistic approach uses a random variable to represent this uncertainty

3.3. DISCRETE RANDOM VARIABLES 55

and considers each measure zi as the consequence of a random outcome ωi . The use

of a random variable zto represent the commute time becomes then a compact (and

approximate) way of modelling the disparate set of causes underlying the uncertainty

of this phenomenon. Whatever its limits, the probabilistic representation provides

us with a computational way to decide when to leave if we want to bound the

probability of missing the start of the game.

•

3.3 Discrete random variables

The probability (mass) function of a discrete r.v. zis the combination of

1. the countable set Zof values that the r.v. can take (also called range),

2. the set of probabilities associated to each value of Z.

This means that we can attach to the random variable some speciﬁc mathemat-

ical function Pz (z ) that gives for each z ∈ Z the probability that zassumes the

value z

Pz ( z) = Prob {z = z}(3.3.27)

This function is called probability function or probability mass function. Note that

henceafter will use P (z ) as a shorthand for Prob {z =z} when the identity of the

random variable is clear from the context.

As depicted in the following example, the probability function can be tabulated

for a few sample values of z. If we toss a fair coin twice, and the random variable

zis the number of heads that eventually turn up, the probability function can be

tabulated as follows

Values of the random variable z 012

Associated probabilities 0.25 0.50 0.25

3.3.1 Parametric probability function

Sometimes the probability function is not precisely known but can be expressed as

a function of zand a quantity θ. An example is the discrete r.v. zthat takes its

value from Z = { 1, 2,3} and whose probability function is

Pz ( z, θ) = θ 2z

θ2 +θ4 +θ6

where θ is some ﬁxed nonzero real number.

Whatever the value of θ ,Pz (z )> 0 for z = 1, 2, 3 and Pz (1) + Pz (2) + Pz (3) = 1.

Therefore z is a well-deﬁned random variable, even if the value of θis unknown.

We call θ a parameter , that is some constant, usually unknown, involved in the

analytical expression of a probability function. We will see in the following that the

parametric form is a convenient way to formalise a family of probabilistic models

and that the problem of estimation can be seen as a parameter identiﬁcation task.

3.3.2 Expected value, variance and standard deviation of a

discrete r.v.

Though the probability function Pz provides a complete description of the uncer-

tainty of z , it is often not practical to use since this requires to keep in mind (or in

memory) as many values as the size of Z. Therefore, it is more convenient to deal

with some compact representation of Pz obtained by computing a functional (i.e.

56 CHAPTER 3. FOUNDATIONS OF PROBABILITY

a function of a function) of Pz . The most common single-number summary of the

distribution Pz is the expected value which is a measure of central tendency4.

Deﬁnition 3.1 (Expected value) . The expected value of a discrete random variable

zis

E[z ] = µ=X

z∈Z

zPz ( z ) (3.3.28)

assuming that the sum is well-deﬁned.

An interesting property of the expected value is that it is the value which mini-

mizes the squared deviation

µ= arg min

mE[(z−m ) 2 ] (3.3.29)

Note that the expected value is not necessarily a value that belongs to the domain

Zof the random variable. It is important also to remark that while the term

mean is used as a synonym of expected value, this is not the case for the term

average. We will discuss in detail the diﬀerence between mean and sample average

in Section 5.3.2.

Example [180]

Let us consider a European roulette with numbers 0, 1,..., 36 and where the number

0 is considered as winning for the house. The gain of a player who places a 1$ bet

on a single number is a random variable zwhose sample space is Z = {− 1,35} . In

other words, only two outcomes are possible: either she wins z1 =− 1$ (or better

he loses 1$) with probability p1 = 36/ 37 or he wins z2 = 35$ with probability

p2 = 1/37. The expected gain is then

E[z ] = p1 z1 +p2z2 =p1 ∗ (− 1) + p2 ∗ 35 = − 36 / 37 + 35 / 37 = − 1 /37 = − 0 . 027

This means that while casinos gain on average 2.7 cents for every staked dollar,

players on average are giving away 2.7 cents (whatever sophisticated their betting

strategy is).

•

A common way to summarise the spread of a distribution is provided by the

variance.

Deﬁnition 3.2 (Variance). The variance of a discrete random variable zis

Var [z ] = σ2 =E [(z−E [z])2 ] = X

z∈Z

(z− E [z])2 Pz (z)

The variance is a measure of the dispersion of the probability function of the

random variable around its mean µ. Note that the following relation holds

σ2 = E[(z− E[z])2 ] = E[z2 − 2z E[z ]+( E[z])2 ] (3.3.30)

=E [z2 ]− (E [z ])2 =E [z2 ]− µ2 (3.3.31)

whatever is the probability function of z. Figure 3.7 illustrate two example discrete

r.v. probability functions that have the same mean but diﬀerent variance. Note

that the variance Var [z] does not have the same dimension as the values of z . For

instance, if z is measured in the unit [m ], Var [z ] is expressed in the unit [m ]2.

Standard deviation is a measure for the spread that has the same dimension as z.

An alternative measure of spread is E [ |z − µ|] but this quantity is less used since

more diﬃcult to be analytically manipulated than the variance.

4This concept was ﬁrst introduced in the 17th century by C. Huygens in order to study the

games of chance

3.3. DISCRETE RANDOM VARIABLES 57

Figure 3.7: Two discrete probability functions with the same mean and diﬀerent

variance

Deﬁnition 3.3 (Standard deviation) . The standard deviation of a discrete random

variable z is the positive square root of the variance.

Std [z ] = p Var [z ] = σ

Example

Let us consider a binary random variable z ∈ Z = { 0,1} where Pz (1) = p , 0 ≤p≤1

and Pz (0) = 1 − p . In this case

E[z ] = p∗1+0∗(1 − p) = p(3.3.32)

E[z2 ] = p∗1+0∗(1 − p) = p(3.3.33)

Var [z ] = E [z2 ]− (E [z ])2 =p− p2 = p (1 −p ) (3.3.34)

•

Deﬁnition 3.4 (Moment) . For any positive integer r , the r th moment of the prob-

ability function is

µr = E [zr ] = X

z∈Z

zr Pz ( z) (3.3.35)

Note that the ﬁrst moment coincides with the mean µ , while the second moment

is related to the variance according to Equation (3.3.30). Higher-order moments

provide additional information, other than the mean and the spread, about the

shape of the probability function.

Deﬁnition 3.5 (Skewness) . The skewness of a discrete random variable zis deﬁned

γ= E[(z− µ)3]

σ3 (3.3.36)

Skewness is a parameter that describes asymmetry in a random variable's prob-

ability function. Probability functions with positive skewness have long tails to the

right, and functions with negative skewness have long tails to the left (Figure 3.8).

Deﬁnition 3.6 (Kurtosis) . The kurtosis of a discrete random variable z is deﬁned

γ= E[(z− µ)4]

σ4 (3.3.37)

Kurtosis is always positive. Its interpretation is that the probability function

of a distribution with large kurtosis has fatter tails, compared with the probability

function of a distribution with smaller kurtosis.

58 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Figure 3.8: A discrete probability function with positive skewness (left) and one

with a negative skewness (right).

3.3.3 Entropy and relative entropy

Deﬁnition 3.7 (Entropy) . Given a discrete r.v. z , the entropy of the probability

function Pz (z ) is deﬁned by

H(z ) = − X

z∈Z

Pz ( z) log Pz ( z)

H(z ) is a measure of the unpredictability of the r.v. z. Suppose that there are

Mpossible values for the r.v. z. The entropy is maximized (and takes the value

log M ) if Pz (z )=1 /M for all z . It is minimized iﬀ P ( z ) = 1 for a single value of z

(i.e. all others probability values are null).

Although entropy measures as well as variance the uncertainty of a r.v., it diﬀers

from the variance since it depends only on the probabilities of the diﬀerent values

and not on the values themselves. In other terms, Hcan be seen as a function of

the probability function Pz rather than of z.

Let us now consider two diﬀerent discrete probability functions on the same set

of values

P0 =Pz 0 ( z) , P1 = Pz 1 ( z)

where P0 (z )> 0 if and only if P1 (z )> 0. The relative entropies (or the Kullback-

Leibler divergences) associated with these two functions are

H( P0 || P1 ) = X

P0 ( z) log P 0 ( z)

P1 ( z)= X

P0 ( z) log P0 ( z)−X

P0 ( z) log P1 ( z)

(3.3.38)

H( P1 || P0 ) = X

P1 ( z) log P 1 ( z)

P0 ( z)= X

P1 ( z) log P1 ( z)−X

P1 ( z) log P0 ( z)

(3.3.39)

where the term

−X

P0 ( z) log P1 ( z) = − Ez [log P1 ] (3.3.40)

is also called the cross-entropy. These asymmetric quantities measure the dissimi-

larity between the two probability functions. A symmetric formulation of the dis-

similarity is provided by the divergence quantity

J( P0 , P1 ) = H( P0 ||P1 ) + H( P1 ||P0 ) .

3.4. CONTINUOUS RANDOM VARIABLE 59

3.4 Continuous random variable

An r.v. zis said to be a continuous random variable if it can assume any of the

inﬁnite values within a range of real numbers. The following quantities can be

deﬁned:

Deﬁnition 4.1 (Cumulative distribution function) . The (cumulative) distribution

function of z is the function Fz :R→ [0,1]

Fz ( z ) = Prob {z ≤ z} (3.4.41)

This function satisﬁes the following two conditions:

1. it is right-continuous: Fz (z ) = limy→z Fz (y),

2. it is non-decreasing: z1 < z2 implies Fz ( z1 )≤Fz (z2 ),

3. it is normalized, i.e

lim

z→−∞ F z (z)=0 ,lim

z→∞ F z (z ) = 1

Deﬁnition 4.2 (Density function) . The density function of a real random variable

zis the derivative of the distribution function

pz ( z) = dF z ( z)

dz (3.4.42)

at all points z where Fz (· ) is diﬀerentiable.

Probabilities of continuous r.v. are not allocated to speciﬁc values but rather to

interval of values. Speciﬁcally

Prob {a ≤z≤b } =Z b

pz ( z) dz, Z Z

pz ( z) dz = 1

Some considerations about continuous r.v. are worthy to be mentioned:

•the quantity Prob {z=z }= 0 for all z,

•the quantity pz (z ) can be bigger than one (since it is a density and not a

probability) and even unbounded,

•two r.v.s z1 and z2 with the same domain Zare equal in distribution if

Fz 1 ( z ) = Fz 1 ( z ) for all z ∈ Z .

Note that hence-after we will use p (z ) as a shorthand for pz (z ) when the identity

of the random variable is clear from the context.

3.4.1 Mean, variance, moments of a continuous r.v.

Consider a continuous scalar r.v. with range Z = (l, h ) and density function p (z).

We may deﬁne the following quantities.

Deﬁnition 4.3 (Expectation or mean) . The mean of a continuous scalar r.v. z is

the scalar value

µ= E[z ] = Z h

zp( z ) dz (3.4.43)

60 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Figure 3.9: Cumulative distribution function and upper critical point.

Deﬁnition 4.4 (Variance). The variance of a continuous scalar r.v. z is the scalar

value

σ2 = E[(z− µ)2 ] = Z h

(z− µ )2 p(z)dz (3.4.44)

Deﬁnition 4.5 (Moments) . The r-th moment of a continuous scalar r.v. zis the

scalar value

µr = E [zr ] = Z h

zr p( z) dz (3.4.45)

Note that the moment of order r= 1 coincides with the mean of z.

Deﬁnition 4.6 (Quantile function) . Given the cumulative function Fz , the quantile

(or inverse cumulative) function is the function F −1

z: [0, 1] →R such that

F−1

z(q) = inf { z: F z (z) > q}

The quantities Fz (1/ 4), Fz (1/ 2), Fz (3/ 4) are called the ﬁrst quartile, the median

and the third quartile, respectively.

Deﬁnition 4.7 (Upper critical point) . For a given 0 ≤α≤1 the upper critical

point of a continuous r.v. zis the value zα such that

1−α = Prob {z ≤ zα } =F ( zα )⇔ zα =F−1 (1 −α )

Figure 3.9 shows an example of cumulative distribution together with the upper

critical point. A compact review of univariate discrete and continuous distributions

is available in Appendix C.1. In what follows we will detail only the univariate

normal case.

3.4.2 Univariate Normal (or Gaussian) distribution

A continuous scalar random variable xis said to be normally distributed with pa-

rameters µ and σ2 (also x ∼ N (µ, σ 2 )) if its probability density function is Normal

(or Gaussian). The analytical form of a Normal probability density function is

px ( x) = 1

√2πσ e − (x−µ)2

2σ 2 (3.4.46)

where the coeﬃcient before the exponential ensures that R px (x ) dx = 1. The mean

of the Normal random variable x is µ and its variance is σ2 . An interesting property

of a normal r.v.is that the probability that an observation xis within 1 (2) standard

deviations from the mean is 0.68 (0.95). You may ﬁnd more probabilistic relation-

ships in Table 3.2. When µ = 0 and σ2 = 1 the distribution is called standard

normal (Figure 3.10) and its distribution function is denoted Fz (z ) = Φ(z ). All

3.5. JOINT PROBABILITY 61

Figure 3.10: Density of a standard r.v. N (0,1)

Prob {µ −σ ≤x≤µ +σ } ≈ 0 .683

Prob {µ − 1.282σ≤x≤ µ + 1. 282σ } ≈ 0 .8

Prob {µ − 1.645σ≤x≤ µ + 1. 645σ } ≈ 0 .9

Prob {µ − 1 . 96σ≤x ≤ µ + 1. 96σ } ≈ 0 .95

Prob {µ − 2σ≤x≤ µ + 2σ } ≈ 0.954

Prob {µ − 2 . 57σ≤x≤ µ + 2. 57σ } ≈ 0.99

Prob {µ − 3σ≤x≤ µ + 3σ } ≈ 0.997

Table 3.2: Some probabilistic relations holding for x ∈ N ( µ, σ 2 )

random variables x ∼ N (µ, σ2 ) are linked to a standard variable zby the following

relation

z= ( x−µ) /σ. (3.4.47)

It follows that z ∼ N (0, 1) ⇒x =µ +σz ∼ N ( µ, σ2 ).

The practitioner might now wonder why the Normal distribution is so ubiqui-

tous in statistics books and literature. There are plenty of reasons both from the

theoretical and the practical side. From a theoretical perspective, the adoption of a

Normal distribution is justiﬁed by the Central Limit theorem (Appendix C.7) which

states that, under conditions almost always satisﬁed in practice, a linear combina-

tion of random variables converges to a Normal distribution. This is particularly

useful if we wish to represent in a compact lumped form the variability that escapes

to a modelling eﬀort (e.g. the regression plus noise form in Section 10.1). Another

relevant property of Gaussian distributions is that they are invariant to linear trans-

formations, i.e. a linear transformation of a Gaussian r.v. is still Gaussian, and its

mean (variance) depends on the mean (variance) of the original r.v.. From a more

pragmatic perspective, an evident asset of a Gaussian representation is that only a

ﬁnite number of parameters (two in the univariate case) are suﬃcient to characterise

the entire distribution.

Exercise

Test yourself the relations in Table 3.2 by random sampling and simulation using

the script norm.R.

•

3.5 Joint probability

So far, we considered scalar random variables only. However, the most interesting

probabilistic (and machine learning) applications are multivariate, i.e. concerning a

number of variables larger than one. Let us consider a probabilistic model described

by n discrete random variables. A fully-speciﬁed probabilistic model gives the joint

62 CHAPTER 3. FOUNDATIONS OF PROBABILITY

probability for every combination of the values of the nr.v.s. In other terms, the

joint probability contains all the information about the random variables.

In the discrete case, the model is speciﬁed by the values of the probabilities

Prob {z1 = z1 ,z2 = z2 ,...,zn = zn } =P (z1 , z2, . . . , zn ) (3.5.48)

for every possible assignment of values z1 , . . . , zn to the variables.

Spam mail example

Let us consider a bivariate probabilistic model describing the relation between the

validity of a received email and the presence of the word Viagra in the text. Let

z1 be the random variable describing the validity of the email (z1 = 0 for no-spam

and z1 = 1 for spam) and z2 the r.v. describing the presence (z2 = 1) or the

absence (z2 = 0) of the word Viagra. The stochastic relationship between these two

variables can be deﬁned by the joint probability distribution given by the table

z1 = 0 z1 = 1 P z 2

z2 = 0 0.8 0.08 0.88

z2 = 1 0.01 0.11 0.12

Pz 1 0.81 0.19 1

•

In the case of ncontinuous random variables, the model is speciﬁed by the joint

distribution function

Prob {z1 ≤ z1 ,z2 ≤ z2 ,...,zn ≤ zn } =F ( z1 , z2, . . . , zn )

which returns a value for every possible assignment of values z1 , . . . , zn to the vari-

ables.

3.5.1 Marginal and conditional probability

Let {z1 ,...,zm } be a subset of size m of the n discrete r.v.s for which a joint

probability function (3.5.48) is deﬁned. The marginal probabilities for the subset

can be derived from expression (3.5.48) by summing over all possible combinations

of values for the remaining variables.

P( z1 , . . . , zm ) = X

˜ zm+1 ··· X

˜ zn

P( z1 , . . . , zm , ˜ zm+1 ,..., ˜ zn ) (3.5.49)

Exercise

Compute the marginal probabilities P (z1 = 0) and P (z1 = 1) from the joint prob-

ability of the spam mail example.

•

For continuous random variables the marginal density is

p(z1 , . . . , zm ) = Z p(z1 , . . . , zm, zm+1 , . . . , zn ) dzm+1 . . . dzn (3.5.50)

This is also known as the sum rule or the marginalisation property.

The following deﬁnition for r.v. derives directly from Equation (3.1.8).

3.5. JOINT PROBABILITY 63

Deﬁnition 5.1 (Conditional probability function) . The conditional probability func-

tion for one subset of discrete variables {zi :i∈ S1 } given values for another disjoint

subset {zj :j∈ S2 } where S1 ∩S2 = ∅ , is deﬁned as the ratio

P({ zi : i∈ S1 }|{zj : j∈ S2 }) = P({zi :i∈ S1 } , {zj :j∈ S2 } )

P({ zj : j∈ S2 })

Deﬁnition 5.2 (Conditional density function) . The conditional density function

for one subset of continuous variables {zi :i∈ S1 } given values for another disjoint

subset {zj :j∈ S2 } where S1 ∩S2 = ∅ , is deﬁned as the ratio

p({ zi : i∈ S1 }|{zj : j∈ S2 }) = p({zi :i∈ S1 } , {zj :j∈ S2 } )

p({ zj : j∈ S2 })(3.5.51)

where p ({ zj :j∈ S2 } ) is the marginal density of the set S2 of variables. When

p({ zj : j∈ S2 } ) = 0 this quantity is not deﬁned.

The simpliﬁed version of (3.5.51) for two r.v.s z1 and z2 is

p(z1 = z1 ,z2 = z2 ) =

=p (z2 =z2 |z1 =z1 )p(z1 =z1 ) = p (z1 =z1 |z2 =z2 )p (z2 =z2 ) (3.5.52)

which is also known as the product rule.

By combining (3.4.43), the sum rule (3.5.50) and the product rule (3.5.52) we

obtain

p( z1 ) = Z p( z1 , z2 ) dz2 =Z p(z1 |z2 ) p( z2 )dz2 = E z 2 [p(z1 |z2 )]

where the subscript z2 makes clear that the expectation is computed with respect

to the distribution of z2 only (while z1 is ﬁxed).

3.5.2 Independence

Having deﬁned the joint and the conditional probability, we can now deﬁne when

two random variables are independent.

Deﬁnition 5.3 (Independent discrete random variables) . Let x and y be two dis-

crete random variables. Two variables x and y are deﬁned to be statistically inde-

pendent (written as x ⊥⊥ y ) if the joint probability

Prob {x = x, y =y} = Prob {x =x} Prob {y =y} , ∀x, y (3.5.53)

The deﬁnition can be easily extended to the continuous case.

Deﬁnition 5.4 (Independent continuous random variables). Two continuous vari-

ables x and y are deﬁned to be statistically independent (written as x ⊥⊥ y ) if the

joint density

p(x = x, y= y ) = p(x = x ) p(y = y) ,∀ x, y (3.5.54)

From the deﬁnition of independence and conditional density it follows that

x⊥⊥ y ⇔p (x =x |y=y ) = p ( x=x ) ∀x, y (3.5.55)

In layman's terms, the independence of two variables means that we do not

expect that the observed outcome of one variable will aﬀect the probability of ob-

serving the other, or equivalently that knowing something about one variable adds

no information about the other. For instance, hair colour and gender are indepen-

dent. Knowing someone's hair colour adds nothing to the knowledge of his gender.

64 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Height and weight are dependent, however. Knowing someone's height does not

determine precisely their weight: nevertheless, you have less uncertainty about his

probable weight after you have been told the height.

Though independence is symmetric

x⊥⊥ y ⇔y ⊥⊥ x

it is neither reﬂexive (i.e. a variable is not independent of itself) nor transitive. In

other terms, if x and y are independent and y and z are independent, then xand

zneed not be independent.

If we consider three instead of two variables, they are said to be mutually inde-

pendent if and only if each pair of rv.s. is independent and

p( x, y, z ) = p ( x ) p ( y) p( z )

Also the relationship

x⊥⊥ (y, z ) ⇒x ⊥⊥ z, x ⊥⊥ y

holds, but not the one in the opposite direction.

Note that in mathematical terms an independence assumption implies that a

bivariate density function can be written in a simple form, i.e. as the product of

two univariate densities. This results in an important beneﬁt in terms of the size of

the parametrisation. For instance, consider two discrete random variables z1 ∈ Z1 ,

z2 ∈ Z2 such that the cardinality of the two ranges is k1 and k2 , respectively. In the

generic case, if z1 and z2 are not independent, the deﬁnition of the joint probability

requires the deﬁnition of k1k2 − 1 terms5(or parameters). In the independent case

because of the property (3.5.54), the deﬁnition requires k1 − 1 terms for z1 and

k2 −1 terms for z2 , so overall k1 + k2 −2. This makes a big diﬀerence in case of

large values of k1 and k2 .

Independence allows an economic parametrisation in the multivariate case as

well. Consider the case of a large number nof binary discrete r.v.s., i.e. each

having a range made of two values. If we need to deﬁne the joint probability, we

require 2n −1 terms (or parameters) in the generic case. If the nvariables are

independent, this number is reduced to n.

Exercise

Check whether the variable z1 and z2 of the spam mail example are independent.

•

Note that hence-after, for the sake of brevity, we will limit to introduce deﬁnitions

for continuous random variables only. All of them can however be extended to the

discrete case too.

3.5.3 Chain rule

Given a set of nrandom variables, the chain rule (also called the general product

rule) returns the joint density as a function of conditional densities:

p( zn , . . . , z1 ) = p(zn |zn−1 , . . . , z1 ) p ( zn−1 | z n−2 , . . . , z1 ). . . p( z2 |z1 )p(z1 ) (3.5.56)

This rule is convenient to simplify the representation of large variate distributions

by describing them in terms of conditional probabilities.

5minus one because of the normalisation constraint

3.5. JOINT PROBABILITY 65

3.5.4 Conditional independence

Independence is not a stable relation. Though x ⊥⊥ y , the r.v. xmay become

dependent with y once we observe the value zof a third variable z. In the same

way, two dependent variables x and y may become independent once the value of

zis known. This leads us to introduce the notion of conditional independence.

Deﬁnition 5.5 (Conditional independence) . Two r.v.s x and y are conditionally

independent given the value z=z ( x ⊥⊥ y |z=z ) iﬀ

p(x = x, y= y |z= z ) = p(x = x |z= z ) p (y = y|z= z) ∀ x, y (3.5.57)

Two r.v.s x and y are conditionally independent given z (x ⊥⊥ y |z ) iﬀ they are

conditionally independent for all values of z.

Since from the chain rule (3.5.56) we may write

p(x = x, y= y|z= z ) = p(x = x|z= z) p(y = y |x= x, z= z )

it follows that x ⊥⊥ y |z=z implies the relation

p(y = y|x= x, z= z ) = p(y = y|z= z) (3.5.58)

In plain words, the notion of conditional dependence makes formal the intuition

that a variable may bring (or not) information about a second one, according to

the context.

Note that the statement x ⊥⊥ y |z=z means that x and y are independent

if z =z occurs but does not say anything about the relation between x and yif

z=z does not occur. It could follow that two variables are independent but not

conditional independent (or the other way round). In general independence does

not imply conditional independence and conditional independence does not imply

independence [27] (as in the example below).

Example: pizzas, dependence and conditional independence

Let y a variable representing the quality of a pizza restaurant and xa variable

quantifying the Italian assonance of the restaurant name. Intuitively, you would

prefer (because of higher quality y) a pizza served in the restaurant "Sole Mio"

(large x ), rather than in the restaurant "Tot Straks" (low x). In probabilistic

terms, this means that x and y are dependent (x 6⊥⊥ y ), i.e. knowing x reduces the

uncertainty we have about y. However, it is not the restaurant owner who makes

your pizza, but the cook (pizzaiolo ). Let z represent the assonance of his name.

Now you would prefer eating a pizza in a Belgian restaurant where the pizzaiolo

has Italian origins rather than in an Italian restaurant with a Flemish cook. In

probabilistic terms x and y become independent once z (the pizzaiolo's name) is

known (x ⊥⊥ y |z ).

•

It can be shown that the following two assertions are equivalent

(x ⊥⊥ (z1 ,z2 ) |y) ⇔ (x ⊥⊥ z1 |(y, z2 )),(x ⊥⊥ z2 |(y, z1 ))

Also

(x⊥⊥ y |z ),(x ⊥⊥ z |y) ⇒ (x ⊥⊥ (y, z ))

If (x ⊥⊥ y |z ), (z ⊥⊥ y | x), ( z ⊥⊥ x |y ) then x ,y ,z are mutually independent.

If z is a random vector, the order of the conditional independence is equal to the

number of variables in z.

66 CHAPTER 3. FOUNDATIONS OF PROBABILITY

3.5.5 Entropy in the continuous case

Consider a continuous r.v. y . The (diﬀerential) entropy of y is deﬁned by

H(y ) = −Z log( p( y ))p( y) dy = Ey [−log( p ( y ))] = Ey  log 1

p( y)

with the convention that 0 log 0 = 0. Entropy is a functional of the distribution of

yand is a measure of the predictability of a r.v. y. The higher the entropy, the

less reliable are our predictions about y. For a scalar normal r.v. y ∼ N (µ, σ2 )

H(y ) = 1

2 1 + ln 2πσ2  =1

2 ln 2πeσ2  (3.5.59)

In the case of a normal random vector Y ={ y1 ,...,yn }∼N (0,Σ)

H(Y ) = 1

2(ln(2πe)n det(Σ))

3.5.5.1 Joint and conditional entropy

Consider two continuous r.v.s x and y and their joint density p (x, y ). The joint

entropy of x and y is deﬁned by

H(x ,y) = − Z Z log( p( x, y)) p( x, y ) dxdy =

=Ex,y [−log(p ( x, y ))] = Ex,y  log 1

p( x, y) 

The conditional entropy is deﬁned as

H(y |x ) = − Z Z log( p( y | x ))p( x, y) dxdy = Ex,y [−log(p( y | x ))] =

=Ex,y  log 1

p( y| x) = E x [ H(y| x)]

This quantity quantiﬁes the remaining uncertainty of y once x is known. Note that

in general H (y |x ) 6 =H (x |y ), H (y )−H (y| x ) = H (x )−H ( x| y ) and that the chain

rule holds

H(y ,x) = H(y| x ) + H(x ) (3.5.60)

Also, conditioning reduces entropy

H(y |x )≤ H(y)

with equality if x and y are independent, i.e. x ⊥⊥ y . This property formalises a

fundamental principle underlying machine learning, data science and prediction in

general, i.e. that by conditioning on some variables x(e.g. inputs) we may reduce

the uncertainty about a variable y(target). Another interesting property is the

independence bound

H(y ,x)≤ H(y ) + H(x)

with equality if x ⊥⊥ y .

3.6. BIVARIATE CONTINUOUS DISTRIBUTION 67

Figure 3.11: 3D visualisation of a bivariate joint density.

3.6 Bivariate continuous distribution

Let us consider two continuous r.v. x and y and their bivariate joint density func-

tion px,y (x, y ). An example of bivariate joint density function is illustrated in

Figure 3.11. From (3.5.50), we deﬁne marginal density the quantity

px ( x ) = Z ∞

−∞

px,y ( x, y)dy

and conditional density the quantity

py|x ( y| x ) = p ( x, y)

p( x)(3.6.61)

which is, in loose terms, the probability that ybelongs to an interval dy about y

assuming that x =x . Note that, if x and y are independent

px,y ( x, y) = px ( x) py ( y ) , p( y| x) = py ( y)

The deﬁnition of conditional expectation is obtained from (3.6.61) and (3.4.43).

Deﬁnition 6.1 (Conditional expectation) . The conditional expectation of y given

x=x is

Ey [y |x = x ] = Z ypy|x ( y | x ) dy = µy|x ( x ) (3.6.62)

From (3.3.29) we may derive that

Ey [y |x = x] = arg min

mE y [(y−m) 2 |x=x ] (3.6.63)

Note that Ey [y| x =x ] is a function of xalso known as the regression function.

The deﬁnition of conditional variance derives from (3.6.61) and (3.4.44).

Deﬁnition 6.2 (Conditional variance).

Var [y| x =x ] = Z (y− µy|x (x))2 py|x (y |x )dy (3.6.64)

68 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Figure 3.12: Bivariate distribution: the ﬁgure shows the two marginal distribu-

tions (beside the axis), the conditional expectation function (dashed line) and some

conditional distributions (dotted).

Note that both these quantities are a function of x. If we replace the given value

xby the r.v. xthe terms Ey [y |x ] and Var [y |x ] are random, too.

Some important results on their expectation are contained in the following the-

orems [192].

Theorem 6.3. For two r.v.s x and y , assuming their expectations exist, we have

that

Ex [Ey [y |x = x]] = Ey [y ] (3.6.65)

and

Var [y ] = Ex [Var [y |x = x ]] + Var [Ey [y |x =x ]] (3.6.66)

where Var [y| x =x ] and Ey [y |x =x ] are functions of x.

We remind that for a bivariate function f (x, y)

Ey [ f( x, y)] = Z f (x, y) py ( y) dy, Ex [ f ( x, y)] = Z f (x, y ) px ( x)dx.

A 2D representation of a bivariate continuous distribution is illustrated in Figure

3.12. It is worthy noting that, although the conditional distribution is bell-shaped,

this is not necessarily the case for the marginal distributions.

3.6.1 Correlation

Consider two random variables x and y with means µx and µy and standard devi-

ations σx and σy .

Deﬁnition 6.4 (Covariance). The covariance between x and y is deﬁned as

Cov[x, y ] = E [(x− µx )(y− µy )] = E [xy ]− µxµy (3.6.67)

3.6. BIVARIATE CONTINUOUS DISTRIBUTION 69

Figure 3.13: Dependent but uncorrelated random variables

A positive (negative) covariance means that the two variables are positively

(inversely) related, i.e. that once one is above its mean, then the other tends to

be above (below) its mean as well. The covariance can take any value in real

numbers. A limitation of covariance is that it depends on variables' scales and

units: for instance, if variables were measured in meters instead of centimetres, this

would induce a change of their covariance. For this reason, it is common to replace

covariance with correlation, a dimensionless measure of linear association.

Deﬁnition 6.5 (Correlation) . The correlation coeﬃcient is deﬁned as

ρ(x ,y) = Cov[ x ,y]

pVar [x ] Var [y ] (3.6.68)

It is easily shown that − 1≤ρ (x, y )≤ 1. For this reason, the correlation is

sometimes expressed as a percentage.

Deﬁnition 6.6 (Uncorrelated variables) . Two r.v.s x and y are said to be uncor-

related if ρ (x, y ) = 0 or equivalently if

E[xy ] = E[x]E[y ] (3.6.69)

Note that if x and y are two independent random variables, then

E[xy ] = Z xyp( x, y ) dxdy =Z xyp( x) p( y) dxdy =Z xp(x) dx Z yp( y) dy = E[x ] E[y ]

This means that independence implies uncorrelation. However, the contrary does

not hold for a generic distribution. The equivalence between independence and

uncorrelation

ρ(x ,y)=0⇔ x⊥⊥ y (3.6.70)

holds only if x and y are jointly Gaussian.

See Figure 3.13 for an example of uncorrelated but dependent variables.

Exercises

1. Let x and y two discrete independent r.v. such that

Px (− 1) = 0 .1 , Px (0) = 0 .8 , Px (1) = 0 . 1

and

Py (1) = 0 .1 , Py (2) = 0 . 8, Py (3) = 0 .1

If z =x +y show that E [z ] = E [ x ] + E [y]

70 CHAPTER 3. FOUNDATIONS OF PROBABILITY

2. Let x be a discrete r.v. which assumes {− 1,0,1} with probability 1/ 3 and

y= x2 . Let z= x+ y. Show that

•E[z ] = E[x ] + E[y ].

•xand yare uncorrelated but dependent random variables.

•

3.7 Normal distribution: the multivariate case

Let z = [z1 ,...,zn ]T be a [n, 1] random vector . The vector is said to be normally

distributed with parameters µ and Σ (also z ∼ N (µ, Σ)) if its probability density

function is given by

pz ( z ) = 1

(√ 2π)n p det(Σ) exp  − 1

2(z− µ )T Σ−1 (z− µ )(3.7.71)

where det(Σ) denotes the determinant of the matrix Σ. It follows that

•the mean E [z ] = µ is an [n, 1] vector,

•the matrix

Σ = E [(z− µ)(z− µ)T ] (3.7.72)

is the [n, n] covariance matrix. This matrix is symmetric and positive semidef-

inite. It has n (n + 1)/ 2 parameters: the diagonal terms Σjj are the variances

Var [zj ] of the vector components and the oﬀ-diagonal terms Σjk , j 6 =k are the

covariance terms Cov[zj ,zk ]. The inverse Σ−1 is also called the concentration

matrix.

The quantity

∆=( z− µ)T Σ −1 ( z− µ) (3.7.73)

which appears in the exponent of pz is called the Mahalanobis distance from zto

µ. It can be shown that the n -dimensional surfaces of constant probability density

•are hyper-ellipsoids on which ∆2is constant;

•their principal axes are given by the eigenvectors uj ,j = 1, . . . , n of Σ which

satisfy

Σuj =λj uj j = 1, . . . , n

where λj are the corresponding eigenvalues.

•the eigenvalues λj give the variances along the principal directions (Figure

3.14).

If the covariance matrix Σ is diagonal then

•the contours of constant density are hyper-ellipsoids with the principal direc-

tions aligned with the coordinate axes.

•the components of z are then statistically independent since the distribution of

zcan be written as the product of the distributions for each of the components

separately in the form

pz ( z) =

j=1

pz j (zj )

•the total number of independent parameters in the distribution is 2n (nfor

the mean vector and nfor the diagonal covariance matrix).

•if σj =σ for all j, the contours of constant density are hyper-spheres.

3.7. NORMAL DISTRIBUTION: THE MULTIVARIATE CASE 71

Figure 3.14: Contour curves of normal distribution for n = 2.

3.7.1 Bivariate normal distribution

Let us consider a bivariate (n= 2) normal density whose mean is µ = [µ1 , µ2 ]Tand

the covariance matrix is

Σ =  σ 2

1σ 12

σ21 σ2

2

The correlation coeﬃcient is

ρ= σ12

σ1σ2

It can be shown that the general bivariate normal density has the form

p(z1 , z2 ) =

2πσ1σ2 p 1−ρ2 exp " − 1

2(1 −ρ2 )" z 1 − µ 1

σ1  2

−2ρ z 1 −µ1

σ1  z 2 − µ 2

σ2  +  z 2 − µ 2

σ2  2 ##

A plot of a bivariate normal density with µ = [0, 0] and Σ = [1. 2919,0. 4546; 0. 4546,1.7081]

and a corresponding contour curve are traced in Figure 3.15 by means of the script

gaussXYZ.R.

We suggest the reader to play with the Shiny dashboard gaussian.R in order

to visualize the impact of the parameters on the Gaussian distribution.

One of the important properties of the multivariate normal density is that all

conditional and marginal probabilities are also normal. Using the relation

p( z2 |z1 ) = p(z1 , z2 )

p(z1 )

we ﬁnd that p (z2 |z1 ) is a normal distribution N (µ2|1 , σ2

2| 1), where

µ2|1 =µ2 + ρσ 2

σ1

(z1 − µ1 )

σ2

2| 1=σ 2

2(1 −ρ 2 )

Note that

•µ2|1 is a linear function of z1 : if the correlation coeﬃcient ρ is positive, the

larger z1 , the larger µ2|1 .

•if there is no correlation between z1 and z2 , the two variables are independent,

i.e. we can ignore the value of z1 to estimate µ2 .

72 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Figure 3.15: Bivariate normal density function

3.7.2 Gaussian mixture distribution

A continuous r.v. zhas a Gaussian mixture distribution with mcomponents if

p(z = z ) =

k=1

wk N( z; µk ,Σk ) (3.7.74)

where N (z ; µk , Σk ) denotes the Normal density with mean µk and covariance Σk,

and the mixture weights wk satisfy

k=1

wk = 1 ,0≤ wk ≤1

A Gaussian mixture is a linear superposition of m Gaussian components and, as

such, has a higher expressive power than a unimodal Gaussian distribution: for

instance, it can be used to model multimodal density distributions.

The script gmm.R samples a bidimensional mixture of Gaussians with 3 compo-

nents with diagonal covariances. The density and the sampled points are in Fig-

ure 3.16. An interesting property of Gaussian mixtures is that they are universal

approximator of densities which means that any smooth density can be approx-

imated with any speciﬁc nonzero amount of error by a Gaussian mixture model

(GMM) with enough components.

3.7.3 Linear transformations of Gaussian variables

If z1 ∼ N (µ1 , Σ1 ) and z2 ∼ N (µ2 , Σ2 ) are independent Gaussian r.v.s., then the

sum z =z1 +z2 is a Gaussian r.v. z ∼ N (µ1 + µ2 , Σ1 + Σ2).

Given two real constants c1 and c2 , the linear combination z =c1 z1 +c2 z2 is a

Gaussian r.v. z ∼ N (c1µ1 + c2µ2 , c2

1Σ 1 +c 2

2Σ 2 ).

If z ∼ N (µ, Σ) is a [n, 1] Gaussian random vector and y =Az , with A a [n, n]

real matrix, then y ∼ N (Aµ, A ΣAT ) is a Gaussian vector.

3.8. MUTUAL INFORMATION 73

Figure 3.16: Density and observations of a bidimensional mixture of Gaussians with

3 components. Each colour corresponds to a diﬀerent component.

3.8 Mutual information

Mutual information is one of the most widely used measures to convey the depen-

dency of variables. It is a measure of the amount of information that one random

variable contains about another random variable. It can also be considered as the

distance from independence between the two variables. This quantity is always non-

negative and zero if and only if the two variables are stochastically independent.

Given two random variables x and y , their mutual information is deﬁned in

terms of their probabilistic marginal density functions px (x ), py (y ) and the joint

p(x, y) ( x, y):

I(x ;y ) = Z Z log p( x, y)

p( x) p( y) p( x, y) dxdy = H (y )− H (y |x ) = H (x )− H (x |y )

(3.8.75)

with the convention that 0 log 0

0= 0. From (3.5.60), we derive

I(x ;y ) = H(y )− H(y| x ) = H(y ) + H(x )− H(x ,y) (3.8.76)

Mutual information is null if and only if x and y are independent, i.e.

I(x ;y )=0⇔x ⊥⊥ y. (3.8.77)

In other words, the larger the mutual information term, the stronger is the degree

of dependency between two variables.

In the Gaussian case, an analytical link between correlation and mutual infor-

mation exists. Let (x, y ) a normally distributed random vector with a correlation

coeﬃcient ρ . The mutual information between x and y is given by

I(x ;y ) = − 1

2log(1 −ρ2 )

74 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Equivalently the correlation coeﬃcient (3.6.68) can be written as

ρ=p 1− exp(− 2 I(x ;y ))

In agreement with (3.8.77) and (3.6.70), it follows that in the Gaussian case

ρ(x ,y)=0⇔ I(x ;y ) = 0 (3.8.78)

3.8.1 Conditional mutual information

Consider three r.v.s x ,y and z . The conditional mutual information is deﬁned by

I(y ;x| z ) = H(y| z )− H(y| x ,z) (3.8.79)

It can also be written as

I(y ;x| z ) = Z Z log p(x, y| z )

p( x| z ) p ( y| z) p( x, y, z ) dxdydz

While mutual information quantiﬁes the degree of (in)dependence between two

variables, conditional mutual information quantiﬁes the degree of conditional (in)dependence

(Section 3.5.4) between three variables. The conditional mutual information is null

iﬀ x and y are conditionally independent given z , i.e.

I(x ;y |z )=0⇔x ⊥⊥ y |z (3.8.80)

Note that I ( x ;y |z ) can be null though I (x ;y )> 0, like in the pizzas example

in Section 3.5.4. Also a symmetric conﬁguration is possible, e.g. I ( x ;y ) = 0 but

I(x ;y| z ) >0 as in the case of complementary variables which will be discussed in

Section 12.8.

3.8.2 Joint mutual information

This section derives the information of a pair of variables (x1 ,x2 ) about a third one

From (3.8.79) and (3.5.60) it follows:

I(x ;y |z ) = H(y |z )− H(y| x ,z) = H(y |z ) + H(x| z )− H((x ,y)| z) =

=H ((x, z )) + H ((y, z )) −H (z )−H ((x, y, z )) (3.8.81)

From (3.8.76) it follows

I((x1 ,x2 ); y) = H(x1 ,x2 ) + H(y )− H(x1 ,x2 ,y)

and

I(x1 ;y ) = H(x1 ) + H(y )− H(x1 ,y)

From (3.8.81) it follows

I(x2 ;y |x1 ) = H(y| x1 )− H(y |x1 ,x2 ) =

=H (y, x1 )−H (x1 )−H (y, x1 ,x2 ) + H (x1 ,x2 )

On the basis of the results above, we derive the chain rule of mutual information

I(x1 ;y ) + I(x2 ;y| x1 ) =

=H (x1 ) + H (y )−H (x1 , y ) + H (y, x1 )−H (x1 )−H (y, x1 ,x2 ) + H (x1 ,x2 ) =

=H (y )−H (y, x1 ,x2 ) + H (x1 ,x2 ) = I ((x1 ,x2 ); y ) (3.8.82)

3.8. MUTUAL INFORMATION 75

This formula shows that the information that a pair of variables (x1 ,x2 ) brings

about a third variable yis not simply the sum of the two mutual information terms

I(x1 ;y ) and I(x2 ;y ) but is the sum of I(x1 ;y ) and the conditional information of

x2 and ygiven x1 . This aspect is particularly important in the feature selection

context (Section 12.8) where simplistic assumptions of monotonicity and additivity

do not hold.

For n > 2 variables X = {x1 ,...,xn } the chain rule formulation is

I(X ;y ) = I(X−i ;y| xi )+I(xi ;y ) = I(xi ;y |X−i )+ I(X−i ;y ) i= 1 , . . . , n (3.8.83)

where X−i denote the X set with the i th term set aside.

3.8.3 Partial correlation coeﬃcient

We have seen in Section 3.6.1 that correlation is a good measure of independence

in the case of Gaussian distributions. The same role for conditional independence

is played by partial correlation.

Deﬁnition 8.1 (First-order partial correlation) . Let us consider three r.v.s x,y

and z . The ﬁrst-order partial correlation is

ρxy|z = ρ xy − ρ xz ρ zy

q(1 −ρ2

xz)(1 −ρ 2

yz)

where ρ xy is deﬁned in (3.6.68).

This quantity returns a measure of the correlation between x and y once the

value of z is known. It is possible to extend the partial correlation to the condition-

ing of two variables.

Deﬁnition 8.2 (Second-order correlation).

ρx 1 y|zx2 = ρ x 1 y|z − ρ x 1 x 2 |z ρ yx 2 |z

q(1 −ρ 2

x1x2 | z )(1 −ρ 2

yx2 | z )

This can be used also to deﬁne a recurrence relationship where qth order partial

correlations can be computed from (q− 1)th order partial correlations.

Another interesting property is the link between partial correlation and con-

centration matrix (Section 3.7). Let Σ and Ω = Σ−1 denote the covariance and

the concentration matrix of the normal set of variables Z ∪ {x, y } . The partial

correlation coeﬃcient ρ xy|Z can be obtained by matrix inversion:

ρxy|Z =− ωxy

√ω xx ωxy

where ωxy is the element of the concentration matrix corresponding tp x and y.

Consider a multivariate normal vector X, such that xi ,xj ∈ X ,XS ⊂ X and s

is the dimension of XS . Then

ρx i x j |XS = 0 ⇔ I(xi ,xj | XS )=0

Note that this is the conditional version of the relation (3.8.78).

76 CHAPTER 3. FOUNDATIONS OF PROBABILITY

3.9 Functions of random variables and Monte Carlo

simulation

For any function g (· ) of the random variable z

E[ g(z )] = Z g( z) pz ( z) dz (3.9.84)

This is also know as the law of the unconscious statistician (LOTUS). Note that in

general E [g (z )] 6 =g (E [z ]), with the exception of the linear function g (z ) = az + b

which will be discussed in the following section.

Exercise

Let z be a scalar r.v. and

g( z) = ( 1 z∈[a, b]

0 else

with a < b . Compute E [g (z)].

•

For a generic g , the analytical computation or numerical integration of (3.9.84) may

be extremely complex. A numerical alternative is represented by the Monte Carlo

simulation which requires a pseudo-random generator of examples according to the

distribution of z. In a nutshell Monte Carlo computes E [g (x )] by

1. generating a large number S of sample points zi ∼ Fz , i = 1, . . . , S ,

2. computing g (zi ),

3. returning the estimation

E[ g(z )] ≈P S

i=1 g(z i )

If S is suﬃciently large, we may consider such approximation as reliable. The same

procedure may be used to approximate other parameters of the distribution (e.g.

the variance). In this book, we will have recourse to Monte Carlo simulation to

provide a numerical illustration of probabilistic formulas or concepts (e.g. bias,

variance and generalisation error), which otherwise might appear too abstract for

the reader.

Monte Carlo computation

The script mcarlo.R contains the Monte Carlo computation of the mean and vari-

ance of z ∼ N (µ, σ 2 ) as well as the computation of E [z2 ] and E [ |z | ].

The Shiny dashboard mcarlo.R visualises the result of some operations on a

single and two random variables by using a Monte Carlo simulation.

•

3.10. LINEAR COMBINATIONS OF R.V. 77

3.10 Linear combinations of r.v.

The expected value of a linear combination of r.v.s is simply the linear combination

of their respective expectation values

E[ ax+ by ] = aE[x ] + bE [y] , a ∈R , b ∈ R

i.e., expectation is a linear statistic. On the contrary, the variance is not a linear

statistic. We have

Var [ax + by ] = a2 Var [x ] + b2 Var [y ]+2 ab ( E[ xy]− E [x] E [y ]) (3.10.85)

=a2 Var [x ] + b2 Var [y ]+2 abCov[x ,y ] (3.10.86)

where the quantity Cov[x, y ] is deﬁned in (3.6.67).

Given n r.v. zj , j = 1, . . . , n

Var 



j=1

cj zj 

=

j=1

jVar [zj ]+2X

i<j

ci cj Cov[zi ,zj ] (3.10.87)

Let us consider now nrandom variables with the same variance σ2 and mutual

correlation ρ . Then the variance of their average is

Var "P n

j=1 z j

n# = nσ2

n2 + 2 1

n( n− 1)

2ρσ 2 =

=σ 2

n+ ρσ 2 − ρσ2

n= (1 − ρ) σ 2

n+ ρσ 2 (3.10.88)

3.10.1 The sum of i.i.d. random variables

Suppose that z1 ,z2 ,. . . ,zN are i.i.d. (identically and independently distributed)

random variables, discrete or continuous, each having a probability distribution

with mean µ and variance σ2 . Let us consider the two derived r.v., that is the sum

SN = z1 +z2 + · ·· + z N

and the average

z= z 1 + z 2 +· ·· +z N

N(3.10.89)

The following relations hold

E[SN ] = Nµ, Var [SN ] = Nσ 2 (3.10.90)

E[¯

z] = µ, Var [¯

z] = σ 2

N(3.10.91)

An illustration of these relations by simulation can be obtained by running the R

script sum rv.R.

3.11 Conclusion

The reader (in particular, if practitioner) might think that a chapter on probabil-

ity theory is an unnecessary frill in a book on machine learning. The author has

a diﬀerent opinion. Probability extends the logical formalism and makes formal

human patterns of reasoning under uncertainty (e.g. abduction). Also, probability

provides an eﬀective language to formalise the task of machine learning, i.e. using

78 CHAPTER 3. FOUNDATIONS OF PROBABILITY

some variables (e.g. inputs) to explain, provide information (or reduce uncertainty)

about other ones (e.g. targets). According to Aristotles, philosophy begins with

wonder. From a scientiﬁc perspective, wonder originates from uncertainty, and sci-

ence has the role of reducing it by explanation. The author hopes that this chapter

showed that uncertainty and information are not only philosophical concepts but

quantities whose nature and relationship can be described in probabilistic terms.

So far, we only considered low variate settings, although the ambition of statis-

tical machine learning is attacking complex high variate problems. For this reason,

the next chapter will provide a probabilistic formalism to deal with high variate

(and then complex) settings. What is still missing for the moment is the second

major ingredient (besides uncertainty) of machine learning: data. Please be patient:

the relation between uncertainty and observations will be discussed in Chapter 5,

which introduces estimation as the statistical way of combining probabilistic models

with real-world data.

3.12 Exercises

1. Suppose you collect a dataset about spam in emails. Let the binary variables x1 ,

x2 and x3 represent the occurrence of the words "Viagra", "Lottery" and "Won",

respectively, in a email. Let the dataset of 20 emails being summarised as follows

Document x1 (Viagra) x2 (Lottery) x3 (Won) y(Class)

E1 0 0 0 NOSPAM

E2 0 1 1 SPAM

E3 0 0 1 NOSPAM

E4 0 1 1 SPAM

E5 1 0 0 SPAM

E6 1 1 1 SPAM

E7 0 0 1 NOSPAM

E8 0 1 1 SPAM

E9 0 0 0 NOSPAM

E10 0 1 1 SPAM

E11 1 0 0 NOSPAM

E12 0 1 1 SPAM

E13 0 0 0 NOSPAM

E14 0 1 1 SPAM

E15 0 0 1 NOSPAM

E16 0 1 1 SPAM

E17 1 0 0 SPAM

E18 1 1 1 SPAM

E19 0 0 1 NOSPAM

E20 0 1 1 SPAM

where

•0 stands for the case-insensitive absence of the word in the email.

•1 stands for the case-insensitive presence of the word in the email.

Let y = 1 denote a spam email and y= 0 a no-spam email.

The student should estimate on the basis of the frequency of the data above

•Prob {x1 = 1, x2 = 1}

•Prob {y= 0 |x2 = 1, x3 = 1}

•Prob {x1 = 0 |x2 = 1}

•Prob {x3 = 1 |y= 0, x2 = 0}

•Prob {y= 0 |x1 = 0,x2 = 0,x3 = 0}

3.12. EXERCISES 79

•Prob {x1 = 0 |y= 0}

•Prob {y= 0}

Solution:

•Prob {x1 = 1, x2 = 1 }= 0.1

•Prob {y= 0 |x2 = 1, x3 = 1 }= 0

•Prob {x1 = 0 |x2 = 1 }= 0 .8

•Prob {x3 = 1 |y= 0, x2 = 0 }= 0.5

•Prob {y= 0 |x1 = 0, x2 = 0, x3 = 0 }= 1

•Prob {x1 = 0 |y= 0 }= 0 .875

•Prob {y= 0 }= 0.4

2. Let us consider a fraud detection problem. Suppose we collect the following trans-

actional dataset where v = 1 means that the transaction came from a suspicious

web site and f= 1 means that the transaction is fraudulent.

f= 1 f= 0

v= 1 500 1000

v= 0 1 10000

Estimate the following quantities by using the frequency as estimator of probability:

•Prob {f= 1}

•Prob {v= 0}

•Prob {f= 1 |v= 1}

•Prob {v= 1 |f= 1}

Use the Bayes theorem to compute Prob {v = 1|f = 1} and show that the result is

identical to the one computed before.

Solution:

•Prob {f= 1 }= 501/ 11501 = 0.043

•Prob {v= 0 }= 10001 /11501 = 0 .869

•Prob {f= 1 |v= 1 }= 500/1500 = 1 /3

•Prob {v= 1 |f= 1 }= 500/501

By Bayes theorem: Prob {v = 1|f = 1} = Prob{f =1|v =1}Prob{v =1}

Prob{f =1 } = 1/3(1500/11501)

501/ 11501 =

500/501

3. Let us consider a dataset with 4 binary variables

x1x2x3 y

1 1 0 1

0 0 1 0

0 1 0 0

1 1 1 1

0 0 0 0

0 1 0 0

0 1 1 0

0 0 1 0

0 0 0 0

0 1 0 0

1 1 1 1

Estimate the following quantities by using the frequency as estimator of probability

80 CHAPTER 3. FOUNDATIONS OF PROBABILITY

•Prob {y= 1}

•Prob {y= 1 |x1 = 0}

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0}

Solution:

•Prob {y= 1 }= 3 /11

•Prob {y= 1 |x1 = 0 }= 0

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0 }= 0

4. Let us consider a task with three binary inputs and one binary target where the

input distribution is

x1x2x3 P( x1 , x2, x3 )

0 0 0 0.2

0 0 1 0.1

0 1 0 0.1

0 1 1 0.1

1 0 0 0.1

1 0 1 0.1

1 1 0 0.1

1 1 1 0.2

and the conditional probability is

x1x2x3 P( y= 1| x1 , x2, x3 )

0 0 0 0.8

0 0 1 0.1

0 1 0 0.5

0 1 1 0.9

1 0 0 0.05

1 0 1 0.1

1 1 0 0.05

1 1 1 0.5

Compute

•Prob {x1 = 1, x2 = 1}

•Prob {y= 0 |x2 = 1, x3 = 0}

•Prob {x1 = 0 |x2 = 1}

•Prob {x3 = 1 |y= 0, x2 = 1}

•Prob {y= 0 |x1 = 0, x2 = 0, x3 = 0}

•Prob {x1 = 0 |y= 0}

Solution:

•Prob {x1 = 1, x2 = 1 }=0.1+0.2=0.3

•By using (3.1.21) (where E0 stands for x2 = 1,x3 = 0) we obtain:

Prob {y = 0|x2 = 1,x3 = 0} = Prob {y = 0| x1 = 0,x2 = 1,x3 = 0}∗ Prob {x1 = 0|x2 = 1, x3 = 0}+

Prob {y = 0|x1 = 1,x2 = 1,x3 = 0}∗ Prob {x1 = 1|x2 = 1,x3 = 0} = 0. 5∗ 0. 5+

0. 95 ∗ 0. 5 = 0.725

•Prob {x1 = 0 |x2 = 1 }= (0 .1 + 0 .1) / (0 .2 + 0.3) = 0 .4

•From the joint four variate distribution computed in the exercise below

Prob { x3 = 1|y = 0,x2 = 1} = Prob {x3 = 1,y = 0,x2 = 1}

Prob {y = 0,x2 = 1} = 0 . 11

0. 255 = 0.4313725

3.12. EXERCISES 81

•Prob {y= 0 |x1 = 0, x2 = 0, x3 = 0 }= 1 − 0. 8 = 0.2

•From the joint four variate distribution computed in the exercise below

Prob {x1 = 0|y = 0} = Prob {x1 = 0,y = 0}

Prob {y = 0} = 0. 19

0. 57 = 0.3333

5. Consider the probability distribution of the previous exercise. Is yconditionally

independent of x1 given x2 ?

Solution:

According to Section 3.5.4, yis conditionally independent of x1 given x2 if for all

values x2 :

Prob {y =y| x1 = x1 ,x2 = x2 } = Prob {y =y|x2 = x2 }

Let us compute Prob {y = 1|x1 = 1,x2 =x2 } and Prob {y = 1| x2 = x2 } for x2 = 0.

From (3.1.21)

Prob {y = 1| x2 = 0,x1 = 1}=

Prob {y = 1|x2 = 0,x1 = 1,x3 = x3 } Prob {x3 =x3 | x2 = 0,x1 = 1}=

= Prob {y = 1|x2 = 0,x1 = 1x3 = 0} Prob { x3 = 0|x2 = 0, x1 = 1}+

+ Prob {y = 1|x2 = 0, x1 = 1,x3 = 1} Prob { x3 = 1|x2 = 0,x1 = 1}=

= 0. 05 ∗ 0. 1/ 0. 2 + 0. 1∗ 0. 1 / 0. 2 = 0.075

and

Prob {y = 1| x2 = 0}=

x1,x3

Prob {y = 1|x2 = 0, x1 =x1 ,x3 = x3 } Prob {x1 =x1 ,x3 =x3 | x2 = 0}=

= Prob {y = 1|x2 = 0,x1 = 0,x3 = 0} Prob {x1 = 0,x3 = 0|x2 = 0}+

+ Prob {y = 1|x2 = 0,x1 = 0,x3 = 1} Prob {x1 = 0,x3 = 1|x2 = 0}+

+ Prob {y = 1|x2 = 0,x1 = 1,x3 = 0} Prob {x1 = 1,x3 = 0|x2 = 0}+

+ Prob {y = 1|x2 = 0,x1 = 1,x3 = 1} Prob {x1 = 1,x3 = 1|x2 = 0}=

= 0. 8∗ 0.2 / 0. 5 + 0. 1∗ 0.1 / 0. 5 + 0. 05 ∗ 0. 1/0 . 5 + 0. 1∗ 0. 1/0 . 5 = 0.37

Since those two values are diﬀerent, the two variables are not conditionally indepen-

dent.

An alternative would be ﬁrst computing the joint distribution of the 4 variables and

then deriving the conditional terms. Since

Prob { y, x1, x2, x3 } = Prob {y | x1 , x2, x3 } Prob {x1 , x2, x3 }

the joint distribution is :

82 CHAPTER 3. FOUNDATIONS OF PROBABILITY

y x1 x2x3 P( y, x1, x2, x3 )

0 0 0 0 (1-0.8)*0.2=0.04

0 0 0 1 (1-0.1)*0.1=0.09

0 0 1 0 0.05

0 0 1 1 0.01

0 1 0 0 0.095

0 1 0 1 0.09

0 1 1 0 0.095

0 1 1 1 0.1

1 0 0 0 0.8*0.2=0.16

1 0 0 1 0.1*0.1=0.01

1 0 1 0 0.05

1 0 1 1 0.09

1 1 0 0 0.005

1 1 0 1 0.01

1 1 1 0 0.005

1 1 1 1 0.1

From the table above we compute the conditional terms as

Prob {y = 1|x2 = 0} = Prob {y = 1, x2 = 0}

Prob { x2 = 0} =

=0. 16 + 0.01 + 0.01 + 0.005

0. 04 + 0.09 + 0.095 + 0.09 + 0.16 + 0.01 + 0.005 + 0. 01 = 0.37

and

Prob {y = 1|x2 = 0, x1 = 1} = Prob {y = 1,x1 = 1,x2 = 0}

Prob {x1 = 1,x2 = 0} =

=0.005 + 0.01

0. 095 + 0.09 + 0.005 + 0. 01 = 0.075

Since the results are (obviously) identical to the ones obtained with the ﬁrst method,

the conclusion is the same, i.e. the variables are conditionally dependent.

6. Let x, y, z be three binary random variables denoting the pathological mutation of

a given gene of the father, mother and child, respectively. The values 0 and 1 stand

for the absence and presence of the mutation, respectively. Suppose that

•the two parents have the same probability 0.5 of having a pathological mutation

in a given gene

•the variables x and y are independent

•the child may inherit the mutation according to this conditional probability

table

Prob {z = 1|x = x, y =y} x y

0 0 0

0.6 0 1

0.4 1 0

0.7 1 1

1. What is the probability that the child has no mutation if both parents are not

aﬀected?

2. What is the probability that the father had a mutated gene if the child has the

mutation and the mother is not aﬀected?

3. What is the probability that the father has a mutated gene if the child has the

mutation and the mother is aﬀected?

3.12. EXERCISES 83

4. What is the probability that the child has the mutation if the father has none?

5. What is the probability that the father has a mutated gene if the child has the

mutation?

6. What is the probability that the father has a mutated gene if the child has no

mutation?

Solution:

Let us derive ﬁrst

P(z = 1|y = 0) =

P(z = 1|y = 0 ,x= 1) P(x = 1|y = 0) + P(z = 1|y = 0 ,x= 0) P(x = 0|y = 0) =

=P (z = 1|y = 0,x = 1)P (x = 1) + P (z = 1|y = 0,x = 0)P (x = 0) =

= 0. 4∗ 0. 5 + 0 ∗ 0 . 5 = 0.2

P(z = 1|y = 1) =

P(z = 1|y = 1 ,x= 1) P(x = 1|y = 1) + P(z = 1|y = 1 ,x= 0) P(x = 0|y = 1) =

=P (z = 1|y = 1,x = 1)P (x = 1) + P (z = 1|y = 1,x = 0)P (x = 0) =

= 0. 7∗ 0 . 5 + 0. 6∗ 0. 5 = 0.65

P(z = 1|x = 1) =

P(z = 1|x = 1 ,y= 0) P(y = 0|x = 1) + P(z = 1|y = 1 ,x= 1) P(y = 1|x = 1) =

= 0. 4∗ 0 . 5 + 0. 7∗ 0. 5 = 0.55

It follows

P(z = 0|x = 0 ,y= 0) = 1

P(x = 1|z = 1 ,y= 0) = P(z = 1|x = 1,y = 0) P(x = 1 |y = 0)

P(z = 1|y = 0 = 0 .4∗ 0 .5

0. 2= 1

P(x = 1|z = 1 ,y= 1) = P(z = 1|x = 1,y = 1) P(x = 1|y = 1)

P(z = 1|y = 1 = 0 .7∗ 0 .5

0. 65 = 0.538

P(z = 1|x = 0) = P(z = 1|x = 0 ,y= 1) P(y = 1|x = 0)+ P(z = 1|x = 0 ,y= 0) P(y = 0|x = 0) =

= 0. 6∗ 0 . 5 + 0 = 0.3

P(x = 1|z = 1) = P(z = 1 |x = 1) P(x = 1)

P(z = 1 = 0 .55 ∗ 0 .5

0. 55 ∗ 0. 5 + 0. 3∗ 0. 5= 0.647

P(x = 1|z = 0) = P(z = 0|x = 1) P(x = 1)

P(z = 0 = 0 .45 ∗ 0 .5

0. 45 ∗ 0. 5 + 0. 7∗ 0. 5 = 0.3913

84 CHAPTER 3. FOUNDATIONS OF PROBABILITY

Chapter 4

Graphical models

Graphical Models combine probability theory and graph theory [113, 136, 119] to

deal with two pervasive issues in applied mathematics and engineering: uncertainty

and complexity. In particular, they rely on the notion of conditional independence

(Section 3.1.6) to simplify the representation of complex high-variate probability

distributions.

4.1 Conditional independence and multivariate dis-

tributions

One of the hardest challenges for machine learning is to model large variate tasks,

i.e. tasks characterised by a large number of variables. Section 3.5.2 shows that an

independence assumption reduces the size of the parameter set needed to describe

a probability distribution with many variables. Unfortunately, the assumption of

independence is very strong and rarely met in real tasks. Nevertheless, it is realistic

to assume the existence of conditional independence (Section 3.5.4) relationships in

large variate settings. This assumption implies sparseness, which is a dependence

pattern where variables tend to interact with few others. If conditional indepen-

dence between some variable holds, thanks to the (3.5.56), we reduce the size of the

parameter set required to describe the joint probability distribution.

Consider for instance the case of n= 4 binary discrete r.v.s. In the generic case,

we need 24 −1 = 35 parameters to encode such probability, i.e. a quantity expo-

nential in the number of variables. This exponential nature makes the probabilistic

modelling unfeasible (i.e. too many parameters to elicit) and unmanageable (i.e.

too large required memory) in case of large n.

Let us now suppose that the 4 binary r.v.s are independent: in this case since

P( z4 , z3, z2, z1 ) = P( z4 ) P( z3 ) P( z2 ) P( z1 ) (4.1.1)

only 4 parameters are necessary to describe the joint distribution. No exponential

explosion of the number of required parameters happens. However, this is a very

simplistic and idealised setting, which rarely occurs in real interesting problems.

Moreover, if all the variables were independent, there would be no need of supervised

learning and predictive modelling since no variable brings information (or reduce

uncertainty) about the other.

A more realistic assumption is to consider some variables as conditionally inde-

pendent of others. For instance, suppose that z4 is conditionally independent of z 1

and z2 given z3 (z4 ⊥⊥ (z1 ,z2 ) |z3 )

P( z4 |z3 , z2, z1 ) = P( z4 |z3 ) (4.1.2)

86 CHAPTER 4. GRAPHICAL MODELS

and z3 is conditionally independent of z1 given z2 (z3 ⊥⊥ z1 |z2 )

P( z3 | z2 , z1 ) = P( z3 |z2 ) (4.1.3)

From the discrete version of (3.5.56) we can write

P( z4 , z3, z2, z1 ) = P( z4 | z3 , z2, z1 ) P( z3 |z2 , z1 ) P( z2 | z1 ) P( z1 )

From the conditional independence relations (4.1.2) and (4.1.3) we obtain the sim-

pliﬁed expression

P( z4 , z3, z2, z1 ) = P( z4 |z3 ) P( z3 |z2 ) P( z2 |z1 ) P( z1 )

Note that the conditional probability P ( zj |zi ) for two binary r.v.s can be now

encoded by a conditional table with two single parameters, e.g. P (zj = 1|zi =

1) and P (zj = 1|zi = 0). It follows that thanks to such assumptions, we may

describe the joint probability with 7 parameters only. The useful compactness of

the representation is still more striking in the case of large n, continuous variables,

or discrete r.v.s with a large range of values.

The representational advantage of conditional independence relationships is evi-

dent in Bayesian Networks, a formalism characterised by a correspondence between

topological properties (e.g. connectivity in a directed graph) and probabilistic ones

(notably independence). This formalism allows a compact, ﬂexible, modular (since

localized and then natural for humans) representation of joint distributions.

4.2 Directed acyclic graphs

Adirected graph G is a pair (V, E ) where V is a ﬁnite non-empty set whose elements

are called nodes , and E is a set of ordered pairs of distinct elements of V . The

elements of E are called edges. A directed cycle is a path from a node to itself. A

directed graph is called a directed acyclic graph (DAG) if it contains no directed

cycles.

Given a DAG and two nodes z1 ∈ V , and z2 ∈ V ,

•z2 is called a parent of z1 if there is an edge from z2 to z 1

•z2 is called a descendent of z1 , and z1 is called an ancestor of z2 if there is a

directed path from z1 to z 2

•z2 is called a non-descendent of z1 if it is not a descendent of z 1

Note that a node is not considered a descendant of itself.

4.3 Bayesian networks

DAGs are an eﬀective way of representing multivariate distributions where the nodes

denote random variables, and the topology (notably the absence of edges) encodes

conditional independence assertions (e.g. elicited from an expert in this domain).

A main advantage of the approach is the notion of modularity (a complex system

is made by combining simpler parts) which makes possible visual interpretability.

A Bayesian Network (BN) is a pair (G , P ) where Gis a Directed Acyclic Graph

(DAG) (i.e. graph with no loops from a variable back to itself) and Pis a joint

probability distribution over Z, which is associated with Gby the Markov condition.

Deﬁnition 3.1 (Markov condition) . Given a DAG graph Gand the associated

joint probability distribution P over Z , the Markov condition (MC) holds if every

variable is independent of its graphical non-descendants conditional on its parents.

4.3. BAYESIAN NETWORKS 87

Figure 4.1: Bayesian Network.

If the Markov condition is satisﬁed (it is also said that G represents P ) the

following theorem holds.

Theorem 3.2. If (G , P ) satisﬁes the Markov condition, then Pis equal to the

product of its conditional distributions of all nodes given values of their parents,

whenever these conditional distributions exist.

This means that if we order the set of r.v. zi , such that if zj is a descendant of

zk ,zj follows zk in the ordering (k < j), we have the product form

P( z1 , . . . , zn ) =

i=1

P( zi | Parents(zi ))

where Parents(zi ) is the set of parents of the node zi in G.

Example

An example of BN is shown in Figure 4.1. Note that the enumeration of the variable

indices satisﬁes the topological ordering mentioned before. Let us consider the node

z4 : the nodes z2 and z3 are its parents, z1 is its ancestor, and z6 is its descendant.

The associate probability distribution may be factorised as follows:

P( Z) = P( z6 |z4 ) P( z5 |z3 ) P( z4 |z3 , z2 ) P( z3 |z1 ) P( z2 |z1 ) P( z1 )

From the DAG, we can derive a number of conditional independent statements on

the basis of the Markov Condition:

z6 ⊥⊥ (z2 ,z3 ,z1 ,z5 ) |z4 (4.3.4)

z4 ⊥⊥ (z1 ,z5 ) | (z2 ,z3 ) (4.3.5)

z2 ⊥⊥ (z3 ,z5 ) |(z1 ) (4.3.6)

Note, for instance, that z4 is not independent of z6 since it is a descendant. May

you write more independence statements?

•

88 CHAPTER 4. GRAPHICAL MODELS

MARY CALLS

BURGLARY EARTHQUAKE

ALARM

JOHN CALLS

Figure 4.2: Alarm BN [166].

Example

This is a well-known example used in [166] to show a practical application of

Bayesian Networks. Suppose you want to model a burglar alarm that is fairly re-

liable at detecting a burglary but also responds on occasion to minor earthquakes.

You also have two neighbours, John and Mary, who promised to call you at work

when they hear the alarm. John always calls when he hears the alarm but some-

times confuses the telephone ringing with the alarm and calls then, too. Mary likes

loud music and sometimes misses the alarm. Given the evidence of who has or has

not called, we would like to estimate the probability of a burglary. We can describe

the problem by using a BN (Figure 4.3) where all the variables are Boolean and

denoted by capital letters to better remember their meaning.

The joint probability can be factorized as follows:

P( J, M, A, B , E) = P( J| A) P( M| A) P( A| B, E ) P( B) P( E)

Suppose that the unconditional probability of a burglar (B) or an earthquake (E)

are quite low, e.g. P (B ) = 0. 001 and P (E )=0 .002. Let the conditional probability

tables be

B E P(A | B,E) P(¬ A| B,E)

T T 0.95 0.05

T F 0.94 0.06

F T 0.29 0.71

F F 0.001 0.999

A P(J| A) P(¬ J |A)

T 0.9 0.1

F 0.05 0.95

A P(M| A) P(¬ M| A )

T 0.7 0.3

F 0.01 0.99

What is the probability Prob {B =T |J =T} (denoted Prob {B |J } below), i.e.

the probability that a burglar entered the house if John calls?

Prob {B |J } = Prob {J |B } Prob {B}

Prob {J } =

=Prob {J|B } Prob {B }

Prob {J|B } Prob {B } + Prob {J|¬B } Prob {¬B}

4.3. BAYESIAN NETWORKS 89

We have

Prob {J|B } = Prob {J|A, B } Prob {A |B } + Prob {J |¬A, B } Prob {¬A |B } =

= Prob {J|A } Prob {A|B } + Prob {J |¬A } Prob {¬A |B }

Since

Prob {A |B } = Prob {A|B , E } ∗ Prob {E } + Prob {A | B , ¬E } ∗ Prob {¬E } =

= 0. 95 · 0 . 002 + 0. 94 · (1 − 0 . 002) = 0.94

Prob {A|¬B } = Prob {A|¬B , E } ∗ Prob {E } + Prob {A|¬B , ¬E } ∗ Prob {¬E } =

= 0. 29 · 0 . 002 + 0. 001 · (1 − 0 . 002) = 0.00158

Prob {¬A|¬B } = 1 − Prob {A|¬B } = 0.9984

it follows

Prob {J |B } = 0. 9· 0 . 94 + 0. 05 · (1 − 0. 94) = 0.8490

Prob {J |¬B } = Prob {J|A } Prob {A |¬B } + Prob {J|¬A } Prob {¬A|¬B}

= 0. 9· 0 . 00158 + 0. 05 · 0. 9984 = 0.0513

and

Prob {B |J } = 0. 8490 · 0 .001

0. 8490 · 0. 001 + 0. 0513 · (1 − 0. 001) = 0.016

You can retrieve the same results by running the script alarm.R which re-

lies on ﬁrst computing the entire joint probability distribution and then the ratio

Prob{B,J }

Prob{J } .

•

Example

An interesting BN topology is known as Naive Bayes (Figure 4.3). In this case all

variables zi , 0 < i ≤ n are conditionally independent given the variable z 0

zi ⊥⊥ zj |z0 ∀i > 0 , j > 0

It follows that the associated joint probability can be written as

P( Z) = P( z0 )

i=1

P( zi |z0 )

Note that in this case, we need overall 2n+1 parameters to encode the distribution,

i.e. two parameters for each conditional distributions and one parameter to encode

P( z0 ).

This probabilistic model is commonly used for probabilistic reasoning in medical

diagnostic where z0 denotes the pathology class (the cause) and z1 ,...,zn represent

nsymptoms (or eﬀects) associated with the pathology. The assumption here is that

symptoms depend only on the underlying pathology. The Naive Bayes principle also

underlies a well-known classiﬁer which will be presented in Section 10.2.3.1.

•

Deﬁnition 3.3 (Minimality condition) . Consider the BN (G , P ) satisfying the MC.

The BN satisﬁes the minimality condition iﬀ for every proper subgraph H of Gthe

pair (H , P ) does not satisfy the MC.

90 CHAPTER 4. GRAPHICAL MODELS

Figure 4.3: Naive Bayes topology.

4.3.1 Bayesian network and d-separation

The Markov Condition induces a set of conditional independence relations in a

Bayesian Network. However, it is not easy to determine which other conditional

independence relationships possibly hold.

In order to show the link between DAG topology and conditional independence,

we introduce the criterion of d-separation. Let Z−(i,j ) the set obtained by removing

zi and zj from Z.

Deﬁnition 3.4 (d-separation) . In a DAG (Directed Acyclic Graph), two nodes z i

and zj are d-separated by the conditioning set S⊆ Z −(i,j ) (denoted by (zi lzj |S ))

if every path from zi to zj is blocked by S.

Two nodes are d-connected if they are not d-separated.

Deﬁnition 3.5 (Blocked path) . A path from zi to zj in a DAG is blocked by the

conditioning set Sif

•at least one diverging or serially connected (i.e. non-collider ) node of the path

is in S, OR

•at least one converging node (collider ) and all its descendants are not in S

Example

Consider the graph Gshown in Figure 4.1. If we consider the path z2 →z4 ←z 3

the node z4 is a collider. If we consider the path z2 →z4 →z6 the node z4 is a

non-collider. It follows then that

•z6 is d-separated from z2 by the conditioning set S= {z4 }

(z6 lz2 |z4 ) (4.3.7)

since the only path is blocked (the serially connected node z4 ∈ S is in the

path z2 →z4 →z6 )

•z2 is not d-separated from z3 by the conditioning set S= {z4 }

(z3 ↔z2 |z4 ) (4.3.8)

since there is at least a path that is not blocked (the collider node z4 ∈ S is

in the path z2 →z4 →z3 )

•

A non-blocked path is also called active . The activity of a path depends on the

activity of its (colliders and non-colliders) nodes:

4.3. BAYESIAN NETWORKS 91

•Non-colliders: when the conditioning set is empty, they are active. When they

belong to the conditioning set they become inactive.

•Colliders: when the conditioning set is empty, they are inactive. They become

active when they or some of their descendants are part of the conditioning

set.

It follows that a path is not blocked or active when all the nodes are active.

R example

The R package bnlearn allows us to encode DAGs and performs checks of d-

separation between sets of variables. This package is used in the script dsep.R to

encode and then visualise the DAG in Figure 4.1. The script also uses the function

dsep provided by the package bnlearn to check the existence of the d-separations

corresponding to the conditional independence statements (4.3.7) and (4.3.8).

•

Note that, for the moment, d-separation is a pure graphical (and non probabilis-

tic) notion related to the topology of the graph G . In what follows, we will discuss

how and when it may be informative about probabilistic properties.

4.3.2 D-separation and I-map

Deﬁnition 3.6 (I-map property) . A Bayesian Network (G , P ) satisﬁes the I-map

property if

∀zi ,zj ∈ Z, ∀S ⊆Z −(i,j ) : (zi lzj |S) ⇒I (zi ;zj |S)=0

i.e. if the d-separation (zi lzj |S ) implies the conditional independence zi ⊥⊥ zj |S

or, equivalently, the null conditional mutual information I (zi ;zj |S ) = 0.

We remind the relation (3.8.80) between conditional mutual information and

conditional independence.

The I-map property implies that the set of d-separations of G(denoted by I (G))

is contained in the set of conditional independencies of P(denoted by I (P )):

I( G) ⊆ I(P)

Note that a completely connected DAG always satisﬁes the property above since

I( G) = ∅.

It can be shown that if (and only if) a probability Psatisﬁes the Markov con-

dition for a given graph G, each d-separation in the graph implies a conditional

independence (MC ⇔I-map).

In other terms, if one takes any instance of a distribution P which factorises

according to the graph structure and I (P ) is the list of all the conditional indepen-

dence statements that can be obtained from P , if z1 and z2 are d-separated by z3 ,

the independence z1 ⊥⊥ z2 |z3 belongs to I (P).

4.3.2.1 D-separation and faithfulness

If MC holds, d-separation implies independence. Can we also say the reverse, i.e.

that d-connection (i.e. absence of d-separation) implies dependence? Do all distri-

butions P factorised according to a graph Gpossess the dependencies related to

the d-connection of the graph? Unless we make additional assumptions, the answer

is no. The required additional assumption is called faithfulness between the graph

and the distribution.

92 CHAPTER 4. GRAPHICAL MODELS

Deﬁnition 3.7 (Faithfulness). A Bayesian Network (G , P ) satisﬁes the Faithfulness

property if

∀zi ,zj ∈ Z, ∀S ⊆Z−(i,j ) : (zi ↔zj | S) ⇒I (zi ;zj | S) 6= 0

or equivalently if the conditional independence zi ⊥⊥ zj |S entails the d-separation

∀zi ,zj ∈ Z, ∀S ⊆Z −(i,j ) :I (zi ;zj |S)=0 ⇒ (zi lzj |S)

Faithfulness means that independence between two variables zi and zj (in in-

formation theory terms I (zi ;zj | S ) = 0) implies d-separation, or equivalently that

d-connection implies dependency.

When both the Markov condition and the faithfulness hold, there is a bijection

between d-separation and conditional independence. The DAG is then said a perfect

map of the joint probability distribution.

If faithfulness holds, we have a probabilistic independence interpretation of the

graphical d-separation. This means, for instance, that when the conditioning set

is empty non-colliders transmit information (dependence) while colliders do not

(independence).

Example

Consider the BN in Figure 4.1 and suppose it is faithful. For an empty conditioning

set, we can derive a number of relations of dependence, such as

z2 6⊥⊥ z6 ,z4 6⊥⊥ z 5

Are there two independent variables?

•

Beware that many distributions have no perfect map in DAGs. The spectrum

of probabilistic dependencies is, in fact, so rich that it cannot be cast into any

representation scheme that uses a polynomial amount of storage ([Verma, 1987]).

So, how strong is the assumption of faithfulness? It is possible to show that

a DAG is a perfect map for almost all distributions that factorise over G (i.e. all

distributions except a set of measure zero) [119]. This means that assuming faith-

fulness is reasonable in most practical settings. There are, however, well-known

counterexamples to faithfulness as in the case of the XOR (eXclusive OR) func-

tion1 : if x1 → y← x2 and y is the output of a XOR function with inputs x1 and

x2 , it follows the y⊥⊥ x1 and y ⊥⊥ x2 , though y is not d-separa ted from x1 and

x2 . In other terms, following the faithfulness assumptions, there should be no edges

connecting x1 , x2 and y , however, there are.

Example

Consider a symptom (e.g. headache) with two possible causes: a serious one (can-

cer) and a less serious one (virus). We can describe the problem by using a BN

(Figure 4.4) where all the variables are Boolean. First of all, if we assume a per-

fect map, from d-separation we obtain that the variable C and V are independent

but conditionally independent (i.e. conditioning on Hthe path from; C to Vis

unblocked).

Suppose that the a priori probability of the serious cause (P (C )=0 .1) is much

lower than the one of the less serious one (P (V ) = 0. 6) and that the conditional

probability table is

1the XOR function returns TRUE if one of the inputs is TRUE and FALSE if both inputs are

TRUE.

4.3. BAYESIAN NETWORKS 93

Figure 4.4: Common eﬀect conﬁguration.

C V P(H | C,V) P(¬ H| C,V)

T T 0.95 0.05

T F 0.8 0.2

F T 0.8 0.2

F F 0.1 0.9

From the script expl away.R we compute the conditional probability of cancer

in three diﬀerent situations: headache only, headache and virus and headache but

no virus:

P( C= T| H)=0 .1597846 (4.3.9)

P( C= T| H, V )=0 .1165644 (4.3.10)

P( C= T| H, ¬ V )=0 .4705882 (4.3.11)

Let us remark that the conditional probability (4.3.9) is higher than the a priori

(P(C )=0 .1) if we observe that the patient has a headache. If we know that the

patient is infected by a virus as well, the probability of cancer decreases to (4.3.10).

We say that the virus explains away the cancer possibility. On the contrary, if we

know that headache is present but the virus is absent, the cancer probability surges

again to (4.3.11).

This non-monotone behaviour is caused by what is called the explaining away

eﬀect, i.e. if we have two common causes of an observed eﬀect, knowing that one

occurs (or not) reduces (or increases) the probability of the other. This is due to the

fact that though virus and cancer are marginally independent (P (C |V ) = P (C)),

they are dependent once we condition on headache (P (C |V, H ) < P (C |H )).

•

4.3.3 Skeleton and I-equivalence

In the case of perfect-map, a BN is fully speciﬁed by its conditional independent

statements. A deﬁnition of equivalence follows, then:

Deﬁnition 3.8 (I-equivalence:) . Two graphs are I-equivalent if they have the same

associated set of independencies.

All distributions that can be factorised on a graph G, can also be factorised on

an equivalent graph G0 .

In order to check the notion of equivalence visually, we introduce the notion of

skeleton.

94 CHAPTER 4. GRAPHICAL MODELS

Deﬁnition 3.9 (Skeleton:) . The skeleton of a Bayesian Network graph G over Zis

an undirected graph that contains an edge for every edge in G.

A suﬃcient (but not necessary) condition for equivalence of two graphs is that

they have the same skeleton and the same set of v-structures. A v-structure occurs in

a DAG when there is a node having two entering edges (e.g. Figure 4.4). Complete

graphs (e.g. completely connected triplets) are equivalent (no independence) but

may have diﬀerent v-structures.

A v-structure with no direct edge between its parents is also called immorality

or unshielded collider. A suﬃcient and necessary condition for the equivalence of

two graphs is that they have the same skeleton and the same set of immoralities.

Deﬁnition 3.10 (Markov equivalence:).Two DAGs are said to be Markov equiv-

alent, if and only if they have the same skeleton and the same v-structures

Observational equivalence places a limit on the ability to infer directionality from

conditional probabilities alone. This means that there are classes of equivalence

of graphs that cannot be distinguished using conditional independence tests. Two

graphs that are I-equivalent cannot be distinguished without resorting to alternative

strategies (e.g. manipulative experimentation or temporal information).

By considering conditional independence only, it is not possible to detect changes

in graphs (e.g. arc reversing) that do not change the skeleton and which do not

introduce or destroy a v-structure. The simplest example is provided by two graphs

z1 →z2 and z2 →z1 . They are Markov equivalent (same skeleton and no v-

structure) and they have associated an empty set of independencies.

At the same time the unshielded collider z1 →z2 ←z3 is a singleton, i.e. the

only member of its equivalence class. This means that independence constraints

alone suﬃce to determine without ambiguity its structure.

4.3.4 Stable distributions

The use of the independence relationships made so far implies that the set of inde-

pendences of the probability distribution associated with a graph depends only on

the structure of the graph and not on the parametrisation.

This restriction is also known as stability: in other terms, we consider distri-

butions whose independencies remain invariant to any change in the parameters.

The stability assumption presumes that unstable independencies (i.e. dependencies

that disappear with a parameter change) are unlikely to occur in the data, so all

the independencies are structural.

In general, it is important to be aware of the limited expressibility of BNs. BNs

cannot necessarily graphically represent all the independence properties of a given

distribution. Consider, for instance, the distribution associated to x1 → y1 ←

z→ y2 ←x2 . If we marginalise the distribution wrt z(i.e. zis not observable),

there is no DAG containing only the vertices x1 ,x2 ,y1 ,y2 , which represents the

independence relations of the original DAG without adding spurious independencies.

4.4 Markov networks

Markov networks (MN) are an undirected graphical representation of conditional

independencies. Let us consider a set Z ={z1 ,...,zn } of n random variables.

Deﬁnition 4.1. The conditional independent graph of Z is the undirected graph

G= (V, E ) where V = {1 , . . . , n } and (i, j) is NOT in the edge set Eiﬀ

zi ⊥⊥ zj |Z −{i,j }

4.4. MARKOV NETWORKS 95

This graph is also called a pairwise Markov graph. Note that for n variables,

there are 2( n

2)potential undirected graphs.

4.4.1 Separating vertices, separated subsets and indepen-

dence

As in the directed case, it is possible to use topological notions (e.g. separation

of vertices in the network) to deduce probabilistic properties (e.g. conditional

(in)dependencies). Given an undirected graph G = (V , E),

•a subset of vertices separates two vertices i and j if every path joining the two

vertices contains at least one vertex from the separating subset

•a subset of vertices separates two subsets Va and Vb in G if it separates every

pair of vertices i∈ Va ,j∈ Vb .

The last property is also called the global Markov property. In general, it can

be show that the set of distributions that satisﬁes the pairwise Markov property

satisﬁes as well the global Markov property.

Given an undirected independence graph G = (V, E ) it can be shown that:

•if V can be partitioned into two subsets Vb and Vc , such that there is no path

between any vertex in Vb and any vertex in Vc then

xi ⊥⊥ xj , forall xi ∈Vb and xj ∈V c

•if Va is any subset of vertices of Gthat separates i and jthen

xi ⊥⊥ xj |X a

4.4.2 Directed and undirected representations

BNs and MNs are two closely related yet diﬀerent, representations and it is im-

portant not to confuse them. So why consider undirected representations besides

directed Bayesian networks? What is their main diﬀerence? First of all, let us

remind the Box golden rule of modelling: no model representation is perfect or

exhaustive, all of them are wrong, but sometimes some of them are useful. Markov

networks visualise conditional independence properties in distributions without hav-

ing recourse to any notion of ordering. In that sense, they are more adequate when

the considered problem is not explicitly associated to a speciﬁc ordering of vari-

ables or is characterised by symmetric relations. At the same time, asymmetric

relationships (e.g. cause and eﬀect, past and future) ﬁt well the BN formalism.

As an example, let us consider two probabilistic distributions of n random vari-

ables. In the ﬁrst case, the n variables represent a quantity measured during n

consecutive time instants. In the second case, the nvariables measure the same

quantity over n diﬀerent spatial locations. In both cases properties of conditional

independence might help in representing and reasoning on the distribution. How-

ever, only the ﬁrst case takes advantage of a Bayesian Network representation which

encodes the explicit and asymmetric time ordering. An undirected representation,

where a notion of symmetric neighbourhood is present, is more suitable to the

spatial distribution task.

A second interesting issue is whether a MN is equivalent to a BN which have

been deprived of the edge directionality. The answer is not so simple. Consider a

DAG G and a faithful probability P . Let U be the undirected skeleton associated

with G and U0 be the undirected conditional independence graph associated with

P. Which relationship exists between Uand U0 ? Uand U0 generally are not the

same, but the relation U⊆ U0 holds. As shown by Wermuth and Lauritzen, Uand

U0 are the same iﬀ Gdoes not contain any unshielded collider.

96 CHAPTER 4. GRAPHICAL MODELS

4.5 Conclusions

The graphical modelling formalism enables a modular representation of large variate

problems thanks to the correspondence between topological properties and prob-

abilistic notions. Graphical models are then eﬀective tools that can be used to

represent and communicate the relations between a large number of variables and

to perform probabilistic reasoning.

In general, an eﬀective modelling approach has to manage the trade-oﬀ between

complexity and ﬁdelity to reality. Graphical modelling uses the notion of conditional

independence to address such issue.

Note that the adoption of conditional independence assumptions to simplify

representations is pervasive in mathematical modelling and human reasoning: think,

for instance, to the notion of state in dynamical systems, which makes the future

behaviour independent from the past given the present. Simplifying by conditioning

is also a peculiar characteristic of human causal reasoning: once we ﬁnd the cause

of a certain phenomenon, we can disregard all other variables as irrelevant.

We will see that in machine learning, graphical modelling is a powerful way

to explain why some variables are more important or relevant than others (Chap-

ter 12). At the same time, machine learning strategies may be used to infer compact,

graphical (and sometimes causal) representations from data (Chapter 13).

Chapter 5

Parametric estimation and

testing

Given the correct probabilistic model of a phenomenon, we may derive the properties

of observable data by logical deduction. The theory of statistics is designed to

reverse the deductive process (Chapter 2). It takes measured data and uses them to

propose a probabilistic model, to estimate its parameters and eventually to validate

it. This chapter will focus on the estimation methodology, intended as the inductive

process which leads from observed data to a probabilistic description of reality.

We will focus here on the parametric approach, which assumes that we know all

about the probabilistic model except the value of a ﬁnite number of parameters.

Parametric estimation algorithms build estimates from data and, more important,

statistical measures to assess their quality. There are two main approaches to

parametric estimation:

Classical or frequentist: it is based on the idea that sample data are the sole

quantiﬁable form of relevant information and that the parameters are ﬁxed

but unknown. It is related to the frequency view of probability (Section 3.1.4).

Bayesian approach: the parameters are supposed to be random variables , having

a distribution prior to data observation and a distribution posterior to data

observation. This approach assumes that there exists something beyond data,

(i.e. a human sense of uncertainty or a subjective degree of belief), and that

this belief can be described in the probabilistic form.

It is well known, however, that in large-sample problems, frequentist and Bayesian

approaches tend to produce similar numerical results and that in small-medium

settings, though the two outcomes may not coincide, their diﬀerence is usually

small. For those reasons and, mainly for reasons of space, we will limit here to

consider the classical approach. It is important, however, not to underestimate the

important role of the Bayesian estimation philosophy, which led recently to a large

amount of research in Bayesian data analysis and important applications in machine

learning [78].

5.1 Classical approach

The classical approach to parameter estimation dates back to the period 1920-35

when J. Neyman and E.S. Pearson, stimulated by problems in biology and industry,

concentrated on the principles for testing hypothesis and R.A. Fisher, interested in

agricultural issues, focused on the estimation from data.

98 CHAPTER 5. PARAMETRIC ESTIMATION

We will introduce estimation by considering a simple univariate setting. Let z

be a continuous r.v. and suppose that

1. we know the analytical form of the distribution family

Fz ( z ) = Fz ( z, θ)

but the parameter vector θ∈ Θ is unknown,

2. we have access to a set DN of N i.i.d. measurements of z , called sample data.

In the general case, few parameters are not enough to describe a function, like

the density function: in that sense, parametric densities are an obvious simpliﬁca-

tion. An example of a parametric distribution function is the Normal distribution

(Section (3.4.2)), where the parameter vector is θ = [µ, σ ]. The goal of the esti-

mation procedure is to ﬁnd a value ˆ

θof the parameter θso that the parameterised

distribution Fz (z, ˆ

θ) closely matches the distribution of data.

The notation i.i.d. stands for identically and i ndependently d istributed. Identi-

cally distributed means that all the observations have been sampled from the same

distribution, that is

Prob {zi =z} = Prob {zj =z} for all i, j = 1, . . . , N and z ∈ Z

Independently distributed means that the fact that we have observed a certain value

zi does not inﬂuence the probability of observing the value zj , that is

Prob {zj =z|zi = zi } = Prob {zj =z }

Example

Here you ﬁnd some examples of estimation problems:

1. Let DN = { 20,31,14,11,19, . . . } be the times in minutes spent the last 2

weeks to go home. What is the mean time to reach my house from ULB?

2. Consider the car traﬃc in the boulevard Jacques. Suppose that the measures

of the inter-arrival times are DN = { 10,11, 1, 21,2, . . . } seconds. What does

this imply about the mean inter-arrival time?

3. Consider the students of the last year of Computer Science. What is the

variance of their grades?

4. Let z be the r.v. denoting tomorrow's temperature. How can I estimate its

mean value on the basis of past observations?

•

Parametric estimation is a mapping from the space of the sample data to the

space of parameters Θ. The two possible outcomes are:

1. some speciﬁc value of Θ. In this case, we have the so-called point estimation.

2. some particular region of Θ. In this case, we obtain an interval of conﬁdence

on the value of the parameter.

5.2. EMPIRICAL DISTRIBUTIONS 99

5.1.1 Point estimation

Consider a random variable zwith a parametric distribution Fz (z, θ ), θ∈ Θ. The

unknown parameter can be written as a function(al) of F

θ= t( F)

This corresponds to the fact that θis a characteristic of the population described

by Fz (· ). For instance the expected value parameter µ =t (F ) = R z dF (z ) is a

functional of F.

Suppose now that we have observed a set of N i.i.d. values DN ={ z1 , z2, . . . , zN } .

Apoint estimate is an example of statistic, where by statistic it is generally meant

any function of the sample data DN . In other terms a point estimate is a function

θ= g( DN ) (5.1.1)

of the sample dataset DN , where g ( · ) stands for the estimation algorithm, that

is the procedure which returns the estimation starting from a dataset DN . Note

that, from a machine learning perspective, it is more appropriate to consider g,

rather than a conventional mathematical function, as a generic algorithm taking

the sample dataset as an input and returning an estimation as output1.

There are two main issues in estimation and, more generally, in data analysis,

statistics and machine learning: how to construct an estimator (i.e. which form

should g take) and how to assess the quality of the returned estimation ˆ

θ. In

Sections 5.3 and 5.8 we will discuss two strategies for deﬁning an estimator; the

plug-in principle and the maximum likelihood. In Section 5.5 we will present the

statistical measures most commonly adopted to assess an estimator accuracy.

Before introducing the plug-in principle, we need, however, to present the notion

of empirical distribution.

5.2 Empirical distributions

Suppose we have observed a i.i.d. random sample of size Nfrom a probability

distribution Fz ( ·)

Fz → { z1 , z2, . . . , zN }

The empirical distribution probability ˆ

Fis deﬁned as the discrete distribution

that assigns probability 1/N to each value zi , i = 1 , . . . , N. In other words, ˆ

assigns to a set Ain the sample space of zits empirical probability

Prob {z ∈A } ≈ # z i ∈ A

that is the proportion of the observations in DN which occur in A.

It can be proved that the vector of observed frequencies in ˆ

Fis a suﬃcient

statistic for the true distribution F ( ·), i.e. all the information about F ( ·) contained

in DN is also contained in ˆ

F(·).

Consider now the distribution function Fz (z ) of a continuous rv zand a set of

Nobservations DN = { z1 , . . . , zN }. Since

Fz ( z) = Prob {z ≤ z}

we deﬁne N (z ) as the number of observations in DN that do not exceed z . We

obtain then the empirical estimate of F (·)

Fz ( z) = N( z)

N=# z i ≤ z

N(5.2.2)

1For instance, an awkward, yet acceptable, estimation algorithm could take the dataset, discard

all the examples except the third one and return it as the estimation.

100 CHAPTER 5. PARAMETRIC ESTIMATION

Figure 5.1: Empirical distribution.

This function is a staircase function with discontinuities at the points zi (Figure

5.1).

Example

Suppose that our dataset is made of the following N = 14 observations

DN ={ 20 , 21 , 22 , 20 , 23 , 25 , 26 , 25 , 20 , 23 , 24 , 25 , 26 , 29}

The empirical distribution function ˆ

Fz (which can be traced by running the

script cumdis.R ) is plotted in Figure 5.1.

5.3 Plug-in principle to deﬁne an estimator

Consider an r.v. zand sample dataset DN drawn from the parametric distribution

Fz ( z, θ). The main issue of estimation is how to deﬁne an estimate of θ . A possible

solution is given by the plug-in principle, that is a simple method of estimating

parameters from observations. The plug-in estimate of a parameter (or target) θis

deﬁned to be:

θ= t(ˆ

F( z)) (5.3.3)

obtained by replacing the distribution function with the empirical distribution in

the analytical expression of the parameter.

The following section will discuss the plug-in estimators of the ﬁrst two moments

of a probability distribution.

5.4. SAMPLING DISTRIBUTION 101

5.3.1 Sample average

Consider an r.v. z∼Fz ( ·) such that

θ= E[z ] = Z zdF ( z )

with θ unknown. Suppose we have available the sample Fz → DN , made of N

observations. The plug-in point estimate of θis given by the sample average

θ=1

i=1

zi = ˆ µ(5.3.4)

Note that the sample average is not a parameter (i.e. it is not a function of the

probability distribution Fz ) but a statistic (i.e. a function of the dataset DN ).

5.3.2 Sample variance

Consider a r.v. z∼ Fz (· ) where the mean µand the variance σ2 are unknown.

Suppose we have available the sample Fz →DN . Once we have the sample average

ˆ µ, the plug-in estimate of σ2 is given by the sample variance

ˆ σ2 =1

N−1

i=1

(zi − ˆ µ)2(5.3.5)

The presence of N− 1 instead of N at the denominator will be explained later.

Note also that the following relation holds for all zi

i=1

(zi − ˆ µ)2 = 1

i=1

i!−ˆ µ2

The expression of the plug-in estimators of other interesting probabilistic pa-

rameters are in the Appendix (D).

5.4 Sampling distribution

Given a dataset DN of N observations sampled from z , let us consider a point

estimate ˆ

θ= g( DN ) (5.4.6)

Note that since DN is the outcome of Nrealisations of a r.v. z, the vector DN

can be considered as the realisation of a random vector DN2 .

By applying the transformation gto the random variable DN we obtain the

random variable ˆ

θ=g (DN ) (5.4.7)

which is called the point estimator of θ . A key point is the following: while θis

an (unknown) ﬁxed value, the estimator ˆ

θis a random variable. For instance, if

we aim to estimate θ =µ (expected value of z) the parameter µis an unknown

and ﬁxed value while the average ˆ

µis a random variable (since it is a function of a

random dataset).

2This is not a mathematical detail but an essential aspect of the data-driven discovery process

under uncertainty. Every model learned from data, or more in general all knowledge acquired from

data, is built on random foundations and, as such, it is a random quantity and has to be assessed

as such.

102 CHAPTER 5. PARAMETRIC ESTIMATION

Figure 5.2: From the parametric parent distribution of Fz ( ·, θ ) (underlying the data

generation) to the sampling distribution of the estimator ˆ

θN . Each dataset has the

same size N.

The probability distribution of the r.v. ˆ

θis called the sampling distribution,

while the distribution of the r.v. z (with parameter θ ) is called the parent dis-

tribution. An example of the process bringing from the parent to the sampling

distribution is plotted in Figure 5.2. Note that the sampling distribution, though

a theoretical quantity, is of great signiﬁcance in estimation since it quantiﬁes the

estimator's accuracy in probabilistic terms, or, in simpler words, the gap between

the estimation and the parameter θ.

5.4.1 Shiny dashboard

The dashboard estimation.R (Appendix G) provides an interactive visualisation

of the sampling distribution of the plug-in estimators of the parameters (mean

and variance) of a Normal parent distribution z. We invite the reader to modify

the values N ,µ and σ and to observe the impact on the sampling distribution.

Note that the sampling distribution is obtained by a Monte Carlo simulation of the

process illustrated in Figure 5.2. The simulation (Algorithm 1 and related R code in

Table 5.1) consists in repeating a number (adjustable) of trials where for each trial

a sample dataset of size Nis generated and the plug-in estimations are computed.

The dashboard shows the histograms of the estimations.

5.5. THE ASSESSMENT OF AN ESTIMATOR 103

Algorithm 1 Monte Carlo simulation to generate a sampling distribution

1: S={}

2: for r = 1 to R do

3: Fz →DN = {z1 , z2, . . . , zN }  // pseudo-random sample generation

4: ˆ

θ= g( DN )  // estimation computation

5: S= S∪ { ˆ

θ}

6: end for

7: Plot histogram of S

8: Compute statistics of S(mean, variance)

9: Study distribution of S with respect to θ (e.g. estimate bias)

mu<-0 # parameter

R<-10000 # number trials

N<-20 # size dataset

S<-numeric(R)

for (r in 1:R){

D<-rnorm(N,mean=mu,sd=10)

# pseudo-random sample generation

S[r]<-mean(D)

# compute estimate

}

hist(S)

# Plot histogram of S

bias=mean(S)-mu

# Estimate bias

Table 5.1: R version of Algorithm 1 pseudo-code to generate the sampling distribu-

tion of ˆ

µ.

5.5 The assessment of an estimator

Once deﬁned an estimator ˆ

θ(e.g. in algorithmic or mathematical form), it is

possible to assess its accuracy from its sampling distribution.

5.5.1 Bias and variance

The following measures rely on the sampling distribution3 to assess the estimator

accuracy.

Deﬁnition 5.1 (Bias of an estimator) . An estimator ˆ

θof θ is said to be unbiased

if and only if

ED N [ˆ

θ] = θ

Otherwise, it is said to be biased with bias

Bias[ˆ

θ] = ED N [ ˆ

θ]−θ (5.5.8)

Deﬁnition 5.2 (Variance of an estimator).The variance of an estimator ˆ

θof θis

the variance of its sampling distribution

Var h ˆ

θi =ED N [(ˆ

θ−E[ˆ

θ])2]

3please note that we refer to the ˆ

θdistribution and not to the zdistribution

104 CHAPTER 5. PARAMETRIC ESTIMATION

Deﬁnition 5.3 (Standard error) . The square root of the variance

σ=r Var h ˆ

θi

is called the standard error of the estimator ˆ

θ.

An unbiased estimator is an estimator that, on average, has the right value

but averaged over what? It is important to retain that this average is over dif-

ferent realisations of the dataset DN as made explicit by the notation ED N [ ˆ

θ],

represented visually by Figure 5.2 and simulated by the Monte Carlo repetitions in

Section (5.4.1).

Note that diﬀerent unbiased estimators may exist for a parameter θ . Also, a

biased estimator with a known bias (i.e. not depending on θ) is equivalent to an

unbiased estimator since we can easily compensate for the bias. We will see in

Section 5.5.3 that for some speciﬁc estimators it is possible to derive analytically

the bias. Unfortunately, in general, the bias is not measurable since this would

require the knowledge of θwhich is in fact the target of our estimation procedure:

nevertheless, the notion of bias is an important theoretical quantity to reason about

the accuracy of an estimation process.

Sometimes we are accurate (e.g. unbiased) in estimating θ though we are inter-

ested in f (θ ). Given a generic transformation f (· ), if ˆ

θis unbiased for θ this does

not imply that that f ( ˆ

θ) is unbiased for f (θ ) as well. This implies, for instance,

that the standard error ˆ

σis not an unbiased estimator of standard deviation σ

despite ˆ

σ2 being an unbiased estimator of σ2 .

5.5.2 Estimation and the game of darts

An intuitive manner of visualising the notion of sampling distribution of an estima-

tor and the related concepts of bias and variance is to use the analogy of the darts

game.

The unknown parameter θ can be seen as the darts game target and the estimator

θas a player. Figure 5.3 shows the target (black dot) together with the distribution

of the draws of two diﬀerent players: the C (cross) player and the R (round) player.

In terms of our analogy the cross player/estimator has small variance but large bias,

while the round one has small bias and large variance. Which one is the best?

Now it's your turn to draw the shot distribution of a player with low bias and

low variance and of a player which large bias and large variance.

5.5.3 Bias and variance of ˆ

This section shows that for a generic r.v. zand an i.i.d. dataset DN , the sample

average ˆ

µis an unbiased estimator of the mean E [z ].

Consider a random variable z∼Fz ( · ). Let µ and σ2 the mean and the variance

of Fz (· ), respectively. Suppose we have observed the i.i.d. sample DN ←Fz .

From (5.3.4) we obtain

ED N [ˆ

µ] = ED N " 1

i=1

zi # =PN

i=1 E[z i ]

N= Nµ

N= µ(5.5.9)

This means that the sample average estimator is not biased, whatever the distribu-

tion Fz (· ) is. And what about its variance? Since according to the i.i.d. assumption

Cov[zi ,zj ] = 0, for i 6 =j , from (3.10.85) we obtain that the variance of the sample

5.5. THE ASSESSMENT OF AN ESTIMATOR 105

Figure 5.3: The dart analogy: the target is the unknown parameter, the round dots

represent some realisations of the estimator R, while the crosses represent some

realisations of the estimator C.

average estimator is

Var [ ˆ

µ] = Var " 1

i=1

zi # =1

N2 Var " N

i=1

zi # =1

N2 Nσ 2 = σ 2

N.(5.5.10)

In fact, ˆ

µacts like the "round player" in darts game (Figure 5.3) with some variance

but no bias.

You can visualise the bias and variance of the sample average estimator by

running the Shiny dashboard estimation.R introduced in Section 5.4.

5.5.4 Bias of the estimator ˆ

σ2

Let us study now the bias of the estimator of the variance of z.

ED N [ˆ

σ2 ] = ED N " 1

N−1

i=1

(zi − ˆ

µ)2 #(5.5.11)

N−1 E D N " 1

i=1

(zi − ˆ

µ)2 #(5.5.12)

N−1 E D N " 1

i=1

i!−ˆ

µ2 # (5.5.13)

Since E [z2 ] = µ2 +σ2 and Cov[zi ,zj ] = 0, the ﬁrst term inside the E [· ] is

ED N " 1

i=1

i!# =1

i=1

ED N  z 2

i=1

NN(µ2 +σ2 )

Since E  P N

i=1 z i  2 =N 2 µ 2 +Nσ 2 the 2nd term is

106 CHAPTER 5. PARAMETRIC ESTIMATION

ED N [ˆ

µ2 ] = 1

N2 E D N 

 N

i=1

zi ! 2 

=1

N2 ( N 2 µ 2 + Nσ 2 ) = µ 2 + σ 2 /N

It follows that

ED N [ˆ

σ2 ] = N

N−1 ( µ 2 + σ 2 )−(µ2 +σ2 /N ) = N

N−1 N−1

Nσ 2  = σ 2

This result justiﬁes our deﬁnition (5.3.5). Once the term N− 1 is inserted at the

denominator, the sample variance estimator is not biased.

Some points are worth considering:

•The results (5.5.9),(5.5.10) and (5.5.11) are independent of the family of the

distribution F (·).

•According to (5.5.10), the variance of ˆ

µis 1/N times the variance of z . This is

a formal justiﬁcation of the reason why taking averages on a large number of

observations is recommended: the larger N , the smaller is Var [ ˆ

µ], so a bigger

Nfor a given σ2 implies a better estimate of µ.

•According to the central limit theorem (Section C.7), under quite general

conditions on the distribution Fz , the distribution of ˆ

µwill be approximately

normal as Ngets large, which we can write as

µ∼ N (µ, σ 2 /N ) for N → ∞

•The standard error p Var [ˆ

µ] = σ

√N is a common measure of statistical ac-

curacy. Roughly speaking, if the estimator is not biased and the conditions

of the central limit theorem apply, we expect ˆ

µto be less than one standard

error away from µabout 68% of the time, and less than two standard errors

away from µabout 95% of the time (see Table 3.2) .

Script

You can visualize the bias and variance of the sample variance estimator by running

the following R script sam dis2.R or by running the Shiny dashboard estimation.R

introduced in Section 5.4..

5.5.5 A tongue-twister exercise

It sounds like a tongue-twister but it is important that the reader takes some time

to reason on the substantial diﬀerence between two quantities like

1. the variance of an estimator and

2. the estimator of the variance.

The ﬁrst quantity is denoted by Var h ˆ

θi , is a real number and measures the accuracy

of an estimator. It has been introduced in Section 5.5.

The second is denoted ˆ

σ2 , is a random quantity since it is an estimator and its

properties (e.g. bias) has been discussed in Section 5.5.4.

Now, if you understand the diﬀerence between the two quantities above, you

could reason on Var  ˆ

σ2  , which is nothing more than the variance of the estimator

of the variance. Clear, isn't it? And what about the estimator of the variance of

the estimator of the variance?

5.5. THE ASSESSMENT OF AN ESTIMATOR 107

5.5.6 Bias/variance decomposition of MSE

Bias and variance are two independent criteria to assess the quality of an estimator.

As shown in Figure 5.3 we could have two estimators behaving in opposite ways:

the ﬁrst has large bias and low variance, while the second has large variance and

small bias. How can we choose among them? We need a measure able to combine or

merge the two to a single criteria. This is the role of the mean-square error (MSE)

measure.

When ˆ

θis a biased estimator of θ, its accuracy is usually assessed by its MSE

rather than simply by its variance. The MSE is deﬁned by

MSE = ED N [(θ− ˆ

θ)2]

For a generic estimator it can be shown that

MSE = (E [ ˆ

θ]−θ )2 + Var h ˆ

θi =h Bias[ˆ

θ]i 2 + Var h ˆ

θi (5.5.14)

i.e., the mean-square error is equal to the sum of the variance and the squared bias

of the estimator . Here it is the analytical derivation

MSE = ED N [(θ− ˆ

θ)2 ] = ED N [(θ− E [ ˆ

θ] + E [ ˆ

θ]− ˆ

θ)2 ] = (5.5.15)

=ED N [(θ− E [ ˆ

θ])2 ] + ED N [(E [ ˆ

θ]− ˆ

θ)2 ] + ED N [2(θ− E [ ˆ

θ])(E [ ˆ

θ]− ˆ

θ)] =

(5.5.16)

=ED N [(θ− E [ ˆ

θ])2 ] + ED N [(E [ ˆ

θ]− ˆ

θ)2 ] + 2(θ− E [ ˆ

θ])(E [ ˆ

θ]−E [ˆ

θ]) =

(5.5.17)

= (E [ ˆ

θ]−θ )2 + Var h ˆ

θi (5.5.18)

This decomposition is typically called the bias-variance decomposition. Note that,

if an estimator is unbiased then its MSE is equal to its variance.

5.5.7 Consistency

Suppose that the sample data contains Nindependent observations z1 , . . . , zN of

a univariate random variable. Let the estimator of θbased on N observations be

denoted ˆ

θN . As N becomes larger, we might reasonably expect that ˆ

θN improves

as estimator of θ(in other terms it gets closer to θ ). The notion of consistency

formalizes this concept.

Deﬁnition 5.4. The estimator ˆ

θN is said to be weakly consistent if ˆ

θN converges

to θ in probability, that is

∀ > 0 lim

N→∞Prob n | ˆ

θN −θ | ≤ o = 1

Deﬁnition 5.5. The estimator ˆ

θN is said strongly consistent if ˆ

θN converges to θ

with probability 1 (or almost surely).

Prob nlim

N→∞

θN =θo = 1

For a scalar θthe property of convergence guarantees that the sampling distribu-

tion of ˆ

θN becomes less disperse as N → ∞. In other terms a consistent estimator

is asymptotically unbiased. It can be shown that a suﬃcient condition for weak

consistency of unbiased estimators ˆ

θN is that Var h ˆ

θN i →0 as N → ∞.

It is important to remark that the property of unbiasedness (for ﬁnite-size sam-

ples) and consistency are largely unrelated.

108 CHAPTER 5. PARAMETRIC ESTIMATION

Exercise

Consider an estimator of the mean that takes into consideration only the ﬁrst 10

sample points, whatever the total number N > 10 of observations is. Is such

estimator consistent?

•

5.5.8 Eﬃciency

Suppose we have two unbiased and consistent estimators. How to choose between

them?

Deﬁnition 5.6 (Relative eﬃciency) . Let us consider two unbiased estimators ˆ

θ1

and ˆ

θ2 . If

Var h ˆ

θ1 i <Var h ˆ

θ2 i

we say that ˆ

θ1 is more eﬃcient than ˆ

θ2 .

If the estimators are biased, typically the comparison is done on the basis of the

mean square error.

Exercise

Suppose z1 , . . . , zN is a random sample of observations from a distribution with

mean θ and variance σ2 . Study the unbiasedness and the consistency of the three

estimators of the mean µ:

θ1 = ˆ µ=PN

i=1 z i

θ2 = N ˆ

θ1

N+ 1

θ3 = z1

•

5.6 The Hoeﬀding's inequality

A probabilistic measure of the discrepancy between the estimator ˆ

µand the quantity

µ= E[z ] to be estimated is returned by the Hoeﬀding's inequality.

Theorem 6.1. [103] Let z1 ,...,zN be independent bounded random variables such

that zi falls in the interval [ai , bi ] with probability one. Let their sum be SN =

i=1 z i . Then for any ε > 0 we have

Prob {|SN −E[ SN ] | > ε } ≤ exp ( −2 ε2/

i=1

(bi − ai )2 )

Corollary 6.2. If the variables z1 ,...,zN are independent and identically dis-

tributed, the following bound on the discrepancy between the sample mean ˆ µ=

i=1 z i

Nand the expected value E[z ] holds

Prob {|ˆ

µ−E[z ] |> ε } ≤ exp  − 2Nε2 / (b −a)2 

5.7. SAMPLING DISTRIBUTIONS FOR GAUSSIAN R.V.S 109

Assume that δ is a conﬁdence parameter, that is we are 100(1 −δ )% conﬁdent

that the estimate ˆ µis within the accuracy εof the true expectation. It is possible

to derive the expression

ε( N) = r ( b−a)2log(2/δ)

which measures with conﬁdence 1 −δ how the sample mean ˆ µ, estimated on the

basis of N points, is close to the expectation E [z ]. We can also determine the

number of observations Nnecessary to obtain an accuracy ε and a conﬁdence δ by

using the relation

N > ( b− a)2 log(2 /δ)

2ε 2

Hoeﬀding's bound is a general bound that only relies on the assumption that

sample points are drawn independently. Bayesian bounds are another example of

statistical bounds which give tighter results under the assumption that the examples

are drawn from a normal distribution.

5.7 Sampling distributions for Gaussian r.v.s

The results in Section 5.5 are independent of the type of distribution function Fz .

Additional results are available in the speciﬁc case of a normal random variable.

Let z1 ,...,zN be i.i.d. realisation of z ∼ N ( µ, σ2 ) and let us consider the

following sample statistics

µ=1

i=1

zi ,c

SS =

i=1

(zi − ˆ

µ)2 , ˆ

σ2 =c

N−1

It can be shown that the following relations hold

•ˆ

µ∼ N (µ, σ 2 /N ) and N(ˆ

µ−µ)2 ∼σ2 χ2

1where the χ 2 distribution is presented

in Appendix C.2.2.

•zi −µ ∼ N (0, σ 2 ), so P N

i=1(z i −µ) 2 ∼σ 2 χ 2

•PN

i=1(z i −µ) 2 =c

SS +N (ˆ µ− µ)2 .

•c

SS ∼σ2 χ 2

N−1 or equivalently (N− 1) ˆ

σ2

σ2 ∼χ2

N−1 . See R script sam dis2.R.

•√ N(ˆ

µ−µ) / ˆ

σ∼ TN−1 where T stands for the Student distribution (Section

C.2.3).

•if E [ |z −µ |4 ] = µ4 then Var  ˆ

σ2  =1

Nµ 4 − N−3

N−1σ 4 .

5.8 The principle of maximum likelihood

Maximum-likelihood is a major strategy used in statistics to design an estimator,

i.e. the algorithm gin (5.4.7). Its rationale is to transform a problem of estimation

into a problem of optimisation. Let us consider

1. a density distribution pz ( z, θ ) which depends on a parameter θ∈ Θ,

2. a dataset DN = {z1 , z2, . . . , zN } i.i.d. drawn from this distribution.

110 CHAPTER 5. PARAMETRIC ESTIMATION

According to (3.5.54), the joint probability density of the i.i.d. dataset is the product

pD N (DN , θ) =

i=1

pz (zi , θ) = LN ( θ ) (5.8.19)

where for a ﬁxed DN , LN ( · ) is a function of θ and is called the empirical likelihood

of θ given DN .

The principle of maximum likelihood was ﬁrst used by Lambert around 1760 and

by D. Bernoulli about 13 years later. It was detailed by Fisher in 1920. The idea

is simple: given an unknown parameter θand a sample data DN , the maximum

likelihood estimate ˆ

θis the value for which the empirical likelihood LN (θ ) has a

maximum ˆ

θml = arg max

θ∈Θ L N (θ)

The estimator ˆ

θml is called the maximum likelihood estimator (m.l.e.). In prac-

tice, it is usual to consider the log-likelihood lN (θ ) instead of LN (θ ). Since log(· ) is

a monotone function, we have

θml = arg max

θ∈Θ L N (θ ) = arg max

θ∈Θlog(L N (θ )) = arg max

θ∈Θl N (θ) (5.8.20)

The likelihood function quantiﬁes the relative abilities of the various parameter

values to explain the observed data. The principle of m.l. is that the value of the

parameter under which the obtained data would have had highest probability of

arising must be intuitively our best estimator of θ. In other terms the likelihood

can be considered a measure of how plausible the parameter values are in light of

the data. Note however that the likelihood function is NOT a probability function:

for instance, in general, it does not integrate to 1 (with respect to θ). In terms

of conditional probability, LN (θ ) represents the probability of the observed dataset

given θ and not the probability of θ (which is not a r.v. in the frequentist approach)

given DN .

Example

Consider a binary variable (e.g. a coin tossing) which takes z = 15 times the value

1 (e.g. "Tail") in N= 40 trials. Suppose that the probabilistic model underlying

the data is Binomial (Section C.1.2) with an unknown probability θ =p . We want

to estimate the unknown parameter θ =p∈ [0, 1] on the basis of the empirical

evidence from the N trials. The likelihood L (p ) is a function of (only) the unknown

parameter p . By applying the maximum likelihood technique we have

θml = ˆ p= arg max

pL N (p ) = arg max

pN

z p z (1−p )(N− z) = arg max

p40

15 p 15 (1−p )(25)

Figure 5.4 plots L (p ) versus p∈ [0, 1] (R script ml bin.R). The most likely value

of p is the value where L (· ) attains its maximum. According to Figure 5.4 this value

is ˆ p= z/N. The log-likelihood for this model is

lN ( p ) = log LN ( p ) = log  N

z + zlog(p )+( N− z) log(1 − p ) =

= log  40

15 + 15 log p+ 25 log(1 − p)

The reader can analytically ﬁnd the maximum of this function by diﬀerentiating

l( p) with respect to p.

•

5.8. THE PRINCIPLE OF MAXIMUM LIKELIHOOD 111

Figure 5.4: Likelihood function

5.8.1 Maximum likelihood computation

In many situations the log-likelihood lN (θ ) is particularly well behaved in being

continuous with a single maximum away from the extremes of the range of variation

of θ . Then ˆ

θml is obtained simply as the solution of

∂lN ( θ )

∂θ = 0

subject to

∂2 lN ( θ)

∂θ 2   ˆ

θml

to ensure that the identiﬁed stationary point is a maximum.

5.8.2 Maximum likelihood in the Gaussian case

Let DN be a random sample from the r.v. z ∼ N (µ, σ2 ). It is possible to derive

analytically the expression of the maximum likelihood estimators of the mean and

variance of z . According to (5.8.19), the likelihood of the N observations is

LN ( µ, σ2 ) =

i=1

pz (zi , µ, σ2 ) =

i=1  1

√2πσ2  exp  −(zi −µ)2

2σ2 

and the log-likelihood is

lN ( µ, σ2 ) = log LN ( µ, σ 2 ) = log " N

i=1

pz (zi , µ, σ2 )# =

i=1

log pz ( zi , µ, σ2 ) = − P N

i=1(z i −µ)2

2σ2 +Nlog  1

√2πσ2 

Note that, for a given σ, maximising the log-likelihood is equivalent to minimising

the sum of squares of the diﬀerence between zi and the mean. Taking the derivatives

with respect to µ and σ2 and setting them equal to zero, we obtain

ˆ µml = P N

i=1 z i

N= ˆ µ(5.8.21)

ˆ σ2

ml = P N

i=1(z i −ˆ µml )2

N6= ˆ σ2 (5.8.22)

112 CHAPTER 5. PARAMETRIC ESTIMATION

Note that the m.l. estimator (5.8.21) of the mean coincides with the sample

average (5.3.4) but that the m.l. estimator (5.8.22) of the variance diﬀers from the

sample variance (5.3.5) in terms of the denominator.

In the multivariate Normal case, where zis a vector with [n, 1] mean µ and [n, n]

covariance matrix Σ, the maximum likelihood estimators are

ˆ µml = P N

i=1 z i

N(5.8.23)

Σml =PN

i=1(z i −ˆ µml )(zi − ˆ µml )T

N(5.8.24)

where zi and ˆ µare [n, 1] vectors.

Exercise

•Let z ∼ U (0, M ) follow a uniform distribution and Fz → DN = {z1 , . . . , zN } .

Find the maximum likelihood estimator of M.

•Let z have a Poisson distribution, i.e.

pz ( z, λ) = e −λ λ z

If Fz ( z, λ )→ DN = {z1 , . . . , zN } , ﬁnd the m.l.e. of λ

•

In case of generic distributions Fz computational diﬃculties may arise: for ex-

ample in some cases no explicit solution might exist for ∂lN (θ) /∂θ = 0. Iterative

numerical methods must be used in this case. The computational cost becomes

heavier if we consider a vector of parameters instead of a scalar θor when there are

several relative maxima of the function lN .

Another complex situation occurs when lN (θ ) is discontinuous, or have a dis-

continuous ﬁrst derivative, or a maximum at an extremal point.

R script

Suppose we know the analytical form of a one dimensional function f (x ) : I→ R

but not the analytical expression of its extreme points. In this case numerical

optimisation methods can be applied. The implementation of some continuous

optimisation routines is available in the R statistical tool.

Consider for example the function f (x ) = (x− 1 / 3)2 and I = [0, 1]. The value

of the point x where f takes a minimum value can be approximated numerically by

this set of R commands

f <- function (x,a) (x-a)^2

xmin <- optimize(f, c(0, 1), tol = 0.0001, a = 1/3)

xmin

These routines may be applied to solve the problem of maximum likelihood

estimation which is nothing more than a particular case of optimisation problem.

Let DN be a random sample drawn from z ∼ N (µ, σ 2 ). The negative log-likelihood

function of the Nobservations can be written in R by

eml <- function(m,D,var) {

N<- length(D)

Lik<-1

5.8. THE PRINCIPLE OF MAXIMUM LIKELIHOOD 113

for (i in 1:N)

Lik<-Lik*dnorm(D[i],m,sqrt(var))

-log(Lik)

}

and the numerical minimisation of −lN ( µ, s2 ) for a given σ =s in the interval

I= [− 10 ,10] can be written in R as

xmin<-optimize( eml,c(-10,10),D=DN,var=s)

In order to run the above code and compute numerically the m.l. solution we invite

the reader to run the R script emp ml.R.

•

5.8.3 Cramer-Rao lower bound

Assume that θ is a scalar parameter, that the ﬁrst two derivatives of LN (θ ) with re-

spect to θ exist for all θand that certain operations of integration and diﬀerentiation

may be interchanged. Let ˆ

θbe an unbiased estimator of θ and lN (θ ) = loge [LN (θ)].

Suppose that the regularity condition

E ∂lN (θ)

∂θ  = 0 (5.8.25)

holds where the quantity ∂l (θ) /∂θ is called score . The Cramer-Rao bound is a lower

bound to the variance of the estimator ˆ

θwhich states that

Var h ˆ

θi ≥1

E ∂lN (θ)

∂θ  2 =− 1

NE h ∂ 2 l N (θ)

∂θ2 i =1

where the denominator term IN is known as the Fisher information. Note that

∂2 lN ( θ)

∂θ2 is the second derivative of l N (·) and, as such, it deﬁnes the curvature of the

log-likelihood function. At the maximum ˆ

θ, the second derivative takes a negative

value. Also, the larger its absolute value the larger is the curvature around the

function peak and then the lower is the uncertainty about the m.l. estimation [145].

An estimator having a variance as low as 1/IN is called a Minimum Variance

Bound (MVB) estimator.

Example

Consider a r.v. z ∼ N (µ, σ2 ) where σ2 is known and the unknown parameter is

θ= µ. Let us consider the bound on the variance of the estimator (5.8.21). Since

∂log p( z, θ)

∂θ = z− θ

σ2

∂2 log p( z, θ)

∂θ 2 =− 1

σ2

It follows that

Var h ˆ

θi ≥1

σ2

=σ 2

From (5.5.10) it derives then that the m.l. estimator (5.8.21) of the mean µis

minimum variance.

•

114 CHAPTER 5. PARAMETRIC ESTIMATION

5.8.4 Properties of m.l. estimators

Under the (strong) assumption that the probabilistic model structure is known, the

maximum likelihood technique features the following properties:

•ˆ

θml is asymptotically unbiased but usually biased in small-size samples (e.g.

ˆ σ2

ml in (5.8.22)).

•ˆ

θml is consistent.

•If ˆ

θml is the m.l.e. of θ and γ (· ) is a monotone function then γ ( ˆ

θml ) is the

m.l.e. of γ (θ).

•If γ ( ·) is a non monotonic function, then even if ˆ

θml is an unbiased estimator

of θ , the m.l.e. γ ( ˆ

θml ) of γ (θ ) is usually biased.

•the variance of ˆ

θml is often diﬃcult to determine. For large-size samples we

can use as approximation

−E ∂ 2 l N

∂θ 2  −1

or  −∂ 2 l N

∂θ 2   ˆ

θml  −1

•ˆ

θml is asymptotically normally distributed, that is

θml ∼ N ( θ, [ IN ( θ )]−1 ) , N → ∞

5.9 Interval estimation

Unlike point estimation which is based on a one-to-one mapping from the space of

data to the space of parameters, interval estimation maps DN to an interval of Θ.

A point estimator is a function which, given a dataset DN generated from Fz (z, θ),

returns an estimate of θ . An interval estimator is a transformation which, given a

dataset DN , returns an interval estimate [θ, ¯

θ] of θ. While an estimator is a random

variable, an interval estimator is a random interval. Let θ and ¯

θbe the random

lower and the upper bounds respectively. While an interval either contains or not

a certain value, a random interval has a certain probability of containing a value.

Suppose that

Prob  θ≤θ ≤ ¯

θ = 1 − α α ∈ [0, 1] (5.9.26)

then the random interval [θ, ¯

θ] is called a 100(1 −α )% conﬁdence interval of θ.

If (5.9.26) holds, we expect that by repeating the sampling of DN and the con-

struction of the conﬁdence interval many times, our conﬁdence interval will contain

the true θat least 100(1 −α )% of the time. Notice, however, that being θa ﬁxed

unknown value, at each realisation DN the interval [θ, ¯

θ] either contains or not the

true θ . Therefore, from a frequentist perspective, it is erroneous to think that 1 − α

is the probability of θbelonging to the interval [θ, ¯

θ] computed for a given DN . In

fact, 1 −α is not the probability of the event θ∈ [θ, ¯

θ] (since θis ﬁxed) but the

probability that the interval estimation procedure returns a (random) interval [θ, ¯

θ]

containing θ.

While a point estimator is characterised by bias and variance (Section 5.5), an

interval estimator is characterised by its endpoints θ and ¯

θ(or its width) and by

its conﬁdence α . In Figure 5.3 we used an analogy between point estimation and

dart game to illustrate the bias/variance notions. In the case of interval estimation

the best analogy is provided by the horseshoes game4 (Figure 5.5). A horseshoe

4https://en.wikipedia.org/wiki/Horseshoes

5.9. INTERVAL ESTIMATION 115

Figure 5.5: Horseshoes game as an analogy of interval estimation

player is like an interval estimator and her interval estimation corresponds to the

tossing of a horseshoe. The horseshoe width corresponds to the interval size and

the probability of encircling the stake corresponds to the conﬁdence α.

5.9.1 Conﬁdence interval of µ

Consider a random sample DN of a r.v. z ∼ N (µ, σ2 ) where σ2 is known. Suppose

we want to estimate µwith the estimator ˆ

µ. From Section 5.7 we have that ˆ

µ∼

N(µ, σ2 /N ) is Gaussian distributed. From (3.4.47) it follows that

µ−µ

σ/√ N ∼ N (0,1)

and consequently, according to the Deﬁnition 4.7

Prob  − zα/2 ≤ ˆ

µ−µ

σ/√ N≤ z α/2  = 1 −α (5.9.27)

Prob  ˆ

µ−zα/2

√N ≤µ ≤ˆ

µ+zα/2

√N  = 1 −α (5.9.28)

where zα is the upper critical point of the standard Gaussian distribution. It follows

that θ = ˆ

µ−zασ/ √ Nis a lower 1 −αconﬁdence bound for µwhile ¯

θ=ˆ

µ+zασ/√N

is an upper 1 −α conﬁdence bound for µ. By varying α we can vary the width and

the conﬁdence of the interval.

Example

Let z ∼ N ( µ, 0. 01) and DN ={ 10,11,12,13,14,15} . We want to estimate the

conﬁdence interval of µ with level α = 0. 1. Since N = 6, ˆ µ= 12. 5, and

= zα/2 σ/√ N = 1 . 645 · 0 . 01 /√ 6=0 .0672

the 90% conﬁdence interval for the given DN is

{µ: | ˆ

µ−µ | ≤  }= { 12. 5 −0. 0672 ≤µ ≤12. 5+0 .0672}

•

116 CHAPTER 5. PARAMETRIC ESTIMATION

Figure 5.6: Fraction of times that the interval of conﬁdence contains the parameter

µvs. the number of repetitions for α= 0 .1

R script

The R script confidence.R allows the test of the formula (5.9.27) by simulation.

The user sets µ ,σ,N,α and a number of iterations N iter.

The script generates N iter times DN ∼ N ( µ, σ 2 ) and computes ˆ µ. The script

returns the percentage of times that

ˆ µ− zα/2 σ

√N<µ< ˆ µ+ zα/2 σ

√N

This percentage versus the number of iterations is plotted in Figure 5.6 (R script

confidence.R). We can easily check that this percentage converges to 100(1 −α )%

for N iter → ∞.

•

Consider now the interval of conﬁdence of µwhen the variance σ2 is not known.

Let ˆ

µand ˆ

σ2 be the estimators of µ and σ2 computed on the basis of the i.i.d.

dataset DN . From Section 5.7, it follows that

ˆ µ− µ

qˆ σ2

∼ TN−1

Analogously to (5.9.28) we have

Prob  ˆ

µ−tα/2

√N ≤µ ≤ˆ

µ+tα/2

√N  = 1 −α (5.9.29)

where tα is the upper critical point of the Student distribution.

Example

Let z ∼ N ( µ, σ 2 ), with σ2 unknown and DN = { 10,11,12,13,14,15} . We want to

estimate the conﬁdence region of µ with level α = 0. 1. We have ˆ µ= 12. 5, ˆ σ2 = 3. 5 .

According to (5.9.29) we have

= t{α/2,N− 1} ˆ σ/√ N = 2.015 ∗ 1 .87/√ 6=1 .53

The (1 −α ) conﬁdence interval of µis

ˆ µ−  < µ < ˆ µ+ 

•

5.10. COMBINATION OF TWO ESTIMATORS 117

Example

We want to estimate θ , the proportion of people who support the politics of Mr.

Berlusconi amongst a very large population. We want to deﬁne how many interviews

are necessary to have a conﬁdence interval of 6% width with a signiﬁcance of 5%.

We interview Npersons and estimate θas

θ= x 1 +· ·· +xN

N= S

where xi = 1 if the i th person supports Berlusconi and xi = 0 otherwise. Note that

Sis a binomial variable. We have

E[ˆ

θ] = θ, Var h ˆ

θi = Var [S/N ] = N ( θ )(1 −θ )

N2 = θ(1 − θ)

N≤ 1

If we approximate the distribution of ˆ

θby N (θ, θ(1−θ )

N) it follows that ˆ

θ−θ

√θ(θ− 1)N ∼

N(0, 1). The following relation holds

Prob n ˆ

θ−0. 03 ≤θ ≤ˆ

θ+ 0. 03o=

Prob ( − 0.03

pθ (1 −θ )/N ≤ ˆ

θ−θ

pθ(1 −θ ) /N ≤ 0.03

pθ(1 −θ )/N ) =

Φ 0.03

pθ (1 −θ )/N ! −Φ − 0.03

pθ(1 − θ) /N ! ≥

≥Φ(0.03 √ 4N ) −Φ( −0 .03 √ 4N)

In order to have this probability to be at least 0.95 we need 0. 03√4N≥ 1 . 96 or

equivalently N≥ 1068.

•

5.10 Combination of two estimators

Consider two unbiased estimators ˆ

θ1 and ˆ

θ2 of the same parameter θ

E[ˆ

θ1 ] = θ E [ ˆ

θ2 ] = θ

having equal and non zero variance

Var h ˆ

θ1 i = Var h ˆ

θ2 i =v

and being uncorrelated, i.e. Cov[ ˆ

θ1 ,ˆ

θ2 ] = 0. Let ˆ

θcm be the combined estimator

θcm =ˆ

θ1 +ˆ

θ2

This estimator has the nice properties of being unbiased

E[ˆ

θcm ] = E [ ˆ

θ1 ] + E [ ˆ

θ2 ]

2=θ(5.10.30)

and with a smaller variance than the original estimators

Var h ˆ

θcm i =1

4Var h ˆ

θ1 +ˆ

θ2 i =

Var h ˆ

θ1 i + Var h ˆ

θ2 i

4= v

2(5.10.31)

This trivial computation shows that the simple average of two unbiased estimators

with a non zero variance returns a combined estimator with reduced variance.

118 CHAPTER 5. PARAMETRIC ESTIMATION

5.10.1 Combination of mestimators

Here, we report the general formula of the linear combination of a number mof

estimators [179, 181]. Assume we want to estimate the unknown parameter θ by

combining a set of m estimators { ˆ

θj },j = 1, . . . , m . Let

E[ˆ

θj ] = µj Var h ˆ

θj i =vj Bias[ˆ

θj ] = bj

be the expected values, the variances and the bias of the m estimators, respectively.

We are interested in estimating θby forming a linear combination

θcm =

j=1

wj ˆ

θj =wT ˆ

θ(5.10.32)

where ˆ

θ= [ˆ

θ1 ,...,ˆ

θm ]T is the vector of estimators and w = [w1 , . . . , wm ]T is the

weighting vector.

The mean-squared error of the combined system is

MSE = E [(ˆ

θcm −θ)2 ] = E[( wT ˆ

θ−E[ wT ˆ

θ])2 ]+( E[ wT ˆ

θ]−θ )2

=E [(wT ( ˆ

θ−E[ˆ

θ]))2 ]+( wT µ− θ)2=

=wT Ωw + (wT µ− θ )2

where Ω is a [m× m ] covariance matrix whose ij th term is

Ωij =E [(ˆ

θi −µi )(ˆ

θj −µj )]

and µ = (µ1 , . . . , µm )T is the vector of expected values. Note that the MSE error

has a variance term (dependent on the covariance of the single estimators) and a

bias term (dependent on the bias of the single estimators).

5.10.1.1 Linear constrained combination

A commonly used constraint is

j=1

wj = 1 , wj ≥ 0 , j = 1 , . . . , m (5.10.33)

This means that the combined estimator is unbiased if the individual estimators are

unbiased. Let us write was

w= ( uT g )−1 g

where u = (1,..., 1)T is an m-dimensional vector of ones, g = (g1 , . . . , gm )T and

gj >0 , ∀ j = 1 , . . . , m.

The constraint can be enforced in minimising the MSE by using the Lagrangian

function

L= wT Ω w+ ( wT µ− θ)2 + λ( wT u−1)

with λ Lagrange multiplier.

The optimum is achieved if we set

g∗ = [Ω + ( µ− θu)( µ− θu)T ]−1 u

With unbiased estimators (µ =θ ) we obtain

g∗ = Ω−1 u

and with uncorrelated estimators

g∗

j=1

j= 1 , . . . , m (5.10.34)

This means that the optimal term g ∗

jof each estimator is inversely proportional to

its own variance.

5.11. TESTING HYPOTHESIS 119

5.11 Testing hypothesis

Hypothesis testing is together with estimation a major area of statistical inference.

Astatistical hypothesis is an assertion or conjecture about the distribution of one

or more random variables. A test of a statistical hypothesis is a rule or procedure

for deciding whether to reject the assertion on the basis of the observed data. The

basic idea is formulate some statistical hypothesis and look to see whether the data

provides any evidence to reject the hypothesis. Examples of hypothesis tests follow:

•Consider the model of the traﬃc in the boulevard. Suppose that the measures

of the inter-arrival times are DN ={ 10,11,1,21, 2 , . . . } seconds. Can we say

that the mean inter-arrival time θis diﬀerent from 10?

•We want to know the eﬀect of a drug on rats' survival to cancer. We randomly

divide some rats in two groups and we administrate a drug only to one of them.

Is the survival rate of the groups the same?

•Consider the grades of two diﬀerent school sections. Section A had { 15,10,12,19, 5, 7}.

Section B had { 14,11,11,12,6,7} . Can we say that Section A had better

grades than Section B?

•Consider two protein coding genes and their expression levels in a cell. Are

the two genes diﬀerentially expressed ?

A statistical test is a procedure that aims to answer such questions.

5.11.1 Types of hypothesis

We start by declaring the working (basic, null) hypothesis H to be tested, in the

form θ =θ0 or θ∈ ω⊂ Θ, where θ0 or ω are given.

The hypothesis can be

simple: this means that it fully speciﬁes the distribution of the r.v. z .

composite: this means that it partially speciﬁes the distribution of z .

For example if DN is a random sample of size N drawn from N (µ, σ2 ) the

hypothesis H :µ = µ0 , σ = σ0 , (with µ0 and σ0 known values) is simple while the

hypothesis H :µ = µ0 is composite since it leaves open the value of σin (0,∞ ).

5.11.2 Types of statistical test

Suppose we have sampled a dataset DN ={ z1 , . . . , zN } from a distribution Fz and

we have declared a null hypothesis H about F . The three most common types of

statistical test are:

Pure signiﬁcance test: data DN are used to assess the inferential evidence against

Signiﬁcance test: the inferential evidence against His used to judge whether H

is inappropriate. This test returns a decision rule for rejecting or not rejecting

Hypothesis test: data DN are used to assess the hypothesis Hagainst a speciﬁc

alternative hypothesis ¯

H.This test returns a rule for rejecting Hin favour of

The three tests will be discussed in the following sections.

120 CHAPTER 5. PARAMETRIC ESTIMATION

5.11.3 Pure signiﬁcance test

Consider a simple null hypothesis H . Let t (DN ) be a statistic (i.e. a function of the

dataset) such that the larger its value the more it casts doubt on H . The quantity

t(DN ) is called test statistic or discrepancy measure. Suppose that the distribution

of t (DN ) under H is known. This is possible since the function t ( · ) is ﬁxed by

the user and the simple hypothesis H entirely speciﬁes the distribution of zand

consequently the distribution of t (DN ). Let tN =t (DN ) the observed value of t

calculated on the basis of the sample data DN . Let us deﬁne the p-value quantity

p= Prob { t( DN ) > tN | H} (5.11.35)

i.e. the probability of observing a statistic greater than tN if the hypothesis H were

true. Note that in the expression (5.11.35), the term t (DN ) is a random variable

having a known distribution, while tN is a value computed on the basis of the

observed dataset.

If the p quantity is small then the sample data DN are highly inconsistent

with H , and p (signiﬁcance probability or signiﬁcance level) is the measure of such

inconsistency. If p is small, then either a rare event has occurred or perhaps His

not true. In other terms, if Hwere true, the quantity p would be the proportion of

situations where we would observe a degree of inconsistency, at least to the extent

represented by tN . The smaller the p-value, the stronger the evidence against H5 .

Note that p depends on DN since diﬀerent DN would yield diﬀerent values of

tN and consequently diﬀerent values of p∈ [0, 1]. Moreover, it can be shown that,

if the null hypothesis is true, the p-value has a Uniform U [0, 1] distribution. Also,

in a frequentist perspective, we cannot say that p is the probability that His true

but rather that p is the probability that the dataset DN is observed given that His

true.

5.11.4 Tests of signiﬁcance

The test of signiﬁcance proposes the following decision rule: if pis less than some

stated value α , we reject H . Once a critical level α is chosen, and the dataset D N

is observed, the rule rejects H at level αif

P{ t( DN ) > tα | H ) = α (5.11.36)

This is equivalent to choosing some critical value tα and to reject H if tN > tα .

This implies the existence of two regions in the space of sample data:

critical region: this is the set of values of DN

S0 = { DN : t (DN ) > tα }

such that if DN ∈ S0 , we reject the null hypothesis H.

non-critical region: this is the set of values of DN such that there is no reason

to reject Hon the basis of the level-αtest.

The principle is that we will accept Hunless what we observed has a too small

probability of happening when H is true. The upper bound of this probability is α,

i.e. the signiﬁcance level αis the highest p-value for which we reject H. Note that

the p-value changes with the observed data (i.e. it is a random variable) while αis

a level ﬁxed by the user.

5It is common habit in life-sceince research to consider a p-value smaller than 0.05 (0.01) a

(very) strong evidence against H

5.11. TESTING HYPOTHESIS 121

Example

Let DN consist of N independent observations of x ∼ N ( µ, σ2 ), with known variance

σ2 . We want to test the hypothesis H :µ = µ0 with µ0 known. Consider as test

statistic the quantity t (DN ) = | ˆ

µ−µ0 |where ˆ

µis the sample average estimator.

If H is true we know from Section 5.4 that ˆ

µ∼ N (µ0 , σ 2 /N ). Let us calculate the

value t (DN ) = | ˆ µ− µ0 | and ﬁx a signiﬁcance level α = 10%. This means that the

decision rule needs the deﬁnition of the value tα such that

Prob { t ( DN ) > tα | H} = Prob {| ˆ

µ−µ0 |> tα |H } =

Prob { ( ˆ

µ−µ0 > tα ) ∪(ˆ

µ−µ0 < −tα ) |H }= 0.1

For a Normal variable z ∼ N ( µ, σ2 ), we have that

Prob {|z − µ |> 1.645σ} = Prob  |z−µ|

σ>1.645 = 2 ∗ 0 .05

It follows that being ˆ

µ∼ N ( µ0 , σ 2 /N )

Prob n | ˆ

µ−µ0 |>1 .645σ/ √ No = 0 .05 + 0.05 = 0 .1

and consequently

tα = 1 .645 σ/√ N (5.11.37)

The critical region is

S0 = nDN :| ˆ µ− µ0 | > 1 . 645σ/√ No •

Example

Suppose that σ = 0. 1 and that we want to test if µ = µ0 = 10 with a signiﬁcance

level 10%. Let N = 6 and DN = { 10,11,12,13,14,15} . From the dataset we

compute

ˆ µ=10 + 11 + 12 + 13 + 14 + 15

6= 12.5

and

t( DN ) = | ˆ µ− µ0 | = 2.5

Since according to (5.11.37) tα = 1. 645 ∗ 0. 1 /√ 6 = 0. 0672, and t ( DN ) > tα ,

the observations DN are in the critical region. The conclusion is: the hypothesis

H: µ= 10 is rejected and the probability that we are making an error by rejecting

His smaller than 0 .1.

•

5.11.5 Hypothesis testing

So far we have dealt with single hypothesis tests. Let us now consider two mutually

exclusive hypothesis: H and ¯

H. Suppose we have a dataset { z1 , . . . , zN } ∼ F drawn

from a distribution F. On the basis of this dataset, one hypothesis will be accepted

and the other one rejected. In this case, given the stochastic setting, two type of

errors are possible.

Type I error. This is the kind of error we make when we reject H but H is true.

For a given critical level tα the probability of making this error is

Prob {t (DN ) > tα | H} =α (5.11.38)

122 CHAPTER 5. PARAMETRIC ESTIMATION

Type II error. This is the kind of error we make when we accept H and His

false. In order to deﬁne this error, we are forced to declare an alternative

hypothesis ¯

Has a formal deﬁnition of what is meant by Hbeing false . The

probability of type II error is

Prob  t ( DN )≤tα | ¯

H (5.11.39)

that is the probability that the test leads to acceptance of Hwhen in fact ¯

holds.

Note that

•when the alternative hypothesis is composite, there could be no unique Type

II error.

•although H and ¯

Hare complementary events, the quantity (5.11.39) cannot

be derived from (5.11.38) (see Equation (3.1.23)).

Example

In order to better illustrate these notions, let us consider the analogy with a murder

trial, where the suspect is Mr. Bean. The null hypothesis His "Mr. Bean is

innocent". The dataset is the amount of evidence collected by the police against

Mr. Bean. The Type I error is the error that we make if, Mr. Bean being innocent,

we send him to death-penalty. The Type II error is the error that we make if,

being Mr. Bean guilty, we acquit him. Note that the two hypotheses have diﬀerent

philosophical status (asymmetry). His a conservative hypothesis, not to be rejected

unless evidence against Mr Bean's innocence is clear. This means that a type I error

is more serious than a type II error (beneﬁt of the doubt ).

•

Example

Let us consider a professor who has to decide, on the basis of empirical evidence,

whether a student copied or not during a class test. The null hypothesis His that

the student is honest. The alternative hypothesis ¯

His that the student cheated.

Let the empirical evidence tN be represented by the number of lines of the classwork

that a student shares with at least one of his classmates.

The decision rule of the professor is the following: a student passes (i.e. the null

hypothesis that she is honest is accepted) if there is not enough empirical evidence

against her (e.g. if tN ≤ tα = 2), otherwise she fails (i.e. the alternative hypothesis

is chosen). Will the professor make any error? why? and does this depend on what?

•

5.11.6 The hypothesis testing procedure

In general terms a hypothesis testing procedure can be decomposed in the following

steps:

1. Declare the null and the alternative hypothesis

2. Choose the numeric value αof the type I error (e.g. the risk I want to run

when I reject the null hypothesis).

3. Deﬁne a test statistic.

4. Determine the critical value tα of the test statistic that leads to a rejection of

Haccording to the Type I error deﬁned in Step 2.

5.11. TESTING HYPOTHESIS 123

5. Among the set of tests of level α, choose the test that minimises the probability

of type II error.

6. Obtain the data and determine whether the observed value of the test statistic

leads to an acceptation or rejection of H.

Note that a number of tests, having a diﬀerent type II error, can guarantee the

same type I error. An appropriate choice of test as a function of the type II error

is therefore required and will be discussed in the following section.

5.11.7 Choice of test

The choice of test and consequently the choice of the partition {S0 , S1 } is based on

two steps

1. Deﬁne a signiﬁcance level α, that is the probability of type I error (or the

probability of incorrectly rejecting H)

Prob { reject H |H } = Prob { DN ∈ S0 | H} = α

2. Among the set of tests {S0 , S1 } of level α, choose the test that minimises the

probability of type II error

Prob  accept H| ¯

H = Prob  DN ∈ S1 | ¯

H

that is the probability of incorrectly accepting H. This is equivalent to max-

imising the power of the test

Prob  reject H| ¯

H = Prob  DN ∈ S0 | ¯

H = 1 − Prob  DN ∈ S1 | ¯

H

which is the probability of correctly rejecting H . Note that for a given signif-

icance level, the higher the power, the better !

Example

In order to reason about the Type II error, let us consider an r.v. z ∼ N (µ, σ2 ),

where σ is known and a set of Niid observations are given. We want to test the

null hypothesis µ = µ0 = 0, with α= 0. 1 Consider three diﬀerent tests and the

associated critical regions S0

1. | ˆ µ− µ0 | > 1 . 645σ/√N

2. ˆ µ− µ0 > 1 . 282σ/√ N (Figure 5.7)

3. | ˆ µ− µ0 | < 0 . 126σ/√ N (Figure 5.8)

Assume that the area blackened in Figure (5.7) equals the area blackened in

Figure (5.8). For all these tests Prob { DN ∈ S0 | H } ≤ α , hence the signiﬁcance

level (i.e. Type I error) is the same. However if ¯

H: µ1 = 10 the type II error of

the three tests is signiﬁcantly diﬀerent. Which test is the best one, that is the one

which guarantees the lowest Type II error?

•

124 CHAPTER 5. PARAMETRIC ESTIMATION

Figure 5.7: On the left: distribution of the test statistic ˆ

µif H : µ0 = 0 is true. On

the right: distribution of the test statistic ˆ

µif ¯

H: µ1 = 10 is true. The interval

marked by S1 denotes the set of observed ˆ µvalues for which His accepted (non-

critical region). The interval marked by S0 denotes the set of observed ˆ µvalues for

which H is rejected (critical region). The area of the black pattern region on the

right equals Prob { DN ∈ S0 | H } , i.e. the probability of rejecting H when H is true

(Type I error). The area of the grey shaded region on the left equals the probability

of accepting H when H is false (Type II error).

Figure 5.8: On the left: distribution of the test statistic ˆ

µif H : µ0 = 0 is true.

On the right: distribution of the test statistic ˆ

µif ¯

H: µ1 = 10 is true. The two

intervals marked by S1 denote the set of observed ˆ µvalues for which His accepted

(non-critical region). The interval marked by S0 denotes the set of observed ˆ µ

values for which His rejected (critical region). The area of the pattern region

equals Prob {DN ∈ S0 |H } , i.e. the probability of rejecting H when H is true (Type

I error). Which area corresponds to the probability of the Type II error?

5.12. PARAMETRIC TESTS 125

5.11.8 UMP level-αtest

Given a signiﬁcance level αwe denote by uniformly most powerful (UMP) test, the

test

1. which satisﬁes

Prob { reject H|H } = Prob {DN ∈ S0 |H } = α

2. for which

Prob  reject H| ¯

H = Prob  DN ∈ S0 | ¯

H

is maximed simultaneously for all θ∈ Θ ¯

How is it possible to ﬁnd UMP tests? In a simple case, an answer is given by

the Neyman-Pearson lemma.

5.11.9 Likelihood ratio test

Consider the simplest case Θ = {θ0 , θ1 } , where H :θ = θ0 and ¯

H: θ= θ1 and

θ0 , θ1 are two diﬀerent values of the parameter of a r.v. z. Let us denote the two

likelihoods by L0 (θ ) and L1 (θ ), respectively.

The idea of Neyman and Pearson was to base the acceptance/rejection of Hon

the relative values L (θ0 ) and L (θ1 ). In other terms we reject Hif the likelihood

ratio L (θ1 )

L( θ0 )

is suﬃciently big.

We reject Honly if the sample data DN are suﬃciently more probable when

θ= θ1 than when θ= θ0 .

Lemma 2 (Neyman-Pearson lemma) . Let H :θ =θ0 and ¯

H: θ= θ1 . If a partition

{S0 , S1 } of the sample space D is deﬁned by

S0 = {DN :L( θ1 ) > kL( θ0 )} S1 ={ DN :L( θ1 ) < kL( θ0 )}

with R S 0 p (DN , θ0 )dDN =α , then {S0 , S1 } is the most powerful level-α test of H

against ¯

This lemma demonstrates that among all tests of level ≤α , the likelihood ratio

test is the optimal procedure, i.e. it has the smallest probability of type II error.

Although, for a generic distribution, the deﬁnition of an optimal test is very

diﬃcult, all the tests that will be described in the following are optimal in the UMP

sense.

5.12 Parametric tests

Suppose we want to test an assertion about a random variable with a known para-

metric distribution F (· , θ ). Besides the distinction between simple and composite

tests presented in Section 5.11.1, there are two more ways of classifying hypothesis

tests:

One-sample vs. two-sample: one-sample tests concern an hypothesis about the

properties of a single r.v. z ∼ N ( µ, σ2 ) while two-sample test concern the

relationship between two r.v. z1 ∼ N (µ1 , σ 2

1) and z 2 ∼ N (µ 2 , σ 2

2).

126 CHAPTER 5. PARAMETRIC ESTIMATION

Single-sided (one-tailed) vs. Two-sided (two-tailed): in single-sided tests the

region of rejection concerns only one tail of the distribution of the null hypoth-

esis. This means that ¯

Hindicates the predicted direction of the diﬀerence.

In two-sided tests the region of rejection concerns both tails of the null distri-

bution. This means that ¯

Hdoes not indicate the predicted direction of the

diﬀerence.

The most common parametric tests rely on hypothesis of normality. A non-

exhaustive list of conventional parametric test is available in the following table:

Name single/two sample known H ¯

z-test single σ2 µ = µ0 µ 6 = µ0

z-test two σ 2

1=σ 2

2µ 1 =µ 2 µ 1 6=µ 2

t-test single µ = µ0 µ 6 = µ0

t-test two µ1 = µ2µ1 6 = µ2

χ2 -test single µ σ2 = σ 2

0σ 2 6=σ 2

χ2 -test single σ2 =σ2

0σ 2 6=σ 2

F-test two σ 2

1=σ 2

2σ 2

16=σ 2

The columns H and ¯

Hcontain the parameter taken into consideration by the

test.

All the parametric test procedures can be decomposed into ﬁve main steps:

1. Deﬁne the null hypothesis and the alternative one.

2. Fix the probability α of having a Type I error.

3. Choose a test statistic t (DN ).

4. Deﬁne the critical value tα that satisﬁes the Type I error constraint.

5. Collect the dataset DN , compute t ( DN ) and decide if the hypothesis is either

accepted or rejected.

Note that the ﬁrst 4 steps are independent of the data and should be carried out

before the collection of the dataset. A more detailed description of some of these

tests is contained in the following sections and Appendix C.3.

5.12.1 z-test (single and one-sided)

Consider a random sample DN ←x ∼ N (µ, σ 2 ) with µ unknown and σ2 known.

Let us see in detail how the ﬁve steps of the testing procedure are instantiated in

this case.

STEP 1:

Consider the null hypothesis and the alternative (composite and one-sided)

H: µ= µ0 ;¯

H: µ > µ0

STEP 2: ﬁx the value αof the type I error.

STEP 3: If H is true then the distribution of ˆ

µis N ( µ0 , σ2 /N ). This means

that the test statistic t (DN ) is

tN = t (DN ) = ( ˆ

µ−µ0 ) √ N

σ∼ N (0,1)

STEP 4: determine the critical value tα .

We reject the hypothesis H if tN > tα = zα where zα is such that Prob {N (0, 1) > zα } =

α.

5.12. PARAMETRIC TESTS 127

Example: for α = 0. 05 we would take zα = 1. 645 since 5% of the standard

normal distribution lies to the right of 1.645. Note that the value zα for a given α

can be obtained by the R command qnorm(alpha,lower.tail=FALSE).

STEP 5: Once the dataset DN is measured, the value of the test statistic is

tN =(ˆ µ− µ0 )√ N

and the hypothesis is either accepted (tN ≤zα ) or rejected.

Example z-test

Consider a r.v. z ∼ N ( µ, 1). We want to test H :µ = 5 against ¯

H: µ > 5 with

signiﬁcance level 0.05. Suppose that the dataset is DN = { 5 . 1,5.5,4.9,5.3} . Then

ˆ µ= 5 .2 and zN = (5. 2− 5) ∗ 2 /1=0 .4. Since this is less than 1.645, we do not

reject the null hypothesis.

•

5.12.2 t-test: single sample and two-sided

Consider a random sample from N ( µ, σ2 ) with σ2 unknown . Let

H: µ= µ0 ;¯

H: µ6= µ0

Let

t(DN ) = tN =√ N (ˆ µ− µ0 )

N−1P N

i=1(z i −ˆ µ)2

=(ˆ µ− µ0 )

qˆ σ2

a statistic computed using the data set DN .

If the hypothesis Hholds, from Sections C.2.3 and 5.7 it follows that t ( DN ) ∼

TN−1 is a r.v. with a Student distribution with N −1 degrees of freedom. The size

αt-test consists in rejecting Hif

|tN | > k = tα/2,N − 1

where t α/2,N−1 is the upper αpoint of a T-distribution on N− 1 degrees of freedom,

i.e.

Prob  tN−1 > tα/2,N−1  = α/ 2, Prob  |tN−1 | > tα/2,N−1  = α.

where tN−1 ∼ TN−1 . In other terms His rejected when tN is too large.

Note that the value t α/2,N−1 for a given N and α can be obtained by the R

command qt(alpha/2,N-1,lower.tail=TRUE).

Example [65]

Suppose we want an answer to the following question: Does jogging lead to a re-

duction in pulse rate?. Let us engage eight non jogging volunteers in a one-month

jogging programme and let us take their pulses before and after the programme

pulse rate before 74 86 98 102 78 84 79 70

pulse rate after 70 85 90 110 71 80 69 74

decrease 4 1 8 -8 7 4 10 -4

Let us assume that the decreases are randomly sampled from N ( µ, σ2 ) where

σ2 is unknown. We want to test H :µ = µ0 = 0 against ¯

H: µ6= 0 with a

signiﬁcance α = 0. 05. We have N = 8, ˆ µ= 2. 75, T= 1 .263, tα/2,N −1 = 2 .365 Since

|T | ≤ t α/2,N −1 , the data is not suﬃcient to reject the hypothesis H . In other terms

the experiment does not provide enough evidence that jogging leads to reduction in

pulse rate.

128 CHAPTER 5. PARAMETRIC ESTIMATION

Figure 5.9: On the left: distribution of the test statistic (number of identical lines)

if H is true, i.e. the student is honest. Typically honest students have very few

lines in common with others though it could happen by chance that such number

is more than 2. On the right: distribution of the test statistic (number of identical

lines) if ¯

His true, i.e. the student is dishonest. Typically dishonest students have

several lines in common with others though some of them are cunning enough to

conceal it.

•

So far we assumed that the distribution of the test statistic is known under the

null hypothesis. In this case it is possible to ﬁx a priori the Type I error. But what

about if we do not know anything about the distribution? Is it possible to assess a

posteriori the quality (in terms of errors of Type I or II) of a certain test (e.g. using

a certain threshold) ?

5.13 A posteriori assessment of a test

Let us consider the professor example (page 122) and the hypothesis test strategy

which leads to the refusal of a student when tN > tα = 2. In this case the distri-

butions of the tN statistic for an honest student (or a dishonest one) has no known

parametric form (Figure 5.9). Moreover, the professor has no information about

such distributions and, consequently, he has no way to measure or control the Type

I error rate (i.e. the grey area in Figure 5.9). Nevertheless, it is possible to estimate

a posteriori the Type I and Type II error rate if we have access to the decisions of

the professor and the real nature of student (honest or dishonest).

Suppose that N students took part in the exam and that NN did not copy while

NP did. According to the professor's decision rule, ˆ

NN were considered honest and

passed the exam, while ˆ

NP were considered dishonest and rejected. Because of

the overlapping of the distributions in Figure 5.9, it happens that FP > 0 honest

students (the ones in the grey area) failed and FN > 0 dishonest students (the ones

in the blue area) passed. Note that the honest students who failed indeed did not

copy but they had by chance more than one line in common with a classmate. At

the same time there are dishonest students who succeeded by copying but who were

clever enough to avoid more than 2 identical lines.

The resulting situation can be summarised in Table 5.2 and Table 5.3 where

we associated the null hypothesis Hto the minus sign (non guilty or honest) and

the hypothesis ¯

Hto the plus sign. In Table 5.2, FP denotes the number of False

Positives, i.e. the number of times that the professor considered the student as guilty

5.14. CONCLUSION 129

Passed Failed

H: Honest student (-) TN FP NN =TN + F P

H: Guilty student (+) FN TPNP = FN + TP

NN = TN + FN ˆ

NP = FP + TPN

Table 5.2: Reality vs. decision: given Nstudents (NN honest and NN dishonest

ones) the table reports the breakdown of the Nprofessor decisions ( ˆ

NN passes and

NP rejections).

H accepted H rejected

H: null hypothesis (-) 1 − α α

H: alternative hypothesis (+) β1− β

Table 5.3: Reality vs. decision: the table reports the probability of correct and bad

decisions in a hypothesis test. In particular α denotes the type I error while 1 − β

the test power.

(+) but in reality she was innocent (-). The ratio FP /N represents an estimate of

the type I error (probability of rejecting the null hypothesis when it is true) which is

denoted by α in Table 5.3. The term FN represents the number of False Negatives,

i.e. the number of times that the professor considered a student to be honest (-),

yet he copied (+). The ratio FN /N is an estimation of the type II error (probability

of accepting the null hypothesis when it is false) which is denoted by βin Table 5.3.

Note that the Type I and II errors are related. For instance, the professor could

decide he does not want to unfairly fail even a single student by setting tN to inﬁnity.

In this case, all honest students, like the dishonest ones, would succeed: this means

we would have a null Type I error (α= 0) at the cost of the highest Type II error

(β≈ NP /N).

5.14 Conclusion

The reader wishing to know more about machine learning could be disappointed.

She has been reading more than one hundred pages and has still the sensation

that she did not learn much about machine learning. All she read seems very far

from intelligent agents, neural networks and fancy applications... Nevertheless, she

already came across the most important notions of machine learning: conditional

probability, estimation and bias/variance trade-oﬀ. Is it all about that? From an

abstract perspective, yes. All the fancy algorithms that will be presented afterwards

(or that the reader is used to hear about) are nothing more (often without the

designer's knowledge) estimators of conditional probability, and as such, submitted

to a bias/variance tradeoﬀ. Such algorithms are accurate and useful only if they

manage well such trade-oﬀ.

But we can go a step further and see the bias/variance tradeoﬀ not only as

a statistical concept but as a metaphor of human attitude towards models and

data, beliefs and experience, ideology and observations, preconceptions and events 6.

Humans deﬁne models (not only in science but also in politics, economics, religion)

to represent the regularity of nature. Now, reality often escapes or diverges from

such regularity. In front of the gap between the Eden of regularity and the natural

Hell of observations, humans waver between two extremes: i) negate or discredit

reality and reduce all divergences to some sort of noise (measurement error) or ii)

6https://tinyurl.com/y25l4xyp

130 CHAPTER 5. PARAMETRIC ESTIMATION

adapt, change their belief, to incorporate discording data and measures in their

model (or preconceptions).

The ﬁrst attitude is exposed to bias (or dogmatism or worse conspiracy think-

ing): the second to variance (or instability). A biased human learner behaves as

an estimator which is insensitive to data: her strength derives from the intrinsic

robustness and coherence, and his weakness is due to the (in)sane attitude of dis-

regarding data and ﬂagrant evidence. On the other side, a highly variant human

learner adapts rapidly and swiftly to data and observations, but he can be easily

criticised for his excessive instability, in simple words for going where the wind

blows.

When the evidence does not conﬁrm your expectations (or what your parents,

teachers or media told you), what is the best attitude to take? Is there an optimal

attitude? Which side are you on?

5.15 Exercises

1. Derive analytically the bias of the sample average estimator in a non i.i.d. setting.

2. Derive analytically the variance of the sample average estimator in an i.i.d. setting.

3. Consider a regression problem where

y= sin( x) + w

and x is uniformly distributed on the interval [0, 2π ] and w =N (1, 1) is a Normal

variable with both mean and variance equal to 1. Let us consider a predictor h ( x)

that is distributed like w. Compute the bias and variance of the predictor in the

following coordinates: x = 0, x =π ,x = π/2.

Solution:

•x= 0 Bias=0, Var=1

•x= πBias=0, Var= 1

•x= π/2 Bias=1, Var=1

4. Let us consider a dataset DN = {z1 ,...,z 20}of 20 observations generated according

to an uniform distribution over the interval [− 1, 1]. Suppose I want to estimate the

expected value of the distribution. Compute the bias and variance of the following

estimators:

•ˆ

θ1 =P 10

i=1 z i

•ˆ

θ2 =ˆ

µ=P 20

i=1 z i

•ˆ

θ3 =−1

•ˆ

θ4 = 1

•ˆ

θ5 =z2

Suppose I want to estimate the variance of the distribution. Compute the bias of

the following estimators:

•ˆ

σ2

1= P (zi − ˆ µ)2

•ˆ

σ2

2= P (zi − ˆ µ)2

•ˆ

σ2

3= 1/3

5.15. EXERCISES 131

Solution: Note that θ = 0 and σ 2

z= 1/3

θ1 :B1 = 0, V1 = 0. 03. Justiﬁcation: E [ ˆ

θ1 ] = θ and Var h ˆ

θ1 i =σ2 / 10

θ2 :B2 = 0, V2 = 0. 015. Justiﬁcation: E [ ˆ

θ2 ] = θ and Var h ˆ

θ2 i =σ2 / 20

θ3 :B3 =− 1, V3 = 0. Justiﬁcation: E [ ˆ

θ3 ] = − 1 and Var h ˆ

θ3 i = 0 since constant

θ4 :B4 = 1, V4 = 0. Justiﬁcation: E [ ˆ

θ4 ] = 1 and Var h ˆ

θ4 i = 0 since constant

θ5 :B5 = 0, V5 = 0. 33. Justiﬁcation: E [ ˆ

θ5 ] = θ and Var h ˆ

θ5 i =σ2

σ2

1:B= 0 .Justiﬁcation: sample variance is unbiased then E[ˆ

σ2

1] = σ 2

σ2

2:B−1 /60 = − 0 .0166 .Justiﬁcation: Note ﬁrst that ˆ

σ2 =19

P(zi − ˆ µ)2

19 . Then

E[ˆ

σ2

2] = 19

20 E P (zi − ˆ µ)2

19  = 19

20 σ 2

then

E[ˆ

σ2

2]−σ 2

z=19

20 σ 2

z−σ 2

z=−σ 2

z/20

σ2

3:B= 0 .Justiﬁcation E[1 /3] = 1/3 = σ 2

5. Let us consider the following observations of the random variable z

DN ={ 0 .1 , −1 , 0 .3 ,1 .4 }

Write the analytical form of the likelihood function of the mean µfor a Gaussian

distribution with a variance σ2 = 1. The student should:

1. Trace the log-likelihood function on the graph paper

2. Determine graphically the maximum likelihood estimator.

3. Discuss the result.

Solution: Since N = 4 and σ= 1

L( µ ) =

i=1

p(zi , µ) =

i=1

√2π exp −(zi − µ ) 2

−2−1 0 1 2

−14 −12 −10 −8−6

log− likelihood

Note that ˆ µml coincides with the sample average ˆ µ= 0 .2 of DN .

6. Suppose you want to estimate the expectation µof the uniform r.v. z ∼ U [ − 2, 3]

by using a dataset of size N= 10. By using R and its random generator, ﬁrst plot

the sampling distribution then estimate the bias and the variance of the following

estimators:

132 CHAPTER 5. PARAMETRIC ESTIMATION

1. ˆ

θ=P N

i=1

2. ˆ

θ= minN

i=1 z i

3. ˆ

θ= maxN

i=1 z i

4. ˆ

θ= z1

5. ˆ

θ= zN

6. ˆ

θ=P i=1 N|zi |

7. ˆ

θ= mediani zi

8. ˆ

θ= maxN

i=1 w iwhere w∼ N (0,1).

9. ˆ

θ= 1

Before each random generation set the seed to zero.

7. The student should ﬁrst create a dataset of N= 1000 observations according to the

dependency

y=g (β0 +β1 x) + w

where x ∼ U [− 1, 1], β0 = 1, β1 =− 1, w ∼ N (µ = 0, σ2 = 0. 1), g (x ) = e x

1+ex .

Then by using the same dataset he should:

•estimate by maximum likelihood the parameters β0 and β1 ,

•plot the contour of the likelihood function, showing in the same graph the

values of the parameters and their estimations.

Hint: use a grid search to perform the maximisation.

8. The student should ﬁrst create a dataset of N= 1000 observations according to the

dependence

Prob {y = 1| x} =g ( β0 +β1 x )

where x ∼ U [0, 1], β0 = 1, β1 = − 1, g (x ) = e x

1+ex and y∈ {0 ,1} .

Then by using the same dataset she should:

•estimate by maximum likelihood the parameters β0 and β1 ,

•plot the contour of the likelihood function, showing in the same graph the

values of the parameters and their estimations.

Hint: use a grid search to perform the maximisation.

9. Let z ∼ N (1, 1), DN a training set of N i.i.d. observations zi and ˆ µN the related

sample average estimator.

1. Compute analytically

Ez,DN [(z− ˆ

µN )2 ]

Hint: consider that z =θ +w where θ =E [z ] and w ∼ N (0,1).

2. Compute analytically

Ez,DN [(z− ˆ

µN )]

3. Validate by Monte Carlo simulation the two theoretical results above.

Solution: Since E [ ˆ

µ] = µ , Var [ˆ

µ] = σ 2

w/N and w is independent of D N:

Ez,DN [(z− ˆ

µN )2 ] = E z,DN [(θ +w− ˆ

µN )2 ] =

=Ez,DN [w2 + 2w (θ− ˆ

µN ) + (θ− ˆ

µN )2 ] =

=Ez [w2 ] + ED N [(θ− ˆ

µN )2 ] = σ 2

w+σ 2

w/N = 1 + 1/N

R code to perform Monte Carlo validation :

5.15. EXERCISES 133

rm(list=ls())

N=5

S=10000

sdw=1 ## noise variance

E=NULL

for (s in 1:S){

DN=rnorm(N,1,sdw)

muhat=mean(DN)

z=rnorm(1,1,sdw)

e=z-muhat

E=c(E,e^2)

}

cat("th=",sdw^2+sdw^2/N, "MC estimation=", mean(E),"\n")

10. Let us supposed that the only measurement of a Gaussian random variable z ∼

N(µ, 1) is the interval [− 3. 5, 1. 5]. Estimate µ by maximum-likelihood and show the

likelihood-function L (µ ). Hint: use the R function pnorm.

11. Let us suppose that 12 of the 31 days of August in Brussels are rainy. Estimate the

probability of a rainy day by maximum likelihood by using the Binomial distribution

(Section C.1.2).

134 CHAPTER 5. PARAMETRIC ESTIMATION

Chapter 6

Nonparametric approaches

to estimation and testing

6.1 Nonparametric methods

In the previous chapter, we considered estimation problems where the probability

distribution is known, parameters' value (e.g. mean and/or variance) aside. Such

estimation methods are called parametric. The meaningfulness of a parametric test

depends entirely on the validity of the assumptions made about the analytical form

of the distribution. However, in real conﬁgurations, it is not uncommon for the

experimenter to question parametric assumptions.

Consider a random sample DN ←z collected through some experimental obser-

vation and for which no hint about the underlying probability distribution Fz ( ·) is

available. Suppose we want to estimate a parameter of interest θof the distribution

of z by using the plug-in estimate ˆ

θ= t(ˆ

F) (Section 5.3). What can we say about

the accuracy of the estimator ˆ

θ? As shown in Section 5.5.3, for some speciﬁc param-

eters (e.g. mean and variance) the accuracy can be estimated independently of the

parametric distribution. In most cases, however, the assessment of the estimator is

not possible unless we know the underlying distribution. What to do, hence, if the

distribution is not available? A solution is provided by the so-called nonparametric

or distribution-free methods that work independently on any speciﬁc assumption

about the probability distribution.

The adoption of these methods enjoyed considerable success in the last decades

thanks to the evolution and parallelisation of computational processing power. In

fact, most techniques for nonparametric estimation and testing are based on re-

sampling procedures, which require a large number of repeated (and almost similar)

computations on the data.

This chapter will deal with two resampling strategies for estimation and two

resampling strategies for hypothesis testing, respectively.

Jacknife: this approach to nonparametric estimation relies on repeated computa-

tions of the statistic of interest for all the combinations of the data where one

or more of the original examples are removed. It will be presented in Section

6.3.

Bootstrap: this approach to nonparametric estimation aims to estimate the sam-

pling distribution of an estimator by sampling (with replacement) from the

original data. It will be introduced in Section 6.4.

Randomisation: This is a resampling without replacement testing procedure. It

135

136 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

consists in taking the original data and either scrambling the order or the

association of the original data. It will be discussed in Section 6.5.

Permutation: This is a resampling two-sample hypothesis-testing procedure based

on repeated permutations of the dataset. It will be presented in Section 6.6.

6.2 Estimation of arbitrary statistics

Consider a set DN of N data points sampled from a scalar r.v. z . Let E [z ] = µthe

parameter to be estimated. In Section 5.3.1 we derived the bias and the variance

of the estimator ˆ

µ:

ˆ µ=1

i=1

zi , Bias[ ˆ

µ] = 0, Var [ ˆ

µ] = σ 2

Consider now another quantity of interest, for example, the median or a mode of the

distribution. While it is easy to design a plug-in estimate of these quantities, their

accuracy is diﬃcult to be computed. In other terms, given an arbitrary estimator

θ, the analytical form of the variance Var h ˆ

θi and the bias Bias[ˆ

θ]is typically not

available.

Example

According to the plug-in principle (Section 5.3) we can design other estimators

besides sampled mean and variance, like:

•Estimation of skewness (3.3.36) of z: see Equation (D.0.2).

•Estimation of correlation (3.6.67) between x and y : : see Equation (D.0.3).

What about the accuracy (e.g. bias, variance) of such estimators?

•

Example

Let us consider an example of estimation taken from an experimental medical

study [65]. The goal of the study is to show bioequivalence between an old and

a new version of a patch designed to infuse a certain hormone in the blood. Eight

subjects take part in the study. Each subject has his hormone levels measured after

wearing three diﬀerent patches: a placebo, an "old" patch and a "new" patch. It is

established by the Food and Drug Administration (FDA) that the new patch will

be approved for sale only if the new patch is bioequivalent to the old one according

to the following criterion:

θ=| E(new) −E (old)|

E(old) − E(placebo) ≤ 0 .2 (6.2.1)

Let us consider the following plug-in estimator (Section 5.3) of (6.2.1)

θ=| ˆ

µnew − ˆ

µold |

µold − ˆ

µplacebo

Suppose we have collected the following data (details in [65])

6.3. JACKKNIFE 137

subj plac old new z=old-plac y=new-old

1 9243 17649 16449 8406 -1200

2 9671 12013 14614 2342 2601

3 11792 19979 17274 8187 -2705

. . . . . . . . . . . . . . . . . .

8 18806 29044 26325 10238 -2719

mean: 6342 -452.3

The estimate is

θ= t(ˆ

F) = | ˆ µnew − ˆ µold |

ˆ µold − ˆ µplacebo

=| ˆ µy |

ˆ µz

=452.3

6342 = 0.07

Can we say on the basis of this value that the new patch satisﬁes the FDA

criterion in (6.2.1)? What about the accuracy, bias or variance of the estimator?

The techniques introduced in the following sections may provide an answer to these

questions.

•

6.3 Jackknife

The jackknife (or leave-one-out ) resampling technique aims at providing a compu-

tational procedure to estimate the variance and the bias of a generic estimator ˆ

θ.

The technique was ﬁrst proposed by Quenouille in 1949 and is based on removing

examples from the available dataset and recalculating the estimator. It is a general-

purpose tool that is easy to implement and able to solve a number of estimation

problems.

6.3.1 Jackknife estimation

In order to show the theoretical foundation of the jackknife, we ﬁrst apply this

technique to the estimator ˆ

µof the mean. Let DN ={ z1 , . . . , zN } be the available

dataset. Let us remove the ith example from DN and let us calculate the leave-

one-out (l-o-o) mean estimate from the N− 1 remaining examples

ˆ µ(i) =1

N−1

j6=i

zj = N ˆ µ− zi

N−1

Observe from above that the following relation holds

zi = Nˆ µ−( N−1)ˆ µ(i) (6.3.2)

that is, we can calculate the i th example zi ,i = 1, . . . , N if we know both ˆ µand

ˆ µ(i) . Suppose now we wish to estimate some parameter θ by using as estimator

some complex statistic of the Ndata points

θ= g( DN ) = g( z1 , z2, . . . , zN )

The jackknife procedure consists in ﬁrst computing

θ(i) = g( z1 , z2, . . . , zi−1 , zi+1 , . . . , zN ) , i = 1 , . . . , N

which is called the i th jackknife replication of ˆ

θ. Then by analogy with the rela-

tion (6.3.2) holding for the mean estimator, we deﬁne the i -th pseudo value by

η(i) = Nˆ

θ−( N−1) ˆ

θ(i) .(6.3.3)

138 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

These pseudo values assume the same role as the zi in calculating the sample aver-

age (5.3.4). Hence the jackknife estimate of θ is given by

θjk =1

i=1

η(i) =1

i=1 N ˆ

θ−( N−1) ˆ

θ(i)  = Nˆ

θ−( N−1) ˆ

θ(·) (6.3.4)

where

θ(·) =P N

i=1 ˆ

θ(i)

The rationale of the jackknife technique is to use the quantity (6.3.4) in order to

estimate the bias of the estimator. Since, according to (5.5.8), θ =E [ ˆ

θ]− Bias[ˆ

θ],

the jackknife approach consists in replacing θ by ˆ

θjk and E[ˆ

θ] by ˆ

θ, thus obtaining

θjk =ˆ

θ−Biasjk [ ˆ

θ].

It follows that the jackknife estimate of the bias of ˆ

θis

Biasjk [ ˆ

θ] = ˆ

θ−ˆ

θjk =ˆ

θ− Nˆ

θ+ ( N−1) ˆ

θ(·) = ( N−1)( ˆ

θ(·) −ˆ

θ) .

Note that in the particular case of a mean estimator (i.e. ˆ

θ=ˆ

µ), we see that we

obtain, as expected, Biasjk [ ˆ

µ] = 0.

A jackknife estimate of the variance of ˆ

θcan be obtained from the sample

variance of the pseudo-values. We deﬁne the jackknife estimate of the variance of

θas

Varjk [ ˆ

θ] = Var h ˆ

θjk i (6.3.5)

Under the hypothesis of i.i.d. η (i)

Var h ˆ

θjk i = Var " P N

i=1 η ( i)

N# =

Var h η(i) i

From (6.3.3) we have

i=1 η ( i)

N= N ˆ

θ−( N−1)

i=1

θ(i)

Since

η(i) = Nˆ

θ−( N−1) ˆ

θ(i) ⇔ η(i) −PN

i=1 η ( i)

N=−(N −1) ˆ

θ(i) −PN

i=1 ˆ

θ(i)

from (6.3.5) and (6.3.4) we obtain

Varjk [ ˆ

θ] = P N

i=1 η ( i)−ˆ

θjk  2

N( N−1) = N−1

i=1  ˆ

θ(i) − ˆ

θ(·)  2 !

Note that in the case of the estimator of the mean (i.e. ˆ

θ=ˆ

µ), since η (i) = zi

and ˆ

θjk = ˆ µ, we ﬁnd again the result (5.5.10)

Varjk [ ˆ

θ] = P N

i=1(z i −ˆ µ) 2

N( N−1) = ˆ σ2

N= Var [ˆ

µ] (6.3.6)

The major motivation for jacknife estimates is that they reduce bias. Also,

it can be shown that under suitable conditions on the type of estimator ˆ

θ, the

quantity (6.3.6) converges in probability to Var h ˆ

θi . However, the jacknife can fail

if the statistic ˆ

θis not smooth (i.e. small changes in data cause small changes in the

statistic). An example of non-smooth statistic for which the jacknife works badly

is the median.

6.4. BOOTSTRAP 139

6.4 Bootstrap

The method of bootstrap was proposed by Efron [62] as a computer-intensive tech-

nique to estimate the accuracy of a generic estimator ˆ

θ. Bootstrap relies on a

data-based simulation method for statistical inference. The term bootstrap derives

from the phrase to pull oneself up by one's bootstrap based on the ﬁctional Adven-

tures of Baron Munchausen. The Baron had fallen to the bottom of a deep lake.

Just when it looked like all was lost, he thought to pick himself up by his own boot-

straps. In general terms, to pull yourself up by your bootstraps means to succeed

in something very diﬃcult without any outside help1.

The idea of statistical bootstrap is very simple, namely that in the absence of

any other information, the sample itself oﬀers the best guide of the sampling dis-

tribution. The method is completely automatic, requires no theoretical calculation,

and is available no matter how mathematically complicated the estimator (5.4.6)

is. By resampling with replacement from DN we can build a set of Bdatasets

D(b) , b = 1, . . . , B . From the empirical distribution of the statistics g ( D(b) ) we can

construct conﬁdence intervals and tests for signiﬁcance.

6.4.1 Bootstrap sampling

Consider a data set DN . A bootstrap data set D(b) , b = 1 , . . . , B is created by

randomly selecting N points from the original set DN with replacement (Figure

6.1).

Since DN itself contains N points, there is nearly always duplication of individual

points in a bootstrap data set. Each point has an equal probability 1/N of being

chosen on each draw. Hence, the probability that a point is chosen exactly ktimes

is given by the binomial distribution (Section C.1.2)

Prob {k } =N !

k!( N− k)!  1

N k  N−1

N N−k

0≤k≤N

Given a set of Ndistinct values, there is a total of  2N−1

Ndistinct bootstrap

datasets. The number is quite large already for N > 10. For example, if N = 3

and DN = {a, b, c} , we have 10 diﬀerent bootstrap sets: { a,b,c} , {a,a,b} , {a,a,c},

{b,b,a}, {b,b,c}, {c,c,a },{c,c,b}, {a,a,a }, {b,b,b }, {c,c,c }.

Under balanced bootstrap sampling, the B bootstrap sets are generated in such a

way that each original data point is present exactly Btimes in the entire collection

of bootstrap samples.

6.4.2 Bootstrap estimate of the variance

Given the estimator (5.4.6), for each bootstrap dataset D(b) ,b = 1, . . . , B , we can

deﬁne a bootstrap replication

θ(b) = g( D(b) ) b= 1 , . . . , B

that is the value of the statistic for the speciﬁc bootstrap sample. The bootstrap

approach computes the variance of the estimator ˆ

θthrough the variance of the set

θ(b) , b= 1 , . . . , B, given by

Varbs [ ˆ

θ] = P B

b=1(ˆ

θ(b) −ˆ

θ(·) )2

(B− 1) where ˆ

θ(·) =PB

b=1 ˆ

θ(b)

B(6.4.7)

1This term has not the same meaning (though the derivation is similar) as the one used in

computer operating systems where bootstrap stands for starting a computer from an hardwired

set of core instructions

140 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

Figure 6.1: Bootstrap replications of a dataset and bootstrap statistic computation

6.4. BOOTSTRAP 141

It can be shown that if ˆ

θ= ˆ µ, then for B → ∞, the bootstrap estimate Varbs [ ˆ

θ]

converges to the variance Var[ ˆ

µ].

6.4.3 Bootstrap estimate of bias

Let ˆ

θbe a plug-in estimator (Equation (5.3.3)) based on the sample DN and

θ(·) =P B

b=1 ˆ

θ(b)

B(6.4.8)

Since Bias[ˆ

θ] = E [ ˆ

θ]−θ , the bootstrap estimate of the bias of the plug-in esti-

mator ˆ

θis obtained by replacing E [ ˆ

θ] with ˆ

θ(·) and θwith ˆ

θ:

Biasbs [ ˆ

θ] = ˆ

θ(·) −ˆ

θ(6.4.9)

Then, since

θ= E[ˆ

θ]− Bias[ˆ

θ]

the bootstrap bias corrected estimate is

θbs =ˆ

θ−Biasbs [ ˆ

θ] = ˆ

θ−(ˆ

θ(·) −ˆ

θ)=2 ˆ

θ−ˆ

θ(·) (6.4.10)

Note that if we want to estimate the bias of a a generic non plug-in estimator

g( DN ), the ˆ

θterm in the right-hand terms of (6.4.9) should anyway refer to the

plug-in estimator t ( ˆ

F) (Equation (5.3.3)).

R script

Run the R ﬁle patch.R for the estimation of bias and variance in the case of the

patch data example.

•

6.4.4 Bootstrap conﬁdence interval

Standard bootstrap conﬁdence limits are based on the assumption that the estima-

tor ˆ

θis normally distributed with mean θ and variance σ2 . Taking the bootstrap

estimate of variance, an approximate 100(1 −α )% conﬁdence interval is given by

θ± zα/2 q Varbs [ ˆ

θ] = ˆ

θ± zα/2 s P B

b=1(ˆ

θ(b) −ˆ

θ(·) )2

(B− 1) (6.4.11)

An improved interval is given by using the bootstrap correction for bias

2ˆ

θ−ˆ

θ(·) ± zα/2 s P B

b=1(ˆ

θ(b) −ˆ

θ(·) )2

(B− 1) (6.4.12)

Another bootstrap approach for constructing a 100(1−α )% conﬁdence interval is

to use the upper and lower α/2 values of the bootstrap distribution. This approach

is referred to as bootstrap percentile conﬁdence interval. If ˆ

θL,α/2 denotes the value

such that only a fraction α/2 of all bootstrap estimates are inferior to it, and

likewise ˆ

θH,α/2 is the value exceeded by only α/2 of all bootstrap estimates, then

the conﬁdence interval is given by

[ˆ

θL,α/2 ,ˆ

θH,α/2 ] (6.4.13)

where the two extremes are also called the Efron's percentile conﬁdence limits.

142 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

6.4.5 The bootstrap principle

Given an unknown parameter θof a distribution Fz and an estimator ˆ

θ, the goal of

any estimation procedure is to derive or approximate the distribution of ˆ

θ−θ. For

example, the calculation of the variance of ˆ

θrequires the knowledge of Fz and the

computation of ED N [(ˆ

θ−E[ˆ

θ])2 ]. Now, in practical contexts, Fz is unknown, and

the calculus of ED N [(ˆ

θ−E[ˆ

θ])2 ] is not possible in an analytical way. The rationale of

the bootstrap approach is (i) to replace Fz by the empirical counterpart (5.2.2) and

(ii) to compute ED N [(ˆ

θ−E[ˆ

θ])2 ] by a Monte Carlo simulation approach (Section 3.9)

where several samples of size Nare generated by resampling DN .

The outcome of a bootstrap technique is a Monte Carlo approximation of the

distribution ˆ

θ(b) −ˆ

θ. In other terms the variability of ˆ

θ(b) (based on the empirical

distribution) around ˆ

θis expected to be similar (or mimic) the variability of ˆ

(based on the true distribution) around θ.

The bootstrap principle relies on the two following properties (i) as Ngets

larger and larger, the empirical distribution ˆ

Fz (· ) converges (almost surely) to Fz (·)

(Glivenko-Cantelli theorem (C.9.17)) and (ii) as Bgets larger, the quantity (6.4.7)

converges (in probability) to the variance of the estimator ˆ

θbased on the empirical

distribution (as stated in (C.8.14)). In other terms

Varbs [ ˆ

θ]B→∞

→Ed

DN [(ˆ

θ−E[ˆ

θ])2 ]N→∞

→ED N [(ˆ

θ−E[ˆ

θ])2 ] (6.4.14)

where E d

DN [(ˆ

θ−E[ˆ

θ])2 ] stands for the plug-in estimate of the variance of ˆ

θbased

on the empirical distribution.

In practice, for a small ﬁnite N, bootstrap estimation inevitably returns some

error. This error is a combination of a statistical error and a simulation error.

The statistical error component is due to the diﬀerence between the underlying

distribution Fz ( · ) and the empirical distribution ˆ

Fz (· ). The magnitude of this error

depends on the choice of the estimator ˆ

θ(DN ) and decreases by increasing the

number N of observations.

The simulation error component is due to the use of empirical (Monte Carlo)

properties of ˆ

θ(DN ) rather than exact properties. Simulation error decreases by

increasing the number Bof bootstrap replications.

Unlike the jackknife method, in the bootstrap, the number of replicates Bcan

be adjusted to the computer resources. In practice, two rules of thumb are typically

used:

1. Even a small number of bootstrap replications, e.g. B= 25, is usually infor-

mative. B = 50 is often enough to give a good estimate of Var h ˆ

θi .

2. Very seldom are more than B= 200 replications needed for estimating Var h ˆ

θi .

Much bigger values of B are required for bootstrap conﬁdence intervals.

Note that the use of rough statistics ˆ

θ(e.g. unsmooth or unstable) can make the

resampling approach behave wildly. Examples of nonsmooth statistics are sample

quantiles and the median.

In general terms, for i.i.d. observations, the following conditions are required

for the convergence of the bootstrap estimate

1. the convergence of ˆ

Fto F(satisﬁed by the Glivenko-Cantelli theorem) for

N→ ∞;

2. an estimator such that the estimate ˆ

θis the corresponding functional of the

empirical distribution.

θ= t( F)→ ˆ

θ= t(ˆ

6.5. RANDOMISATION TESTS 143

This is satisﬁed for sample means, standard deviations, variances, medians

and other sample quantiles.

3. a smoothness condition on the functional. This is not true for extreme order

statistics such as the minimum and the maximum values.

But what happens when the dataset DN is not i.i.d. sampled from a distribution

F? In such non conventional conﬁgurations, the most basic version of bootstrap

might fail. Examples are incomplete data (survival data, missing data), dependent

data (e.g. variance of a correlated time series) and dirty data (outliers) conﬁgura-

tions. In these cases, speciﬁc adaptations of the bootstrap procedure are required.

For reason of space, we will not discuss them here. However, for a more exhaustive

discussion on the limits of bootstrap, we invite the reader to refer to [123].

6.5 Randomisation tests

Randomisation tests were introduced by R.A. Fisher in 1935. The goal of a ran-

domisation test is to help to discover some regularity (e.g. a non random property

or pattern) in a complicated data set. A classic example is to take a pack of poker

play-cards and check whether they were well shuﬄed by our poker opponent. Ac-

cording to the hypothesis testing terminology, randomisation tests make the null

hypothesis of randomness and test this hypothesis against data. In order to test

the randomness hypothesis, several random transformations of data are generated.

Suppose we are interested in some property which is related to the order of data.

Let the original data set DN ={x1 , . . . , xN } and t (DN ) some statistic which is a

function of the order in the data DN . We want to test if the value of t ( DN ) is due

only to randomness.

•An empirical distribution is generated by scrambling (or shuﬄing )R times

the N elements at random. For example the j th, j = 1, . . . , R scrambled data

set could be D (j)

N={x 23 , x 4 , x 343 , . . . }

•For each of the j th scrambled sets we compute a statistic t (i) . The resulting

distribution is called the resampling distribution.

•Suppose that the value of t ( DN ) is only exceeded by k of the R values of the

resampling distribution.

•The probability of observing t (DN ) under the null hypothesis (i.e. random-

ness) is only pt = k/R . The null hypothesis can be accepted/rejected on the

basis of pt .

The quantity pt plays the role of nonparametric p-value (Section 5.11.3) and it can

be used, like its parametric counterpart, both to assess the evidence of the null

hypothesis and to perform a decision test (e.g. refuse to play if we think cards were

not suﬃciently shuﬄed).

A bioinformatics example

Suppose we have a DNA sequence and we think that the number of repeated se-

quences (e.g. AGTAGTAGT) in the sample is greater than expected by chance. Let

t= 17 be the number of repetitions. How to test this hypothesis? Let us formulate

the null hypothesis that the base order is random. We can construct an empirical

distribution under the null hypothesis by taking the original sample and randomly

scrambling the bases R= 1000 times. This creates a sample with the same base fre-

quencies as the original sample but where the order of bases is assigned at random.

144 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

Suppose that only 5 of the 1000 randomised samples has a number of repetition

higher or equal than 17. The p-value (i.e. the probability of seeing t = 17 under

the null hypothesis) which is returned by the randomisation test amounts to 0.005.

You can run the randomisation test by using the R script ﬁle randomiz.R.

•

6.5.1 Randomisation and bootstrap

Both bootstrap and randomisation rely on resampling. But what are their peculiar-

ities? A randomised sample is generated by scrambling the existing data (sampling

without replacement) while a bootstrap sample is generated by sampling with re-

placement from the original sample. Also, randomisation tests are appropriate when

the order or association between parts of data are assumed to convey important in-

formation. They test the null hypothesis that the order or the association is random.

On the other side, bootstrap sampling aims to characterise the statistical distribu-

tion of some statistics t ( DN ) where the order makes no diﬀerence in the statistics

(e.g. mean). Randomisation would be useless in that case since t (D (1)

N) = t( D (2)

if D (1)

Nand D (2)

Nare obtained by resampling D N without replacement.

6.6 Permutation test

Permutation test is used to perform a nonparametric two-sample test. Consider a

random sample {z1 , . . . , zM } drawn from an unknown distribution z∼ Fz (· ) and a

random sample {y1 , . . . , yN } from an unknown distribution y∼Fy (· ). For example,

in a bioinformatics task the two datasets could be expression measures of a gene

under M normal and N pathological conditions. Let the null hypothesis be that the

two distributions are the same regardless of the analytical forms of the distributions.

Consider a (order-independent) test statistic for the observed data and call

it t (DN , DM ). The rationale of the permutation test is to locate the statistic

t( DN , DM ) with respect to the distribution which could be obtained if the null

hypothesis were true. In order to build the null hypothesis distribution, all the pos-

sible R = M+N

Mpartitionings of the N+ M observations in two subsets of size N

and M are considered. If the null hypothesis were true, all the partitionings would

be equally likely. Then for each i-th permutation (i = 1, . . . , R ) the permutation

test computes the t (i) statistic. Eventually, the value t ( DN , DM ) is compared with

the set of values t(i) . If the the value t (DN , DM ) falls in the α/2 tails of the t (i)

distribution, the null hypothesis is rejected with type I error α.

The permutation procedure will involve substantial computation unless Mand

Nare small. When the number of permutations is too large a random sample of a

large number R of permutations can be taken.

Note that when observations are drawn according to a normal distribution, it

can be shown that the use of a permutation test gives results close to those obtained

using the ttest.

Example

Let us consider D4 = [74, 86,98,102, 89] and D3 = [10, 25, 80]. We run a permu-

tation test (R = 8

4= 70 permutations) to test the hypothesis that the two sets

belong to the same distribution (R script s perm.R).

Let t (DN ) = ˆ µ(D4 )− ˆ µ( D3 ) = 51 . 46. Figure 6.2 shows the position of t (DN )

with respect to the null sampling distribution.

•

6.7. CONSIDERATIONS ON NONPARAMETRIC TESTS 145

Figure 6.2: Null distribution returned by the permutation test and position (vertical

red line) of the observed statistic

6.7 Considerations on nonparametric tests

Nonparametric tests are a worthy alternative to parametric approaches when no

assumptions about the probability distribution may be made (e.g. in bioinformat-

ics). It is risky, however, to consider them as a panacea, and a critical attitude

towards them has to be preferred. In short terms, here you ﬁnd some of the ma-

jor advantages and disadvantages concerning the use of a nonparametric approach.

Advantages:

•If the sample size is very small, there may be no alternative to using a nonpara-

metric test unless the nature of the population distribution is known exactly.

•Nonparametric tests make fewer assumptions about the data.

•Nonparametric tests are available to analyse data that are inherently in ranks

(e.g. taste of food), classiﬁcatory or categorical.

•Nonparametric tests are typically more intuitive and easier to implement.

Disadvantages:

•They involve high computational costs.

•The large availability of statistical software makes possible the potential mis-

use of statistical measures.

•A nonparametric test is less powerful than a parametric one when the as-

sumptions of the parametric test are met.

•Assumptions are associated with most nonparametric statistical tests, namely,

that the observations are independent.

146 CHAPTER 6. NONPARAMETRIC ESTIMATION AND TESTING

6.8 Exercises

1. Suppose you want to estimate the skewness γof the uniform r.v. z ∼ U [ − 2, 3] by

using a dataset of size N = 10. By using R and its random generator, ﬁrst plot

the sampling distribution then estimate the bias and the variance of the following

estimators:

1. ˆ γ=1

NP i(z i −ˆ µ)3

ˆ σ3

2. ˆ γ=1

NP i|z i −ˆ µ| 3

ˆ σ3

3. ˆ γ= 1

Before each random generation set the seed to zero. Hint: the skewness of a uniform

continuous variable is equal to 0.

2. Suppose you want to estimate the skewness γof the uniform r.v. z ∼ U [− 2, 3] by

using a dataset of size N= 10. By using R and its random generator, ﬁrst generate

a dataset DN with N = 10. By using the jacknife, plot the sampling distribution,

then estimate the bias and the variance of the following estimators,

1. ˆ γ=1

NP i(z i −ˆ µ)3

ˆ σ3

2. ˆ γ=1

NP i|z i −ˆ µ| 3

ˆ σ3

3. ˆ γ= 1

Compare the results with the ones of the exercise before. Before each random

generation set the seed to zero.

3. Suppose you want to estimate the skewness γof the uniform r.v. z ∼ U [− 2, 3] by

using a dataset of size N= 10. By using R and its random generator, ﬁrst generate

a dataset DN with N = 10. By using the bootstrap method, plot the sampling

distribution, then estimate the bias and the variance of the following estimators,

1. ˆ γ=1

NP i(z i −ˆ µ)3

ˆ σ3

2. ˆ γ=1

NP i|z i −ˆ µ| 3

ˆ σ3

3. ˆ γ= 1

Compare the results with the ones of the two exercises before. Before each random

generation set the seed to zero.

4. Let us consider a r.v. z such that E [ z ] = µ and Var[z ] = σ2 . Suppose we want to

estimate from i.i.d. dataset DN the parameter θ =µ2 = (E [z ])2 . Let us consider

three estimators:

θ1 = PN

i=1 z i

N! 2

θ2 = P N

i=1 z 2

θ3 =( P N

i=1 z i ) 2

•Are they unbiased?

•Compute analytically the bias of the three estimators. Hint: use (3.3.30).

•By using R, verify the result above by Monte Carlo simulation using diﬀerent

values of N.

•By using R, estimate the bias of the three estimators by bootstrap.

Solution: See the ﬁle Exercise1.pdf in the directory gbcode/exercises of the

companion R package (Appendix F).

Chapter 7

A statistical framework of

supervised learning

7.1 Introduction

A supervised learning problem can be described in statistical terms by the following

elements:

1. A vector of nrandom input variables x ∈ X ⊂ Rn , whose values are

i.i.distributed according to an unknown probabilistic distribution Fx (·).

2. A target operator which transforms the input values into outputs y ∈ Y

according to an unknown conditional probability distribution Fy (y|x =x ).

3. A collection DN of N input/output data points hxi , yi i ,i = 1, . . . , N , called

the training set and drawn according to the joint input/output density F x,y ( x, y).

4. A learning machine or learning algorithm which, on the basis of the training

set DN , returns an estimation (or prediction) of the target for an input x . The

input/output function estimated by the learning machine is called hypothesis

or model.

Note that in this deﬁnition we encounter most of the notions presented in the

previous chapters: probability distribution, conditional distribution, estimation.

Examples

Several practical problems can be seen as instances of a supervised learning problem:

•Predict whether a patient, hospitalised due to a heart attack, will have a sec-

ond heart attack, on the basis of demographic, diet and clinical measurements.

•Predict the price of a stock in 6 months from now, on the basis of company

performance measures and economic data.

•Identify the risk factors for breast cancer, based on clinical, demographic and

genetic variables.

•Classify the category of a text email (spam or not) on the basis of its text

content.

•Characterise the mechanical property of a steel plate on the basis of its phys-

ical and chemical composition.

147

148 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.1: The supervised learning setting. The target operator returns an output

for each input according to a ﬁxed but unknown probabilistic law. The hypothesis

predicts the value of the target output when entered with the same input.

In the case of the spam categorisation problem, the input vector may be a vector of

size n where n is the number of the most used English words and the ith component

of x represents the frequency of the ith word in the email text. The output yis

a binary class which takes two values: { SPAM,NO.SPAM} . The training set is a

set of emails previously labeled by the user as SPAM and NO.SPAM. The goal of

the learning machine is to create a classiﬁcation function which, once a vector xof

word frequencies is presented, should be able to classify correctly the nature of the

email.

•

A learning machine is nothing more than a particular instance of an estima-

tor (5.4.7) whose goal is to estimate the parameters of the joint distribution F x,y ( y, x)

(or sometimes of the conditional distribution Fy (y|x =x )) on the basis of a training

set DN , i.e. a set of i.i.d. realisations of the pair x and y . The goal of a learn-

ing machine is to return a hypothesis with low prediction error, i.e. a hypothesis

which computes an accurate estimate of the output of the target when the same

test value is an input to the target and the predictor (Fig. 7.1). The prediction

error is also usually called generalisation error, since it measures the capacity of

the learned hypothesis to generalise to previously unseen test samples. A learning

algorithm generalises well if it returns an accurate prediction for i.i.d. test data, i.e.

input/output pairs which are independent from the training set yet are generated by

the same joint distribution Fx,y ( x, y) . We insist on the importance of the two "i"

in the i.i.d. assumption: test data are supposed i) to be generated by the same

distribution underlying the training set but ii) to be independent from the training

set.

We will only consider hypotheses in the form h ( ·, α ) where α∈ Λ∗ is a vector of

model parameters1or weights. Therefore, henceforth, we will denote an hypothesis

h(· , α) by the corresponding vector α∈ Λ∗ . As we will see later, examples of

hypothesis are linear models h ( x, α ) = xT α (Section 9.1) where αrepresents the

coeﬃcients of the model, or feed-forward neural networks (Section 10.1.1) where α

is the set of values taken by the weights of the neural architecture.

1It is important to remark that by model parameter we refer here to a tunable/trainable weight

of the hypothesis function and not to the target of the estimation procedure as in Section 5.1.1

7.1. INTRODUCTION 149

Let αN be the hypothesis returned by the learning machine on the basis of

the training set, and deﬁne GN its generalisation error. The goal of the learning

machine is then to seek the hypothesis αN which minimises the value GN .

In these terms, the learning problem could appear as a simple problem of op-

timisation which consists of searching the hypothesis αwhich yields the lowest

generalisation error. Unfortunately the reality is not that simple, since the learning

machine cannot measure directly GN but only return an estimate of this quantity,

denoted by ˆ

GN . Moreover, what makes the problem still more complex is that the

same ﬁnite training set is employed both to select αN and to estimate GN , thus

inducing a strong correlation between these two quantities.

The common supervised learning practice to minimise the quantity GN consists

1. decomposing the set of hypothesis Λ∗into a nested sequence of hypothesis

classes (or model structures ) Λ1 ⊂ Λ2 ⊂ ··· ⊂ ΛS of increasing capacity (or

expressiveness) s with Λ∗ = ∪S

s=1Λ s

2. implementing a search procedure at two nested levels [125] (Fig. 7.2). The

inner level, also known as parametric identiﬁcation, considers a single class of

hypotheses Λsand uses a method or algorithm to select a hypothesis h (·, αs

from this class. The algorithm typically implements a procedure of multivari-

ate optimisation in the space of model parameters of the class Λs, which can

be solved by (conventional) optimisation techniques. Examples of paramet-

ric identiﬁcation procedures which will be presented in subsequent chapters

are linear least-squares for linear models or back-propagated gradient-descent

for feedforward neural networks [165]. The outer level, also called structural

identiﬁcation, ranges over nested classes of hypotheses Λs , (s = 1, . . . , S ), and

executes for each of them the parametric routine returning the vector α s

N. The

outcome of the parametric identiﬁcation is used to assess the class Λsthrough

avalidation procedure which returns the estimate ˆ

Non the basis of the ﬁ-

nite training set. It is common to use nonparametric techniques to assess the

quality of a predictor like the bootstrap (Section 6.4) or cross-validation [176]

based on the jacknife strategy (Section 6.3).

3. selecting the best hypothesis in the set {α s

N}, with s = 1, . . . , S , according

to the assessments n ˆ

Noproduced by the validation step. This ﬁnal step,

which returns the model to be used for prediction, is usually referred to as the

model selection procedure. Instances of model selection include the problem

of choosing the degree of a polynomial model or the problem of determining

the best number of hidden nodes in a neural network [25].

The outline of the chapter is as follows. Section 7.2 introduces the supervised

learning problem in statistical terms. We will show that classiﬁcation (Section 7.3)

and regression (Section 7.4) can be easily cast in this framework. Section 7.5 intro-

duces the statistical assessment of a learning machine while Section 7.6 reports some

results from the work of Prof. Vapnik on statistical learning and in particular the

formalisation of the notion of capacity of a learning machine. Section 7.7 discusses

the notion of generalisation error and its bias/variance decomposition. Section 7.9

introduces the supervised learning procedure and its decomposition in structural

and parametric identiﬁcation. Model validation and in particular cross validation,

a technique for estimating the generalisation error on the basis of a ﬁnite number

of data, are introduced in Section 7.10.

150 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.2: The learning problem and its decomposition in parametric and struc-

tural identiﬁcation. The larger is the class of hypothesis Λs, the large is its expressive

power in terms of functional relationships.

7.2 Estimating dependencies

This section details the main actors of the supervised learning problem:

•A data generator of random input vectors x ∈ X ⊂ Rn independently and

identically distributed (i.i.d) according to some unknown (but ﬁxed) probabil-

ity distribution Fx (x ). The variable x is called the independent variable. It is

helpful to distinguish between cases in which the experimenter has a complete

control over the values of xand those cases in which she does not. When the

nature of inputs is completely random, we consider xas a realisation of the

random variable x having probability law Fx ( · ). When the experimenter's

control is complete, we can regard Fx (· ) as describing the relative frequencies

with which diﬀerent values for xare set.

•Atarget operator, which transforms the input xinto the output value y ∈ Y

according to some unknown (but ﬁxed) conditional distribution

Fy ( y|x= x) (7.2.1)

(this includes the simplest case where the target implements some determin-

istic function y =f (x )). The conditional distribution (7.2.1) formalizes the

stochastic dependency between inputs and output.

•Atraining set DN = {hx1 , y1 i,hx2 , y2 i ,..., hxN , yN i} made of N pairs (or

training examples) hxi , yi i∈Z = X × Y independent and identically dis-

tributed (i.i.d) according to the joint distribution

Fz ( z) = Fx,y (h x, yi) (7.2.2)

Note that, as in Section 5.4, the observed training set DN ∈ ZN = (X × Y ) N

is considered here as the realisation of a random variable DN .

•Alearning machine having three components:

1. A class of hypothesis functions h (·, α ) with α∈ Λ. We consider only the

case where the functions h (· , α ) ∈ Y are single valued mappings.

7.2. ESTIMATING DEPENDENCIES 151

2. A loss function L (·, · ) associated with a particular y and a particular h (x),

whose value L (y, h(x )) measures the discrepancy between the output y

and the prediction h (x ). For a given hypothesis h ( ·, α ), the functional

risk is the loss average over the XY -domain

R( α) = Exy [L ] =

ZX,Y

L( y, h(x, α)) dFx,y ( x, y) = Z X,Y

L( y, h( x, α)) p (x, y) dxdy (7.2.3)

Note that L is random since x and y are random test points (i.i.drawn

from the same distribution (7.2.2) of the training set) while the hypoth-

esis h (·, α ) is given. This is the expected loss if we test the hypothesis

h( · , α) over an inﬁnite amount of i.i.d. input/output pairs generated

by (7.2.2). For the class Λ of hypothesis we deﬁne

α0 = arg min

α∈ΛR(α ) (7.2.4)

as the hypothesis in the class Λ which has the lowest functional risk.

Here, we assume for simplicity that there exists a minimum value of

R( α) achievable by a function in the class Λ. We deﬁne with R (α0 ) the

functional risk of the class Λof hypotheses.

3. If instead of a single class of hypothesis we consider the set Λ∗ containing

all possible single valued mappings h : X → Y , we may deﬁne the

quantity

α∗ = arg min

α∈ Λ∗ R(α ) (7.2.5)

and

R∗ =R( α∗ ) (7.2.6)

as the absolute minimum rate of functional risk. Note that this quan-

tity is ideal since it requires the complete knowledge of the distribution

underlying the data. In a classiﬁcation setting, the optimal model with

parameters α∗ is called the Bayes classiﬁer and R (α∗ ) the Bayes error

(Section 7.3.1). In a regression setting (Section 7.4) where y =f ( x ) + w

and the loss function is quadratic, h (·, α∗ ) = f (· ) and R (α∗ ) amounts to

the variance of w.

4. An algorithm L of parametric identiﬁcation which takes as input the

training set DN and returns as output one hypothesis function h ( ·, αN )

with αN ∈ Λ. Here, we will consider only the case of deterministic and

symmetric algorithms. This means respectively that they always give

the same h (· , αN ) for the same data set DN and that they are insensitive

to the ordering of the examples in DN .

The parametric identiﬁcation of the hypothesis is done according to ERM

(Empirical Risk Minimisation) inductive principle [186] where

αN = α (DN ) = arg min

α∈Λ R emp (α ) (7.2.7)

minimizes the empirical risk (also know as training error or apparent

error)

Remp ( α ) = 1

i=1

L( yi , h( xi , α)) (7.2.8)

constructed on the basis of the data set DN .

152 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

This formulation of a supervised learning problem is quite general, given that it

includes two basic statistical problems:

1. the problem of classiﬁcation (also known as pattern recognition),

2. the problem of regression estimation.

These two problems and their link with supervised learning will be discussed in the

following sections.

7.3 Dependency and classiﬁcation

Classiﬁcation is one of the most common problem in statistics. It consists in ex-

ploring the association between categorical dependent variables and independent

variables which can take either continuous or discrete values. The problem of clas-

siﬁcation is formulated as follows: consider an input/output stochastic dependence

which can be described by a joint distribution F x,y ( · ), such that once an input

vector x is given, y ∈ Y = {c1 , . . . , cK } takes a value among Kdiﬀerent classes. In

the example of spam email classiﬁcation, K = 2 and c1 =SPAM, c2 =NO.SPAM.

We assume that the dependence is described by a conditional discrete probability

distribution Prob {y = ck |x =x} that satisﬁes

k=1

Prob {y = ck | x} = 1

This means that observations are noisy and follow a probability distribution. In

other terms, given an input x ,y does not always take the same value. Pretending

to have a zero-error classiﬁcation in this setting is then completely unrealistic.

Example

Consider a stochastic dependence where xrepresents a year's month and yis a

categorical variable representing the weather situation in Brussels. Suppose that y

may take only the two values { RAIN,NO.RAIN} . The setting is stochastic since

you might have rainy August and some rare sunny December days. Suppose that

the conditional probability distribution of y is represented in Figure 7.3. This ﬁgure

plots Prob {y = RAIN|x = month} and Prob {y = NO.RAIN|x = month} for each

month. Note that for each month the probability constraint is respected:

Prob {y = RAIN|x = month} + Prob {y = NO.RAIN|x = month}= 1

•

A classiﬁer is a particular instance of estimator which for a given x is expected

to return an estimate ˆ y= ˆ c= h( x, α) which takes a value in { c1 , . . . , cK }. Once a

cost function is deﬁned, the problem of classiﬁcation can be expressed in terms of

the formalism introduced in the previous section. An example of cost function is

the indicator function (taking only two values: zero and one)

L( c, ˆ c) = ( 0 if c= ˆ c

1 if c 6 = ˆ c(7.3.9)

also called the 0/1 loss . However, we can imagine situations where some misclassi-

ﬁcations are worse than others. In this case, it is better to introduce a loss matrix

L(K×K) where the element L(jk) =L(cj ,ck ) denotes the cost of the misclassiﬁcation

7.3. DEPENDENCY AND CLASSIFICATION 153

Figure 7.3: Conditional distribution Prob {y |x } where x is the current month and

yis the random weather state. For example the column corresponding to x=Dec

and y =RAIN returns the conditional probability of RAIN in December.

when the predicted class is ˆ c( x ) = cj and the correct class is ck . This matrix must

be null on the diagonal and non negative everywhere else. In practical cases the

deﬁnition of a loss matrix could be quite challenging since it should take into ac-

count and combine several criteria, some easy to quantify (e.g. ﬁnancial costs) and

some much less (e.g. ethical considerations)2. Note that in the case of the 0-1 loss

function (Equation 7.3.9) all the elements outside the diagonal are equal to one.

The goal of the classiﬁcation procedure for a given xis to ﬁnd the predictor

ˆ c( x) = h (x, α ) that minimises the quantity

k=1

L(ˆ c( x),ck ) Prob {y= c k|x}(7.3.10)

which is an average of the ˆ c( x) row of the loss matrix weighted by the conditional

probabilities of observing y = ck . Note that the average of the above quantity over

the Xdomain

k=1

L(ˆ c( x),ck ) Prob {y= c k|x} dF x=Z X ,Y

L( y, h( x, α)) dFx,y = R ( α ) (7.3.11)

corresponds to the functional risk (7.2.3).

The problem of classiﬁcation can then be seen as a particular instance of the

more general supervised learning problem described in Section 7.2.

2By default, any automatic classiﬁer (and the associated decision maker) implicitly or explicitly

embeds a loss function weighting often highly heterogeneous criteria. For instance, the Tesla

automatic braking systems (implicitly or explicitly) assigns a cost to false positives (e.g. a bag

wrongly identiﬁed as a pedestrian) and false negatives (e.g. a pedestrian mistaken for a bag).

154 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

7.3.1 The Bayes classiﬁer

It can be shown that the optimal classiﬁer h (·, α0 ) where α0 is deﬁned as in (7.2.4)

is the one that returns for all x

c∗ ( x) = h( x, α0 ) = arg min

cj ∈{c1,...,cK }

k=1

L(j,k) Prob {y = ck | x} (7.3.12)

The optimal classiﬁer is also known as the Bayes classiﬁer . In the case of a 0-1 loss

function the optimal classiﬁer returns

c∗ ( x) = arg min

cj ∈{c1 ,...,cK } X

k=1: K,k6= j

Prob {y =ck | x} (7.3.13)

= arg min

cj ∈{c1 ,...,cK }(1 − Prob {y = c j |x}) (7.3.14)

= arg min

cj ∈{c1,...,cK } Prob {y6=c j |x}= arg max

cj ∈{c1,...,cK } Prob {y= c j |x}

(7.3.15)

The Bayes decision rule selects the j ,j = 1, . . . , K , that maximizes the posterior

probability Prob {y =cj |x}.

Example

Consider a classiﬁcation task where X ={ 1,2,3,4, 5 } ,Y = {c1 , c2, c3 } and the loss

matrix and the conditional probability values are given in the following ﬁgures.

Let us focus on the optimal classiﬁcation for x= 2. According to (7.3.12) the Bayes

classiﬁcation rule for x = 2 returns

c∗ (2) = arg min

k=1 ,2,3{L 11 Prob {y=c 1 |x= 2} + L 12 Prob {y=c 2 |x= 2} +L 13 Prob {y= c 3 |x= 2} ,

L21 Prob {y = c1 |x = 2} + L22 Prob {y = c2 |x = 2 }+ L23 Prob {y = c3 |x = 2 } ,

L31 Prob {y = c1 |x = 2 }+ L32 Prob {y = c2 |x = 2} + L33 Prob {y = c3 |x = 2}}

= arg min

k=1 ,2 ,3{0∗ 0. 2+1 ∗ 0. 8+5 ∗ 0.0, 20 ∗ 0. 2+0∗ 0. 8 + 10 ∗ 0.0,

2∗ 0. 2+1 ∗ 0. 8+0 .0∗ 0} = arg min

k=1 ,2 ,3{1, 4,1.2} = 1

What would have been the Bayes classiﬁcation in the 0-1 case?

•

7.3. DEPENDENCY AND CLASSIFICATION 155

Figure 7.4: Class conditional distributions: the green class is distributed as a mix-

ture of two gaussians while the red class as a gaussian.

7.3.2 Inverse conditional distribution

An important quantity, often used in classiﬁcation algorithms, is the inverse condi-

tional distribution. According to the Bayes theorem (3.1.20) we have that

Prob {y = ck |x =x} = Prob {x= x |y=ck } Prob {y =ck }

k=1 Prob {x=x |y= c k }Prob {y =c k }(7.3.16)

and that

Prob {x = x|y =ck } = Prob {y=ck |x=x } Prob {x=x }

Px Prob {y=ck |x=x } Prob {x =x} . (7.3.17)

The above relation means that by knowing the a-posteriori conditional distribution

Prob {y = ck |x =x} and the a-priori distribution Prob {x =x} , we can derive the

inverse conditional distribution Prob {x =x |y =ck } . This distribution is replaced

by a density if x is continuous and is also known as the class conditional density.

This distribution characterises the values of the inputs xfor a given class ck .

Shiny dashboard

The Shiny dashboard classif2.R illustrates a binary classiﬁcation task where

x∈R2 and the two classes are green and red. The green and the class condi-

tional distributions (7.3.17) are a mixture of two gaussians (Section 3.7.2) and a

unimodal gaussian, respectively (Figure 7.4). Figure 7.5 illustrates the associated

conditional distribution (7.3.16) if the two classes have an equal a-priori proba-

bility (Prob {y = red} = Prob {y = green} ). Figure 7.6 shows the scattering of a

set of N = 500 points sampled according to the class-conditional distributions in

Figure 7.4.

•

156 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.5: The a-posteriori conditional distribution associated to the class-

conditional distributions (equal a-priori probability) in Figure 7.4.

Figure 7.6: Dataset sampled according to the class-conditional distributions (equal

a-priori probability) in Figure 7.4.

7.4. DEPENDENCY AND REGRESSION 157

Figure 7.7: Inverse conditional distribution of the distribution in Figure 7.3

Example

Suppose we want to know during which months it is most probable to have rain.

This boils down to have the distribution of x for y = RAIN . Figure 7.7 plots the in-

verse conditional distributions Prob {x = month|y = RAIN}and

Prob {x = month|y = NO.RAIN} according to (7.3.17) when we assume that the a

priori distribution is uniform (i.e. Prob {x =x} = 1/ 12 for all x).

Note that

month

Prob {x = month|y = NO.RAIN}=

month

Prob {x = month|y = RAIN}= 1

•

7.4 Dependency and regression

Consider the stochastic relationship between two continuous random variables x ∈

Rn and y∈ Rdescribed by

Fx,y ( x, y) (7.4.18)

This means that to each vector xsampled according to the Fx (x ) there corresponds

a scalar ysampled from Fy (y |x = x ). Assume that a set of Ninput/output obser-

vations is available. The estimation of the stochastic dependence on the basis of the

empirical dataset requires the estimation of the conditional distribution Fy (y |x ).

This is known to be a diﬃcult problem but for prediction purposes, most of the

time, it is suﬃcient to estimate the conditional expectation

f( x) = Ey [y | x] = Z Y

ydFy ( y| x ) (7.4.19)

also known as the regression function.

158 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

The regression function is also related to the functional risk

R( α) = Z L( y, h(x, α)) dFx,y ( x, y) = Z ( y− h( x, α ))2 dFx,y ( x, y) (7.4.20)

for the quadratic loss L ( y, h )=( y− h)2 . From (3.6.63) it can be shown that the

minimum (7.2.4) is attained by the regression function h ( ·, α0 ) = f ( · ) if the function

fbelongs to the set h( x, α), α∈ Λ.

Once deﬁned the regression function f , the input/output stochastic depen-

dency (7.4.18) is commonly represented in the regression plus noise form

y=f (x ) + w=Ey [ y|x ] + w(7.4.21)

where w denotes the noise term and satisﬁes E [w ] = 0 and E [w2 ] = σ 2

w. The role of

the noise is to make explicit that some variability of the target cannot be explained

by the regression function f. Notice that the assumption of an additive noise w

independent of x is common in statistical literature and is not overly restrictive. In

fact, many other conceivable signal/noise models can be transformed into this form.

The problem of estimating the regression function (7.4.19) is then a particular

instance of the supervised learning problem described in Section 7.2, where the

learning machine is assessed by a quadratic cost function. Examples of learning

algorithms for regression will be discussed in Section 9.1 and Section 10.1.

7.5 Assessment of a learning machine

A learning machine works well if it exhibits good generalisation, i.e. if it is able

to perform good predictions for unseen input values, which are not part of the

training set but that are generated by the same input/output distribution (7.2.2)

underlying the training set. This ability is commonly assessed by the amount of

bad predictions, measured by the generalisation error. The generalisation error of

a learning machine can be evaluated at two levels:

Hypothesis: Let αN be the hypothesis returned by a learning algorithm for a

training set DN according to the ERM principle (Eq. (7.2.7)). The functional

risk R (αN ) in (7.2.3) represents the generalisation error of the hypothesis αN .

This quantity is also known as conditional error rate [98] since it is conditional

on a given training set DN .

Algorithm: Let us deﬁne the average of the loss Lfor a given input xover the

ensemble of training sets of size Nas

gN ( x ) = ED N ,y [ L| x= x ] = Z Z N ,Y

L( y, h(x, αN )) dFy ( y| x ) dF N

z(D N )

(7.5.22)

where F N

z(D N ) is the distribution of the i.i.d. dataset D N . In this expression

Lis a function of the random variables DN (through h ) and y, while the

test input xis ﬁxed. In the case of a quadratic loss function, this quantity

corresponds to the mean squared error (MSE) deﬁned in Section 5.5.6. By

averaging the quantity (7.5.22) over the X domain we have

GN = Z X

gN ( x) dFx ( x ) = ED N E x,y [L(y , h(x ,αN ))] (7.5.23)

that is the generalisation error of the algorithm L (also known as expected

error rate [66] or expected test error [98]).

7.5. ASSESSMENT OF A LEARNING MACHINE 159

STOCHASTIC

PROCESS

LEARNING MACHINE

αN

R( )

αN

R( )

G =E[R( )] αNN

Figure 7.8: Functional risk vs. MISE

From (7.2.3) and (7.5.23) we obtain that

GN =ED N [ R(αN )]

where R (αN ) is random because of the dependence on DN (Figure 7.8).

In the case of a quadratic loss function, the quantity

MISE = ED NEx,y [(y− h(x,αN ))2 ] (7.5.24)

takes the name of mean integrated squared error (MISE).

The two criteria correspond to two diﬀerent ways of assessing the learning machine:

the ﬁrst is a measure to assess the speciﬁc hypothesis (7.2.7) chosen by ERM, the

second assesses the average performance of the algorithm over training sets with N

observations. According to the hypothesis-based approach the goal of learning is to

ﬁnd, on the basis of observations, the hypothesis that minimises the functional risk.

According to the algorithmic-based approach the goal is to ﬁnd, on the basis of

observations, the algorithm which minimises the generalisation error. The two cri-

teria will be detailed in Section 7.6 and 7.7, respectively. Note that both quantities

requires the knowledge of Fx,y which is unfortunately unknown in real situations. A

key issue in machine learning is then to take advantage of observable quantities, i.e.

quantities that may be computed on the basis of the observed dataset, to estimate

or approximate the measures discussed above. An important quantity in this sense

is the empirical risk (7.2.8) which has however to be carefully considered in order

to avoid too optimistic evaluations of the learning machine accuracy.

7.5.1 An illustrative example

The notation introduced in Section 7.2 and 7.5 is rigorous but it may appear hostile

to the practitioner. In order to make the statistical concepts more aﬀordable we

160 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.9: Training set (dots) obtained by sampling uniformly in the interval [− 2,2]

an input/output distribution with regression function f (x ) = x3 and unit variance.

present a simple example to illustrate these concepts. We consider a supervised

learning regression problem where :

•The input is a scalar random variable x ∈Rwith a uniform probability

distribution over the interval [− 2,2].

•The target is distributed according to a conditional Gaussian distribution

py ( y |x= x) = N ( x3 , 1) (7.5.25)

where the conditional expected value E [y |x ] is the regression function f (x ) =

x3 and the noise w has a unit variance.

•The training set DN = {hxi , yi i}, i = 1, . . . , N consists of N = 100 i.i.d. pairs

(Figure 7.9) generated according to the distribution 7.5.25. Note that this

training set can be easily generated with the following R commands

## script regr.R

N<-100

X<-runif(N,-2,2)

Y=X^3+rnorm(N)

plot(X,Y)

•The learning machine is characterised by the following three components:

1. A class of hypothesis functions h (x, α ) = αx consisting of all the linear

models passing through the origin. The class Λ is then the set of real

numbers.

7.5. ASSESSMENT OF A LEARNING MACHINE 161

Figure 7.10: The empirical risk for the training set DN vs. the model parameter

value (x-axis). The minimum of the empirical risk is attained in α = 2.3272.

2. A quadratic loss L (y, h(x )) = (y− h (x))2.

3. An algorithm of parametric identiﬁcation based on the least-squares tech-

nique, which will be detailed later in Section 9.1.2. The empirical risk is

the quantity

Remp ( α ) = 1

100

i=1

(yi −αxi )2(7.5.26)

The empirical risk is a function of αand the training set. For the given

training set DN , the empirical risk as a function of αis plotted in Fig.

7.10.

For the dataset DN in Figure 7.9, it is possible to obtain αN by minimising the

empirical risk (7.5.26)

αN = arg min

α∈ΛR emp (α) = arg min

α∈ Λ

100

i=1

(yi −αxi )2 = 2. 3272 (7.5.27)

The selected hypothesis is plotted in the input/output domain in Fig. 7.11.

If the joint distribution (e.g. its conditional expectation and variance) were to

be known, it would also be possible to compute the risk functional (7.2.3) as

R( α) = 1

4Z 2

−2

(x3 − αx )2 dx + 1 = 4α 2

3− 32

5α + 71/7 (7.5.28)

where the derivation of the equality is sketched in Appendix C.13. For the given

joint distribution, the quantity R (α ) is plotted as a function of α in Fig. 7.12. The

function takes a global minimum in α0 = 2. 4 as can be derived from the analytical

expression in (7.5.28).

The computation of the quantity (7.5.22) requires however an average over all

the possible realisations of the random variable αN for datasets of N = 100 points.

Figure 7.13 shows 6 diﬀerent realisations of the training set for the same conditional

distribution (7.5.25) and the corresponding 6 values of αN . Note that those six

values may be considered as 6 diﬀerent realisations of the sampling distribution

(Section 5.4) of αN .

It is important to remark that both the quantities (7.2.3) and (7.5.22) may be

computed only if we know a priori the data joint distribution. Unfortunately, in

real cases this knowledge is not accessible and the goal of learning theory is to study

the problem of estimating these quantities from a ﬁnite set of data.

162 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.11: Training set (dotted points) and the linear hypothesis function h (·, αN )

(straight line). The quantity αN , which represents the slope of the straight line, is

the value of the model parameter αwhich minimizes the empirical risk.

Figure 7.12: The functional risk (7.5.28) vs. the value of model parameter α(x-

axis). The minimum of the functional risk is attained in α0 = 2.4.

7.5. ASSESSMENT OF A LEARNING MACHINE 163

Figure 7.13: Six diﬀerent realisations of a training set with N = 100 points (dots)

and the relative hypotheses (solid straight lines) chosen according to the ERM

principle (7.5.27).

164 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Monte Carlo computation of generalisation error

The script functRisk.R computes by Monte Carlo the functional risk (7.5.28) for

diﬀerent values of αand returns the value of α0 = 2. 4 which minimises it. Note

that the functional risk is computed by generating a very large number of i.i.d. test

examples.

The script gener.R computes by Monte Carlo the generalisation error (7.5.23).

Unlike the previous script which considers only the predictive value of diﬀerent hy-

pothesis (with diﬀerent α ), this script assesses the average accuracy of the empirical

risk minimisation strategy (7.5.27) for a ﬁnite number N= 100 of examples.

•

7.6 Functional and empirical risk

This section reports some results from the pioneering work of Prof. Vladimir Vap-

nik [188, 186, 187] on statistical learning. He deﬁnes the learning problem as the

problem of ﬁnding the hypothesis which minimises the functional risk (7.2.3) on

the basis of a ﬁnite set of observed data and without any speciﬁc assumption about

the data distribution. For details and mathematical derivations, we refer the reader

to his books [186, 187]. Here we will limit to report some of his most signiﬁcant

results. We start by rewriting the functional risk notation (7.2.3) as

R( α) = Z L( y, h(x, α) dFhx, yi ( x, y) = Z Q( z , α) dFz ( z) α∈ Λ (7.6.29)

where z = hx, yi , Q (z, α ) = L ( y, h ( x, α )), the probability measure Fz (· ) is unknown

but an i.i.d. sample z1 , . . . , zN is given. Analogously, the empirical risk may be

rewritten as

Remp ( αN ) = 1

i=1

Q(zi , αN )

Let us deﬁne with Λ∗the set of all possible single valued mappings f : X → Y and

consider the quantity

α∗ = arg min

α∈ Λ∗ R(α )

where R (α∗ ) is the absolute minimum rate of functional risk (7.2.6).

We can write the equality

R( αN )− R( α∗ )=( R( αN )− R (α0 )) + ( R (α0 )− R ( α∗ )) =

= Errestim (αN ) + Errapprox (αN )

where α0 is the hypothesis with lowest risk in Λ (Equation (7.2.4)).

The ﬁrst right-hand term is the estimation error while the second is the ap-

proximation error (Figure 7.14). The estimation error represents the discrepancy

between the generalisation error of the best hypothesis in the class (R ( α0 )) and the

one learned from DN (R ( αN )). The approximation error is non null when the best

hypothesis in the class Λ (h ( ·, α0 )) is diﬀerent from h (·, α∗ ).

The trade-oﬀ between approximation and estimation error is controlled by the

size of Λ: when the size of Λ is large, R (α0 ) is close to R (α∗ ) but the estimation

error could be large. On the other way round, if the size of Λ is small, the estimation

error is limited but the approximation error could be non negligible.

7.6. FUNCTIONAL AND EMPIRICAL RISK 165

Err

estim

αN

α*

α0

Err approx

Figure 7.14: Decomposition of the functional risk into estimation and approximation

error.

7.6.1 Consistency of the ERM principle

Functional and empirical risk are two key quantities in statistical learning (Fig-

ure 7.15). The functional risk represents the generalisation accuracy of the hypoth-

esis once tested with new data while R emp ( ·) measures the accuracy of the ﬁtting

to the training set. A main issue is that Remp (· ) could be a very bad estimator of

the functional risk, e.g. when the class of hypothesis is too rich with respect to the

size of the observed sample.

According to Vapnik it is important to characterise the relation between those

two quantities, i.e. to deﬁne the (necessary and suﬃcient) conditions for the em-

pirical risk R emp (αN ) to converge for N → ∞ to the best functional risk R (α0 )

in the class Λ. This is known as the problem of consistency of the Empirical Risk

Minimisation (ERM) principle.

In formal terms, the ERM principle is consistent for the set of functions Q ( z, α)

and for the probability distribution Pz (z ) if the following two sequences converge

in probability to the same limit

R( αN ) P

−−−−→

N→∞ R(α 0 )

Remp (αN ) P

−−−−→

N→∞ R(α 0 )

The following lemma shows that both convergences may be studied by consid-

ering the quantity supα∈Λ |R emp (α )−R (α ) |.

Lemma 3 (Devroye 1988).

R( αN )− inf

α∈ΛR(α ) = R ( α N )−R( α 0 )≤2 sup

α∈Λ|R emp (α)− R( α)|

|Remp (αN ) −R( αN ) | ≤ sup

α∈Λ |R emp (α)− R (α)|

Setting an upper bound for supα∈Λ |R emp (α )−R (α )| , we obtain an upper bound

for three quantities:

166 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

α0 αN

Remp

(

(α )

Figure 7.15: Functional and empirical risk.

1. the estimation error R (αN )−R (α0 ) which returns the sub-optimality of the

the model chosen by the ERM principle within the class α∈ Λ

2. |R emp (αN )−R (αN )| that is the error committed when the empirical risk is

used to estimate the functional risk of the selected model

3. |R emp (αN )−R (α0 )| that is the error made when the empirical risk is used to

estimate the functional risk of the best model in the class Λ.

It can be shown that bounding supα∈Λ | Remp (α )−R (α )| is not only a suﬃcient but

also a necessary condition for consistency of the ERM principle.

7.6.2 Key theorem of learning

Theorem 6.1 (Vapnik,Chervonenkis, 1991). Let Q ( z, α )α∈ Λ be a set of functions

that satisfy the condition

a≤Z Q( z, α) dP ( z)≤ b

Condition necessary and suﬃcient for the ERM principle to be consistent is that

the empirical risk Remp ( α) converges uniformly to the actual risk R (α ) over the set

Q( z, α), α∈Λ that is

lim

N→∞ Prob  sup

α∈Λ

(R(α )−Remp (α )) > ε = 0 ∀ε > 0

This theorem rephrases the problem of ERM consistency as a problem of uniform

convergence, which ensures that the empirical risk is a good approximation of the

functional risk over all functions of Λ (i.e. including the worst-case).

The uniform convergence is trivial and guaranteed by the Law of Large Numbers

if the set of functions Q (z, α ) contains a single element: in fact, this is nothing more

than the convergence of the average to expectation for increasing N. For a real-

valued bounded function a≤ Q (z, α )≤ b , by Hoeﬀding's inequalities (Section 5.6)

we have

Prob (  ZQ( z, α) dP ( z)− 1

i=1

Q( zi , α)   > ε) < exp  − 2 ε 2 N

(b− a )2 

7.6. FUNCTIONAL AND EMPIRICAL RISK 167

Then the probability of a deviation between empirical and functional risk converges

to zero for N → ∞. It is easy to generalise to the case where Q (z , α ) has a ﬁnite

number K of elements:

Prob (sup

1≤k ≤K   ZQ(z, α ) dP (z )− 1

i=1

Q(zi , α)  > ε)<

Kexp  − 2 ε 2 N

(b− a )2  = exp  ln K

N− 2 ε 2

(b− a )2  N 

In order to obtain uniform convergence for any ε, the expression

lim

N→∞

ln K

N= 0 (7.6.30)

has to be satisﬁed. A problem arises when the set of functions is inﬁnite, like in

machine learning where the most common classes of hypothesis are uncountable.

In this case we need to generalise the classical law of large numbers to functional

spaces. Consider the sequence of random variables

ξN = sup

α∈ Λ

(R (α )−R emp (α )) = sup

α∈Λ Z Q( z, α) dF ( z)− 1

i=1

Q(zi , α)!

where the set of functions Q (z, α ), α∈ Λ, has an inﬁnite number of elements. Unlike

the ﬁnite case, the sequence ξN does not necessarily converge to zero. The problem

of learning is then strongly related to the problem of deﬁning which properties of

the class of functions Q (z, α ), α∈ Λ, guarantee the convergence in probability of the

sequence ξN to zero. In the following section we show some theoretical results from

Vapnik about the relation between ERM consistency and the topological properties

(notably the diversity ) of the class of hypothesis.

7.6.2.1 Entropy of a set of functions

In what follows we limit to consider the binary classiﬁcation setting though similar

results can be shown for regression. In this setting the functions Q ( z, α ), α∈ Λ are

indicator functions since they may take only 0 or 1 values. In order to characterise

the diversity of the set of functions Q (z, α ), α∈ Λ, on the dataset DN , let NΛ (DN )

be the number of possible separations of DN using the functions Q (z, α ), α∈ Λ.

Note that NΛ (DN ) is a random variable since DN is a random variable.

An example of this concept is presented in Figure 7.163 where N = 3 and the

functions h (· ) implement linear separators of the 2D (n= 2) input space. This class

of functions is able to perform all possible (i.e. 2N= 8) separations of the dataset.

It is also said that the class Λ of functions shatters the dataset of size N = 3. In

other words, a set of Npoints is said to be shattered by a class of hypothesis Λ if,

no matter how we assign a binary label to each point, there exists a hypothesis in

Λ that separates them. Note that a set of N= 4 points is not shattered by a class

of linear separators.

The quantity

HΛ ( N) = Eln NΛ ( DN )

is called the entropy of the set of functions on the given data and measures the

diversity of the class of hypothesis for a given number of observations.

The following theorem from Vapnik shows that this quantity is related to the

consistency of the ERM principle.

3Taken from https://datascience.stackexchange.com/questions/16140/

how-to- calculate-vc- dimension/16146

168 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.16: Number of linear separations of a dataset of N = 3 points.

Theorem 6.2. A necessary and suﬃcient condition for the two-sided uniform con-

vergence of the functional risk to the empirical risk is that

lim

N→∞

HΛ ( N)

N= 0 (7.6.31)

In other words, the ratio of the entropy to the number of observations should

decrease to zero with increasing number of observations. Note that this condition

depends on the underlying probability distribution Fz (· ) and that the entropy plays,

for uncountable classes, the role played by the number of functions in the ﬁnite case

(compare (7.6.30) with (7.6.31)).

7.6.2.2 Distribution independent consistency

Vapnik has been also able to extend the distribution-dependent result of the previous

Section to a distribution-free setting.

Theorem 6.3. Necessary and suﬃcient condition for consistency of ERM for any

probability measure is

lim

N→∞

GΛ ( N )

N= 0

where

GΛ ( N ) = ln max

DN N Λ (D N )

is the growth function.

Vapnik proved that in the pattern recognition case

Prob sup

α∈ Λ

(R(α )−Remp (α )) > ε ≤ 4 exp  G Λ (2N)

N− ε 2  N  (7.6.32)

7.6. FUNCTIONAL AND EMPIRICAL RISK 169

This means that, provided that GΛ (N ) does not grow linearly in N , it is actu-

ally possible to bound the (unknown) functional risk R (αN ) on the basis of the

(observable) empirical risk Remp (αN ).

If we set the probability in (7.6.32) to δ > 0 and we solve for ε, then the following

inequality holds with probability 1 −δ :

R( αN )≤ R emp (αN ) + √ E

2(7.6.33)

where the right-hand side is called the guaranteed risk and

E= 4 G Λ (2N)− ln(δ/4)

Several other bounds have been derived for diﬀerent class of hypothesis in [186].

7.6.3 The VC dimension

Vapnik and Chervonenkis showed that either the relation GΛ (N ) = N ln 2 holds

true for all N, or there exists some maximal Nfor which this relation is satisﬁed.

In this case, this maximal Nis called the VC (Vapnik and Chervonenkis) dimension

and denoted by D. By construction, the VC dimension is the maximal number of

points which can be shattered by functions in Λ.

Theorem 6.4. Any growth function either satisﬁes the equality

GΛ ( N) = Nln 2

or is bounded by the inequality

GΛ ( N) ≤ D  ln N

D+ 1

where D is an integer such that when N= D

GΛ (D ) = D ln 2 , GΛ (D + 1) < (D + 1) ln 2

The VC dimension of a set of indicator functions Q (z, α ) is inﬁnite if the growth

function is linear. It is ﬁnite and equal to Dif the growth function is bounded by

a logarithmic function with coeﬃcient D.

The VC dimension quantiﬁes the richness or capacity of a set of functions. If for

any N an hypothesis function h ( ·, α ), α∈ Λ can shatter Npoints (i.e. separate them

in all 2Npossible ways) then GΛ (N ) = N ln 2. In this case, the class of function has

an inﬁnite capacity and there is no ERM convergence (the empirical risk is always

zero whatever is the functional risk): no learning from data is possible4.

The ﬁniteness of the the VC dimension is a necessary and suﬃcient condition for

distribution independent consistency of ERM learning machines. The VC dimension

of the set of linear functions with n + 1 model parameters is equal to D =n + 1.

Note that, though for this speciﬁc class the VC dimension equals the number of free

parameters, this is not necessarily true for other family of functions. For instance,

it can be shown that the VC dimension of the highly wiggly set of functions

h( x, α) = sin αx, α ∈ R

4Note that, in Popper terminology (Section 2.6) this corresponds to a non scientiﬁc situation

where no dataset may falsify the hypothesis, or equivalently it is always possible to ﬁnd a hypothesis

justifying what we observe. Since the class of hypothesis is too rich, no falsiﬁcation (and then no

generalisation or scientiﬁc discovery) is possible.

170 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

is inﬁnite though it has a single parameter. At the same time, you can have set of

functions with inﬁnite number of parameters yet a ﬁnite VC dimension.

Generally speaking, the VC dimension of a set of functions can be either larger

than or smaller than the number of parameters. The VC dimension of the set of

functions (rather than the number of parameters) is responsible for the generalisa-

tion ability of learning machines.

Once deﬁned D, the relation between empirical and functional risk of a class of

function with ﬁnite VC dimension is now made explicit by the bound (7.6.33) where

the second summand is

E( D, N, δ )=4 D ln 2N

D+ 1 − ln(δ/4)

The reliability of the empirical risk as approximation of the functional risk depends

on the ratio N/D . If N/D is large (i.e. sample size much larger than the VC

dimension), the E term is small and the empirical risk is a good approximation

of the functional risk. In other terms, minimising the empirical risk guarantees a

small value of the (expected) risk. On the contrary, if N/D is small (i.e. number of

samples comparable to the VC dimension), a small empirical risk Remp (αN ) does

not guarantee a small value of the actual risk. In other terms a small empirical

risk could be an optimistic (then biased) estimator of the associated functional risk.

In those conﬁgurations, to minimise the actual risk R (αN ) it is recommended to

address both terms of the conﬁdence interval (e.g. by considering alternative classes

of hypothesis).

7.7 Generalisation error

In the previous section we presented how Vapnik [185, 186, 187] formalised the

learning task as the minimisation of functional risk R ( αN ) in a situation where the

joint distribution is unknown. This section focuses on the algorithm-based criterion

GN (Equation (7.5.24)) as a measure of the generalisation error of the learning

machine.

In particular we will study how the generalisation error can be decomposed in

the regression formulation and in the classiﬁcation formulation.

7.7.1 The decomposition of the generalisation error in re-

gression

Let us focus now on of the gN measure (Equation (7.5.22)) of the generalisation

error in the case of regression. In the case of a quadratic loss

L( y ( x) , h(x, α)) = ( y ( x)− h( x, α))2(7.7.34)

the quantity gN is often referred to as the mean squared error (MSE) and its

marginal (7.5.24) as the mean integrated squared error (MISE). If the regression

dependency is described in the regression plus noise form (7.4.21), the conditional

target density can be written as

py ( y− f( x )| x ) = py ( y− Ey [y | x ]| x ) = pw ( w ) (7.7.35)

where w is a noisy random variable with zero mean and variance σ 2

This supervised learning problem can be seen as a particular instance of the

estimation problem discussed in Chapter 5, where, for a given x , the unknown

parameter θ to be estimated is the quantity f (x ) and the estimator based on the

7.7. GENERALISATION ERROR 171

training set is ˆ

θ=h (x, αN ). The MSE quantity, deﬁned in (5.5.14) coincides, apart

from an additional term, with the term (7.5.22) since

gN ( x ) = ED N ,y [ L| x ] = (7.7.36)

=ED N ,y  (y− h(x, αN ))2  = (7.7.37)

=ED N ,y  (y− Ey [y |x ] + Ey [y |x ]− h (x, αN ))2  = (7.7.38)

=ED N ,y  (y−Ey [y |x ])2 + 2w (Ey [y|x ]− h(x, αN ))+ (7.7.39)

+ (Ey [y |x ]− h(x, αN ))2  = (7.7.40)

=Ey  (y−Ey [y|x ])2  +ED N (h (x, αN )−Ey [y|x ])2  = (7.7.41)

=Ey  w2  +ED N  (h (x, αN )−Ey [y |x ])2 (7.7.42)

=σ2

w+E D N[(f( x)− h (x, α N )) 2 ] = σ 2

w+E D N[(θ− ˆ

θ)2 ] = (7.7.43)

=σ2

w+ MSE (7.7.44)

Note that y =f (x ) + w = Ey [y |x ] + w ,f is ﬁxed but unknown and that the noise

term w is independent of DN and satisﬁes E [w ] = 0 and E [w2 ] = σ 2

We can then apply bias/variance decomposition (5.5.14) to the regression prob-

lem where θ =f (x ) and ˆ

θ=h (x, αN ):

gN ( x ) = ED N ,y [L( x, y)] =

=σ2

w+E D N(h( x, α N )−E y [y|x ]) 2 =

=σ2

w+noise variance

+ (ED N [h ( x, αN )] −Ey [y |x ])2 + squared bias

+ED N  (h(x, αN )−ED N [h(x, αN )])2  = model variance

=σ2

w+B 2 (x) + V( x)

(7.7.45)

In a regression task, the bias B (x ) measures the diﬀerence in xbetween the aver-

age of the outputs of the hypothesis functions over the set of possible DN and the

regression function value f (x ) = Ey [y |x ]. The variance V ( x ) reﬂects the variabil-

ity of the guessed h ( x, αN ) as one varies over training sets of ﬁxed dimension N.

This quantity measures how sensitive the algorithm is to changes in the data set,

regardless of the target. So by Eq. (7.5.24) by averaging (7.7.45) over Xwe obtain

MISE = GN =σ 2

w+Z X

B2 ( x) dFx + Z X

V( x) dFx (7.7.46)

where the three terms are

1. the intrinsic noise term reﬂecting the target alone,

2. the integrated squared bias reﬂecting the target's relation with the learning

algorithm and

3. the integrated variance term reﬂecting the learning algorithm alone.

As the aim of a learning machine is to minimise the quantity GN and the com-

putation of (7.7.46) requires the knowledge of the joint input/output distribution,

this decomposition could appear as a useless theoretical exercise. In practical set-

tings, the designer of a learning machine does not have access to the term GN but

can only estimate it on the basis of the training set. Nevertheless, the bias/variance

decomposition is relevant in practical learning too since it provides a useful hint

about how to control the error GN . In particular, the bias term measures the lack

of representational power of the class of hypotheses. This means that to reduce the

172 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Capacity

Underﬁtting

Overﬁtting

Noise variance

Model

variance

Squared bias

MISE

Figure 7.17: Bias/variance/noise tradeoﬀ in regression: this is a qualitative rep-

resentation of the relationship between the hypothesis' bias and variance and the

capacity of the class of functions. The MISE generalisation error is the sum of

the three terms (squared bias, hypothesis variance and noise variance) as shown

in (7.7.46). Note that the variance of the noise is supposed to be target indepen-

dent and then constant.

bias term of the generalisation error we should consider classes of hypotheses with

a large capacity s, or in other words hypotheses which can approximate a large

number of input/output mappings. On the other side, the variance term warns us

against an excessive capacity (or complexity) s of the approximator. This means

that a class of too powerful hypotheses runs the risk of being excessively sensitive to

the noise aﬀecting the training set; therefore, our class Λscould contain the target

but it could be practically impossible to ﬁnd it out on the basis of the available

dataset.

In other terms, it is commonly said that an hypothesis with large bias but low

variance underﬁts the data while an hypothesis with low bias but large variance

overﬁts the data. In both cases, the hypothesis gives a poor representation of the

target and a reasonable trade-oﬀ needs to be found.

A graphical illustration of the bias/variance/noise tradeoﬀ (7.7.46) is made in

Figure 7.17. The left side of the ﬁgure corresponds to an underﬁtting conﬁguration

where the model has too low capacity (i.e. high bias) to capture the nonlinearity

of the regression function. The right side of the ﬁgure corresponds to an overﬁtting

conﬁguration where the model capacity is too large (i.e. high variance) leading then

to high instability and poor generalisation. Note that Figure 7.17 requires a formal

deﬁnition of the notion of capacity and that it is only a qualitative visualisation of

the theoretical link between the hypothesis' properties and the capacity of the class

of functions. Nevertheless it provides useful hints about the impact of the learning

procedure on the ﬁnal generalisation accuracy. The task of the model designer is to

search for the optimal trade-oﬀ between the variance and the bias terms (ideally the

capacity s∗ in Figure 7.17), on the basis of the available training set. Section 7.9

will discuss how this search proceeds in practice in a real setting.

7.7. GENERALISATION ERROR 173

Two naive predictors

Consider a regression task y =f (x ) + w , where Var [w ] = σ 2

wand two naive

predictors:

1. h(1) (x )=0

2. h(2) (x ) = P N

i=1 y i

What about their generalisation errors in x = ¯ x? By using (7.7.45) we obtain

1. g (1)

N(¯ x) = σ2

w+f(¯ x)2

2. g (2)

N(¯ x) = σ2

w+ (f (¯ x)− E[y])2 + Var [y ] /N

The script naive.R executes a Monte Carlo validation of the formulas above.

•

7.7.2 The decomposition of the generalisation error in clas-

siﬁcation

Let us consider a classiﬁcation task with Koutput classes and a loss function L.

For a given input x, we denote by ˆy the class predicted by the classiﬁer h ( x, αN )

trained with a dataset DN . We derive the analytical expression of gN (x ), usually

referred to as the mean misclassiﬁcation error (MME).

MME(x ) = Ey,DN [ L (y , h(x, αN ))|x ] = Ey,DN [L (y, ˆy)] = (7.7.47)

=Ey,DN [

k,j=1

L(j,k) 1(ˆy =cj | x )1( y= ck | x )] = (7.7.48)

k,j=1

L(j,k) ED N [1(ˆy = cj | x )] Ey [1( y=ck |x )]] = (7.7.49)

k,j=1

L(j,k) Prob { ˆy = cj | x} Prob { y=ck |x } (7.7.50)

where 1(·) is the indicator function which returns zero when the argument is false

and one otherwise. Note that the distribution of ˆy depends on the training set DN

while the distribution of yis the distribution of a test set (independent of DN ).

For zero-one loss function, since y and ˆy are independent, the MME expression

simpliﬁes to

MME(x ) =

k,j=1

1(cj 6 =ck )Prob { ˆy =cj |x } Prob { y= ck | x} =

= 1 −

k,j=1

1(cj =ck )Prob { ˆy =cj |x } Prob { y= ck | x} =

= 1 −

Prob { ˆy =ck |x } Prob { y= ck | x} = Prob { y 6= ˆy} (7.7.51)

174 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

A decomposition of a related quantity was proposed in [198]. Let us consider

the squared sum:

j=1

(Prob {y = cj } − Prob { ˆy =cj })2=

2



j=1

Prob {y =cj }2 

+1

2



j=1

Prob { ˆy = cj }2 

−

j=1

Prob {y = cj } Prob { ˆy =cj }

By adding one to both members and by using (7.7.47) we obtain a decomposition

analogous to the one in (7.7.45)

gN ( x ) = MME( x ) =

2

1− 



j=1

Prob {y = cj | x}2 



+"noise"

j=1

(Prob {y = cj | x } − Prob { ˆy = cj | x })2 + "squared bias"

2

1− 



j=1

Prob { ˆy = cj |x}2 



"variance"

(7.7.52)

The noise term measures the degree of uncertainty of yand consequently the degree

of stochasticity of the dependance. It equals zero if and only if there exists a class

csuch that Prob {y = c| x}= 1 and zero otherwise. Note that this quantity does

not depend on the learning algorithm nor on the training set.

The variance term measures how variant the classiﬁer prediction ˆy = h( x, αN )

is. This quantity is zero if the predicted class is always the same regardless of the

training set.

The squared bias term measures the squared diﬀerence between the yand the

ˆy probability distributions on the domain Y.

7.8 The hypothesis-based vs the algorithm-based

approach

In the previous sections we introduced two diﬀerent manners of assessing the accu-

racy of a learning machine. The reader could logically raise the following question:

which approach is the most adequate in practice?

Instead of providing a direct answer to such question, we prefer to conduct a

short comparison of the assumptions and limitations related to the two approaches.

The hypothesis-based approach formulates learning as the problem of ﬁnding

the hypothesis which minimises the functional risk. Vapnik reformulates this prob-

lem into the problem of consistency of a learning process based on ERM. The main

result is that it is possible to deﬁne a probabilistic distribution-free bound on the

functional risk which depends on the empirical risk and the VC dimension of the

class of hypothesis. Though this achievement is impressive from a theoretical and

scientiﬁc perspective (it was published in a Russian book in the 60s), its adop-

tion in practical settings is not always easy for several reasons: results derive from

asymptotic considerations though learning by deﬁnition deals with ﬁnite samples,

the computation of the VC dimension is explicit only for speciﬁc classes of hypoth-

esis functions and the bound, derived from worst-case analysis, is not always tight

enough for practical purposes.

7.9. THE SUPERVISED LEARNING PROCEDURE 175

The algorithm-based approach relies on the possibility of emulating the stochas-

tic process underlying the dataset by means of resampling procedures like cross-

validation or bootstrap. Note that this approach is explicitly criticised by Vapnik

and others who consider it inappropriate to reason in terms of data generation once

a single dataset is available. According to [58] "averaging over the data would be

unnatural, because in a given application, one has to live with the data at hand. It

would be marginally useful to known the number GN as this number would indicate

the quality of an average data sequence, not your data sequence". Nevertheless,

though it is hard to guarantee formally the accuracy of a resampling strategy, its

general-purpose nature, simplicity and ease of implementation have been, along

years, key ingredients of its success.

Whatever the degree of realism of the hypothesis made by the two approaches

is, it is worth making a pragmatic and historical consideration. Though the Vap-

nik results represent a major scientiﬁc success and underlie the design of powerful

learning machines (notably SVM), in a wider perspective it is fair to say that cross-

validation is the most common and successful workhorse of practical learning appli-

cations. This means that, though most data scientists have been eager to formalise

the consistency of their algorithms in terms of Vapnik bounds, in practice they had

recourse to intensive cross-validation tricks to make it work in the real world. Now,

more than 60 years after the ﬁrst computational version of learning processes, we

have enough evidence to say that cross-validation is a major element of the machine

learning success story. This is the reason why in the following sections we will focus

on an algorithm-based approach aiming to assess (and minimise) the generalisation

error by means of a resampling strategy.

7.9 The supervised learning procedure

The goal of supervised learning is to return the hypothesis with the lowest gen-

eralisation error. Since we assume that data samples are generated in a random

way, there is no hypothesis which gives a null generalisation error. Therefore, the

generalisation error GN of the hypothesis returned by a learning machine has to

be compared to the minimal generalisation error that can be attained by the best

single-valued mapping. Let us deﬁne by Λ∗the set of all possible single valued

mappings h : X → Y and consider the hypothesis

α∗ = arg min

α∈ Λ∗ R(α ) (7.9.53)

where R (α ) has been deﬁned in (7.2.3).

Thus, R (α∗ ) represents the absolute minimum rate of error obtainable by a single

valued approximator of the unknown target. For maintaining a simple notation, we

put G∗ =R (α∗ ). For instance, in our illustrative example in Section 7.5.1, α ∗

denotes the parameters of the cubic function and G∗ amounts to the unit variance

of the Gaussian noise.

In theoretical terms, a relevant issue is to demonstrate that the generalisation

error GN of the model with parameters αN learned from the dataset DN converges

to the minimum G∗ for N going to inﬁnity. Unfortunately, in real learning settings,

two problems must be dealt with. The ﬁrst is that the error GN cannot be computed

directly but has to be estimated from data. The second is that a single class Λ could

not be large enough to contain the hypothesis α∗ .

A common practice to handle these problems is to decompose the learning pro-

cedure in the following sequence of steps:

1. A nested sequence of classes of hypotheses

Λ1 ⊆ ··· ⊆ Λs ⊆ . . . ΛS (7.9.54)

176 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Capacity

Underﬁtting Overﬁtting

Figure 7.18: Bias/variance/noise tradeoﬀ and model selection: since the generali-

sation error (e.g. MISE) is not accessible in practical settings, model selection is

performed on the basis of an estimation (dotted line) which may induce an error

(and a variability) in the selection (7.9.55) of the best capacity.

is deﬁned so that Λ∗ = ∪S

s=1Λ s where s denotes the capacity of the class. This

guarantees that the set of hypotheses taken into consideration will necessarily

contain the best hypothesis α∗ .

A priori informations as well as considerations related to the bias/variance

dilemma can help in the design of this sequence.

2. For each class in the sequence, a hypothesis h (· , αs

N), s = 1, . . . , S , is selected

by minimising the empirical risk (7.2.8). This step is deﬁned as the parametric

identiﬁcation step of the learning procedure.

3. For each class in the sequence, a validation procedure returns ˆ

Nwhich esti-

mates the generalisation error Gs

Nof the hypothesis α s

N. This step is called

the validation step of the learning procedure.

4. The hypothesis h ( ·, α¯ s

N)∈Λ ¯ swith

¯ s= arg min

N(7.9.55)

is returned as the ﬁnal outcome. This ﬁnal step is called the model selection

step.

In order to accomplish the learning procedure, and speciﬁcally the selection in (7.9.55),

we need an estimation of the generalisation error (Section 7.10). However, since the

estimator of the generalisation error may be aﬀected by an error (as any estima-

tor), this may induce an error and a variability in the model selection step (7.9.55)

(Figure 7.18).

7.10. VALIDATION TECHNIQUES 177

7.10 Validation techniques

This section discusses validation methods to estimate the generalisation error GN

from a ﬁnite set of N observations.

The empirical risk (also called apparent error)R emp ( αN ) introduced in (7.2.7)

could be the most intuitive estimator of GN . However, it is generally known that the

empirical risk is a biased (and optimistic) estimate of GN and that Remp (αN ) tends

to be smaller than GN , because the same data have been used both to construct

and to evaluate h (·, αN ). A demonstration of the biasedness of the empirical risk

for a quadratic loss function in a regression setting is available in Appendix C.14.

In Section 9.1.16 we will analytically derive the biasedness of the empirical risk in

case of linear regression models.

The study of error estimates other than the apparent error is of signiﬁcant

importance if we wish to obtain results applicable to practical learning scenarios.

There are two main ways to obtain better, i.e. unbiased, estimates of GN : the ﬁrst

requires some knowledge on the distribution underlying the data set, the second

makes no assumptions on the data. As we will see later, an example of the ﬁrst

approach is the FPE criterion (presented in Section 9.1.16.2) while examples of the

second approach are the resampling procedures.

7.10.1 The resampling methods

Cross-validation [176] is a well-known method in sampling statistics to circumvent

the limits of the apparent error estimate. The basic idea of cross-validation is that

one builds a model from one part of the data and then uses that model to predict

the rest of the data. The dataset DN is split ltimes in a training and a test subset,

the ﬁrst containing N tr examples, the second containing Nts =N− N tr examples.

Each time, N tr examples are used by the parametric identiﬁcation algorithm Lto

select a hypothesis α i

Ntr ,i= 1, . . . , l , from Λ and the remaining N ts examples are

used to estimate the error of h ( ·, αi

Ntr ) (Fig. 7.19)

Rts ( αi

Ntr ) =

Nts

j=1

L(yj , h( xj , αi

Ntr )) (7.10.56)

The resulting average of the l errors ˆ

Rts ( α i

Ntr ), i = 1, . . . , l , is the cross-validation

estimate

Gcv =1

i=1

Rts ( αi

Ntr ) (7.10.57)

A common form of cross-validation is the "leave-one-out" (l-o-o). Let D (i) be the

training set with zi removed, and h ( x, αN(i ) ) be the corresponding prediction rule.

The l-o-o cross-validated error estimate is

Gloo =1

i=1

L yi , h( xi , αN(i ) )(7.10.58)

In this case lequals the number of training points and N ts = 1.

Bootstrap (Section 6.4) is also used to return a nonparametric estimate of GN ,

by repeatedly sampling the training cases with replacement. Since empirical risk is

a biased optimistic estimation of generalisation error and bootstrap is an eﬀective

method to assess bias (Section 6.4.3), it follows that bootstrap plays a role in a

validation strategy.

A bootstrap sample D(b) is a "fake" dataset {z1b , z2b , . . . , zN b } ,b = 1, . . . , B

randomly selected from the training set { z1 , z2, . . . , zN } with replacement.

178 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

Figure 7.19: Partition of the training dataset in the i th fold of cross-validation. The

quantity N i

tr is the number of training points while N i

ts is the number of test points.

Efron and Tibshirani [65] proposed to use bootstrap to correct the bias (or

optimism) of empirical risk by adopting a strategy similar to Section 6.4.3. Equa-

tion (6.4.9) estimates the bias of an estimator by computing the gap between the

average bootstrap (6.4.8) estimate and the sample estimation. In the case of gener-

alisation, the sample estimation ˆ

θis the empirical risk and the bootstrap estimate

θ(·) may be computed as follows

G(·) =1

B" N

i=1

(P ib L yi , h (xi , α(b) )# (7.10.59)

where P ib indicates the proportion of the bootstrap sample D (b) ,b = 1, . . . , B

containing the ith training point zi ,

Pib =# N

j=1(z j b =z i )

N(7.10.60)

and α(b) is the output of the parametric identiﬁcation performed on the set D(b) .

The diﬀerence between empirical risk and (7.10.59)

Biasbs = 1

b=1

i=1  (P ib −1

N) L  y i , h(xi , α(b) ) (7.10.61)

is the bias correction term to be subtracted to empirical risk to obtain a bootstrap

bias corrected estimate (6.4.10) of the generalisation error.

An alternative consists in using the holdout principle in combination with the

bootstrap one [65]. Since each bootstrap set is a resampling of the original training

set, it may happen that some of the original examples (called out-of-bag ) do not

belong to it: we can then use them to have an independent holdout set to be used

7.11. CONCLUDING REMARKS 179

for generalisation assessment. The bootstrap estimation of the generalisation error

(also known as E0) is then

Gbs =1

i=1

|B (i) | X

b∈ B(i)

L yi , h( xi , α(b) )(7.10.62)

where B (i) is the set of bootstrap samples which do not contain the ith point and

|B(i) |is its size. The terms where |B (i) | = 0 are discarded.

7.11 Concluding remarks

The goal of a learning procedure is to return a hypothesis which is able to predict

accurately the outcome of an input/output probabilistic mapping on the basis of

past observations. In order to achieve this goal, the learning procedure has to deal

with three major diﬃculties.

Minimisation of the empirical risk: in a general case ﬁnding the global mini-

mum of the empirical risk as in (7.2.7) demands the resolution of a multivari-

ate and nonlinear optimisation problem for which no analytical solution could

exist. Some heuristics to address this issue are discussed in Section 8.6.

Finite number of data: in real problems, a single random realisation of the sta-

tistical process, made of a ﬁnite number of input/output pairs, is accessible to

the learning machine. This means, that the hypothesis generated by a learn-

ing algorithm is a random variable as well. In theory, it would be required

to have access to the underlying process and to generate several times the

training set, in order to have a reliable assessment of the learning algorithm.

In practice, the use of repeated realisations is not viable in a real learning

problem.

The validation procedure copes with this problem by trying to assess a random

variable on the basis of a single realisation. In particular we focused on cross-

validation, a resampling method which works by simulating the stochastic

process underlying the data.

No a priori knowledge: we consider a setting where no knowledge about the

process underlying the data is available. This lack of a priori knowledge puts

no constraints on the complexity of the class of hypotheses to consider, with

the consequent risk of using an inadequate type of approximator. The model

selection deals with this issue by considering classes of hypotheses of increasing

complexity and selecting the one which behaves the best according to the

validation criteria. This strategy ensures the covering of the whole spectrum of

approximators, ranging from low bias/high variance to high bias/low variance

models, making easier the selection of a good trade-oﬀ on the basis of the

available data.

So far, the learning problem has been introduced and discussed for a generic class

of hypotheses, and we did not distinguish on purpose between diﬀerent learning

machines. The following chapter will show the parametric and the structural iden-

tiﬁcation procedure as well as the validation phase for some speciﬁc learning ap-

proaches.

7.12 Exercises

1. Consider an input/output regression task where n = 1, E [y |x ] = sin(πx/ 2) and

p( y| x) = N (sin( πx/2) , σ2 ), σ = 0 . 1 and x ∼ U (− 2 , 2). Let N be the size of the

180 CHAPTER 7. STATISTICAL SUPERVISED LEARNING

training set and consider a quadratic loss function.

Let the class of hypothesis be hM (x ) = α0 + P M

m=1 α m x m with α j ∈[− 2, 2], j =

0,...,M.

For N = 20 generate S = 50 replicates of the training set. For each replicate,

estimate the value of the parameters that minimise the empirical risk, compute the

empirical risk and the functional risk.

1. Plot the evolution of the distribution of the empirical risk for M = 0, 1, 2.

2. Plot the evolution of the distribution of the functional risk for M = 0, 1, 2.

Hints: to minimise the empirical risk, perform a grid search in the space of pa-

rameter values, i.e. by sweeping all the possible values of the parameters in the

set [− 1,− 0. 9,− 0 . 8,..., 0. 8, 0. 9, 1]. To compute the functional risk generate a set of

Nts = 10000 i.i.d. input/output testing examples.

Solution: See the ﬁle Exercise6.pdf in the directory gbcode/exercises of the

companion R package (Appendix F).

Chapter 8

The machine learning

procedure

8.1 Introduction

Raw data is rarely of direct beneﬁt. Its true value resides in the amount of informa-

tion that a model designer can extract from it. Modelling from data is often viewed

as an art form, mixing the insight of the expert with the information contained in the

observations. Typically, a modelling process is not a sequential process but is better

represented as a sort of loop with a lot of feedback and a lot of interactions with

the designer. Diﬀerent steps are repeated several times aiming to reach, through

continuous reﬁnements, a good model description of the phenomenon underlying

the data.

This chapter reviews the practical steps constituting the process of constructing

models for accurate prediction from data. Note that the overview is presented with

the aim of not distinguishing between the diﬀerent families of approximators and

of showing which procedures are common to the majority of modelling approaches.

The following chapters will be instead devoted to the discussion of the peculiarities

of speciﬁc learning approaches.

We partition the data modelling process into two phases: a preliminary phase

which leads from the raw data to a structured training set, and a learning phase,

which leads from the training set to the ﬁnal model. The preliminary phase is

made of a problem formulation step (Section 8.2) where the designer selects the

phenomenon of interest and deﬁnes the relevant input/output features, an experi-

mental design step (Section 8.3) where input/output data are collected, and a data

preprocessing step (Section 8.4) where preliminary conversion and ﬁltering of data

is performed.

Once the numeric dataset has been formatted, the learning procedure begins.

In qualitative terms, this procedure can be described as follows. First, the designer

deﬁnes a set of models (e.g. polynomial models, neural networks) characterised by

acapacity (or complexity) index (or hyper-parameter) (e.g. degree of the polyno-

mial, number of neurons, VC dimension) which controls the approximation power

of the model. According to the capacity index, the set of models is consequently

decomposed in a nested sequence of classes of models (e.g. classes of polynomials

with increasing degree). Hence, a structural identiﬁcation procedure loops over the

set of classes, ﬁrst by identifying a parametric model for each class (parametric

identiﬁcation) and then by assessing the prediction error of the identiﬁed model on

the basis of the ﬁnite set of points (validation ). Finally, a model selection procedure

selects the ﬁnal model to be used for future predictions. A common alternative to

181

182 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

model selection is a model combination, where a combination (e.g. averaging) of

the most promising models is used to return a meta-model, presumably with better

accuracy properties.

The problem of parametric identiﬁcation is typically a problem of multivariate

optimisation [2]. Section 8.6 introduces the most common optimisation algorithms

for linear and nonlinear conﬁgurations. Structural identiﬁcation is discussed in

Section 8.8, which focuses on the existing methods for model generation, model

validation and model selection. The last section concludes and resumes the whole

modelling process with the support of a diagram.

8.2 Problem formulation

The problem formulation is the preliminary and somewhat the most critical step of

a learning procedure. The model designer chooses a particular application domain

(e.g. ﬁnance), a phenomenon to be studied (e.g. the credit risk of a customer)

and hypothesises the existence of an unknown dependency (e.g. between the ﬁnan-

cial situation of the customer and the default risk) which is to be estimated from

experimental data. First, the modeller speciﬁes a set of constructs , i.e. abstract

concepts or high-level topics which are potentially relevant for the study (e.g. the

proﬁle and the ﬁnancial situation of a client). Second, a set of variables (e.g. the

client age and her salary) is deﬁned by grounding the constructs into a measurable

form. Eventually, an operationalisation, i.e. the deﬁnition of how to measure those

variables (e.g. by accessing a bank database), is proposed.

In this step, domain-speciﬁc knowledge and experience are the most crucial

requirements to come up with a meaningful problem formulation. Note that the

time spent in this phase is usually highly rewarding and can save vast amounts

of modelling time. There is often no substitute for physical intuition and human

analytical skills.

8.3 Experimental design

The most precious thing in data-driven modelling is the data itself. No matter how

powerful a learning method is, the resulting model would be ineﬀective if the data

are not informative enough. Hence, it is necessary to devote a great deal of attention

to the process of observing and collecting the data. In input/output modelling it is

essential that the training set be a representative sample of the phenomenon and

cover the input space adequately. To this aim, it is relevant to consider the relative

importance of the various areas of the input space. Some regions are more relevant

than others, as in the case of a dynamical system whose state has to be regulated

about some speciﬁed operating point.

The discipline of creating an optimal sampling of the input space is called exper-

imental design [72]. The study of experimental design is concerned with locating

training input data in the space of input variables so that the performance of the

modelling process is maximised. However, in some cases, the designer cannot ma-

nipulate the process of collecting data, and the modelling process has to deal with

what is available. This conﬁguration, which is common to many real problems, is

called the observational setting [42]. Though this setting seems the most adequate

for a learning approach ("just learn from what you observe"), it is worth reminding

that most of the time, behind an observation setting, there is the strong implicit

assumption that the observations are i.i.d. samples of a stationary (i.e. invariant)

stochastic process. Now, in most realistic cases, this assumption is not valid (at

least not for a long time), and considerations of nonstationarity, drift should be

8.4. DATA PRE-PROCESSING 183

integrated in the learning process. Other problems are related to the poor causal

value of inferences made in an observational setting, e.g. in situations of sampling

bias or non-observable variables. Nevertheless, given the introductory nature of

this book, in what follows, we will limit to consider the simplest observational and

stationary setting.

8.4 Data pre-processing

Once data have been recorded, it is common practice to pre-process them. The hope

is that such treatment might make learning easier and improve the ﬁnal accuracy.

Pre-processing includes a large set of actions on the observed data, and some of

them are worth being discussed:

Numerical encoding. Some interesting data for learning might not be in a nu-

meric format (e.g. text, image). Since, in what follows, we will assume that

all data are numeric, a preliminary conversion or encoding step is needed.

Given that most encoding procedures are domain-speciﬁc, we will not further

discuss them here.

Missing data treatment. In real applications, it often happens that some input

values are missing. If the quantity of data is suﬃciently large, the simplest

solution is to discard the examples having missing features. When the amount

of data is too restricted or there are too many partial examples, it becomes

important to adopt some speciﬁc technique to deal with the problem. Various

heuristics [93], as well as methods based on the Expectation Maximisation

(EM) algorithm [79], have been proposed in the literature. Note that any

missing data treatment strategy makes assumptions about the process that

caused some observations to be missing (e.g. missing at random or not) [124]:

it is recommended to be aware of such assumptions before applying them (see

also Section 13.7.4 on selection bias).

Categorical variables. It may be convenient to treat categorical variables, specif-

ically in situations when they may take a very large number of values (e.g.

names of retailers in a business intelligence application). Two common ways

to deal with are: i) replace them with dummy variables encoding the dif-

ferent values in binary terms (e.g. K bits for Kcategories) ii) replace each

category with numerical values informative about the conditional distribution

of the target given such category: for instance, in regression (binary clas-

siﬁcation) we could replace a category x = "black " with an estimation of

E[y| x= " black"] (Prob {y = 1| x= " black"}).

Feature selection. The feature selection problem consists in selecting a relevant

subset of input variables in order to maximise the performance of the learning

machine. This approach is useful if there are inputs that carry only little useful

information or are strongly correlated. In these situations a dimensionality

reduction improves the performance reducing the variance of the estimator at

the cost of a slight increase in the bias.

Several techniques exist for feature selection, such as conventional methods

in linear statistical analysis [59], principal component analysis [139] and the

general wrapper approach [117]. For more details, we refer the reader to

Chapter 12.

Outliers removal. Outliers are unusual data values that are not consistent with

most of observations. Commonly, outliers are due to wrong measurement

procedures, storage errors and coding malfunctioning. There are two common

184 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

strategies to deal with outliers: the ﬁrst is performed at the preprocessing

stage [71] and consists in their detection and consequent removal, the second

is to delay their treatment at the model identiﬁcation level by adopting robust

methodologies [105] that are by design insensitive to outliers.

Other common preprocessing operations are pre-ﬁltering to remove noise eﬀects,

anti-aliasing to deal with sampled signals, variable scaling to standardise all vari-

ables to have a zero mean and a unity variance 1 , compensation of nonlinearities or

the integration of some domain-speciﬁc information to reduce the distorting eﬀects

of measurable disturbances.

8.5 The dataset

The outcome of the pre-processing phase is a dataset in a tabular numeric form

where each row represents a particular observation (also called instance, example

or data point), and each column a descriptive variable (also called feature, attribute

or covariate). We denote the dataset

DN = {z1 , z2, . . . , zN }

where N is the number of examples, nis the number of features, the i th example is

an input/output pair zi =hxi , yi i, i = 1, . . . , N ,xi is a [n× 1] input vector and yi

is a scalar output.

Note that hereafter, for the sake of simplicity, we will restrict ourselves to a

regression setting. We will assume the input/output data to be i.i.d. generated by

the following stochastic dependency:

y=f ( x) + w, (8.5.1)

where E [w ] = 0 and σ 2

wis the noise variance.

The noise term wis supposed to lump together all the unmeasured contributions

to the variability of y , like, for instance, missing or non-observable variables. There

are two main assumptions underlying Equation (8.5.1). The ﬁrst is that the noise is

independent of the input and has a constant variance. This assumption also called

homeskedasticity in econometrics, is typically made in machine learning because of

the primary focus on the dependency fand the lack of eﬀective methodologies to

assess it a priori in nonlinear and high dimensional settings. The reader should

be aware, however, that heteroskedastic conﬁgurations may have a strong impact

on the ﬁnal model accuracy and that some a priori output variable transformation

and/or a posteriori assessment is always recommended (e.g. study of the residual

distribution after ﬁtting). The second assumption is that noise enters additively

to the output. Sometimes the measurements of the inputs to the system may also

be noise corrupted; in system identiﬁcation literature, this is what is called the

error-in-variable conﬁguration [9]. As far as this problem is concerned, we adopt

the pragmatic approach proposed by Ljung [125], which assumes that the measured

input values are the actual inputs and that their deviations from the correct values

propagate through fand lump into the noise w.

In the following, we will refer to the set of vectors xi and yi through the following

matrices:

1. the input matrix X of dimension [N× n ] whose ith row is the vector x T

2. the output vector Y of dimension [N× 1] whose ith component is the scalar

yi .

1This can be easily done with numeric inputs by the R command scale

8.6. PARAMETRIC IDENTIFICATION 185

8.6 Parametric identiﬁcation

Assume that a class of hypotheses h ( ·, α ) with α∈ Λ has been ﬁxed. The problem of

parametric identiﬁcation from a ﬁnite set of data consists in seeking the hypothesis

whose vector of parameters αN ∈ Λ minimises the loss function

Remp ( α) = 1

i=1

L( yi , h( xi , α)) (8.6.2)

This phase of the learning procedure requires the resolution of the optimisation

task (7.2.7). In this section, we will review some of the most common algorithms

that address the problem of parametric identiﬁcation in linear and nonlinear cases.

To make the notation more readable, henceforth we will deﬁne the error function

J( α) = Remp ( α)

and we will formulate the optimisation problem (7.2.7) as

αN = arg min

α∈ΛJ(α ). (8.6.3)

Also, we will use the term model as a synonymous of hypothesis.

8.6.1 Error functions

The choice of an optimisation algorithm is strictly dependent on the form of the

error function J (α ). The function J (α ) is directly determined by two factors

1. the form of the model h (·, α ) with α∈ Λ,

2. the loss function L (y, h ( x, α )) for a generic x.

As far as the cost function is concerned, there are many possible choices depending

on the type of data analysis problem. In regression problems, the goal is to model

the conditional distribution of the output variable conditioned on the input vari-

able (see Section 7.4) whose mean is the value minimising the mean squared error

(Equation 3.3.29). This motivates the use of a quadratic function

L( y, h( x, α)) = ( y− h( x, α))2(8.6.4)

which gives to J (α ) the form of a sum-of-squares.

For classiﬁcation problems, the goal is to model the posterior probabilities of

class membership, again conditioned on the input variables. Although the sum-of-

squares J can also be used for classiﬁcation, there are more appropriate error func-

tions to be considered [60]. The most used is cross-entropy which derives from the

adoption of the maximum-likelihood principle (Section 5.8) for supervised classiﬁca-

tion. Consider a classiﬁcation problem where the output variable ytakes values in

the set {c1 , . . . , cK } and Prob {y =cj | x} , j = 1, . . . , K is the conditional probabil-

ity. Given a training dataset and a set of parametric models ˆ

Pj ( x, α), j = 1, . . . , K

of the conditional distribution, the classiﬁcation problem boils down to the minimi-

sation of the quantity

J( α) = −

i=1

log ˆ

Py i ( xi , α) (8.6.5)

Note that the models ˆ

Pj ( x, α), j = 1, . . . , K must satisfy two important constraints:

1. ˆ

Pj ( x, α) > 0

186 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

2. P K

j=1 ˆ

Pj ( x, α )=1

In the case of 0/1 binary classiﬁcation problem, the cross-entropy is written as

J( α) = −

i=1

[yi log ˆ

P1 ( xi , α) + (1 − yi ) log(1 − ˆ

P1 ( xi , α))] (8.6.6)

where ˆ

P1 ( xi , α) is the estimation of the conditional probability of the class y = 1.

Since this chapter will focus mainly on regression problems, we will limit to

consider the case of a quadratic loss function.

8.6.2 Parameter estimation

8.6.2.1 The linear least-squares method

The parametric identiﬁcation of a linear model

h( x, α) = αT x

is obtained by minimising a quadratic function J (α ) by the well-known linear least-

squares method. In Chapter 9 we will see in detail the linear least-squares minimisa-

tion. Here we just report that, in the case of non-singularity of the matrix (XT X ),

J( α) has a single global minimum in

αN = ( XT X) −1 XT Y (8.6.7)

8.6.2.2 Iterative search methods

In general cases, when either the model is not linear, or the cost function is not

quadratic, J (α ) can be a highly nonlinear function of the parameters α , and there

may exist many minima, all of which satisfy

∇J= 0 (8.6.8)

where ∇ denotes the gradient of Jin parameter space. We will deﬁne as stationary

points all the points which satisfy condition (8.6.8). They include local maxima,

saddle points and minima. The minimum for which the value of the error function

is the smallest is called the global minimum, while other minima are called local

minima. As a consequence of the non-linearity of the error function J (α ), it is not

in general possible to ﬁnd closed-form solutions for the minima. For more details

on multivariate optimisation we refer the reader to [2].

We will consider iterative algorithms, which involve a search through the pa-

rameter space consisting of a succession of steps of the form

α(τ +1) =α(τ) + ∆α(τ) (8.6.9)

where τ labels the iteration step.

Iterative algorithms diﬀer for the choice of the increment ∆α (τ) .

In the following, we will present some gradient-based and non-gradient-based it-

erative algorithms. Note that each algorithm has a preferred domain of application

and that it is not possible, or at least fair, to recommend a single universal opti-

misation algorithm. We consider it much more interesting to highlight the relative

advantages and limitations of the diﬀerent approaches.

8.6. PARAMETRIC IDENTIFICATION 187

8.6.2.3 Gradient-based methods

In some cases, the analytic form of the error function makes it possible to evaluate

the gradient of the cost function Jwith respect to the parameters α, increasing the

rate of convergence of the iterative algorithm. Some examples of gradient-based

methods are reported in the following sections. Those methods require the deriva-

tives of the cost function with respect to the model parameters. Such computation

is not always easy in complex nonlinear mappings, but it has been recently facil-

itated by the appearance of automatic diﬀerentiation functionalities [17], like the

ones made available by libraries like TensorFlow or PyTorch.

8.6.2.4 Gradient descent

It is the simplest of the gradient-based optimisation algorithms, also known as the

steepest descent. This algorithm starts with some initial guess α (0) for the parameter

vector (often chosen at random). Then, it iteratively updates the parameter vector

such that, at the τ th step, the estimate is updated by moving a short distance in

the direction of the negative gradient evaluated in α (τ) :

∆α(τ) =−µ ∇J (α(τ) ) (8.6.10)

where µ is called the learning rate. The updates are repeatedly executed until

convergence, i.e. when further improvements are considered to be too small to be

useful.

The gradient descent method is known to be a very ineﬃcient procedure. One

drawback is the need for a suitable value of the learning rate µ. In fact, a decrease

of the cost function is guaranteed by (8.6.10) only for learning rates of inﬁnitesimal

size: if its value is suﬃciently small, it is expected that the value of J (α(τ) ) will

decrease at each successive step, eventually leading to a parameter vector at which

the condition (8.6.8) is satisﬁed. Too small learning rates may considerably delay

the convergence, while too large rates might result in numerical overﬂows.

Further, at most points in the parameter space, the local gradient does not point

directly towards the minimum: gradient descent then needs many small steps to

reach a stationarity point.

Example of gradient-based univariate optimisation

Let us consider the univariate function

J( α) = α2 −2 α+ 3

visualised in Figure 8.1. By running the script optim.R you can visualise the gra-

dient search of the minimum of the function. Note that the gradient is obtained by

computing analytically the derivative

J0 ( α) = 2 α−2

We invite the reader to assess the impact of the learning rate µon the conver-

gence of the minimisation process.

The function

J( α) = α4 /4− α" /3− α2 + 2

with two local minima is shown in Function 8.2 and minimised in the script optim2.R.

We invite the reader to assess the impact of the initial value α (0) of the solution on

the result of the minimisation process.

•

188 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

Figure 8.1: Gradient-based minimisation of the function J (α ) = α2 − 2α + 3 with

a single global minimum.

Figure 8.2: Gradient-based minimisation of the function J (α ) = α4 / 4−α" / 3−α2 +2.

8.6. PARAMETRIC IDENTIFICATION 189

Figure 8.3: Contour plot: gradient-based minimisation of the bivariate function

J( α) = α2

1+α 2

2−2α 1 −2α 2 + 6 with a single global minimum.

Example of gradient-based bivariate optimisation

Let us consider the bivariate function

J( α) = α2

1+α 2

2−2α 1 −2α 2 + 6

whose contour plot is visualised in Figure 8.3. By running the script optim2D.R

you can visualise the gradient search of the minimum of the function. Note that

the gradient is obtained by computing analytically the gradient vector

∇J( α) = [2 α1 −2, 2α2 −2]T

We invite the reader to assess the impact of the learning rate µ on the conver-

gence of the minimisation process.

The contour plot of the function

J( α) = α 4

1+α 4

4− α 3

1+α 3

3−α2

1−α 2

2+ 4

with three local minima is shown in Function 8.4 and minimised in the script

optim2D2.R. We invite the reader to assess the impact of the initial value α (0)

of the solution on the result of the minimisation process.

•

In alternative to the simplest gradient descent, there are many iterative methods

in the literature, as the momentum -based method [155], the enhanced gradient

descent method [191] and the conjugate gradients techniques [157], which make

implicit use of second-order derivatives of the error function.

In the following section, we present instead a class of algorithms that make

explicit use of second-order information.

8.6.2.5 The Newton method

The Newton's method is a well-known example in optimisation literature. It is an

iterative algorithm which uses at the τ th step a local quadratic approximation in

190 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

Figure 8.4: Contour plot: gradient-based minimisation of the function J (α ) =

α4

1+α 4

4− α3

1+α 3

3−α 2

1−α 2

2+ 4.

the neighbourhood of α (τ)

J( α) = J α(τ)  + α− α(τ)  T ∇ J α(τ)  +1

2 α−α(τ)  T H( α(τ) ) α− α(τ) 

(8.6.11)

where H (α (τ) ) is the Hessian matrix of J (α ) computed in α (τ) and

H( α) = 





∂2 J

∂α2

∂2 J

∂α1∂α2 . . . ∂ 2 J

∂α1∂αp

... . .

∂2 J

∂αp∂α1

∂2 J

∂αp∂α2 . . . ∂ 2 J

∂α2







is a [p, p] square matrix of second-order partial derivatives if α∈Rp .

The minimum of (8.6.11) satisﬁes

αmin =α(τ) − H−1 ( α(τ) )∇ J α(τ)  (8.6.12)

where the vector H−1 (α (τ) )∇J α(τ)  is denoted as the Newton direction or the

Newton step and forms the basis for the iterative strategy

α(τ +1) =α(τ) − H−1 ∇ J α(τ)  (8.6.13)

There are several diﬃculties with such an approach, mainly related to the pro-

hibitive computational demand. Alternative approaches, known as quasi-Newton or

variable metric methods, are based on (8.6.12) but instead of calculating the Hes-

sian directly, and then evaluating its inverse, they build up an approximation to the

inverse Hessian. The two most commonly used update formulae are the Davidson-

Fletcher-Powell (DFP) and the Broyden-Fletcher-Goldfarb-Shanno (BFGS) proce-

dures [128].

8.6.2.6 The Levenberg-Marquardt algorithm

This algorithm is designed speciﬁcally for minimising a sum-of-squares error func-

tion

J( α) = 1

i=1

L( yi , h( xi , α)) = 1

i=1

2kek2 (8.6.14)

8.6. PARAMETRIC IDENTIFICATION 191

where ei is the error for the ith training case, eis the [N× 1] vector of errors and

k·k is a 2-norm. Let us consider an iterative step in the parameter space

α(τ) →α(τ+1) (8.6.15)

If the quantity (8.6.15) is suﬃciently small, the error vector ecan ban expanded in

a ﬁrst-order Taylor series form:

e α(τ+1)  = e α(τ)  + E α(τ+1) −α(τ)  (8.6.16)

where the generic element of the matrix Eis in the form

Eij = ∂e i

∂αj

(8.6.17)

and αj is the j th element of the vector α. The error function can then be approxi-

mated by

J α(τ+1)  =1

2



e  α (τ)  +E  α (τ +1) −α (τ)  



2(8.6.18)

If we minimise with respect to α(τ+1) we obtain:

α(τ +1) =α(τ) −( ETE)−1 ET e( α(τ) ) (8.6.19)

where (ET E )−1 ET is the pseudo-inverse of the matrix E . For the sum-of-squares

error function (8.6.14), the elements of the Hessian take the form

Hjk = ∂ 2 E

∂αjαk

i=1  ∂e i

∂αj

∂ei

∂αk

+ei

∂2 ei

∂αjαk  (8.6.20)

Neglecting the second term, the Hessian can be written in the form

H' ET E(8.6.21)

This relation is exact in the case of linear models, while in the case of nonlinearities

it represents an approximation that holds exactly only at the global minimum of the

function [25]. The update formula (8.6.19) could be used as the step of an iterative

algorithm. However, the problem with such an approach could be a too large step

size returned by (8.6.19), making the linear approximation no longer valid.

The idea of the Levenberg-Marquardt algorithm is to use the iterative step, at

the same time trying to keep the step size small so as to guarantee the validity of

the linear approximation. This is achieved by modifying the error function in the

form

Jlm =1

2



e( α (τ) ) + E (α (τ +1) −α(τ) ) 



2+λ



α (τ +1) −α (τ) 



2(8.6.22)

where λ is a parameter that governs the step size. The minimisation of the error

function (8.6.22) ensures, at the same time, the minimisation of the sum-of-square

cost and a small step size. Minimising (8.6.22) with respect to α (τ +1) we obtain

α(τ+1) =α(τ) −( ET E+ λI ) −1 ET e( α(τ) ) (8.6.23)

where I is the unit matrix. For very small values of λwe have the Newton formula,

while for large values of λ we recover the standard gradient descent.

A common approach for setting λis to begin with some arbitrary low value

(e.g. λ = 0. 1) and at each step (8.6.23) check the change in J . If J decreases, the

new parameter is retained, λis decreased (e.g. by a factor of 10), and the process

repeated. Otherwise, if J increased after the step (8.6.23), the old parameter is

restored, λ decreased, and a new step performed. The procedure is iterated until a

decrease in Jis obtained [25].

192 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

8.6.3 Online gradient-based algorithms

The algorithms above are called batch since they compute the gradient of the quan-

tity (8.6.2) over the entire training set. In the case of very large datasets or se-

quential settings, this procedure is not recommended since it requires the storage

of the entire dataset. For this reason, online modiﬁcation of the batch algorithms

have been proposed in the literature. The idea consists of replacing the gradient

computed on the entire training set with a gradient computed on the basis of a

single data point

α(τ +1) =α(τ) − µ(τ) ∇α J(τ) ( α(τ) ) (8.6.24)

where

J(τ) ( α(τ) )=( yτ − h( xτ , α(τ) ))2

and zτ = hyτ , xτ i the input/output observation at the τth instant.

The underlying assumption is that the training error obtained by replacing the

average with a single term will not perturb the average behaviour of the algorithm.

Note also that the dynamics of µ(τ) plays an important role in the convergence.

This algorithm can be easily used in an adaptive online setting where no training

set needs to be stored, and observations are processed immediately to improve

performance. A linear version of the iteration (8.6.24) is the Recursive Least Squares

regression algorithm presented in Section 9.1.20. Note also that the earliest machine

learning algorithms were based on sequential gradient-based minimisation. Well-

known examples are Adaline and LMS [195].

8.6.4 Alternatives to gradient-based methods

Virtually no gradient-based method is guaranteed to ﬁnd the global optimum of

a complex nonlinear error function. Additionally, all descent methods are deter-

ministic in the sense that they inevitably lead to convergence to the nearest local

minimum. As a consequence, the way a deterministic method is initialised is decisive

for the ﬁnal result.

Further, in many practical situations, the gradient-based computation is time-

consuming or extremely diﬃcult due to the complexity of the objective function.

For these reasons, a lot of derivative-free and stochastic alternatives to gradient-

based methods have been explored in the literature. We will limit to cite the most

common solutions:

Random search methods. They are iterative methods that are primarily used

for continuous optimisation problems. Random search methods explore the

parameter space of the error function sequentially in a random fashion to ﬁnd

the global minimum. Their strength lies mainly in their simplicity, which

makes these methods easily understood and conveniently customised for spe-

ciﬁc applications. Moreover, it has been demonstrated that they converge

to the global optimum with probability one on a compact set. However, the

theoretical result of convergence to the minimum is not really important here

since the optimisation process could take a prohibitively long time.

Genetic algorithms. They are derivative-free stochastic optimisation methods

based loosely on the concepts of natural selection and evolutionary processes

[82]. Important properties are the strong parallelism and the possibility to

be applied to both continuous and discrete optimisation problems. Typically,

Genetic Algorithms (GA) encode each parameter solution into a binary bit

string (chromosome) and associate each solution with a ﬁtness value. GAs

usually keep a set of solutions (population ) which is evolved repeatedly toward

a better overall ﬁtness value. In each generation, the GA constructs a new

8.7. REGULARISATION 193

population using genetic operators such as crossover or mutation; members

with higher ﬁtness values are more likely to survive and to participate in fu-

ture operations. After a number of generations, the population is expected to

contain members with better ﬁtness values and to converge, under particular

conditions, to the optimum.

Simulated annealing. It is another derivative-free method suitable for continu-

ous and discrete optimisation problems. In Simulated Annealing (SA), the

value of the cost function J (α ) to be minimised is put in analogy to the en-

ergy in a thermodynamic system at a certain temperature T[115]. At high

temperatures T(τ) , the SA technique allows function evaluations at points far

away from α (τ) , and it is likely to accept a new parameter value with a higher

function value. The decision whether to accept or reject a new parameter

value α (τ +1) is based on the value of an acceptance function, generally shaped

as the Boltzmann probability distribution. At low temperatures, SA evaluates

the objective function at more local points, and the likelihood of accepting

a new point with a higher cost is much lower. An annealing schedule regu-

lates how rapidly the temperature Tgoes from high values to low values as a

function of time or iteration counts.

R script

The script grad.R compares four parameter identiﬁcation algorithms in the case

of a univariate linear model: least-squares, random search, gradient-based and

Levenberg-Marquardt.

•

8.7 Regularisation

Parameter identiﬁcation relies on Empirical Risk Minimisation to return an estima-

tor in supervised learning problems. In Section 7.7, we stressed that the accuracy

of such an estimator depends on the bias/variance trade-oﬀ, which is typically con-

trolled by capacity related hyper-parameters. There is, however, another important

strategy, called regularisation , to control the bias/variance trade-oﬀ by constraining

the ERM problem. The rationale consists then in restricting the set of possible so-

lutions by transforming the unconstrained problem (8.6.3) into a constrained one.

An example of constrained minimisation is

αN = arg min

α∈ΛJ(α ) + λk αk (8.7.25)

where λ > 0 is the regularisation parameter. By adding the squared norm term,

solutions with large values of components of αare penalised unless they play a

major role in the J (α ) term. An alternative version is

αN = arg min

α∈ΛJ(α ) + λS (h ( ·, α )) (8.7.26)

where the term S penalises non-smooth wiggling hypothesis functions. An example

S( α) = Z ( h"( x, α))2 dx

where the integral of the second derivative is a measure of lack of smoothness.

Regularisation is a well-known strategy in numerical analysis and optimisation

to avoid or limit the ill-conditioning of the solution. In estimation, regularisation

194 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

is an additional way to control the variance of the estimator resulting from the

optimisation procedure. This is particularly eﬀective in learning problems with a

number of observations comparable or even smaller than the input dimension.

Note, however, that variance reduction occurs at the cost of both increased bias

and additional complexity of the optimisation problem. For instance, according

to the nature of the regularisation and the nonlinearity of the hypothesis function

we could lose some interesting properties like the closed-form of the solution. In

Chapter12, we will show some examples of regularisation to address the curse of

dimensionality problem.

8.8 Structural identiﬁcation

Once a class of models Λ is given, the identiﬁcation procedure, described above,

returns a model h (,· , αN ) deﬁned by the set of parameters αN ∈ Λ .

The choice of an appropriate class or model structure [125] is, however, the most

crucial aspect for a successful modelling application. The procedure of selecting the

best model structure is called structural identiﬁcation. The structure of a model

is made of a series of features that inﬂuence the generalisation power of the model

itself. Among others, there are:

•The type of the model. We can distinguish, for example, between nonlinear

and linear models, between physically based and black box representations,

between continuous-time or discrete-time systems.

•The size of the model. This is related to features like the number of inputs

(regressors), the number of parameters, the degree of the polynomials in a

polynomial model, the number of neurons in a neural network, the number of

nodes in a classiﬁcation tree, etc.

In general terms, structural identiﬁcation requires (i) a procedure for proposing

a series of alternative model structures, (ii) a method for assessing each of these

alternatives and (iii) a technique for choosing among the available candidates.

We denote the ﬁrst issue as model generation. Some techniques for obtaining

diﬀerent candidates to the ﬁnal model structure are presented in Section 8.8.1.

The second issue concerns the important problem of model validation , and will

be extensively dealt with in Section 8.8.2.

Once models have been generated and validated, the last step is the model

selection, which will be discussed in Section 8.8.3

It is important to remark that a selected model structure should never be ac-

cepted as a ﬁnal and true description of the phenomenon. Rather, it should be

regarded as a good enough description, given the available dataset.

8.8.1 Model generation

The goal of the model generation procedure is to generate a set of candidate model

structures among which the best one is to be selected. The more this procedure

is eﬀective, the easier will be the selection of a powerful structure at the end of

the whole identiﬁcation. Traditionally there have been a number of popular ways

to search through a large collection of model structures. Maron and Moore [131]

distinguish between two main methods of model generation:

Brute force. This is the exhaustive approach. Every possible model structure is

generated in order to be evaluated.

Consider, for instance, the problem of selecting the best structure in a 3-

layer Feed Forward Neural Network architecture. The brute force approach

8.8. STRUCTURAL IDENTIFICATION 195

consists of enumerating all the possible conﬁgurations in terms of the number

of neurons.

The exhaustive algorithm runs in a time that is generally unacceptably slow

for complex architectures with a large number of structural parameters. The

only advantage, however, is that this method is guaranteed to return the best

learner according to a speciﬁed assessment measure.

Search methods. These methods treat the collection of models as a continuous

and diﬀerentiable surface. They start at some point on the surface and search

for the model structure that corresponds to the minimum of the generalisation

error until some stop condition is met. This procedure is much faster than

brute force since it does not need to explore all the space. It only needs to

validate those models that are on the search path. Gradient-based techniques

and/or non-gradient based methods can be used for the search in the model

space. Besides the well-known problem related to local minima in the gradi-

ent base case, a more serious issues derives from the structure of the model

selection procedure. At every step of the search algorithm, we need to ﬁnd

a collection of models that are near or related to the current model. Both

gradient-based and non-gradient-based techniques require some metric in the

search space. This implies a notion of model distance, diﬃcult to deﬁne in a

general model selection problem. Examples of search methods in model gen-

eration are the growing and pruning techniques in Neural Networks structural

identiﬁcation [18].

8.8.2 Validation

The output of the model generation procedure is a set of model structures Λs,

s= 1 , . . . , S. Once the parametric identiﬁcation is performed on each of these model

structures, we have a set of models h (·, αs

N) identiﬁed according to the Empirical

Risk Minimisation principle.

Now, the prediction quality of each one of the model structures Λs ,s = 1, . . . , S,

has to be assessed on the basis of the available data. In principle, the assessment

procedure, known as model validation , could measure the goodness of a structure

in many diﬀerent ways: how the model relates to our a priori knowledge, how the

model is easy to be used, to be implemented or to be interpreted. In this book, as

stated in the introduction, we will focus only on criteria of accuracy, neglecting any

other criterion of quality.

In the following, we will present the most common techniques to assess a model

on the basis of a ﬁnite set of observations.

8.8.2.1 Testing

An obvious way to assess the quality of a learned model is by using a testing sequence

Dts = (h xN+1 , yN+1 i ,...,h xN+Nts , yN+Nts i) (8.8.27)

that is a sequence of i.i.d. pairs, independent of DN and distributed according

to the probability distribution P (x, y ) deﬁned in (7.2.2). The testing estimator is

deﬁned by the sample mean

Rts ( α s

N) = 1

Nts

N+ Nts

j= N+1

(yj − h (xj , αs

N)) 2 (8.8.28)

This estimator is clearly unbiased in the sense that

ED ts [ˆ

Rts (α s

N)] = R (α s

N) (8.8.29)

196 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

When the number of available examples is suﬃciently high, the testing technique

is an eﬀective validation technique at a low computational cost. A serious problem

concerning the practical applicability of this estimate is that it requires a large,

independent testing sequence. In practice, unfortunately, an additional set of in-

put/output observations is rarely available.

8.8.2.2 Holdout

The holdout method, sometimes called test sample estimation, partitions the data

DN into two mutually exclusive subsets, the training set D tr and the holdout or

test set DN ts . It is common to design 2/3 of the data as training set and the

remaining 1/3 as test set. However, when the training set has a reduced number of

cases, the method can present a series of shortcomings, mainly due to the strong

dependence of the prediction accuracy on the repartition of the data between the

training and validation set. Assuming that the error R (αN tr ) decreases as more

cases are inserted in the training set, the holdout method is a pessimistic estimator

since only a reduced amount of data is used for training. The larger the number

of points used for test set, the higher the bias of the estimate αN tr ; at the same

time, fewer test points implies a larger conﬁdence interval of the estimate of the

generalisation error.

8.8.2.3 Cross-validation in practice

In Chapter 7 we focused on the theoretical properties of cross-validation and boot-

strap. Here we will see some more practical details on these validation proce-

dures, commonly grouped under the name of computer-intensive statistical methods

(CISM) [101].

Consider a learning problem with a training set of size N.

In l -fold cross-validation the available points are randomly divided into l mutu-

ally exclusive test partitions of approximately equal size. The examples not found

in each test partition are independently used for selecting the hypothesis, which

will be tested on the partition itself (Fig. 7.19). The average error over all the l

partitions is the cross-validated error rate.

A special case is the leave-one-out (l-o-o). For a given algorithm and a dataset

DN , an hypothesis is generated using N− 1 observations and tested on the single

remaining one. In leave-one-out, cross-validation is repeated l =N times, each data

point is used as a test case, and each time nearly all the examples are used to design

a hypothesis. The error estimate is the average over the N repetitions.

In a general nonlinear case, leave-one-out is computationally quite expensive.

This is not true for a linear model where the PRESS l-o-o statistic is computed as

a by-product of the least-squares regression (Section 9.1.17).

8.8.2.4 Bootstrap in practice

Bootstrap is a resampling technique that samples the training set with replacement

to return a nonparametric estimate of the desired statistic.

There are many bootstrap estimators, but two are the most commonly used in

model validation: the E0 and the E632 bootstrap.

The E0 bootstrap estimator, denoted by ˆ

Gb in (7.10.62), samples with replace-

ment from the original training set Bbootstrap training sets, each consisting of N

cases. The cases not found in the training group form the test groups. The average

of error rate on the Btest groups is the E0 estimate of the generalisation error.

The rationale for the E632 technique is given by Efron [63]. He argues that while

the resubstitution error R emp is the error rate for patterns that are "zero" distance

8.8. STRUCTURAL IDENTIFICATION 197

from the training set, patterns contributing to the E0 estimate can be considered as

too far out from the training set. Since the asymptotic probability that a pattern

will not be included in a bootstrap sample is

(1 − 1/N)N ≈e −1 ≈ 0 .368

the weighted average of R emp and E0 should involve patterns at the "right" distance

from the training set in estimating the error rate:

GE632 = 0 . 368 ∗ Remp + 0.632 ∗ ˆ

Gbs (8.8.30)

where the quantity ˆ

Gbs is deﬁned in (7.10.62). The choice of B is not critical as

long as it exceeds 100. Efron [63] suggests, however, that Bneed not be greater

than 200.

There a lot of experimental results on comparison between cross-validation and

bootstrap methods for assessing models [107], [116]. In general terms, only some

guidelines can be given to the practitioner [194]:

•For training set size greater than 100, use cross-validation; either 10-fold cross-

validation or leave-one-out is acceptable.

•For training set sizes less than 100, use leave-one-out.

•For very small training sets (N < 50), in addition to the leave-one-out esti-

mator, the ˆ

GE632 and the ˆ

Gboot estimates may be useful measures.

8.8.2.5 Complexity based criteria

In conventional statistics, various criteria have been developed, often in the context

of linear models, for assessing the generalisation performance of the learned hypoth-

esis without the use of further validation data. Such criteria aim to understand the

relationship between the generalisation performance and the training error. Gen-

erally, they take the form of a prediction error, which consists of the sum of two

terms ˆ

GPE = R emp + complexity term (8.8.31)

where the complexity (or capacity) term represents a penalty that grows as the

number of free parameters in the model grows.

This expression quantiﬁes the qualitative consideration that simple models re-

turn high empirical risk with a reduced complexity term while complex models have

a low empirical risk thanks to the high number of parameters. The minimum for

the criterion (8.8.31) represents a trade-oﬀ between performance on the training set

and complexity. Note that the bound (7.6.33) derived from the Vapnik learning

theory agrees with the relation (8.8.31) if we take the functional risk as a measure

of generalisation.

Let us consider a quadratic loss function and the quantity

MISEemp = Remp (αN ) = min

αP N

i=1(y i −h(x i , α))2

If the input/output relation is linear and nis the number of input variables, well-

known examples of complexity based criteria are:

1. the Final Prediction Error (FPE) (see Section 9.1.16.2 and [6])

FPE = \

MISEemp

1 + p/N

1−p/N (8.8.32)

with p =n + 1,

198 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

2. the Predicted Squared Error (PSE) (see Section 9.1.16.2)

PSE = \

MISEemp + 2ˆ σ2

N(8.8.33)

where ˆ σ2

wis an estimate of the noise variance. This quantity is also known as

the Mallows' Cp statistics [130]

3. the Generalised Cross-Validation (GCV) [48]

GCV = \

MISEemp

(1 − p

N) 2 (8.8.34)

A comparative analysis of these diﬀerent measures is reported in [14].

These estimates are computed assuming a linear model underlying the data.

Moody [133] introduced the Generalised Prediction Error (GPE) as an estimate of

the prediction risk for generic biased nonlinear models. The algebraic form is:

GPE = \

MISEemp + 2

Ntr ˆ

Vˆ

R(8.8.35)

where tr(·) denotes the trace, ˆ

Vis a nonlinear generalisation of the estimated noise

covariance matrix of the target and ˆ

Ris the estimated generalised inﬂuence matrix.

GPE can be expressed in an equivalent form as:

GPE = \

MISEemp + 2ˆ σeﬀ

ˆ peﬀ

N(8.8.36)

where ˆ peﬀ = tr ˆ

Ris the estimated eﬀective number of model parameters, and

ˆ σeﬀ =tr ˆ

Vˆ

tr ˆ

Ris the estimated eﬀective noise variance in the data. For nonlinear

models, ˆ peﬀ is generally not equal to the number of parameters (e.g. number of

weights in a neural network). When the noise in the target variables is assumed

to be independent with uniform variance and the squared error loss function is

used, (8.8.36) simpliﬁes to:

GPE = \

MISEemp + 2ˆ σ2

ˆ peﬀ

N(8.8.37)

In neural network literature another well-known form of complexity-based criterion

is the weight decay technique

U( λ, α, DN ) =

i=1

(yi − h (xi , α ))2 + λg (α ) (8.8.38)

where the second term penalizes either small, medium or large weights of the neurons

depending on the form of g ( · ). Two common examples of weight decay functions

are the ridge regression form g (α ) = α2 which penalizes large weights, and the

Rumelhart form g (α ) = α 2

α0 + α2 which penalizes weights of intermediate values near

α0 .

Several roughness penalties likeZ [h"(x)]2 dx

have been proposed too. Their aim is penalising hypothesis functions that vary too

rapidly by controlling large values of the second derivative of h.

Another important method for model validation is the minimum-description

length principle proposed by Rissanen [161]. This method proposes to choose the

model which induces the shortest description for the data available. Rissanen and

Barron [14] have each shown a qualitative similarity between this principle and the

complexity based approaches. For further details refer to the cited works.

8.8. STRUCTURAL IDENTIFICATION 199

8.8.2.6 A comparison of validation methods

Computer intensive statistical methods are relatively new and must be measured

against more established statistical methods, as the complexity based criteria. In

the following we summarize some practical arguments on behalf of one or the other

method. The beneﬁts of a CISM method are:

•All the assumptions of prior knowledge on the process underlying the data

are discarded.

•The validation technique replaces theoretical analysis by computation.

•Results are generally much easier to grasp for non-theorist.

•No assumption on the statistical properties of noise is required.

•They return an estimate of the model precision and an interval of conﬁdence.

Arguments on behalf of complexity criteria are:

•The whole dataset can be used for estimating the prediction performance and

no partitioning is required.

•Results valid for linear models remain valid to the extent that the nonlin-

ear model can be approximated by some ﬁrst-order Taylor expansion in the

parameters.

Some results in literature show the relation existing between resampling and com-

plexity based methods. For example, an asymptotic relation between a kind of

cross-validation and the Akaike's measure was derived by Stone [177], under the

assumptions that the real model α∗ is contained in the class of hypothesis Λ and

that there is a unique minimum for the log-likelihood.

Here we will make the assumption that no a priori information about the correct

structure or the quasi-linearity of the process is available. This will lead us to con-

sider computer intensive methods as the preferred method to validate the learning

algorithms.

8.8.3 Model selection criteria

Model selection concerns the ﬁnal choice of the model structure in the set that

has been proposed by model generation and assessed by model validation. In real

problems, this choice is typically a subjective issue and is often the result of a

compromise between diﬀerent factors, like the quantitative measures, the personal

experience of the designer and the eﬀort required to implement a particular model

in practice.

Here we will reduce the subjectivity factors to zero, focusing only on a quan-

titative criterion of choice. This means that the structure selection procedure will

be based only on the indices returned by the methods of Section 8.8.2. We distin-

guish between two possible quantitative approaches: the winner-takes-all and the

combination of estimators approach.

8.8.3.1 The winner-takes-all approach

This approach chooses the model structure that minimises the generalisation error

according to one of the criteria described in Section 8.8.2.

Consider a set of candidate model structures Λs ,s = 1, . . . , S , and an associated

measure ˆ

G(Λs ) which quantiﬁes the generalisation error.

200 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

The winner-takes-all method simply picks the structure

¯ s= arg min ˆ

G(Λs ) (8.8.39)

that minimises the generalisation error. The model which is returned as ﬁnal out-

come of the learning process is then h (· , α¯ s

N).

From a practitioner perspective it may be useful to make explicit the entire

winner-takes-all procedure in terms of pseudo-code . Here below you will ﬁnd a

compact pseudo-code detailing the structural, parametric, validation and selection

steps in the case of a leave-one-out assessment .

1. for s = 1, . . . , S : (Structural loop)

•for j = 1, . . . , N

(a) Inner parametric identiﬁcation (for l-o-o):

αs

N−1= arg min

α∈ Λs X

i=1:N,i6= j

(yi − h (xi , α ))2

(b) ej = yj −h (xj , αs

N−1)

•\

MISELOO(s ) = 1

NP N

j=1 e 2

2. Model selection: ¯ s= arg mins=1,...,S \

MISELOO ( s)

3. Final parametric identiﬁcation:

α¯ s

N= arg min α∈Λ ˜ sPN

i=1(y i −h( x i , α))2

4. The output prediction model is h (· , α ¯ s

8.8.3.2 The combination of estimators approach

The winner-takes-all approach is intuitively the approach which should work the

best. However, recent results in machine learning [153] show that the performance

of the ﬁnal model can be improved not by choosing the model structure which is ex-

pected to predict the best but by creating a model whose output is the combination

of the output of models having diﬀerent structures.

The reason for that non intuitive result is that in reality any chosen hypothesis

h(· , αN ) is only an estimate of the real target (Figure 7.18) and, like any estimate,

is aﬀected by a bias and a variance term.

Section 5.10 presented some results on the combination of estimators. The

extension of these results to supervised learning is the idea which underlies the

ﬁrst results in combination [16] and that has led later to more enhanced forms of

averaging diﬀerent models.

Consider m diﬀerent models h ( ·, αj ) and assume they are unbiased and uncor-

related. By (5.10.32), (5.10.33) and (5.10.34) we have that the combined model

h( ·) = P m

j=1 1

ˆ vj h(·, α j )

j=1 1

ˆ vj

(8.8.40)

where ˆ vj is an estimation of the variance of h (· , αj ). This is an example of the

generalised ensemble method (GEM) [153].

More advanced applications of the combination principle to supervised learning

will be discussed in Chapter 11.

8.9. PARTITION OF DATASET IN TRAINING, VALIDATION AND TEST 201

8.9 Partition of dataset in training, validation and

test

The main challenge of machine learning consists of using a ﬁnite size dataset for i)

learning several predictors, ii) assessing them, iii) selecting the most promising one

and ﬁnally iv) returning it together with a reliable estimate of its generalisation

error.

Section 7.10 discussed the need of avoiding correlation between training and

validation examples. While the training set is used for parametric identiﬁcation, a

non-overlapping portion of the dataset (validation set) should be used to estimate

the generalisation error of model candidates.

The use of validation (or cross-validation) does not prevent, however, a risk of

overﬁtting inherent to the winner-take-all model selection. If we take the minimum

generalisation error ˆ

G(Λ¯ s) in (8.8.39) as the generalisation error of the winning

model, we have an optimistic estimation again. This is known as selection bias , i.e.

the bias that occurs when we make a selection in a stochastic setting and due to

the fact the expectation of minimum is lower than the minimum of expectations

(Appendix C.11).

Anested cross-validation strategy [40] is recommended to avoid such bias. If

we have enough observations (i.e. large N), the strategy consists in randomly

partitioning (e.g. 50%, 25%, 25%) the labelled dataset into three parts: a training

set, a validation set, and a test set. The test portion is supposed to be used for

the unbiased assessment of the generalisation error of the model ¯ sin (8.8.39). It is

important to use only this set to assess the generalisation accuracy of the chosen

model. For this reason, the test set should be carefully made inaccessible to the

learning process (and ideally forgotten) and considered only at the very end of the

data analysis. Any other use of the test-set during the analysis (e.g. before the ﬁnal

assessment) would "contaminate" the procedure and make it irreversibly biased.

Selection bias

A Monte Carlo illustration of the selection bias eﬀect in a univariate regression task

is proposed in the R script selectionbias.R. The script estimates the generalisa-

tion errors of a constant model (h1 ), a linear model (h2 ) and a third model which is

nothing more than the winner-takes-all of the twos in terms of leave-one-out valida-

tion. It appears that the winner-takes all model is not better than the best between

h1 and h2 : in other terms it has a generalisation error larger than the minimum

between h1 and h2 .

•

8.10 Evaluation of a regression model

Let us consider a test set of size N ts where Yts ={ y1 , . . . , yN ts } is the target and

Y={ ˆ y1 ,..., ˆ yNts }is the prediction returned by the learner. The canonical way to

assess a regression model by using a testing set is to measure the mean-squared-

error (8.8.28) (MSE):

MSE = P N ts

i=1(y i −ˆ yi )2

Nts

Let us suppose that the test of a learning algorithm returns a mean-squared-

error of 0.4. Is that good or bad? Is that impressive and/or convincing? How may

we have a rapid and intuitive measure of the quality of a regression model?

202 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

A recommended way is to compare the learner to a baseline, e.g. the simplest

(or naive) predictor we could design. This is the rationale of the Normalised Mean-

Squared-Error measure which normalises the accuracy of the learner with respect

to the accuracy of the average predictor, i.e. the simplest predictor we could learn

from data. Then

NMSE = P N ts

i=1(y i −ˆ yi )2

PN ts

i=1(y i −¯ y)2 (8.10.41)

where

¯ y=P Nts

i=1 y i

Nts

(8.10.42)

is the prediction returned by an average predictor. NMSE is then the ratio between

the MSE of the learner and the MSE of the baseline naive predictor (8.10.42).

As for the MSE, the lower the NMSE the better. At the same time, we should

target a NMSE (signiﬁcantly) lower than one, if we wish to claim that the complex

learning procedure is eﬀective. NMSE values close to (yet smaller than) one are

either indicators of a bad learning design or, more probably, of a high noise to signal

ratio (e.g. large σ 2

win (8.5.1)) which makes any learning eﬀort ineﬀective.

Our recommendation is always to measure the NMSE of a regression model

before making too enthusiastic claims about the success of the learning procedure.

A very small NMSE could be irrelevant if not signiﬁcantly smaller than what we

could obtain by a simple naive predictor.

Another common way to assess the MSE of a predictor is to normalise it with

respect to the MSE of the same learning algorithm, yet trained on a randomly

shuﬄed version of the training set. For instance it is enough to shuﬄe the training

target to cancel any dependency between inputs and outputs. Again in this case

we expect that the NMSE is much lower than one. Otherwise any claim that our

prediction is better than a random one would be unfounded.

8.11 Evaluation of a binary classiﬁer

The most popular measure of performance is error rate or misclassiﬁcation rate,

i.e. the proportion of test examples misclassiﬁed by the rule. However, misclassiﬁ-

cation error is not necessarily the most appropriate criterion in real settings since

it implicitly assumes that the costs of diﬀerent types of misclassiﬁcation are equal.

When there are only a few or a moderate number of classes, the confusion matrix is

the most complete way of summarising the classiﬁer performance. In the following,

we will focus on evaluating a binary classiﬁer.

Suppose to use the classiﬁer to make Ntest classiﬁcations and that among the

values to be predicted there are NP examples of class 1 and NN examples of class

0. The confusion matrix is

Negative (0) Positive (1)

Classiﬁed as negative TN FN ˆ

Classiﬁed as positive FP TP ˆ

NNNP N

where FP is the number of False Positives and FN is the number of False Negatives.

The confusion matrix contains all the relevant information to assess the general-

isation capability of a binary classiﬁer. From its values it is possible to derive a

number of commonly used error rates or measures of accuracy. For instance the

misclassiﬁcation error rate is

ER = F P +F N

N(8.11.43)

8.11. EVALUATION OF A BINARY CLASSIFIER 203

8.11.1 Balanced Error Rate

In a setting where the two classes are not balanced the misclassiﬁcation error

rate (8.11.43) can lead to a too optimistic interpretation of the rate of success.

For instance if NP = 90 and NN = 10, a naive classiﬁer returning always the

positive class would have a misclassiﬁcation ER = 0. 1 since FN = 0 and FP = 10.

This low value of misclassiﬁcation gives a false sense of accuracy since humans tend

to associate a 50% error to random classiﬁers. This is true in balanced settings

while in an unbalanced setting (as the one above) this generalisation performance

may be obtained with a trivial classiﬁer making no use of the input information.

In these cases, it is preferable to adopt the balanced error rate which is the

average of the errors on each class:

BER = 1

2 F P

TN +FP

+F N

FN + TP 

Note that in the example above BER= 0. 5, normalising the misclassiﬁcation error

rate to a value correctly interpretable by humans.

8.11.2 Speciﬁcity and sensitivity

In many research works on classiﬁcation, it is common usage to assess the classiﬁer

in terms of sensitivity and speciﬁcity.

Sensitivity is a synonymous of True Positive Rate (TPR)

SE = TPR = T P

TP +FN

=TP

=N P −FN

= 1 −F N

,0≤ SE ≤ 1 (8.11.44)

It is a quantity to be maximised, and it increases by reducing the number of

false negatives. Note that it is also often called the recall in information retrieval.

Speciﬁcity stands for the True Negative Rate (TNR)

SP = TNR = T N

FP +TN

=TN

=N N −FP

= 1 − F P

,0≤ SP ≤1

It is a quantity to be maximised and it increases by reducing the number of false

positive.

In other terms, sensitivity is the proportion of positive examples classiﬁed as

positive while speciﬁcity is the proportion of negative examples classiﬁed as negative.

There exists a trade-oﬀ between these two quantities. This is the reason why

both quantities have to be calculated to have a thorough assessment of the classiﬁer

accuracy. In fact, it is trivial to maximise one of those quantities at the detriment

of the other.

For instance, for a naive classiﬁer who return always 0 we have ˆ

NP = 0, ˆ

NN = N,

FP = 0, TN = NN . This means that a naive classiﬁer may attain maximal speciﬁcity

(SP = 1) but at the cost of minimal sensitivity (SE = 0).

Analogously in the case of a naive classiﬁer who returns always 1 we have ˆ

NP =

N,ˆ

NN = 0, FN = 0, TP = NP , i.e. maximal sensitivity ( SE = 1) but null speciﬁcity

(SP = 0).

8.11.3 Additional assessment quantities

Other commonly used quantities which can be derived by the confusion matrix are

•False Positive Rate:

FPR=1-SP = 1 − T N

FP +TN

=FP

FP + TN

=FP

,0≤ FPR ≤1

It decreases by reducing the number of false positive.

204 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

•False Negative Rate:

FNR = 1-SE = 1 − T P

TP + F N

=F N

TP +FN

=F N

0≤ FNR ≤1

It decreases by reducing the number of false negatives.

•Positive Predictive value: the ratio (to be maximised)

PPV = T P

TP +FP

=TP

,0≤ PPV ≤ 1 (8.11.45)

This quantity is also called precision in information retrieval.

•Negative Predictive value: the ratio (to be maximised)

PNV = T N

TN +FN

=TN

,0≤ PNV ≤1

•False Discovery Rate: the ratio (to be minimised)

FDR = F P

TP +FP

=FP

= 1 − PPV, 0≤ FDR ≤1

8.11.4 Receiver Operating Characteristic curve

All the assessment measures discussed so far make the assumption that the classiﬁer

returns a class for each test point. However, since most binary classiﬁers compute an

estimation of conditional probability, a class may be returned as outcome provided

a threshold on the conditional probability. In other terms, the confusion matrix,

as well as its derived measured depends on a speciﬁc threshold. The choice of a

threshold is related to Type I error and Type II errors (Section 5.13) that we are

ready to accept in a stochastic setting.

In order to avoid conditioning our assessment on a speciﬁc threshold, it is in-

teresting to assess the overall accuracy for all possible thresholds. This is possible

by plotting curves, like the Receiver Operating Characteristic (ROC) which plots

the true positive rate (i.e. sensitivity or power) against the false positive rate (1-

speciﬁcity) for diﬀerent classiﬁcation thresholds.

In other terms, ROC visualises the probability of detection vs. the probability

of false alarm. Diﬀerent points on the curve correspond to diﬀerent thresholds used

in the classiﬁer.

The ideal ROC curves would follow the two axes. In practice, real-life classiﬁ-

cation rules produce ROC curves which lie between these two extremes. It can be

show that a classiﬁer with a ROC curve following the bissetrix line would be useless.

For each threshold, we would have TP/NP =FP /NN , i.e. the same proportion of

true positives and false positives. In other terms, this classiﬁer would not separate

the classes at all.

A common way to summarise a ROC curve is to compute the area under the

curve (AUC). By measuring the AUC of diﬀerent classiﬁers, we have a compact way

to compare classiﬁers without setting a speciﬁc threshold.

8.11.5 Precision-recall curves

Another commonly used curve to visualise the accuracy of a binary classiﬁer is

the precision-recall (PR) curve. This curve shows the relation between preci-

sion (8.11.45) (probability that an example is positive given that it has been clas-

siﬁed as positive) vs recall (8.11.44) (probability that an example is classiﬁed as

positive given that is positive).

8.11. EVALUATION OF A BINARY CLASSIFIER 205

Figure 8.5: ROC and PR curves of a binary classiﬁer.

Since precision is dependent on the probability a priori of the positive class, in

largely unbalanced problems (e.g. few positive classes like in fraud detection), the

PR curve is more informative than the AUC.

R script: visual assessment of a binary classiﬁer

The R script roc.R illustrates the assessment of a binary classiﬁer for a task where

x∈R, p( x |y = +) ∼ N (1 , 1) and p( x|y= −) ∼ N ( − 1 , 1). Suppose that the

classiﬁer categorises the examples as positive if t >Th and negative if t < Th, where

Th∈R is a threshold. Note that if Th= −∞ , all the examples are classed as positive:

TN =FN = 0 which implies SE =T P

NP = 1 and FPR = F P

FP + TN = 1. On the other

way round, if Th= ∞ , all the examples are classed as negative: TP =FP = 0 which

implies SE = 0 and FPR = 0.

By sweeping over all possible values of Th we obtain the ROC and the PR curves

in Figure 8.5. Each point on the ROC curve, associated to a speciﬁc threshold, has

an abscissa FPR =FP /NN and an ordinate TPR = TP/NP . Each point on the

PR curve, associated to a speciﬁc threshold, has an abscissa TPR = TP/NP and

an ordinate PR=T P / (T P + F P ).

•

Fraud detection example

Let us consider a fraud detection problem [52] with NP = 100 frauds out of

N= 2 · 106 transactions. Since one of the two classes (in this case the fraud)

is extremely rare, the binary classiﬁcation setting is called unbalanced [51]. Un-

balanced classiﬁcation settings are very common in real-world tasks (e.g. churn

detection, spam detection, predictive maintenance).

Suppose we want to compare two algorithms: the ﬁrst returns 100 alerts, 90 of

which are frauds. Its confusion matrix is

Genuine (0) Fraudulent (1)

Classiﬁed as genuine 1, 999, 890 10 1, 999,900

Classiﬁed as fraudulent 10 90 100

1, 999, 900 100 2 · 106

206 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

The second algorithm returns a much larger number of alerts (1000) 90 of which

are actual frauds. Its confusion matrix is then

Genuine (0) Fraudulent (1)

Classiﬁed as genuine 1, 998, 990 10 1, 999,000

Classiﬁed as fraudulent 910 90 1000

1,999, 900 100 2 · 106

Which of two algorithms is the best? In terms of TPR and FPR we have

1. TPR=TP/NP = 90/100 = 0. 9, FPR= FP/NN = 10 / 1 , 999 , 900 = 0 . 00000500025

2. TPR=90/ 100 = 0. 9, FPR=910/ 1,999, 900 = 0.00045502275

The FPR diﬀerence between the two algorithms is then negligible. Nevertheless,

though the recalls of the two algorithms are almost identical, the ﬁrst algorithm is

deﬁnitely better in terms of false positives (much higher precision):

1. A1: PR=TP/ (TP + FP ) = 90/100 = 0. 9, recall=0.9

2. A2: PR=90/ 1000 = 0. 09, recall= 0.9

The example shows that, in strongly unbalanced settings, the performance of a

classiﬁcation algorithm may be highly sensitive to the adopted cost function.

•

8.12 Multi-class problems

So far we limited to consider binary classiﬁcation tasks. However, real-world clas-

siﬁcation tasks (e.g. in bioinformatics or image recognition) are often multi-class.

Some classiﬁcation strategies (detailed in following chapters) may be easily adapted

to the multi-class setting, like the Naive Bayes (Section 10.2.3.1) or the KNN clas-

siﬁer (Section 10.2.1.1).

Suppose, however, to have a binary classiﬁcation strategy that we want to use

in a multi-class context. There are three main strategies to extend binary classiﬁers

to handle multi-class tasks y ∈ {c1 , . . . , ck } .

1. One-versus-the rest (or one-versus-all, OVA): it builds for each class ck a

binary classiﬁer that separates this class from the rest. To predict the class

of a query point q, the outputs of the kclassiﬁers are considered. If there

is a unique class label which is consistent with the kpredictions, the query

point is labeled with such a class. Otherwise, one of the kclasses is selected

randomly.

2. Pairwise (or one-versus-one, OVO): it trains a classiﬁer for each pair of classes,

requiring in total the independent learning of k (k− 1)/ 2 binary classiﬁers. To

predict a query point class, the output of the k (k− 1)/ 2 classiﬁers is calculated

and a majority vote is considered. If there is a class which receives the largest

number of votes, the query point is assigned to such a class. Otherwise each

tie is broken randomly.

3. Coding: it ﬁrst encodes each class by a binary vector of size d, then it trains a

classiﬁer for each component of the vector. The aggregation of the outputs of

the d classiﬁers returns an output word , i.e. a binary of size d. Given a query

q, the output word is compared against the codeword of each class, and the

class having the smallest Hamming distance (the number of disagreements)

to the output word is returned.

8.13. CONCLUDING REMARKS 207

Suppose that we have a task with k = 8 output classes. According to the coding

strategy, d log2 8e = 3 binary classiﬁers can be used to handle this problem.

ˆ c1 ˆ c2 ˆ c3

c1 0 0 0

c2 0 0 1

c3 0 1 0

c4 0 1 1

c5 1 0 0

c6 1 0 1

c7 1 1 0

c8 1 1 1

The table columns denote the classiﬁers while the rows contain the coding of the

associated class. For instance, the ˆ c3 classiﬁer will i) encode the training points

labeled with the classes { c2 , c4, c6, c8 } as ones ii) encode all the remaining examples

as zeros and iii) learn the corresponding binary classiﬁer.

Note that, though each strategy requires the learning of more than a single

classiﬁer, the number of trained classiﬁers is not the same. Given k > 2 classes, the

number of classiﬁers trained for each method is mentioned here below.

•One-versus-the rest: k binary classiﬁers

•Pairwise: k (k −1)/ 2 binary classiﬁers

•Coding: dlog2 k ebinary classiﬁers where d·e denote the ceiling operator.

8.13 Concluding remarks

The chapter presented the most important steps to learn a model on the basis of a

ﬁnite set of input/output data. Though the entire procedure was globally depicted

as a waterfall process (Figure 8.6), it should be kept in mind that a learning pro-

cess, like any modelling eﬀort, is better represented by a spiral model characterised

by feedbacks, iterations and adjustments. An example is the identiﬁcation step,

composed of two nested loops, the inner one returning the parameters of a ﬁxed

structure, and the external one searching for the best conﬁguration.

The chapter focused on the core of the learning process which begins once the

data are in a tabular format. Nevertheless, it is worth reminding that the upstream

steps, sketched in Section 8.2-8.4 play a very important role as well. However, since

those steps are often domain and task-dependent, we considered them beyond the

scope of this book.

In the following chapters we will quit the general perspective and we will delve

into the speciﬁcities of the most known learning algorithms.

8.14 Exercises

1. Consider an input/output regression task where n = 1, E [y | x ] = sin(x ) and p (y| x ) ∼

N(sin(x ), 1). Let N = 100 be the size of the training set and consider a quadratic

loss function.

Let the class of hypothesis be h3 (x ) = α0 + P 3

m=1 α m x m .

1. Estimate the parameter by least-squares.

2. Estimate the parameter by gradient-based search and plot the evolution of the

training error as a function of the number of iterations. Show in the same

ﬁgure the least-squares error.

208 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

FORMULATION

PHENOMENON

EXPERIMENTAL

DESIGN

RAW DATA

PREPROCESSING

MODEL

GENERATION

PARAMETRIC

IDENTIFICATION

VALIDATION

MODEL

SELECTION

DATA

MODEL

PROBLEM

TEST ERROR

ASSESSMENT

TRAINING SET

VALIDATION SET

TEST SET

SELECTED

MODEL

ASSESSMENT

TR+VAL

TEST

DATASET

Figure 8.6: From the phenomenon to the predictive model: overview of the steps

constituting the modelling process.

8.14. EXERCISES 209

3. Plot the evolution of the gradient-based parameter estimations as a function of

the number of iterations. Show in the same ﬁgure the least-squares parameters.

4. Discuss the impact of the gradient-based learning rate on the training error

minimisation.

5. Estimate the parameter by Levenberg-Marquardt search and plot the evolution

of the training error as a function of the number of iterations. Show in the

same ﬁgure the least-squares error.

6. Plot the evolution of the Levenberg-Marquardt parameter estimations as a

function of the number of iterations. Show in the same ﬁgure the least-squares

parameters.

2. Consider an input/output regression task where n = 1, E [y|x ] = 3x +2 and p (y| x ) ∼

N(3x + 2, 1). Let N = 100 be the size of the training set and consider a quadratic

loss function. Consider an iterative gradient-descent procedure to minimize the

empirical error.

1. Show in a contour plot the evolution ˆ

β(τ) of the estimated parameter vector

for at least 3 diﬀerent learning rates.

2. Compute the least-squares solution and show the convergence of the iterated

procedure to the least-squares solution.

3. Let us consider the dependency where the conditional distribution of yis

y= sin(2πx1x2 x3 ) + w

where w∼N (0, σ2 ), x∈R3 has a 3D normal distribution with an identity covari-

ance matrix, N = 100 and σ= 0.25.

Consider the following families of learners:

1. constant model returning always zero

2. constant model h ( x ) = β0

3. linear model h (x ) = xTβ

4. K nearest neighbour for K = 1, 3, 5, 7 where the distance is Euclidean

Implement for each learner above a function:

learner<-function(Xtr,Ytr,Xts){

####

## Xtr [N,n] input training set

## Ytr [N,1] output training set

## Xts [Nts,n] input test set

return(Yhat)

}

which returns a vector [N ts ,1] of predictions for the given input test set.

By using Monte Carlo simulation (S= 100 runs) and by using a ﬁxed-input test set

of size Nts = 1000

•compute the average squared bias of all the learners,

•compute the average variance of all the learners,

•check the relation between squared bias, variance, noise variance and MSE

•deﬁne what is the best learner in terms of MSE,

•discuss the results.

4. The student should prove the following equality concerning the quantities deﬁned

in Section 8.11:

FPR = p

1−p

1− PPV

PPV (1 −FNR)

where p = Prob {y = +}.

Hint: use the Bayes theorem.

210 CHAPTER 8. THE MACHINE LEARNING PROCEDURE

Chapter 9

Linear approaches

The previous chapters distinguished between two types of supervised learning tasks

according to the type of output:

Regression when we predict quantitative outputs, e.g. real or integer numbers.

Predicting the weight of an animal on the basis of its age and height is an

example of regression problem.

Classiﬁcation (or pattern recognition) where we predict qualitative or cate-

gorical outputs which assume values in a ﬁnite set of classes (e.g. black, white

and red) where there is no explicit ordering. Qualitative variables are also

referred to as factors. Predicting the class of an email on the basis of English

words frequency is an example of classiﬁcation task.

This chapter will consider learning approaches to classiﬁcation and regression

where the hypothesis functions are linear combinations of the input variables.

9.1 Linear regression

Linear regression is a very old technique in statistics and traces back to the work

of Gauss.

9.1.1 The univariate linear model

The simplest regression model is the univariate linear regression model where the

input is supposed to be a scalar variable and the stochastic dependency between

input and output is described by

y=β0 +β1 x + w(9.1.1)

where x∈R is the regressor (or independent) variable, yis the measured response

(or dependent) variable, β0 is the intercept, β1 is the slope and wis called noise

or model error. We will assume that E [w ] = 0 and that its variance σ 2

wis inde-

pendent of the xvalue. The assumption of constant variance is often referred to as

homoscedasticity. From (9.1.1) we obtain

Prob {y =y| x} = Prob {w =y− β0 −β1 x} , E [y |x ] = f (x ) = β0 +β1 x

The function f (x ) = E [y |x ], also known as regression function, is a linear func-

tion in the parameters β0 and β1 (Figure 9.1). In the following we will intend as

linear model each input/output relationship which is linear in the parameters but

211

212 CHAPTER 9. LINEAR APPROACHES

Figure 9.1: Conditional distribution and regression function for stochastic linear

dependence

not necessarily in the dependent variables. This means that: i) any value of the

response variable yis described by a linear combination of a series of parameters

(regression slopes, intercept) and ii) no parameter appears as an exponent or is

multiplied or divided by another parameter. According to this deﬁnition of linear

model, then

•y= β0 +β1 xis a linear model

•y= β0 +β1 x2 is again a linear model. Simply by making the transformation

X= x2 , the dependency can be put in in the linear form (9.1.1).

•y= β0 xβ 1 can be studied as a linear model between Y= log( y) and X=

log(x ) thanks to the equality

log(y ) = β0 +β1 log(x )⇔Y =β0 +β1 X

•the relationship y =β0 +β1 β x

2is not linear since there is no way to linearize

it.

9.1.2 Least-squares estimation

Suppose that Npairs of observations (xi , yi ), i = 1, . . . , N are available. Let us

assume that data are generated by the following stochastic dependency

yi = β0 +β1 xi + wi , i = 1 , . . . , N (9.1.2)

where

1. the wi ∈R are i.i.d realisations of the r.v. whaving mean zero and constant

variance σ 2

w(homoscedasticity),

2. the xi ∈R are non random and observed with negligible error.

The unknown parameters (also known as regression coeﬃcients)β0 and β1 can be

estimated by the least-squares method. The method of least squares is designed to

provide

1. the estimations ˆ

β0 and ˆ

β1 of β0 and β1 , respectively

9.1. LINEAR REGRESSION 213

2. the ﬁtted values of the response y

ˆ yi =ˆ

β0 +ˆ

β1 xi , i = 1 , . . . , N

so that the residual sum of squares (which is Ntimes the empirical risk )

SSEemp =N· \

MISEemp =

i=1

(yi − ˆ yi )2=

i=1

(yi − ˆ

β0 −ˆ

β1 xi )2

is minimised. In other terms

{ˆ

β0 ,ˆ

β1 }= arg min

{b0,b1 }

i=1

(yi − b0 − b1 xi )2

It can be shown that the least-squares solution is

β1 = Sxy

Sxx

β0 = ¯ y−ˆ

β1 ¯ x

if Sxx 6 = 0, where

¯ x=PN

i=1 x i

N,¯ y=PN

i=1 y i

and

Sxy =

i=1

(xi − ¯ x) yi

Sxx =

i=1

(xi − ¯ x)2 =

i=1

(x 2

i−2x i ¯ x+ ¯ x2 ) =

i=1

(x 2

i−x i ¯ x− xi ¯ x+ ¯ x2 ) =

i=1

[(xi − ¯ x)xi ] +

i=1

[¯ x(¯ x− xi )] =

i=1

(xi − ¯ x)xi

It is worth noting that if ¯ x= 0 and ¯ y= 0 then ˆ

β0 = 0 and

Sxy = h X, Y i , Sxx = h X, X i (9.1.3)

where X and Y are the [N, 1] vectors of x and y observations, respectively, and the

inner product h·, ·i of two vectors is deﬁned in Appendix B.2.

Also it is possible to write down the relation between the least squares estimation

β1 and the sample correlation coeﬃcient (D.0.3):

ˆ ρ2 =ˆ

β1 Sxy

Syy

(9.1.4)

R script

The script lin uni.R computes and plots the least-squares solution for N = 100

observations generated according to the dependency (9.1.2) where β0 = 2 and β1 =

−2.

•

214 CHAPTER 9. LINEAR APPROACHES

If the dependency underlying the data is linear then the estimators are unbiased.

We show this property for ˆ

β1 :

ED N [ˆ

β1 ] = ED N  S xy

Sxx  =

i=1

(xi − ¯ x) E[yi ]

Sxx

i=1

(xi − ¯ x)( β0 +β1 xi ) =

Sxx " N

i=1

[(xi − ¯ x) β0 ] +

i=1

[(xi − ¯ x) β1 xi ]# = β 1 Sxx

Sxx

=β1

Note that the analytical derivation relies on the relation P N

i=1(x i −¯ x) = 0 and the

fact that xis not a random variable. Also it can be shown [139] that

Var h ˆ

β1 i =σ 2

Sxx

(9.1.5)

E[ˆ

β0 ] = β0 (9.1.6)

Var h ˆ

β0 i =σ2

w1

N+¯ x2

Sxx  (9.1.7)

Another important result in linear regression is that the quantity

ˆ σ2

w=P N

i=1(y i −ˆ yi )2

N−2(9.1.8)

is an unbiased estimator of σ 2

wunder the (strong) assumption that observations

have been generated according to (9.1.1). The denominator is often referred to as

the residual degrees of freedom, also denoted by df. The degree of freedom can be

be seen as the number Nof observations reduced by the numbers of parameters

estimated (slope and intercept). The estimate of the variance σ 2

wcan be used in

Equations (9.1.7) and (9.1.5) to derive an estimation of the variance of the intercept

and slope, respectively.

9.1.3 Maximum likelihood estimation

The properties of least-squares estimators rely on the only assumption that the

wi = yi − β0 −β1 xi (9.1.9)

are i.i.d. realisations with mean zero and constant variance σ 2

w. Therefore, no

assumption is made concerning the probability distribution of w(e.g. Gaussian or

uniform). On the contrary, if we want to use the maximum likelihood approach

(Section 5.8), we have to deﬁne the distribution of w. Suppose that w ∼ N (0, σ2

w).

By using (9.1.9), the likelihood function can be written as

LN ( β0 , β1 ) =

i=1

pw ( wi ) = 1

(2π)N/2 σ N

exp ( − P N

i=1(y i −β 0 −β 1 x i ) 2

2σ2

w)(9.1.10)

It can be shown that the estimates of β0 and β1 obtained by maximising LN (·)

under the normal assumption are identical to the ones obtained by least squares

estimation.

9.1.4 Partitioning the variability

An interesting way of assessing the quality of a linear model is to evaluate which

part of the output variability the model is able to explain. We can use the following

relation

9.1. LINEAR REGRESSION 215

i=1

(yi − ¯ y)2=

i=1

(ˆ yi − ¯ y)2 +

i=1

(yi − ˆ yi )2

i.e.

SSTot =SSMod +S SRes

where SSTot (which is also Ntimes the sample variance of y) represents the total

variability of the response, SSMod is the variability explained by the model and

SSRes is the variability left unexplained. This partition helps to determine whether

the variation explained by the regression model is real or is no more than chance

variation. It will be used in the following section to perform hypothesis test on the

quantities estimated by the regression model.

9.1.5 Test of hypotheses on the regression model

Suppose that we want to answer the question whether the regression variable xtruly

inﬂuences the distribution Fy (· ) of the response yor, in other words, that they are

linearly dependent. We can formulate the problem as an hypothesis testing problem

on the slope β1 where

H: β1 = 0 ,¯

H: β1 6= 0

If H is true this means that the regressor variable does not inﬂuence the response

(at least not through a linear relationship). Rejection of Hin favor of ¯

Hleads to

the conclusion that xsigniﬁcantly inﬂuences the response in a linear fashion. It

can be shown that, under the assumption that w is normally distributed, if the null

hypothesis H (null correlation) is true then

SSMod

SSRes /( N− 2) ∼ F 1,N −2 .

Large values of the Fstatistic (Section C.2.4) provide evidence in favor of ¯

H(i.e. a

linear trend exists). The test is a two-sided test. In order to perform a single-sided

test, typically T -statistics are used.

9.1.5.1 The t-test

We want to test whether the value of the slope is equal to a predeﬁned value ¯

β:

H: β1 =¯

β, ¯

H: β1 6=¯

Under the assumption of normal distribution of w , the following relation holds

β1 ∼ N (β1 ,σ 2

Sxx

) (9.1.11)

It follows that

(ˆ

β1 −β1 )

ˆ σp Sxx ∼ TN−2

where ˆ σ2 is the estimation of the variance σ2

w. This is a typical t-test applied to

the regression case. Note that this statistic can be used also to test a one-sided

hypothesis, e.g.

H: β1 =¯

β, ¯

H: β1 >¯

216 CHAPTER 9. LINEAR APPROACHES

9.1.6 Interval of conﬁdence

Under the assumption of normal distribution, according to (9.1.11)

Prob ( − tα/2,N−2 < ( ˆ

β1 −β1 )

ˆ σp Sxx < tα/2,N −2 ) = 1 − α

where t α/2,N−2 is the upper α/2 critical point of the T -distribution with N− 2

degrees of freedom. Equivalently we can say that with probability 1 −α , the real

parameter β1 is covered by the interval described by

β1 ± tα/2,N−2 s ˆ σ2

Sxx

(9.1.12)

Note that the interval (9.1.12) may be used to test the hypothesis of input irrele-

vance. If the value 0 is outside the interval above, we can reject the input irrelevance

hypothesis with 100(1 −α )% conﬁdence.

Similarly from (9.1.7) we obtain that the 100(1 −α )% conﬁdence interval of β 0

β0 ± tα/2,N−2 ˆ σs 1

N+¯ x2

Sxx

9.1.7 Variance of the response

Let

ˆy =ˆ

β0 +ˆ

β1 x

be the estimator of the regression function value in x. If the linear dependence (9.1.1)

holds, we have for an arbitrary x =x 0

E[ˆy|x0 ] = E [ ˆ

β0 ] + E [ ˆ

β1 ]x0 =β0 +β1 x0 =E [y|x0 ]

This means that the prediction ˆy is an unbiased estimator of the value of the

regression function in x0 . Under the assumption of normal distribution of w , the

variance of ˆy in x 0

Var [ˆy|x0 ] = σ 2

w1

N+( x 0 −¯ x)2

Sxx 

where ¯ x=PN

i=1 x i

N. This quantity measures how the prediction ˆy would vary if

repeated data collections from (9.1.1) and least-squares estimations were conducted.

R script

Let us consider a data set DN = {xi , yi }i=1,...,N where

yi = β0 +β1 xi + w i

where β0 and β1 are known and w ∼ N (0, σ2

w) with σ 2

wknown. The R script bv.R

may be used to:

•Study experimentally the bias and variance of the estimators ˆ

β0 ,ˆ

β1 and

σwhen data are generated according to the linear dependency (9.1.2) with

β0 =− 1, β1 = 1 and σw = 4.

•Compare the experimental values with the theoretical results.

•Study experimentally the bias and the variance of the response prediction.

•Compare the experimental results with the theoretical ones.

•

9.1. LINEAR REGRESSION 217

R script

Consider the medical dataset available in the R script medical.R. This script may be

used to: i) estimate the intercept and slope of the linear model ﬁtting the dataset,

ii) plot the ﬁtted linear model, iii) estimate the variance of the estimator of the

slope, iv) test the hypothesis β1 = 0, v) compute the conﬁdence interval of β1 and

compare your results with the output of the R command lm().

•

9.1.8 Coeﬃcient of determination

The coeﬃcient of determination, also known as R2 ,

R2 = SS Mod

SSTot

=PN

i=1(ˆ yi − ¯ y)2

i=1(y i −¯ y)2 = 1 − SSRes

SSTot

is often used as a measure of the ﬁt of the regression line.

This quantity, which satisﬁes the inequality 0 ≤R2 ≤ 1, represents the propor-

tion of variation in the response data that is explained by the model. The coeﬃcient

of determination is easy to interpret and can be understood by most experimenters

regardless of their training in statistics. However, it is a dangerous criterion for com-

parison of candidate models because any additional model terms (e.g. a quadratic

term) will decrease SSRes and thus increase R2 . In other terms R2 can be made

artiﬁcially high by a practice of overﬁtting (Section 7.7) since it is not merely the

quality of ﬁt which inﬂuences R2 .

9.1.9 Multiple linear dependence

Consider a linear relation between an independent vector x ∈ X ⊂ Rn and a

dependent random variable y ∈ Y ⊂ R

y=β0 +β1 x·1 +β2 x·2 +· ·· +βn x·n + w(9.1.13)

where w represents a random variable with mean zero and constant variance σ 2

Note that it is possible to establish a link between the partial regression coeﬃcients

βi and partial correlation terms (Section 3.8.3) showing that βi is related to the

conditional information of xi about y once ﬁxed all the other terms (ceteris paribus

eﬀect) [139].

In matrix notation1the equation 9.1.13 can be written as:

y=xT β + w(9.1.14)

where x stands for the [p× 1] vector x = [1, x·1 , x·2 , . . . , x·n ]T and p =n + 1 is the

total number of model parameters.

9.1.10 The multiple linear regression model

Consider N observations DN = {hxi , yi i :i = 1, . . . , N} generated according to the

stochastic dependence (9.1.14) where xi = [1, xi1 , . . . , xin ]T . We suppose that the

following multiple linear relation holds

Y= Xβ +W

1We use the notation x ·jto denote the jth variable of the non random vector x , while xi =

[1, xi1 , xi2 ,...,x in] T denotes the ith observation of the vector x. This extension of notation is

necessary when the input is not considered a random vector. In the generic case xj will be used

to denote the jth variable

218 CHAPTER 9. LINEAR APPROACHES

where Y is the [N× 1] response vector, Xis the [N× p ] data matrix , whose j th column

of X contains readings on the j th regressor, β is the [p× 1] vector of parameters

Y=









X= 





1x11 x 12 . . . x1n

1x21 x 22 . . . x2n

1xN1 xN2 . . . xN n





= 











β=





β0

β1

βn





W= 











Here wi are assumed uncorrelated, with mean zero and constant variance σ 2

w(ho-

mogeneous variance). Then Var [w1 ,...,wN ] = σ 2

wI N .

9.1.11 The least-squares solution

We seek the least-squares estimator ˆ

βsuch that

β= arg min

i=1

(yi − x T

ib) 2 = arg min

b(Y− Xb ) T (Y− Xb )(9.1.15)

where

SSEemp =N· \

MISEemp = (Y− Xb )T (Y− Xb ) =eT e (9.1.16)

is the residual sum of squares (which is Ntimes the empirical risk (7.2.8) with

quadratic loss) and

e= Y− Xb

the [N× 1] vector of residuals. The quantity SSEemp is quadratic function in the p

parameters. In order to minimise

(Y− X ˆ

β)T ( Y− Xˆ

β) = ˆ

βT XT Xˆ

β−ˆ

βT XT Y− YT Xˆ

β+ YT Y

the vector ˆ

βmust satisfy

∂

∂ˆ

β[( Y−Xˆ

β)T ( Y− Xˆ

β)] = − 2 XT ( Y− Xˆ

β) = 0 (9.1.17)

Assuming X is of full column rank, the second derivative

∂2

∂ˆ

β∂ ˆ

βT [( Y−Xˆ

β)T ( Y− Xˆ

β)] = 2 XT X

is deﬁnite positive and the SSEemp attains its minimum in the solution of the least-

squares normal equations

(XTX ) ˆ

β= XT Y

As a result ˆ

β= ( XT X)−1 XT Y= X† Y(9.1.18)

where the XT X matrix is a symmetric [p× p ] matrix (also known as Gram ma-

trix) and X† = (XT X ) −1 XT is called the pseudo-inverse of X since X† X = IN .

Note that the computation of ˆ

βrepresents the parametric identiﬁcation step of the

supervised learning procedure (Section 7.9) when the class of hypothesis is linear.

9.1. LINEAR REGRESSION 219

9.1.12 Least-squares and non full-rank conﬁgurations

A full-rank Xis required to ensure that the matrix XT X is invertible in (9.1.18).

However, for numerical reasons it is recommended that XT X is not only invertible

but also well-conditioned, or equivalently non ill-conditioned [2]. An ill-conditioned

matrix is an almost singular matrix: its inverse may contain very large entries

and sometimes numeric overﬂows. This means that small changes in the data may

cause large and unstable changes in the solution ˆ

β. Such sensibility of the solution

to the dataset should evoke in the attentive reader the notion of estimator variance

(Section 5.5). In fact, in the following sections we will show that the variance of

least-squares estimators is related to the inverse of XT X (e.g. Equation (9.1.19)).

But what to do in practice if Xis not full-rank (or rank-deﬁcient) or ill-

conditioned? A ﬁrst numerical ﬁx consists in computing the generalised QR de-

composition (Appendix B.4)

X= QR

where Q is an orthogonal [N, p0 ] matrix and Ris a [p0 , p ] upper-triangular matrix

of full row rank with p0 < p . Since RRT is invertible, the pseudo-inverse in (9.1.18)

can be written as X† =RT (RRT ) −1 QT (details in Section 2.8.1 of [2]). A second

solution consists in regularising the optimisation, i.e. constraining the optimisation

problem (9.1.15) by adding a term which penalises solutions ˆ

βwith a too large

norm. This leads to the ridge regression formulation which will be discussed in

Section 12.5.1.1. In more general terms, since non invertible or ill-conditioned con-

ﬁgurations are often due to highly correlated (multicollinear) or redundant inputs,

the use of feature selection strategies (Chapter 12) before the parametric identiﬁ-

cation step may be beneﬁcial.

9.1.13 Properties of least-squares estimators

Under the condition that the linear stochastic dependence (9.1.14) holds, it can be

shown [139] that:

•If E [w ] = 0 then the random vector ˆ

βis an unbiased estimator of β.

•The residual mean square estimator

σ2

w=(Y− X ˆ

β)T (Y− X ˆ

β)

N− p

is an unbiased estimator of the error variance σ 2

•If the wi are uncorrelated and have common variance, the variance-covariance

matrix of ˆ

βis given by

Var[ ˆ

β] = σ 2

w(X T X) −1(9.1.19)

It can also be shown (Gauss-Markov theorem) that the least-squares estima-

tion ˆ

βis the "best linear unbiased estimator" (BLUE) i.e. it has the lowest

variance among all linear unbiased estimators.

From the results above it is possible to derive the conﬁdence intervals of model

parameters, similarly to the univariate case discussed in Section 9.1.6.

R script

A list of the most important least-squares summary statistics is returned by the

summary of the R command lm. See for instance the script ls.R.

220 CHAPTER 9. LINEAR APPROACHES

summary(lm(Y~X))

Call:

lm(formula = Y ~ X)

Residuals:

Min 1Q Median 3Q Max

-0.40141 -0.14760 -0.02202 0.03001 0.43490

Coefficients:

Estimate Std. Error t value Pr(>|t|)

(Intercept) 1.09781 0.11748 9.345 6.26e-09

X 0.02196 0.01045 2.101 0.0479

(Intercept) ***

X *

---

Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 0.2167 on 21 degrees of freedom

Multiple R-Squared: 0.1737, Adjusted R-squared: 0.1343

F-statistic: 4.414 on 1 and 21 DF, p-value: 0.0479

•

9.1.14 Variance of the prediction

Since the estimator ˆ

βis unbiased, this is also the case for the the prediction ˆy =x T

0ˆ

for a generic input value x =x0 . Its variance is

Var [ˆy| x0 ] = σ 2

wx T

0(X T X) − 1 x 0 (9.1.20)

Assuming that w is normally distributed, the 100(1 −α )% conﬁdence bound for the

regression value in x0 is given by

ˆ y( x0 )± tα/2,N − p ˆ σw q xT

0(X T X) − 1 x 0

where t α/2,N− p is the upper α/ 2 percent point of the t-distribution with N− p

degrees of freedom and the quantity ˆ σw p xT

0(X T X) − 1 x 0 , obtained from (9.1.20),

is the standard error of prediction for multiple regression.

R script

The R script bv mult.R validates by Monte Carlo simulation the properties of least-

squares estimation mentioned in Section 9.1.11 and 9.1.14.

In order to assess the generality of the results, we invite the reader to run the

script for diﬀerent input sizes n, diﬀerent number of observations Nand diﬀerent

values of the parameter β.

•

9.1.15 The HAT matrix

The Hat matrix is deﬁned as

H= X( XTX)−1 XT (9.1.21)

9.1. LINEAR REGRESSION 221

It is a symmetric, idempotent [N× N ] matrix that transforms the output values Y

of the training set in the regression predictions ˆ

Y= Xˆ

β= X(XT X)−1 XT Y= HY

Using the above relation, the vector of residuals can be written as:

e= Y− Xˆ

β= Y− X(XT X)−1 XT Y= [ I− H]Y

and the residual sum of squares as

eT e= YT [ I− H ]2 Y= YT [ I− H ] Y= YT P Y (9.1.22)

where P is a [N× N ] matrix, called the projection matrix.

if X has full rank, by commutativity of the trace operator it follows that

tr(H ) = tr(X (XT X )−1 XT ) = tr(XTX (XT X ) −1 ) = tr(Ip ) = p(9.1.23)

If we perform a QR decomposition of X(Appendix B.4) then we obtain

H= X(XT X)−1 XT = QR( RT QTQR)−1 RT QT =QRR−1 ( RT )−1 RT QT =QQT

(9.1.24)

Note that, in this case the input matrix Xis replaced by the matrix Q which

contains an orthogonalised transformation of the original inputs.

9.1.16 Generalisation error of the linear model

Given a training dataset DN = {hxi , yi i :i = 1, . . . , N } and a query point x , it is

possible to return a linear prediction

ˆ y= h( x, α) = xT ˆ

where ˆ

βis returned by least-squares estimation (9.1.18). From an estimation per-

spective, the ˆ

βis a realisation of the random estimator ˆ

βfor the speciﬁc dataset

DN . But which precision can we expect from ˆ

y=xT ˆ

βif we average the pre-

diction error over all ﬁnite-size datasets DN that can be generated by the linear

dependency (9.1.13)?

A quantitative measure of the quality of the linear predictor on the whole domain

Xis the Mean Integrated Squared Error (MISE) deﬁned in (7.5.24). But how can

we estimate this quantity in the linear case? Also, is the empirical risk \

MISEemp

in (9.1.16) a reliable estimate of MISE?

9.1.16.1 The expected empirical error

This section derives analytically that the empirical risk \

MISEemp (deﬁned in (9.1.16))

is a biased estimator of the MISE generalisation error.

Let us ﬁrst compute the expectation of the residual sum of squares which is

equal to N times the empirical risk. According to (9.1.22) and Theorem 4.2 the

expectation can be written as

ED N [SSEemp ] = ED N [eTe ] = ED N [YT PY] = σ2

wtr(P ) + E [Y T ]P E[Y]

Since tr(ABC ) = tr(CAB)

tr(P ) = tr(I− H ) = N− tr(X(XT X ) −1 XT )

=N− tr(XT X (XT X )−1 ) = N− tr(Ip ) = N− p

222 CHAPTER 9. LINEAR APPROACHES

and we have

ED N [eTe ]=( N− p) σ 2

w+ (Xβ) T P(Xβ ) = (9.1.25)

= (N− p )σ 2

w+β T X T (I− X(X T X) −1X T)Xβ = (9.1.26)

= (N− p )σ 2

w(9.1.27)

It follows that

ED N [\

MISEemp ] = ED N  SSE emp

N = E D N  e T e

N = (1 − p/N )σ2

w(9.1.28)

is the expectation of the error made by a linear model trained on DN to predict the

value of the output in the same dataset DN .

In order to obtain the MISE term we derive analytically the expected sum of

squared errors of a linear model trained on DN and used to predict for the same

training inputs X a set of outputs Yts distributed according to the same linear

law (9.1.13) but independent of the training output Y.

ED N ,Y ts [(Y ts − X ˆ

β)T(Y ts −X ˆ

β)] =

=ED N ,Y ts [(Y ts − Xβ + Xβ − X ˆ

β)T (Y ts − Xβ + Xβ − X ˆ

β)] =

=ED N ,Y ts [(W ts + Xβ − X ˆ

β)T ( Wts + Xβ − X ˆ

β)] =

=Nσ2

w+E D N[(Xβ − X ˆ

β)T (Xβ − X ˆ

β)]

Since

Xβ − X ˆ

β= Xβ − X ( XTX )−1 XT Y =

=Xβ − X (XT X )−1 XT (Xβ +W ) = −X (XTX)−1 XT W

we obtain

Nσ2

w+E D N[(Xβ − X ˆ

β)T(Xβ − X ˆ

β)]

=Nσ2

w+E D N[(W T X(X T X) −1X T )(X(X T X) −1X TW)]

=Nσ2

w+E D N[W T X(X T X) −1X TW]

=Nσ2

w+σ 2

wtr(X(X T X) −1X T)

=Nσ2

w+σ 2

wtr(X T X(X T X) −1)

=Nσ2

w+σ 2

wtr(I p ) = σ 2

w(N+ p)

By dividing the above quantity by N

MISE = (1 + p/N )σ 2

w(9.1.29)

From (9.1.28) and (9.1.29) it follows that that the empirical error \

MISEemp is a

biased estimate of MISE:

ED N [ \

MISEemp ] = ED N  e T e

N = σ 2

w(1 −p/N) 6= MISE = σ 2

w(1 + p/N) (9.1.30)

As a consequence, if we replace \

MISEemp with

eTe

N+ 2 σ 2

N(9.1.31)

we correct the bias and we obtain an unbiased estimator of the MISE generalisation

error. Nevertheless, this estimator requires an estimate of the noise variance.

9.1. LINEAR REGRESSION 223

R script

The R script ee.R performs a Monte Carlo validation of (9.1.30).

•

Example

Let { y1 , . . . , yN } ← Fy be the training set. Consider the simplest linear predictor

of the output variable: the average ˆ

µy (i.e. p = 1). This means that

ˆ yi = P N

i=1 y i

N= ˆ µy , i = 1 , . . . , N

We want to show that, even for this simple estimator, the empirical error is a biased

estimator of the quality of this predictor. Let µbe the mean of the r.v. y . Let us

write yas

y=µ + w

where E [w ] = 0 and Var [w] = σ2 . Let {z1 , . . . , zN } ← Fy a test set coming from

the same distribution underlying DN . Let us compute the expected empirical error

and the mean integrated square error.

Since E [ ˆ

µy ] = µ and Var  ˆ

µy  =σ2 /N

N·MISE = ED N ,Y ts [

i=1

(zi − ˆ

µy )2 ] = ED N ,w [

i=1

(µ +wi − ˆ

µy )2]

=Nσ2 +

i=1

ED N [( µ− ˆ

µy )2 ]

=Nσ2 +N (σ2 /N ) = (N + 1)σ 2

Instead, since ˆ σy = (P N

i=1(y i −µ y ) 2 )/(N− 1) and E [ ˆ

σ2

y] = σ 2

ED N [

i=1

(yi − ˆ

µy )2 ] = ED N [(N− 1) ˆ

σ2

y]=( N−1) σ2 6= N·MISE

It follows that, even for a simple estimator like the estimator of the mean, the

empirical error is a biased estimate of the accuracy (see R ﬁle ee mean.R).

•

9.1.16.2 The PSE and the FPE

In the previous section we derived that \

MISEemp is a biased estimate of MISE and

that the addition of the correction term 2σ 2

wp/N makes it unbiased.

Suppose we have an estimate ˆ σ2

wof σ 2

w. By replacing it into the expres-

sion (9.1.31) we obtain the so-called Predicted Square Error (PSE) criterion

PSE = \

MISEemp + 2ˆ σ2

wp/N (9.1.32)

In particular, if we take as estimate of σ 2

wthe quantity

ˆ σ2

w=1

N− pSSE emp = N

N− p

MISEemp

we obtain the so-called Final Prediction Error (FPE)

FPE = 1 + p/N

1−p/N

MISEemp (9.1.33)

224 CHAPTER 9. LINEAR APPROACHES

Figure 9.2: Estimation of the generalisation error of hm ,m = 2,..., 7 returned by

the empirical error.

The PSE and the FPE criteria allow us to replace the empirical risk with a

more accurate estimate of the generalisation error of a linear model. Although their

expression is easy to compute, it is worth reminding that their derivation relies on

the assumption that the stochastic input/output dependence has the linear form

9.1.14.

R script

Let us consider an input/output dependence

y=f (x ) + w= 1 + x +x2 +x3 + w(9.1.34)

where w ∼ N (0, 1) and x ∼ U ( − 1, 1). Suppose that a dataset DN of N = 100

input/output observations is drawn from the joint distribution of hx, y i . The R

script fpe.R assesses the prediction accuracy of 7 diﬀerent models having the form

hm ( x ) = ˆ

β0 +

j=1

βj xj (9.1.35)

by using the empirical risk and the FPE measure. These results are compared with

the generalisation error measured by

MISEm = 1

i=1

(hm (xi )−f (xi ))2(9.1.36)

The empirical risk and the FPE values for m = 2,..., 7 are plotted in Figure 9.2

and 9.3, respectively. The values MISEmare plotted in Figure 9.4. It is evident, as

conﬁrmed by Figure 9.4, that the best model should be h3 (x ) since it has the same

analytical structure as f (x ). However, the empirical risk is not able to detect this

and returns as the best model the one with the highest complexity (m = 7). This

is not the case for FPE which, by properly correcting the \

MISEemp value, is able to

select the optimal model.

•

9.1. LINEAR REGRESSION 225

Figure 9.3: Estimation of the generalisation error of hm ,m = 2,..., 7 returned by

the FPE.

Figure 9.4: Computation of the generalisation error of hm ,m = 2,..., 7 by (9.1.36).

226 CHAPTER 9. LINEAR APPROACHES

Figure 9.5: Leave-one-out for linear models. The leave-one-out error can be com-

puted in two equivalent ways: the slowest way (on the right) which repeats Ntimes

the training and the test procedure; the fastest way (on the left) which performs

only once the parametric identiﬁcation and the computation of the PRESS statistic.

9.1.17 The PRESS statistic

Section 7.10.1 introduced cross-validation to provide a reliable estimate of the gener-

alisation error GN . The disadvantage of this approach is that it requires the training

process to be repeated l times, implying a large computational eﬀort. However, in

the linear case the PRESS (Prediction Sum of Squares) statistic [7] returns the leave-

one-out cross-validation error at a reduced computational cost (Fig. 9.5). PRESS

relies on a simple formula which returns the leave-one-out (l-o-o) as a by-product

of the parametric identiﬁcation of ˆ

βin Eq. (10.1.41). Consider a training set DN

in which for Ntimes

1. we set aside the ith observation (i = 1, . . . , N )hxi , yi i from the training set,

2. we use the remaining N− 1 observations to estimate the linear regression

coeﬃcients ˆ

β−i ,

3. we use ˆ

β−i to predict the target in xi .

The leave-one-out residual is

eloo

i=y i −ˆ y−i

i=y i −x T

iˆ

β−i , i = 1 , . . . , N (9.1.37)

The PRESS statistic is an eﬃcient way to compute the l-o-o residuals on the basis of

the simple regression performed on the whole training set. This allows a fast cross-

validation without repeating Ntimes the leave-one-out procedure. The PRESS

procedure can be described as follows:

1. use the whole training set to estimate the linear regression coeﬃcients ˆ

β. This

procedure is performed only once and returns as by product the Hat matrix

(see Section 9.1.15)

H= X(XT X)−1 XT (9.1.38)

2. compute the residual vector e, whose ith term is ei = yi − x T

iˆ

β,

3. use the PRESS statistic to compute e loo

ias

eloo

i=e i

1−Hii

(9.1.39)

9.1. LINEAR REGRESSION 227

where H ii is the i th diagonal term of the matrix H.

Note that (9.1.39) is not an approximation of (9.1.37) but simply a faster way of

computing the leave-one-out residual e loo

Let us now derive the formula of the PRESS statistic. Matrix manipulations

show that

XT X− xixT

i=X T

−i X −i (9.1.40)

where X T

−iX −i is the X T Xmatrix obtained by putting the i th row aside. Using

the relation (B.9.13) we have

(XT

−i X −i) −1 = (X T X−x i x T

i) −1 = (X T X) −1+(X T X) −1x i x T

i(X T X) −1

1−Hii

(9.1.41)

and

β−i = ( XT

−iX −i) −1X 0

−iy −i= (X T X) −1+(X T X) −1x i x T

i(X T X) −1

1−Hii  X T

−iy −i

(9.1.42)

where y−i is the target vector with the ith example set aside.

From (9.1.37) and (9.1.42) we have

eloo

i=y i −x T

iˆ

β−i

=yi − x T

i(X T X) −1+(X T X) −1x i x T

i(X T X) −1

1−Hii  X T

−iy −i

=yi − x T

i(X T X) −1X T

−iy −i−H ii x T

i(X T X) −1X T

−iy −i

1−Hii

=(1 −Hii ) yi −xT

i(X T X) −1X T

−iy −i

1−Hii

=(1 −Hii ) yi −xT

i(X T X) −1(X Ty−x i y i )

1−Hii

=(1 −Hii ) yi − ˆ yi +Hii yi

1−Hii

=y i − ˆ yi

1−Hii

=e i

1−Hii

(9.1.43)

where X T

−iy −i+x iy i=X T yand x T

i(X T X) −1X Ty= ˆ yi . Thus, the leave-one-out

estimate of the local mean integrated squared error is:

Gloo =1

i=1 y i −ˆ yi

1−Hii  2

(9.1.44)

Since from (9.1.23) the sum of the diagonal terms of the H matrix is p , the

average value of Hii is p/N . It follows that the PRESS may be approximated by

Gloo ≈ 1

i=1 y i −ˆ yi

1−p/N  2

which leads us to the GCV formula (8.8.34).

9.1.18 Dual linear formulation

Consider a linear regression problem with [N, n] input matrix Xand [N , 1] out-

put vector y. The conventional least-squares solution is the [p, 1] parameter vec-

tor (9.1.18) where p =n +1. This formulation is common in conventional statistical

settings where the number of observations is supposed to be much larger than the

number of variables.

228 CHAPTER 9. LINEAR APPROACHES

However, machine learning may be confronted with high-dimensional settings

where the ratio of observations to features is low: this would imply a very large

value of p and risks of ill-conditioning of the numerical solution. In this case it

is interesting to consider a dual formulation of the least-squares problem based

on (B.9.14). In this formulation

β= ( X0 X)−1 X0 y= X0X(X0 X)−2 X0 y

| {z }

=X0 α =

i=1

αi xi

where α is a [N, 1] vector and xi is the [n, 1] vector which represents the ith obser-

vation. It follows that if N << p, the dual formulation has less parameters than the

conventional one with advantages in terms of storage requirements and numerical

conditioning.

9.1.19 The weighted least-squares

The assumption of homogeneous variance of the noise wmade in Eq. (9.1.14) is often

violated in practical situations. Suppose we relax the assumption that Var(w ) =

σ2

wI N with I N the identity matrix and assume instead that there is a positive

deﬁnite matrix Vfor which Var(w ) = V . We may wish to consider

V= diag[ σ2

1, σ 2

2, . . . , σ 2

N] (9.1.45)

in which case we are assuming uncorrelated errors with error variances that vary

from observation to observation. As a result it would seem reasonable that the

estimator of β should take this into account by weighting the observations in some

way that allows for the diﬀerences in the precision of the results. Then the function

being minimised is no more (9.1.16) but depends on Vand is given by

(y− X ˆ

β)T V−1 ( y− Xˆ

β) (9.1.46)

The estimate of βis then

β= ( XT V−1 X)−1 XT V−1 y(9.1.47)

The corresponding estimator is called the generalized least-squares estimator and

has the following properties: i) it is unbiased, that is E [ ˆ

β] = β , ii) under the as-

sumption w ∼ N (0, V ) it is the minimum variance estimator among all the unbiased

estimators.

9.1.20 Recursive least-squares

In many analytics tasks, data records are not statically available but have to be

processed and analysed continuously rather than in batches. Examples are the data

streams generated from sensors (notably IoT), ﬁnancial, business intelligence or

adaptive control applications. In those cases it is useful not to restart from scratch

the model estimation but simply to update the model on the basis of the newly

collected data. One appealing feature of least-squares estimates is that they can be

updated at a lower cost than their batch counterpart.

Let us rewrite the least-squares estimator (9.1.18) for a training set of Nobser-

vations as: ˆ

β(N) = ( XT

(N ) X (N) ) − 1 X T

(N ) Y (N)

where the subscript (N) is added to denote the number of observations used for

the estimation. Suppose that a new data point hxN+1 , yN+1 i becomes available.

9.1. LINEAR REGRESSION 229

Instead of recomputing the estimate ˆ

β(N+1) by using all the N+ 1 available data

we want to derive ˆ

β(N+1) as an update of ˆ

β(N) . This problem is solved by the

so-called recursive least-squares (RLS) estimation [26].

If a single new example hxN+1 , yN+1 i , with xN+1 a [1, p] vector, is added to the

training set the X matrix acquires a new row and ˆ

β(N +1) can be written as:

β(N +1) =  X (N)

xN+1  T  X (N)

xN+1 ! −1  X (N)

xN+1  T  Y (N)

yN+1 

By deﬁning the [p, p] matrix

S(N) = ( X T

(N ) X (N) )

we have

S(N +1) = ( X T

(N+1)X (N+1) ) = h X T

(N )x T

N+1i X ( N)

xN+1 

= XT

(N )X (N)+x T

N+1x N+1 =S (N )+x T

N+1x N+1

(9.1.48)

Since  X (N)

xN+1  T  Y (N)

yN+1  = X T

(N ) Y (N)+x T

N+1y N+1

and

S(N) ˆ

β(N) = ( XT

(N ) X (N))h (X T

(N ) X (N) ) − 1 X T

(N ) Y (N)i=X T

(N ) Y (N)

we obtain

S(N+1) ˆ

β(N +1) = X (N)

xN+1  T  Y (N)

yN+1  = S (N) ˆ

β(N) + xT

N+1y N+1

= S(N +1) − x T

N+1x N+1  ˆ

β(N) + xT

N+1y N+1

=S(N +1) ˆ

β(N) − xT

N+1x N+1 ˆ

β(N) + xT

N+1y N+1

or equivalently

β(N +1) =ˆ

β(N) + S−1

(N +1)x T

N+1(y N+1 −x N+1 ˆ

β(N) ) (9.1.49)

9.1.20.1 1st Recursive formulation

From (9.1.48) and (9.1.49) we obtain the following recursive formulation











S(N +1) =S(N) +xT

N+1x N+1

γ(N +1) = S−1

(N+1)x T

N+1

e= yN+1 −xN+1 ˆ

β(N)

β(N +1) =ˆ

β(N) + γ(N+1) e

where the term ˆ

β(N +1) can be expressed as a function of the old estimate ˆ

β(N) and

the new observation hxN+1 , yN+1 i. This formulation requires the inversion of the

[p× p ] matrix S(N+1) . This operation is computationally expensive but, fortunately,

using a matrix inversion theorem, an incremental formula for S−1 can be found.

230 CHAPTER 9. LINEAR APPROACHES

9.1.20.2 2nd Recursive formulation

Once deﬁned

V(N) = S−1

(N ) = (X T

(N ) X (N) ) − 1

we have (S(N+1) ) −1 = (S(N) +x T

N+1x N+1 ) − 1 and

V(N+1) = V (N) − V (N) x T

(N+1)(I+ x N +1 V (N)x T

N+1) − 1 x N+1 V ( N)(9.1.50)

=V(N) − V (N) x T

N+1x N+1 V ( N)

1 + xN+1 V (N) x T

N+1

(9.1.51)

From (9.1.50) and (9.1.49) we obtain a second recursive formulation:











V(N+1) = V (N) −V (N)x T

N+1x N+1 V (N)

1+xN+1 V (N) x T

N+1

γ(N +1) = V(N+1) xT

N+1

e= yN+1 − xN+1 ˆ

β(N)

β(N +1) =ˆ

β(N) + γ(N+1) e

(9.1.52)

9.1.20.3 RLS initialisation

Both recursive formulations presented above require the initialisation values ˆ

β(0)

and V(0) . One way to avoid choosing these initial values is to collect the ﬁrst N

data points, to solve ˆ

β(N) and V(N) directly from

V(N) = ( X T

(N ) X (N) ) − 1

β(N) = V(N) X T

(N )Y (N)

and to start iterating from the N + 1th point. Otherwise, in case of a generic

initialisation ˆ

β(0) and V(0) we have the following relations

V(N) = (V(0) + XT

(N )X (N ) ) −1

β(N) = V(N) ( XT

(N ) Y (N)+V − 1

(0) ˆ

β(0) )

A common choice is to put

V(0) = aI, a > 0

Since V(0) represents the variance of the estimator to choose a very large ais equiv-

alent to consider the initial estimation of βas very uncertain. By setting aequal to

a large number the RLS algorithm will diverge very rapidly from the initialisation

β(0) . Therefore, we can force the RLS variance and parameters to be arbitrarily

close to the ordinary least-squares values, regardless of ˆ

β(0) .

In any case, in absence of further information, the initial value ˆ

β(0) is usually

put equal to a zero vector.

9.1.20.4 RLS with forgetting factor

In some adaptive conﬁgurations it can be useful not to give equal importance to all

the historical data but to assign higher weights to the most recent data (and then

to forget the oldest one). This may happen when the phenomenon underlying the

data is non stationary or when we want to approximate a nonlinear dependence by

using a linear model which is local in time. Both these situations are common in

adaptive control problems.

9.2. LINEAR APPROACHES TO CLASSIFICATION 231

Figure 9.6: RLS ﬁtting of a nonlinear function where the arrival order of data is

from left to right

RLS techniques can deal with these situations by a modiﬁcation of the formu-

lation (9.1.52) obtained by adding a forgetting factor µ < 1.











V(N +1) =1

µV ( N)− V (N) xT

N+1x N+1 V ( N)

1+xN+1 V (N) x T

N+1 

γ(N +1) = V(N+1) x T

N+1

e= yN+1 − xN+1 ˆ

β(N)

β(N+1) =ˆ

β(N) + γ(N+1) e

Note that: (i) the smaller µ, the higher the forgetting, (ii) for µ= 1 we have the

conventional RLS formulation.

R script

The R script lin rls.R implements the RLS ﬁtting of a nonlinear univariate

function. The simulation shows that the ﬁtting evolves as long as data xi , yi ,

i= 1 , . . . , N are collected. Note that the values xi , i= 1 , . . . , N are increas-

ingly ordered. This means that xis not random and that the oldest collected values

are the ones with the lowest xi .

The ﬁnal ﬁtting for a forgetting factor µ = 0. 9 is shown in Figure 9.6. Note that

the linear ﬁtting concerns only the rightmost points since the values on the left,

which are also the oldest ones, are forgotten.

•

9.2 Linear approaches to classiﬁcation

The methods presented so far deal with linear regression tasks. Those methods

may be easily extended to classiﬁcation once we consider that in a binary 0/1

classiﬁcation case the conditional expectation coincides with conditional probability:

E[y| x ]=1· Prob {y = 1| x}+ 0 ·Prob {y = 0| x}= Prob {y = 1| x}(9.2.53)

232 CHAPTER 9. LINEAR APPROACHES

In other words, by encoding the two classes in 0/ 1 values and estimating the con-

ditional expectation with regression techniques, we estimate as well the conditional

probability. Such value may be used to return the most probable class associated

to a query point x.

This section will present some additional strategies to learn linear boundaries

between classes. The ﬁrst strategy relies on modelling the class conditional densities

and derive from them the equation of the boundary region. The other strategies

aim to learn directly the equations of separating hyperplanes.

9.2.1 Linear discriminant analysis

Let x∈Rn denote a real valued random input vector and ya categorical random

output variable that takes values in the set {c1 , . . . , cK } such that

k=1

Prob {y = ck |x} = 1

A classiﬁer can be represented in terms of a set of Kdiscriminant functions

gk ( x ), k = 1 , . . . , K such that the classiﬁer applies the following decision rule [61]:

assigns a feature vector xto a class ˆ y( x) = ck if

k= arg max

jg j (x ) (9.2.54)

Section 7.3 showed that in the case of a zero-one loss function (Equation (7.3.13)),

the optimal classiﬁer corresponds to a maximum a posteriori discriminant function

gk ( x ) = Prob {y = ck | x }. This means that if we are able to deﬁne the K functions

gk (· ), k = 1 , . . . , K and we apply the classiﬁcation rule (9.2.54) to an input x , we

obtain a classiﬁer which is equivalent to the Bayes one.

The discriminant functions divide the feature space into K decision regions Dk ,

where a decision region Dk is a region of the input space Xwhere the discriminant

classiﬁer returns the class ck for each x∈ Dk . The regions are separated by decision

boundaries, i.e. surfaces in the domain of xwhere ties occur among the largest

discriminant functions.

Example

Consider a binary classiﬁcation problem where y can take values in {c1 , c2 } and

x∈R2 . Let g1 ( x )=3 x1 +x2 + 2 and g2 ( x)=2 x1 + 2 the two discriminant

functions associated to the class x1 and x2 , respectively. The classiﬁer will return

the class c1 if

3x1 +x2 + 2 > 2x1 + 2 ⇔x1 > −x 2

The decision regions D1 and D2 are depicted in Figure 9.7.

•

We can multiply all the discriminant functions by the same positive constant

or shift them by the same additive constant without inﬂuencing the decision [61].

More generally, if we replace every gk (z ) by f (gk (z )), where f (· ) is a monotonically

increasing function, the resulting classiﬁcation is unchanged.

For example in the case of a zero/one loss function, any of the following choices

gives identical classiﬁcation result:

gk ( x ) = Prob {y = ck | x}= p(x|y= ck ) P (y = ck )

k=1 p(x|y = c k )P(y = c k )(9.2.55)

gk ( x ) = p( x |y= ck ) P (y = ck ) (9.2.56)

gk ( x ) = ln p ( x|y= ck ) + ln P(y = ck ) (9.2.57)

9.2. LINEAR APPROACHES TO CLASSIFICATION 233

Figure 9.7: Decision boundary and decision regions for the binary discrimination

functions g1 (x )=3 x1 +x2 + 2 and g2 ( x)=2 x1 + 2

and returns a Bayes classiﬁer.

9.2.1.1 Discriminant functions in the Gaussian case

Let us consider a binary classiﬁcation task where the inverse conditional densities

are multivariate normal (Section 3.7), i.e. p (x =x|y =ck ) ∼ N (µk , Σk ) where

x∈Rn , µk is a [ n, 1] vector and Σkis a [ n, n] covariance matrix. Since

p(x = x |y= ck ) = 1

(√2π )n pdet(Σk ) exp  − 1

2(x− µk )T Σ −1

k(x− µ k )

from (9.2.57) we obtain

gk ( x ) = ln p( x |y= ck ) + ln P (y = ck ) (9.2.58)

=− 1

2(x− µk )T Σ −1

k(x− µ k )−n

2ln 2π− 1

2ln det(Σ k ) + ln P (y =ck )

(9.2.59)

If we make no assumptions about Σkthe discriminant function is quadratic. Now

let us consider a simpler case where all the distributions have the same diagonal

covariance matrix Σk =σ2 I where I is the [n, n] identity matrix. It follows that

det(Σk ) = σ2n , Σ−1

k= (1/σ 2 )I

are independent of k and can be ignored by the decision rule (9.2.54). From (9.2.58),

we obtain the simpler discriminant function

gk ( x ) = −k x− µ k k 2

2σ2 + ln P (y=ck )

=− (x− µk )T(x− µk )

2σ2 + ln P(y = ck )

=− 1

2σ2 [xT x−2 µT

kx+µ T

kµ k ] + ln P(y = c k )

However, since the quadratic term xT x is the same for all kthis is equivalent to a

linear discriminant function

gk ( x ) = w T

kx+w k0(9.2.60)

where wk is a [n, 1] vector

wk =1

σ2 µ k (9.2.61)

234 CHAPTER 9. LINEAR APPROACHES

and

wk0 =− 1

2σ2 µT

kµ k + ln P (y =c k ) (9.2.62)

In the two-classes problem, the decision boundary (i.e. the set of points where

g1 ( x ) = g2 ( x )) can be obtained by solving the identity

1x+w 10 =w T

2x+w 20 ⇔(w 1 −w 2 ) T x−(w 20 −w 10 )=0

We obtain a hyperplane having equation

wT ( x− x0 ) = 0 (9.2.63)

where

w= µ 1 − µ2

σ2

and

x0 =1

2(µ1 +µ2 )− σ 2

kµ1 −µ2 k2 ln Prob {y=c1 }

Prob {y = c2 } (µ1 −µ2 )

This can be veriﬁed by the fact that wT x0 = w20 −w10 . The equation (9.2.63)

deﬁnes a hyperplane through the point x0 and orthogonal to the vector w.

9.2.1.2 Uniform prior case

If the prior probabilities P (y = ck ) are identical for the Kclasses, then the term

ln P (y =ck ) is a constant that can be ignored. In this case, it can be shown that

the optimum decision rule is a minimum distance classiﬁer [61]. This means that

in order to classify an input x, it measures the Euclidean distance kx − µk k2 from x

to each of the Kmean vectors, and assign xto the category of the nearest mean. It

can be shown that for the more generic case Σk= Σ, the discriminant rule is based

on minimising the Mahalanobis distance

ˆ c( x ) = arg min

k(x− µ k ) T Σ −1(x− µ k ) (9.2.64)

R script

The R script discri.R considers a binary classiﬁcation task (c1 =red, c2 =green)

where x∈R2 and the inverse conditional distributions of the two classes are

N(µ1 , σ2 I ) and N(µ2 , σ2 I ), respectively. Suppose that the two a priori proba-

bilities are identical, that σ = 1, µ1 = [− 1,− 2]T and µ2 = [2, 5]T . The positions of

100 points randomly drawn from N (µ1 , σ2 I ), of 100 points drawn from N ( µ2 , σ2 I )

together with the optimal decision boundary computed by (9.2.63) are plotted in

Figure 9.8.

The R script discri2.R shows instead the limitations of the LDA approach

when the assumption of Gaussian unimodal class-conditional distributions is not

respected. Suppose that the two a priori probabilities are identical, but that the

class conditional distribution of the green class is a mixture of two Gaussians. The

positions of 1000 points randomly drawn from the two class-conditional distributions

together with the LDA decision boundary computed by (9.2.63) are plotted in Figure

9.9.

•

9.2. LINEAR APPROACHES TO CLASSIFICATION 235

Figure 9.8: Binary classiﬁcation problem: distribution of inputs and linear decision

boundary

Figure 9.9: Binary classiﬁcation problem where one class distribution is bimodal:

distribution of inputs and linear decision boundary. Since the classiﬁcation task is

not linearly separable the LDA classiﬁer performs poorly.

236 CHAPTER 9. LINEAR APPROACHES

Figure 9.10: Several hyperplanes separating the two classes (blue and red).

9.2.1.3 LDA parameter identiﬁcation

In a real setting, we do not have access to the quantities µk , Σ and Prob {y =ck }

to compute the boundary (9.2.63) Before applying the discrimination rule above,

we need to estimate those quantities from the dataset DN :

[

Prob {y = ck } =N k

N(9.2.65)

ˆ µk = P i:yi =ck x i

(9.2.66)

Σ = P K

k=1 P i: yi = ck (x i −ˆ µk )( xi − ˆ µk )T

N− K(9.2.67)

where Nk is the number of observations labeled with the class ck and (9.2.67) is

also known as pooled covariance [99].

9.2.2 Perceptrons

Consider a binary classiﬁcation task (Figure 9.10) where the two classes are denoted

by +1 and −1. The previous section presented a technique to separate input data

by a linear boundary by making assumptions on the class conditional densities and

their covariances. In a generic setting, however, the problem is ill-posed and there

are inﬁnitely many possible separating hyperplanes (Figure 9.10) characterized by

the equation

β0 + xT β= 0 (9.2.68)

If x∈R2 , this equation represents a line. In a generic case (x∈Rn ) some properties

hold for all hyperplanes

•Since for any two points x1 and x2 lying on the hyperplane we have

(x1 −x2 )T β = 0

9.2. LINEAR APPROACHES TO CLASSIFICATION 237

Figure 9.11: Bi-dimensional space (n = 2): vector β∗ normal to the hyperplane and

distance of a point from an hyperplane.

the vector normal to the hyperplane (Figure 9.11) is given by

β∗ = β

kβk

•The signed distance of a point x to the hyperplane (Figure 9.11) is called the

geometric margin and is given by

β∗T ( x− x0 ) = x T β−βxT

kβ k= 1

kβ k(xT β+ β0 )

Aperceptron is a classiﬁer that uses the sign of the linear combination h (x, ˆ

β) =

β0 +ˆ

βT xto perform classiﬁcation [98]. The class returned by a perceptron for a

given input xq is

(1 if ˆ

β0 + xT

qˆ

β=ˆ

β0 +Pn

j=1 x qj ˆ

βj >0

−1 if ˆ

β0 + xT

qˆ

β=ˆ

β0 +Pn

j=1 x qj ˆ

βj <0

In other terms the decision rule is given by

h( x ) = sgn( ˆ

β0 + xT ˆ

β) (9.2.69)

For all well classiﬁed points in the training set the following relation holds

γi = yi ( x T

iˆ

β+ˆ

β0 ) >0

where the quantity γi is called the functional margin of the pair hxi , yi i with respect

to the hyperplane (9.2.68). Misclassiﬁcations in the training set occur when

(y i = 1 but ˆ

β0 +ˆ

βT xi <0

yi =− 1 but ˆ

β0 +ˆ

βT xi >0⇔ y i (ˆ

β0 +ˆ

βT xi ) <0

The parametric identiﬁcation step of a perceptron learning procedure aims at

ﬁnding the values { ˆ

β, ˆ

β0 }that minimise the quantity

SSEemp ( ˆ

β, ˆ

β0 ) = − X

i∈M

yi ( x T

iˆ

β+ˆ

β0 )

238 CHAPTER 9. LINEAR APPROACHES

where M is the subset of misclassiﬁed points in the training set. Note that this

quantity is non-negative and proportional to the distance of the misclassiﬁed points

to the hyperplane. Since the gradients are

∂SSEemp ( ˆ

β, ˆ

β0 )

∂ˆ

β=−X

i∈M

yi xi ,∂ SSE emp ( ˆ

β, ˆ

β0 )

∂ˆ

β0

=−X

i∈M

a batch gradient descent minimisation procedure (Section 8.6.2.3) or the online

version (Section 8.6.3) can be adopted. This procedure is guaranteed to converge

provided there exists a hyperplane that correctly classiﬁes the data: this conﬁgura-

tion is called linearly separable.

Although the perceptron set the foundations for much of the following research

in machine learning, a number of problems with this algorithm have to be men-

tioned [98]:

•When the data are separable, there are many possible solutions, and which

one is found depends on the initialisation of the gradient method.

•When the data are not separable, the algorithm will not converge.

•Also for a separable problem the convergence of the gradient minimisation

can be very slow.

R script

The script hyperplane.R visualises the evolution of the separating hyperplane dur-

ing the perceptron learning procedure. We invite the reader to run the script for

diﬀerent number of points and diﬀerent data distributions (e.g. by changing the

mean and the variance of the 2D gaussians).

•

A possible solution to the separating hyperplane problem has been proposed by the

SVM technique.

9.2.3 Support vector machines

This technique relies on an optimisation approach to compute the separating hy-

perplane.

Let us deﬁne as geometric margin of a hyperplane with respect to a training

dataset the minimum of the geometric margin of the training points. Also, the

margin of a training set is the maximum geometric margin over all hyperplanes.

The hyperplane attaining such maximum is known as a maximal margin hyperplane.

The SVM approach [186] computes the maximal margin hyperplane for a train-

ing set. In other words, the SVM optimal separating hyperplane is the one which

separates the two classes by maximising the distance to the closest point from both

classes. This approach provides a unique solution to the separating hyperplane

problem and was shown to lead to good classiﬁcation performance on real data.

The search for the optimal hyperplane is modelled as the optimisation problem

max

β,β0

C(9.2.70)

subject to 1

kβ k y i ( x T

iβ+β 0 )≥Cfor i= 1 , . . . , N (9.2.71)

where the constraint ensures that all the points are at least a distance Cfrom the

decision boundary deﬁned by β and β0 . The SVM parametric identiﬁcation step

seeks the largest Cthat satisﬁes the constraints and the associated parameters.

9.2. LINEAR APPROACHES TO CLASSIFICATION 239

Since the hyperplane (9.2.68) is equivalent to the original hyperplane where the

parameters β0 and β have been multiplied by a constant, we can set kβ k = 1/C.

The maximisation problem can be reformulated in a minimisation form

min

β,β0

2kβk2 (9.2.72)

subject to yi (x T

iβ+β 0 )≥1 for i = 1, . . . , N (9.2.73)

where the constraints impose a margin around the linear decision of thickness 1/k βk .

This problem is a convex optimisation problem (Appendix ) where the primal La-

grangian is

LP ( β, β0 ) = 1

2kβk2 −

i=1

αi [ yi (xT

iβ+β 0 )−1] (9.2.74)

and αi ≥ 0 are the Lagrangian multipliers.

Setting the derivatives wrt β and β0 to zero we obtain:

β=

i=1

αiyi xi , 0 =

i=1

αiyi (9.2.75)

Substituting these in the primal form (9.2.74) we obtain

LD =

i=1

αi − 1

i=1

k=1

αiαk yiyk x T

ix k (9.2.76)

subject to αi ≥ 0.

The dual optimisation problem is now

max

i=1

αi − 1

i=1

k=1

αiαk yiyk x T

ix k = max

i=1

αi − 1

i=1

k=1

αiαk yi yk h xi , xk i

(9.2.77)

subject to 0 =

i=1

αi yi , (9.2.78)

αi ≥ 0 , i = 1 , . . . , N (9.2.79)

where hxi , xk i is the inner product of xi and xk .

Note that the problem formulation requires the computation of all the inner

products hxi , xk i, i = 1, . . . , N, k = 1, . . . , N . This boils down to the computation

of the Gram matrix

G= XXT (9.2.80)

It can be shown that the optimal solution must satisfy the Karush-Kuhn-Tucker

(KKT) condition

αi [yi ( x T

iβ+β 0 )−1] = 0, ∀i

The above condition means that we are in either of these two situations

1. yi (x T

iβ+β 0 ) = 1, i.e. the point is on the boundary of the margin, then α i >0

2. yi (x T

iβ+β 0 )>1, i.e. the point is not on the boundary of the margin, then

αi = 0

240 CHAPTER 9. LINEAR APPROACHES

The training points having an index isuch that αi > 0 are called the support vectors

Given the solution α and obtained βfrom (9.2.75), the term β0 is obtained by

β0 =− 1

2[βx∗ (1) + βx∗ ( −1)]

where we denote by x∗ (1) some (any) support vector belonging to the ﬁrst class

and we denote by x∗ (− 1) a support vector belonging to the second class.

Now, the decision function can be written as

h( x, β, β0 ) = sign[ xT β+ β0 ]

or equivalently

h( x, β, β0 ) = sign[ X

support vectors

yi αi h xixi+ β0 ] (9.2.81)

This is an attractive property of support vector machines: the classiﬁer can

be expressed as a function of a limited number of points of the training set, the

so called support vectors which are on the boundaries. This means that in SVM

all the points far from the class boundary do not play a major role, unlike the

linear discriminant rule where the mean and the variance of the class distributions

determine the separating hyperplane (see Equation (9.2.63)). It can be shown, also,

that in the separable case

C=1

kβ k= 1

qP N

i=1 α i

(9.2.82)

R script

The R script svm.R considers a binary classiﬁcation problem. It generates sets of

separable data and builds a separating hyperplane by solving the problem (9.2.74).

The training points belonging to the two classes (in red and blue), the separating

hyperplane, the boundary of the margin and the support vectors (in black) are

plotted for each training set (see Figure 9.12 ).

•

A modiﬁcation of the formulation (9.2.70) occurs when we suppose that the

classes are nonlinearly separable. In this case the dual problem (9.2.77) is un-

bounded. The idea is still to maximise the margin but by allowing some points to

be misclassiﬁed. For each example hxi , yi i we deﬁne the slack variable ξi and we

relax the constraints (9.2.71) into

kβ ky i (xT

iβ+β 0 )≥C(1 −ξ i ) for i = 1, . . . , N (9.2.83)

ξi ≥0 (9.2.84)

i=1

ξi ≤ γ(9.2.85)

The value ξi represents the proportional amount by which the quantity yi (x T

iβ+

β0 ) can be lower than Cand the norm k ξkmeasures how much the training set fails

to have a margin C. Note that since misclassiﬁcations occur when ξi > 1, the upper

bound γ of P N

i=1 ξ i represents the maximum number of allowed misclassiﬁcations

9.2. LINEAR APPROACHES TO CLASSIFICATION 241

Figure 9.12: Maximal margin hyperplane for a binary classiﬁcation task with the

support vectors in black.

in the training set. It can be shown [98] that the maximisation (9.2.70) with the

above constraints can be put in the equivalent quadratic form

max

i=1

αi − 1

i=1

k=1

αiαkyiyk x T

ix k (9.2.86)

subject to 0 =

i=1

αiyi , (9.2.87)

0≤αi ≤γ, i = 1, . . . , N (9.2.88)

The decision function takes again the form (9.2.81) where β0 is chosen so that

yi h( xi ) = 1 for any i such that 0 < αi< γ. The geometric margin takes the value

C= N

k=1

αi αk yi yk x T

ix k ! −1/ 2

(9.2.89)

Note that the set of points for which the corresponding slack variables satisfy

ξi >0 are also the points for which αi = γ.

R script

The R script svm.R solves a non separable problem by setting the boolean variable

separable to FALSE. Figure 9.13 plots: the training points belonging to the two

classes (in red and blue), the separating hyperplane, the boundary of the margin,

the support vectors (in black), the points of the red class for which the slack variable

is positive (in yellow) and the points of the blue class for which the slack variable

is positive (in green).

•

Once the value γis ﬁxed, the parametric identiﬁcation in the SVM approach

boils down to a quadratic optimisation problem for which a large amount of meth-

ods and numerical software exists. The value γ plays the role of capacity hyper-

parameter which bounds the total proportional amount by which classiﬁcations fall

242 CHAPTER 9. LINEAR APPROACHES

Figure 9.13: Maximal margin hyperplane for a non separable binary classiﬁcation

task for diﬀerent values of C: support vectors are in black, the slack points of the

red class are in yellow and the slack points of the blue class are in green.

on the wrong side of the margin. In practice, the choice of this parameter requires a

structural identiﬁcation loop where the parameter γis varied through a wide range

of values and assessed through a validation strategy.

9.3 Conclusion

In this chapter we considered input/output regression problems where the rela-

tionship between input and output is linear and classiﬁcation problems where the

optimal decision boundaries are linear.

The advantage of linear models are numerous:

•the least-squares ˆ

βestimate can be expressed in an analytical form and can

be easily calculated through matrix computation.

•statistical properties of the estimator can be easily deﬁned.

•recursive formulation for sequential updating are available.

Unfortunately, in real problems, it is extremely unlikely that the input and output

variables are linked by a linear relation. Moreover, the form of the relationship is

often unknown, and only a limited amount of observations is available. For this

reason, machine learning proposed a number of nonlinear approaches to address

nonlinear tasks.

9.4 Exercises

1. Consider an input/output regression task where n = 1, E [y | x ] = sin(x ) and p (y| x ) ∼

N(sin(x ), 1). Let N = 100 be the size of the training set and consider a quadratic

loss function.

Let the class of hypothesis be hM (x ) = α0 + P M

m=1 α m x m .

1. Estimate the parameter by least-squares.

2. Compute the error by leave-one-out and by using the PRESS statistic.

3. Plot the empirical error as a function of the degree M for M = 0, 1,...,7.

4. Plot the leave-one-out error as a function of the degree M for M = 0, 1,..., 7.

9.4. EXERCISES 243

2. Consider a univariate linear regression problem. Write a R script which, using

Monte Carlo simulation, validates the formula (9.1.7) for at least three regression

tasks diﬀering in terms of

•parameters β0 , β1 ,

•variance σ2 ,

•number N of observations.

3. Consider a univariate linear regression problem. Write a R script which, using

Monte Carlo simulation, validates the formula (9.1.8) for at least three regression

tasks diﬀering in terms of

•parameters β0 , β1 ,

•variance σ2 ,

•number N of observations.

4. Consider a univariate linear regression problem. Write a R script which, using Monte

Carlo simulation, shows that the least-squares estimates of β0 and β1 minimise the

quantity (9.1.10) for at least three regression tasks diﬀering in terms of

•parameters β0 , β1 ,

•variance σ2 ,

•number N of observations.

5. Consider a regression task with input x and output y . Suppose we observe the

following training set

X Y

0 .1 1

0 0.5

-0.3 1.2

0.2 1

0.4 0.5

0.1 0

-1 1.1

1. Fit a linear model to the dataset.

2. Trace the data and the linear regression function on graph paper.

3. Are the two variables positively or negatively correlated?

Hint:

A= a 11 a12

a12 a22  ⇒ A −1 = 1

a11a22 −a2

12  a 22 −a 12

−a12 a11 

Solution:

1. Once we set X=







1 0.1

1 0

1− 0. 3

1 0.2

1 0.4

1 0.1

1−1







we have

X0 X= 7 .0− 0 .50

−0. 5 1. 31 

and

β= ( X0X)−1 X0 Y= 0 .725

−0. 456

244 CHAPTER 9. LINEAR APPROACHES

−1.0 − 0.8 −0.6 − 0.4 −0.2 0.0 0.2 0.4

0.0 0.2 0.4 0.6 0.8 1.0 1.2

3. Since ˆ

β1 <0 the two variables are negatively correlated.

6. Let us consider the dependency where the conditional distribution of yis

y= 1 −x +x2 −x3 + w

and w∼N (0, σ2 ) and σ = 0. 5. Suppose that x∈R takes the values seq (− 1,1,length. out = N)

(with N = 50).

Consider the family of regression models

h(m) ( x) = β0 +

j=1

βjxj

where p denote the number of weights of the polynomial model h(m) of degree m.

Let \

MISE(m)

emp denote the least-squares empirical risk and MISE the mean integrated

empirical risk. By using Monte Carlo simulation and for m = 0,...,6

•plot E [ \

MISE(m)

emp] as a function of p,

•plot MISE(m) as a function of p,

•plot the diﬀerence E [ \

MISE(m)

emp]−MISE (m) as a function of p and compare it

with the theoretical result seen during the class.

For a single observed dataset:

•plot \

MISE(m)

emp as a function of the number of model parameters p,

•plot PSE as a function of p,

•discuss the relation between

arg min

MISE(m)

emp

and

arg min

mPSE ( m)

Solution: See the ﬁle Exercise2.pdf in the directory gbcode/exercises of the

companion R package gbcode (Appendix F).

Chapter 10

Nonlinear approaches

This chapter will present several algorithms proposed in machine learning literature

to deal with nonlinear regression and nonlinear classiﬁcation tasks. Along the years

statisticians and machine learning researchers have proposed a number of nonlinear

approaches, with the aim of ﬁnding approximators able to combine high generalisa-

tion with eﬀective learning procedures. The presentation of these techniques could

be organised according to several criteria and principles. In this chapter, we will

focus on the distinction between global and divide-and-conquer approaches.

A family of models traditionally used in supervised learning is the family of

global models which describes the relationship between the input and the output

values as a single analytical function over the whole input domain (Fig. 10.1). In

general, this makes sense when it is reasonable to believe that a physical-like law

describes the data over the whole set of operating conditions. Examples of well-

known global parametric models in the literature are the linear models discussed in

the previous chapter, generalised linear models and neural networks which will be

presented in Section 10.1.1.

A nice property of global modelling is that, even for huge datasets, the storage of

a parametric model requires a small amount of memory. Moreover, the evaluation

of the model requires a short program that can be executed in a reduced amount

of time. These features have undoubtedly contributed to the success of the global

approach in years when most computing systems imposed severe limitations on

users.

However, for a generic global model, the parametric identiﬁcation (Section 7.2)

consists of a nonlinear optimisation problem (see Equation 7.2.7) which is not an-

alytically tractable due to the numerous local minima and for which only a sub-

optimal solution can be found through a slow iterative procedure. Similarly, the

problem of selecting the best model structure in a generic nonlinear case cannot be

handled in analytical form and requires time-consuming validation procedures.

For these reasons, alternatives to global modelling techniques, as the divide-

and-conquer approach, gained popularity in the modelling community. The divide-

and-conquer principle consists in attacking a complex problem by dividing it into

simpler problems whose solutions can be combined to yield a solution to the original

problem. This principle presents two main advantages. The ﬁrst is that simpler

problems can be solved with simpler estimation techniques: in statistical language,

this means to adopt linear techniques, well studied and developed over the years.

The second is that the learning method can better adjust to the properties of the

available dataset. Training data are rarely distributed uniformly in the input space.

Whenever the distribution of patterns in the input space is uneven, a proper local

adjustment of the learning algorithm can signiﬁcantly improve the overall perfor-

mance.

245

246 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.1: A global model (solid line) which ﬁts the training set (dotted points)

for a learning problem with one input variable (x-axis) and one output variable

(y-axis).

Figure 10.2: Function estimation (model induction + model evaluation) vs. value

estimation (direct prediction from data).

We will focus on two main instances of the divide-and-conquer principle: the

modular approach, which originated in the ﬁeld of system identiﬁcation, and the

local modelling approach, which was ﬁrst proposed in the nonparametric statistical

literature.

Modular architectures are input/output approximators composed of a number

of modules which cover diﬀerent regions of the input space. This is the idea of

operating regimes which propose a partitioning of the operating range of the system

as a more eﬀective way to solve modelling problems (Section 10.1.3).

Although these architectures are a modular combination of local models, their

learning procedure is still performed on the basis of the whole dataset. Hence,

learning in modular architectures remains a functional estimation problem, with the

advantage that the parametric identiﬁcation can be made simpler by the adoption

of local linear modules. However, in terms of structural identiﬁcation, the problem

is still nonlinear and requires the same procedures used for generic global models.

A second example of divide-and-conquer methods are local modelling techniques

(Section 10.1.11), which turn the problem of function estimation in a problem of

value estimation. The goal is not to model the whole statistical phenomenon but

to return the best output for a given test input hereafter called the query . The

motivation is simple: why should the problem of estimating the values of an un-

known function at given points of interest be solved in two stages? Global modelling

techniques ﬁrst estimate the function (induction) and second estimate the values

of the function using the estimated function (deduction). In this two-stage scheme

one actually tries to solve a relatively simple problem (estimating the values of a

function at given points of interest) by ﬁrst solving, as an intermediate problem, a

much more diﬃcult one (estimating the function).

Local modelling techniques take an alternative approach, deﬁned as transduc-

tion by Vapnik [186] (Fig. 10.2). They focus on approximating the function only

10.1. NONLINEAR REGRESSION 247

Figure 10.3: Local modelling of the input/output relationship between the input

variable x and the output variable y, on the basis of a ﬁnite set of observations

(dots). The value of the variable y for x =q is returned by a linear model (solid

line) which ﬁts the training points in a neighbourhood of the query point (bigger

dots).

in the neighbourhood of the point to be predicted. This approach requires to keep

in memory the dataset for each prediction, instead of discarding it as in the global

modelling case. At the same time, local modelling requires only simple approxi-

mators, e.g. constant and/or linear, to model the dataset in a neighbourhood of

the query point. An example of local linear modelling in the case of a single-input

single-output mapping is presented in Fig. 10.3.

Many names have been used in the past to label variations of the local modelling

approach: memory-based reasoning [174], case-based reasoning [121], local weighted

regression [44], nearest neighbor [47], just-in-time [49], lazy learning [5], exemplar-

based, instance based [4],... These approaches are also called nonparametric in the

literature [96, 170], since they relax the assumptions on the form of a regression

function, and let the data search for a suitable function that describes well the

available data.

In the following, we will present in detail some machine learning techniques for

nonlinear regression and classiﬁcation.

10.1 Nonlinear regression

A general way of representing the unknown input/output relation in a regression

setting is the regression plus noise form (7.4.21) where f (· ) is a deterministic func-

tion and the term wrepresents the noise or random error. It is typically assumed

that w is independent of x and E [w ] = 0. Suppose that we collect a training

set {hxi , yi i:i = 1, . . . , N } with xi = [xi1 , . . . , xin ]T , generated according to the

model (7.4.21). The goal of a learning procedure is to ﬁnd a model h (x ) which is

able to give a good approximation of the unknown function f (x ).

Example

Consider an input/output mapping represented by the Dopler function

f( x) = 20p x(1 − x) sin(2 π1 .05

x+ 0 .05 ) (10.1.1)

distorted by additive Gaussian noise wwith unit variance.

248 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.4: Training set obtained by sampling uniformly in the input domain of a

Dopler function distorted with Gaussian noise.

The training set is made of N= 2000 points obtained by sampling the input

domain X = [0. 12, 1] through a random uniform distribution (Fig. 10.4). This

stochastic dependency and the related training dataset (see R script dopler.R ) will

be used to assess the performance of the techniques we are going to present.

•

10.1.1 Artiﬁcial neural networks

Artiﬁcial neural networks (ANN) (aka neural nets ) are parallel, distributed infor-

mation processing computational models which draw their inspiration from neurons

in the brain. However, one of the most important trends in recent neural computing

has been to move away from a biologically inspired interpretation of neural networks

to a more rigorous and statistically founded interpretation based on results deriving

from statistical pattern recognition theory.

The main class of neural network used in supervised learning for classiﬁcation

and regression is the feed-forward network, aka as multi-layer perceptron (MLP).

Feed-forward ANNs (FNNs) have been applied to a wide range of prediction tasks

in such diverse ﬁelds as speech recognition, ﬁnancial prediction, image compression,

adaptive industrial control.

10.1.1.1 Feed-forward architecture

Feed-forward NN have a layered architecture, with each layer comprising one or

more simple processing units called artiﬁcial neurons or nodes (Figure 10.5). Each

node is connected to one or more other nodes by real-valued weights (in the following

we will refer to them as parameters) but not to nodes in the same layer. All FNN

have an input layer and an output layer. FNNs are generally implemented with an

additional node, called the bias1unit , in all layers except the output layer. This

1Note that this has nothing to do with the estimator bias concept. In neural network literature,

bias is used to denote the intercept term

10.1. NONLINEAR REGRESSION 249

Figure 10.5: Two-layer feed-forward NN

node plays the role of the intercept term β0 in linear models.

For simplicity, henceforth, we will consider only FNN with one single output.

Let

•nbe the number of inputs,

•Lthe number of layers,

•H(l) the number of hidden units of the lth layer (l = 1, . . . , L ) of the FNN,

•w(l)

kv denote the weight of the link connecting the kth node in the l− 1 layer

and the vth node in the l layer,

•z(l)

v,v= 1 , . . . , H ( l)the output of the vth hidden node of the lth layer,

•z(l)

0denote the bias for the l, l = 1, . . . , L layer.

•Let H (0) =n and z (0)

v=x v,v= 0 , . . . , n.

For l≥ 1 the output of the v th, v = 1, . . . , H (l) , hidden unit of the lth layer,

is obtained by ﬁrst forming a weighted linear combination of the H (l−1) outputs of

the lower level

a(l)

H(l−1)

k=1

w(l)

kv z ( l −1)

k+w ( l)

0vz (l− 1)

0, v = 1, . . . , H (l)

and then by transforming the sum using an activation function to give

z(l)

v=g ( l)(a (l )

v), v = 1 , . . . , H ( l)

The activation function g (l) (· ) is typically a nonlinear transformation like the

logistic or sigmoid function

g(l) ( z) = 1

1 + e −z (10.1.2)

250 CHAPTER 10. NONLINEAR APPROACHES

For L = 2 (i.e. single hidden layer or two-layer feed-forward NN), the in-

put/output relation is given by

ˆ y= h( x, αN ) = g (2) ( a(2)

1) = g (2) H

k=1

w(2)

k1z k +w (2)

01 z 0 !

where

zk = g (1) 



j=1

w(1)

jk x j +w (1)

0kx 0 

, k = 1, . . . , H

Note that if g (1) (· ) and g (2) (· ) are linear mappings, this functional form becomes

linear.

Once given the number of inputs and the form of the function g ( ·) two are the

parameters which remain to be chosen: the value of weights w (l) ,l = 1, 2 and the

number of hidden nodes H. Note that the set of weights of an FNN represents

the set of parameters αN introduced in Section 7.1 when the hypothesis function

h(· ) is modelled by a FNN. The calibration procedure of the weights on the basis

of a training dataset represents the parametric identiﬁcation procedure in neural

networks. This procedure is normally carried out by a back-propagation algorithm

which will be discussed in the following section.

The number Hof hidden nodes represents the complexity sin Equation (7.9.54).

By increasing the value H, we increase the class of input/output functions that can

be represented by the FNN. In other terms, the choice of the number of hidden

nodes aﬀects the representation power of the FNN approximator and constitutes

the structural identiﬁcation procedure in FNN (Section 10.1.1.3) .

10.1.1.2 Back-propagation

Back-propagation is an algorithm which, once the number of hidden nodes His

given, estimates the weights αN ={w (l) , l = 1, 2} on the basis of the training set

DN . It is a gradient-based algorithm which aims to minimise the non-convex cost

function

SSEemp(αN ) =

i=1

(yi − ˆ yi )2 =

i=1

(yi − h (xi , αN ))2

where αN ={w(l) , l = 1, 2} is the set of weights.

The back-propagation algorithm exploits the network structure and the diﬀeren-

tiable nature of the activation functions in order to compute the gradient recursively.

The simplest (and least eﬀective) back-prop algorithm is an iterative gradient

descent which is based on the iterative formula

αN ( k+ 1) = αN ( k)− η∂ SSE emp ( α N ( k ))

∂αN ( k)(10.1.3)

where αN (k ) is the weight vector at the k th iteration and η is the learning rate

which indicates the relative size of the change in weights.

The weights are initialised with random values and are changed in a direction

that will reduce the error. Some convergence criterion is used to terminate the

algorithm. This method is known to be ineﬃcient since many steps are needed to

reach a stationary point, and no monotone decrease of SSEemp is guaranteed. More

eﬀective versions of the algorithm are based on the Levenberg-Marquardt algorithm

(Section 8.6.2.6). Note that this algorithm presents all the typical drawbacks of the

gradient-based procedures discussed in Section 8.6.4, like slow convergence, local

minima convergence, sensitivity to the weights initialisation.

10.1. NONLINEAR REGRESSION 251

Figure 10.6: Single-input single-output neural network with one hidden layer, two

hidden nodes and no bias units.

In order to better illustrate how the derivatives are computed in (10.1.3), let us

consider a simple single-input (i.e. n= 1) single-output neural network with one

hidden layer, two hidden nodes and no bias units (Figure 10.6). Since

a1 ( x ) = w (2)

11 z 1 +w (2)

21 z 2

the FNN predictor takes the form

ˆ y( x) = h ( x, αN ) = g( a1 ( x)) = g( w(2)

11 z 1 +w (2)

21 z 2 ) = g (w (2)

11 g(w (1)

11 x) + w (2)

21 g(w (1)

12 x))

where αN = [w (1)

11 , w (1)

12 , w (2)

11 , w (2)

21 ] The backprop algorithm needs the derivatives of

SSEemp wrt to each weight w∈ αN . Since for each w∈ α N

∂SSEemp

∂w =−2

i=1

(yi − ˆ y( xi )) ∂ˆ y( xi )

∂w

and the terms (yi − ˆ y( xi )) are easy to be computed, we focus on ∂ˆ y

∂w .

As far as the weights {w (2)

11 , w (2)

21 }of the hidden/output layer are concerned, we

have

∂ˆ y( x)

∂w (2)

=∂g

∂a(2)

∂w (2)

=g0 (a (2)

1(x))z v (x) , v = 1 ,..., 2 (10.1.4)

where

g0 ( z) = e −z

(1 + e−z ) 2

As far as the weights {w (1)

11 , w (1)

12 }of the input/hidden layer

∂ˆ y( x)

∂w (1)

=∂g

∂a(2)

∂zv

∂a(1)

∂w (1)

=g0 (a(2)

1(x))w (2)

v1g 0 (a (1)

v(x))x(10.1.5)

252 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.7: Neural network ﬁtting with s= 2 hidden nodes. The red continuous

line represents the neural network estimation of the Dopler function.

where the term g0 ( a2

1(x)) has been already obtained during the computation of (10.1.4).

The computation of the derivatives with respect to the weights of the lower layers

relies on terms which have been used in the computation of the derivatives with

respect to the weights of the upper layers. In other terms, there is a sort of back-

propagation of numerical terms from the upper layer to the lower layers that justiﬁes

the name of the procedure.

R example

Tensorﬂow [1] is a open-source library developed by Google2 which had a great

success in recent years as ﬂexible environment for building and training of neural

network architectures. In particular this library provides automatic diﬀerentiation

functionalities to speed up the backpropagation implementation.

The script tf nn.R uses the R tensorflow package (wrapper over the python li-

brary) to compute the derivatives (10.1.5) and (10.1.4) for the network in Figure 10.6

and checks that the TensorFlow result coincides with the one derived analytically.

•

R example

The FNN learning algorithm for a single-hidden layer architecture is implemented

by the R library nnet . The script nnet.R shows the prediction accuracy for diﬀerent

number of hidden nodes (Figure 10.7 and Figure 10.8).

•

2https://www.tensorflow.org

10.1. NONLINEAR REGRESSION 253

Figure 10.8: Neural network ﬁtting with s = 7 hidden nodes. The continuous red

line represents the neural network estimation of the Dopler function.

10.1.1.3 Approximation properties

Let us consider a two-layer FNN with sigmoidal hidden units. This has proven to

be an important class of network for practical applications. It can be shown that

such networks can approximate arbitrarily well any functional (one-one or many-

one) continuous mapping from one ﬁnite-dimensional space to another, provided

the number H of hidden units is suﬃciently large. Note that although this result

is remarkable, it is of no practical use. No indication is given about the number of

hidden nodes to choose for a ﬁnite number of observations and a generic nonlinear

mapping.

In practice, the choice of the number of hidden nodes requires a structural identi-

ﬁcation procedure (Section 8.8) which assesses and compares several diﬀerent FNN

architectures before choosing the ones expected to be the closest to the optimum.

Cross-validation techniques or regularisation strategies based on complexity-based

criteria (Section 8.8.2.5) are commonly used for this purpose.

Example

This example presents the risk of overﬁtting when the structural identiﬁcation of

a neural network is carried out on the basis of the empirical risk and not on less

biased estimates of the generalisation error.

Consider a dataset DN = {xi , yi } ,i = 1, . . . , N where N = 50 and

x∈ N 

[0,0,0], 

100

010

001 





is a 3-dimensional vector. Suppose that y is linked to x by the input/output rela-

254 CHAPTER 10. NONLINEAR APPROACHES

tionship

y= x2

1+ 4 log(|x 2 |)+5 x3

where xi is the i th component of the vector x. Consider as non-linear model a single-

hidden-layer neural network (implemented by the R package nnet ) with s = 15

hidden neurons. We want to estimate the prediction accuracy on a new i.i.d dataset

of Nts = 50 examples. Let us train the neural network on the whole training set by

using the R script cv.R. The empirical prediction MISE error is

MISEemp = 1

i=1

(yi −h(xi , αN ))2 = 1. 6∗ 10−6

where αN is obtained by the parametric identiﬁcation step. However, if we test

h(· , αN ) on the test set we obtain

MISEts = 1

Nts

i=1

(yi − h (xi , αN ))2 = 22.41

This neural network is seriously overﬁtting the dataset. The empirical error is a

very bad estimate of the MISE.

We perform now a K-fold cross-validation in order to have a better estimate of

MISE, where K = 10. The K = 10 cross-validated estimate of MISE is

MISECV = 24.84

This ﬁgure is a much more reliable estimation of the prediction accuracy.

The leave-one-out estimate K =N = 50 is

MISEloo = 19.47

It follows that the cross-validated estimate could be used to select a more ap-

propriate number of hidden neurons.

•

10.1.2 From shallow to deep learning architectures

Until 2006, FNNs with more than two layers have been rarely used in literature

because of poor training and large generalisation errors. The common belief was

that the solutions returned by deep neural networks were worse solutions than the

ones obtained with shallower networks. This was mainly attributed to two aspects:

i) gradient-based training of deep supervised FNN gets stuck in local minima or

plateaus, and ii) the higher the number of layers in a neural network, the smaller

the impact of the back-propagation on the ﬁrst layers.

However, an incredible resurgence of the domain occurred from 2006 on when

some teams (notably the Bengio team in Montreal, the Hinton team in Toronto and

the Le Cun team in Facebook)3were able to show that some adaptation of the FNN

algorithm could bring remedy to the above-mentioned problems and lead to major

accuracy improvements with respect to other learning machines. In particular deep

architectures (containing up to hundreds of layers) showed a number of advantages:

•some highly nonlinear functions can be represented much more compactly

with deep architectures than with shallow ones,

3Yoshua Bengio, Geoﬀrey Hinton, and Yann LeCun were awarded the 2018 Turing Award,

known as the Nobel Prize of computing

10.1. NONLINEAR REGRESSION 255

•the XOR parity function for n -bit inputs4 can be coded by a feed-forward

neural network with O (log n ) hidden layers and O (n ), neurons, while a feed-

forward neural network with only one hidden layer needs an exponential num-

ber of the same neurons to perform the same task,

•DL allows automatic generation and extraction of new features in large di-

mensional tasks with spatial dependency and location invariance,

•DL allows easy management of datasets where inputs and outputs are stored

in tensors (multidimensional matrices),

•DL relies on the learning of successive layers of increasingly meaningful rep-

resentations of input data (layered representation learning) and is a powerful

automatic alternative to time-consuming human crafted feature engineering,

•portions of DL pre-trained networks may be reused for similar, yet diﬀerent,

tasks (transfer learning) or calibrated in online learning pipelines (continuous

learning),

•iterative gradient optimisation (Section 8.6.3) is a very eﬀective manner of

ingesting huge amount of data in large networks.

•new activation functions and weight-initialisation schemes (e.g. layer-wise

pretraining) improve the training process.

Also, new network architectures were proposed, like auto-encoders or convolu-

tional networks. An auto-encoder is a multi-input multi-output neural network that

maps its input to itself. It has a hidden layer that describes a code used to repre-

sent the input and is composed of two parts: an encoder function and a decoder

that produces a reconstruction. They can be used for dimensionality reduction or

compression (if the number of hidden nodes is smaller than the number of inputs).

Convolutional networks are biologically inspired architectures imitating the pro-

cessing of cortical cells. They are ideal for taking into consideration local and spatial

correlation and consist of a combination of convolution, pooling and normalisation

steps applied to inputs taking the generic form of tensors. The convolution phase

applies a number of ﬁlters with shared weights to the same image. It ensures trans-

lation invariance since the weights depend on spatial separation and not on absolute

positions. Pooling is a way to take large images and shrink them down while pre-

serving the most important information in them. This step allows the creation

of new features as the combination of previous level features. The normalisation

ensures that every negative value is set to zero.

Those works had such a major impact on theoretical and applied research that

nowadays, deep learning is a de-facto synonymous of the entire machine learning

domain and more generally of AI. This comeback has been supported by a number

of headlines in the news like the success of a deep learning solution in the ImageNet

Large-Scale Visual Recognition Competition (2012): (bringing down the state-of-

the-art error rate from 26.1% to 15.3%) or the DL program AlphaGo, developed

by the company DeepMind, beating the no.1 human GO player. Other impres-

sive applications of deep learning are near-human-level speech recognition, near-

human-level handwriting transcription, autonomous cars (e.g. traﬃc sign recog-

nition, pedestrian detection), image segmentation (e.g. face detection), analysis

of particle accelerator data in physics, prediction of mutation eﬀects in bioinfor-

matics and machine translation (LSTM model sequence-to-sequence relationships).

4This is a canonical challenge in classiﬁcation where the target function is a boolean function

whose value is one if and only if the n-dimensional input vector has an odd number of ones.

256 CHAPTER 10. NONLINEAR APPROACHES

The breakthroughs of DL in the AI community have been acknowledged by the

attribution of the 2019 ACM Turing prize to Bengio, Hinton and Le Cun.

The domain is so large and rich that the most honest recommendation of the

author is to refer the reader, for more details, to seminal books [85] and articles [122]

authored by the pioneers in this domain. Nevertheless, we would like to make a

number of pedagogical considerations about the role of deep learning with respect

to other learning machines:

•DL models are not faithful models of the brain.

•the astonishingly success of DL makes of it a privileged approach in recent

years, but deﬁnitely, it should not be considered a machine learning panacea.

•DL, like all machine learning techniques, relies on a number of hyper-parameters

which aﬀect its capacity, its bias/variance trade-oﬀ and the expected gener-

alisation power. The setting of those parameters has a major impact on the

generalisation power. An important factor of the recent success of deep net-

works learning is the eﬀective integration of computational strategies already

adopted in other learning approaches, like regularisation, averaging, resam-

pling.

•the success of DL, though fulgurant, if often restricted to some speciﬁc percep-

tual tasks, e.g. convolutional networks has been explicitly designed to process

data that come in the form of multiple arrays (1D for signals and sequences,

including language; 2D for images or audio spectrograms; and 3D for video

or volumetric images.)

•there is no evidence that representation learning is by default a better strat-

egy than feature engineering: it is surely less biased but very probably more

variant.

•DL is particularly successful in tasks where it is possible to collect (and label)

huge amounts of examples: nevertheless, there are still a number of chal-

lenging tasks where the number of examples is typically low or scarce (e.g.

bioinformatics or tie series forecasting),

•the success of DL has been ampliﬁed by the advent of fast parallel graphics

processing units (GPUs), tensor processing units (TPU) and related libraries

(e.g. TensorFlow, Keras, PyTorch) that are convenient to program and allow

researchers to train networks 10 or 20 times faster.

•any assumption of a priori superiority of DL over other techniques for a given

learning task is more often due to hype consideration than to a scientiﬁc

attitude that should instead rely on the validation of a number of alternative

strategies and the pondering of diﬀerent criteria (accuracy, computational

cost, energy consumption, interpretability).

Exercise

The script keras.regr.R compares a Keras [43] implementation of a DNN and a

Random Forest (Section 11.4) in a very simple nonlinear regression task where a

single input out of nis informative about the target. The default setting of the

DNN is very disappointing in terms of NMSE accuracy (8.10.41) with respect to

the Random Forest. We invite the reader to spend some time performing DNN

model selection (e.g. by changing the architecture, tuning the number of layers

and/or the number of nodes per layers) or increasing the number of training points

to bring the DNN accuracy closer to the RF one. Is that easy? Is that fast? What

is your opinion?

•

10.1. NONLINEAR REGRESSION 257

10.1.3 From global modelling to divide-and-conquer

Neural networks are a typical example of global modelling. Global models have

essentially two main properties. First, they make the assumption that the relation-

ship between the inputs and the output values can be described by an analytical

function over the whole input domain. Second, they solve the problem of learning

as a problem of function estimation: given a set of data, they extract the hypothesis

which is expected to approximate the best the whole data distribution (Chapter 7).

The divide-and-conquer paradigm originates from the idea of relaxing the global

modelling assumptions. It attacks a complex problem by dividing it into simpler

problems whose solutions can be combined to yield a solution to the original prob-

lem. This principle presents two main advantages. The ﬁrst is that simpler problems

can be solved with simpler estimation techniques; in statistics, this means to adopt

linear techniques, well studied and developed over the years. The second is that the

learning method can better adjust to the properties of the available dataset.

The divide-and-conquer idea evolved in two diﬀerent paradigms: the modular

architectures and the local modelling approach.

Modular techniques replace a global model with a modular architecture where

the modules cover diﬀerent parts of the input space. This is the idea of operating

regimes which assume a partitioning of the operating range of the system in order

to solve modelling and control problems [111]. The following sections will introduce

some examples of modular techniques.

10.1.4 Classiﬁcation and Regression Trees

The use of tree-based classiﬁcation and regression dates back to the work of Morgan

and Sonquist in 1963. Since then, methods of tree induction from samples have

been an active topic in the machine learning and the statistics community. In

machine learning the most representative methods of decision-tree induction are

the ID3 [158] and the C4 [159] algorithms. Similar techniques were introduced

in statistics by Breiman et al. [37], whose methodology is often referred to as the

CART (Classiﬁcation and Regression Trees) algorithm.

A decision tree (see Fig. 10.9) partitions the input space into mutually exclusive

regions, each of which is assigned a procedure to characterise its data points (see

Fig. 10.10)

The nodes of a decision tree can be classiﬁed in internal nodes and terminal

nodes. An internal node is a decision-making unit that evaluates a decision function

to determine which child node to visit next. A terminal node or leaf has no child

nodes and is associated with one of the partitions of the input space. Note that

each terminal node has a unique path that leads from the root to itself.

In classiﬁcation trees each terminal node contains a label that indicates the

class for the associated input region. In regression trees the terminal node con-

tains a model that speciﬁes the input/output mapping for the corresponding input

partition.

Hereafter we will focus only on the regression case. Let mbe the number of

leaves and hj (· , αj ) the input/output model associated with the j th leaf. Once a

prediction in a query point qis required, the output evaluation proceeds as follows.

First, the query is presented to the root node of the decision tree; according to the

associated decision function, the tree will branch to one of the root's children. The

procedure is iterated recursively until a leaf is reached, and an input/output model

is selected. The returned output will be the value hj (q, αj ).

Consider for example the regression trees in Fig. 10.9, and a query point

q= ( xq , yq ) so that xq < x1 and yq > y1. The predicted output will be

258 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.9: A binary decision tree.

Figure 10.10: Input space partitioning induced on the input space by the binary

tree in Fig. 10.9

10.1. NONLINEAR REGRESSION 259

yq = h2 ( q, α2 ) where α2 is the vector of parameters of the model localised in region

R2.

When the terminal nodes contain only constant models, the input/output map-

ping results in a combination of several constant-height planes put together with

crisp boundaries. In the case of linear terms, the resulting approximator is instead

a piecewise linear model.

10.1.4.1 Learning in Regression Trees

10.1.4.2 Parameter identiﬁcation

A regression tree partitions the input space into mutually exclusive regions. In terms

of parametric identiﬁcation, this requires a two-step procedure. First, the training

dataset is partitioned into m disjoint sets D Nj ; second, a local model hj (· , αj ) is

ﬁtted to each subset DNj . The nature of the local model determines the kind of

procedure (linear or nonlinear) to be adopted for the parameter identiﬁcation (see

Section 8.6).

R implementation

A regression tree with constant local models is implemented by the R library tree.

The script tree.R shows the prediction accuracy for diﬀerent minimum number of

observations per leaf. (Figure 10.11 and Figure 10.12).

•

10.1.4.3 Structural identiﬁcation

This section presents a summary of the CART procedure [37] for structural identiﬁ-

cation in binary regression trees. In this case the structural identiﬁcation procedure

addresses the problem of choosing the optimal partitioning of the input space.

To construct an appropriate decision tree, CART ﬁrst grows the tree on the

basis of the training set, and then prunes the tree back based on a minimum cost-

complexity principle. This is an example of the exploratory approach to model

generation described in Section 8.8.1.

Let us see in detail the two steps of the procedure:

Tree growing. CART makes a succession of splits that partition the training data

into disjoint subsets. Starting from the root node that contains the whole

dataset, an exhaustive search is performed to ﬁnd the split that best reduces

a certain cost function.

Let us consider a certain node t and let D (t ) be the corresponding subset of

the original DN . Consider the empirical error of the local model ﬁtting the

N( t) data contained in the node t:

Remp ( t) = min

αt

N( t)

i=1

L( yi , ht ( xi , αt )) (10.1.6)

For any possible split s of node tinto the two children tr and tl , we deﬁne the

quantity

∆E (s, t ) = Remp (t )− (R emp (tl ) + R emp (tr ))

with N ( tr ) + N (tl ) = N ( t ) (10.1.7)

260 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.11: Regression tree ﬁtting with a minimum number of points per leaf

equal to s = 7.

Figure 10.12: Regression tree ﬁtting with a minimum number of points per leaf

equal to s = 30.

10.1. NONLINEAR REGRESSION 261

that represents the change in the empirical error due to a further partition of

the dataset. The best split is the one that maximizes the decrease ∆E

s∗ = arg max

s∆E( s, t) (10.1.8)

Once the best split is attained, the dataset is partitioned into the two disjoint

subsets of length N (tr ) and N ( tl ), respectively. The same method is recur-

sively applied to all the leaves. The procedure terminates either when the

error measure associated with a node falls below a certain tolerance level, or

when the error reduction ∆Eresulting from further splitting does not exceed

a threshold value.

The tree that the growing procedure yields is typically too large and presents a

serious risk of overﬁtting the dataset (Section 7.7). For that reason, a pruning

procedure is often adopted.

Tree pruning. Consider a fully expanded tree Tmax characterized by Lterminal

nodes.

Let us introduce a complexity-based measure of the tree performance

Rλ ( T ) = Remp ( T ) + λ | T| (10.1.9)

where λ is a parameter that accounts for the tree's complexity and |T | is the

number of terminal nodes of the tree T. For a ﬁxed λwe deﬁne with T (λ)

the tree structure which minimizes the quantity (10.1.9).

The parameter λ is gradually increased in order to generate a sequence of tree

conﬁgurations with decreasing complexity

TL =Tmax ⊃TL−1 ⊃ ··· ⊃ T2 ⊃T1 (10.1.10)

where Ti has i terminal nodes. In practice, this requires a sequence of shrinking

steps where for each step we select the value of λleading from a tree to a tree

of inferior complexity. When we have a tree Tthe next inferior tree is found

by computing for each admissible subtree Tt ⊂ T the value λt which makes of

it the minimizer of (10.1.9). For a generic subtree Tt this value must satisfy

Rλ t (Tt )≤ R λ t ( T ) (10.1.11)

that is

Remp ( Tt ) + λt | Tt | ≤ Remp ( T) + λt | T|

which means

λt ≥ R emp ( T)−Remp (Tt )

|T |−| Tt | (10.1.12)

Hence, λt =R emp (T)−R emp (Tt )

|T|−|Tt | makes of T t the minimising tree. Therefore we

choose among all the admissible subtrees Tt the one with the smallest right-

hand term in Eq. (10.1.12). This implies a minimal increase in λ toward the

next minimising tree.

At the end of the shrinking process we have a sequence of candidate trees

that have to be properly assessed to perform the structural selection. As

far as validation is concerned, either a procedure of cross-validation or of

independent testing can be used. The ﬁnal structure is then obtained through

one of the selection procedures described in Section 8.8.3.

262 CHAPTER 10. NONLINEAR APPROACHES

Regression trees are a very easy-to-interpret representation of a nonlinear in-

put/output mapping. However, these methods are characterised by rough disconti-

nuity at the decision boundaries, which might bring undesired eﬀects to the overall

generalisation. Dividing the data by partitioning the input space shows typically

small estimator bias but at the cost of increased variance. This is particularly prob-

lematic in high-dimensional spaces where data become sparse. One response to the

problem is the adoption of simple local models (e.g. constant or linear). These

simple functions minimise the variance at the cost of an increased bias.

Another trick is to make use of soft splits, allowing data to lie simultaneously

in multiple regions. This is the approach taken by BFN.

10.1.5 Basis Function Networks

Basis Function Networks (BFN) are a family of modular architectures which are

described by a linear basis expansion, i.e. the weighted linear combination

j=1

ρj (x)hj (10.1.13)

where the weights are returned by the activations of m local nonlinear basis functions

ρj and where the term hj is the output of a generic module of the architecture.

The basis or activation function ρj is a function

ρj :X → [0 ,1] (10.1.14)

usually designed so that its value monotonically decreases towards zero as the input

point moves away from its centre cj .

The basis function idea arose almost at the same time in diﬀerent ﬁelds and

led to similar approaches, often denoted with diﬀerent names. Examples are the

Radial Basis Function in machine learning, the Local Model Networks in system

identiﬁcation and the Neuro-Fuzzy Inference Systems in fuzzy logic. These three

architectures are described in the following sections.

10.1.6 Radial Basis Functions

A well-known example of basis functions are the Radial Basis Functions (RBF) [156].

Each basis function in RBF takes the form of a kernel

ρj = K( x, cj, Bj ) (10.1.15)

where cj is the centre of the kernel and Bj is the bandwidth. An example of

kernel functions is illustrated in Fig. 10.13. Other examples of kernel functions are

available in Appendix E. Once we deﬁne with ηj the set {cj , Bj } of parameters of

the basis function, we have

ρj =ρj (· , ηj ) (10.1.16)

If the basis ρj have localised receptive ﬁelds and a limited degree of overlap

with their neighbours, the weights hj in Eq. (10.1.13) can be interpreted as locally

piecewise constant models, whose validity for a given input is indicated by the

corresponding activation function for a given input.

10.1.7 Local Model Networks

Local Model Networks (LMN) were ﬁrst introduced by Johansen and Foss [111].

They are a generalised form of Basis Function Network in the sense that the constant

10.1. NONLINEAR REGRESSION 263

Figure 10.13: A Gaussian kernel function in a two-dimensional input space.

weights hj associated with the basis functions are replaced by local models hj (· , αj ).

The typical form of a LMN is then

j=1

ρj ( x, ηj ) hj ( x, αj ) (10.1.17)

where the ρj are constrained to satisfy

j=1

ρj ( x, ηj ) = 1 ∀ x ∈ X (10.1.18)

This means that the basis functions form a partition of unity [137]. This ensures

that every point in the input space has equal weight, so that any variation in the

output over the input space is due only to the models hj .

The smooth combination provided by the LMN formalism enables the repre-

sentation of complex nonlinear mappings on the basis of simpler modules. See the

example in Fig. 10.14 which shows the combination in a two-dimensional input

space of three local linear models whose validity regions is represented by Gaussian

basis functions.

In general, the local models hj (· , α ) in Eq. (10.1.17) can be of any form: linear,

nonlinear, physical models or black-box parametric models.

Note that, in the case of local linear models

hj ( x, αj ) =

i=1

aji xi + bj (10.1.19)

where the vector of parameters of the local model is αj = [aj1 , . . . , ajn , bj ] and x i

is the i th term of the vector x , a LMN architecture returns one further information

about the input/output phenomenon: the local linear approximation hlin of the

input/output mapping about a generic point x

hlin ( x ) =

j=1

ρj ( x, ηj ) n

i=1

aji xi + bj ! (10.1.20)

10.1.8 Neuro-Fuzzy Inference Systems

Fuzzy modelling consists of describing relationships between variables by means of

if-then rules, such as

If x is high then y is low

264 CHAPTER 10. NONLINEAR APPROACHES

(a) (b)

(c)

Figure 10.14: A Local Model Network with m= 3 local models: the nonlinear

input/output approximator in (c) is obtained by combining the three local linear

models in (a) according to the three basis functions in (b).

10.1. NONLINEAR REGRESSION 265

where the linguistic terms, as high and low , are described by fuzzy sets [200].

The ﬁrst part of each rule is called the antecedent while the second part is called

the consequent. Depending on the particular form of the consequent proposition,

diﬀerent types of rule-based fuzzy models can be distinguished [11].

Here we will focus on the fuzzy architecture for nonlinear modelling introduced

by Takagi and Sugeno [178]. A Takagi-Sugeno (TS) fuzzy inference system is a set

of m fuzzy if-then rules having the form:











If x1 is A 11 and x2 is A 21 . . . and xn is An1 then y =h1 (x1 , x2 , ..., xn , α1 )

. . .

If x1 is A 1m and x2 is A 2m . . . and xn is A nm then y =hm (x1 , x2 , ..., xn , αm )

(10.1.21)

The antecedent is deﬁned as a fuzzy AND proposition where A kj is a fuzzy set on

the k th premise variable deﬁned by the membership function µkj : < → [0, 1]. The

consequent is a function hj ( ·, αj ), j = 1, . . . , m , of the input vector [x1 , x2, . . . , xn ].

By means of the fuzzy sets A kj , the input domain is softly partitioned into m

regions where the mapping is locally approximated by the models hj ( ·, αj ).

If the TS inference system uses the weighted mean criterion to combine the local

representations, the model output for a generic query xis computed as

y=Pm

j=1 µ j (x) h j (x, α j )

j=1 µ j (x)(10.1.22)

where µj is the degree of fulﬁlment of the j th rule, commonly obtained by

µj ( x ) =

k=1

µkj ( x )

This formulation makes of a TS fuzzy system a particular example of LMN

where

ρj ( x) = µ j ( x)

j=1 µ j (x)(10.1.23)

is the basis function and hj (· , αj ) is the local model of the LMN architecture.

In a conventional fuzzy approach, the membership functions and the consequent

models are ﬁxed by the model designer according to a priori knowledge. In many

cases, this knowledge is not available; however a set of input/output data has been

observed. Once we put the components of the fuzzy system (memberships and

consequent models) in a parametric form, the TS inference system becomes a para-

metric model which can be tuned by a learning procedure. In this case, the fuzzy

system turns into a Neuro-Fuzzy approximator [108]. For a thorough introduction

to Neuro-Fuzzy architecture see [109] and the references therein. Further work on

this subject was presented by the author in [21, 22, 29, 20].

10.1.9 Learning in Basis Function Networks

Given the strong similarities between the three instances of BFN discussed above,

our discussion on the BFN learning procedure does not distinguish between these

approaches.

The learning process in BFN is divided in structural (see Section 8.8) and para-

metric identiﬁcation (see Section 8.6). The structural identiﬁcation aims to ﬁnd

the optimal number and shape of the basis functions ρj ( · ). Once the structure of

the network is deﬁned, the parametric identiﬁcation searches for the optimal set of

parameters ηj of the basis functions (e.g. centre and width in the Gaussian case)

266 CHAPTER 10. NONLINEAR APPROACHES

and the optimal set of parameters αj of the local models (e.g. linear coeﬃcients in

the case of local linear models).

Hence, the classes of parameters to be identiﬁed are two: the parameters of the

basis function and the parameters of the local model.

10.1.9.1 Parametric identiﬁcation: basis functions

The relationship between the model output and the parameters ηj of the basis func-

tion is typically nonlinear, hence, methods for nonlinear optimisation are currently

employed. A typical approach consists in decomposing the identiﬁcation procedure

into two steps: ﬁrst, an initialisation step, which computes the initial location and

width of the basis functions, then a nonlinear optimisation procedure which uses

the outcome η (0)

jof the previous step as initial value.

Since the methods for nonlinear optimisation have already been discussed in

Section 8.6.2.2, here we will focus on the diﬀerent initialisation techniques for Basis

Function Networks.

One method for placing the centres of the basis functions is to locate them at the

interstices of some coarse lattice deﬁned over the input space [39]. If we assume the

lattice to be uniform with d divisions along each dimension, and the dimensionality

of the input space to be n, a uniform lattice requires dn basis functions. This

exponential growth makes the use of such a uniform lattice impractical for high

dimensional space.

Moody and Darken [134] suggested a K-means clustering procedure in the input

space to position the basis functions. The K-means method, described into detail in

Appendix A.2, takes as input the training set and returns mgroups of input vectors

each parameterized by a centre cj and a width σj . This method generally requires

a much smaller number of basis functions than the uniform partition, nevertheless

the basis location concerns only that part of the input space actually covered by

data. The assumption underlying this method is that similar inputs should produce

similar outputs and that these similar input pairs should be bundled together into

clusters in the training set. This assumption is reasonable but not necessarily

true in real problems. Therefore, the adoption of K-means clustering techniques

for supervised learning is essentially a heuristic technique and ﬁnding a dataset to

which this technique cannot be applied satisfactorily is not uncommon.

An alternative to K-means clustering for initialisation has been proposed in the

Neuro-Fuzzy literature [11, 12]. The initialisation of the architecture is provided by

ahyperellipsoidal fuzzy clustering procedure. This procedure clusters the data in

the input/output domain, obtaining a set of hyperellipsoids which are a preliminary

rough representation of the mapping. The parameters of the ellipsoids (eigenval-

ues) are used to initialise the parameters αj of the consequent models, while the

projection of their barycenters on the input domain determines the initial positions

of the membership functions (see Fig. 10.15).

10.1.9.2 Parametric identiﬁcation: local models

A common approach to the optimisation of the parameters αj of local models is the

least-squares method (see Eq. (8.6.2) and (8.6.4)).

If the local models are nonlinear, some nonlinear optimisation technique is re-

quired (Section 8.6.2.2). Such a procedure is typically computationally expensive

and does not guarantee the convergence to the global minimum.

However, in the case of local linear models (Eq. 10.1.19), the parametric identi-

ﬁcation can take advantage of linear techniques. Assume that the local models are

linear, i.e.

hj ( x, αj ) = hj ( x, βj ) = xT βj (10.1.24)

10.1. NONLINEAR REGRESSION 267

Figure 10.15: The hyperellipsoidal clustering initialisation procedure for a single-

input single-output mapping. The training points (dots) are grouped in three el-

lipsoidal clusters after a procedure of fuzzy clustering in the input/output domain.

The projection of the resulting clusters in the input domain (x-axis) determines the

centre and the width of the triangular membership functions.

There are two possible variants for the parameter identiﬁcation [137, 138]:

Local optimisation. The parameters of each local model are estimated indepen-

dently.

A weighted least squares optimisation criterion can be deﬁned for each local

model, where the weighting factor is the current activation of the correspond-

ing basis function. The parameters of each model hj ( ·, βj ), j = 1, . . . , m , are

then estimated using a set of locally weighted estimation criteria

Jj ( βj ) = 1

N( y−Xβj )T Qj ( y− Xβj ) (10.1.25)

where Qj is a [N× N ] diagonal weighting matrix, having as diagonal elements

the weights ρj (x1 , ηj ), . . . , ρj (xN , ηj ). The weight ρj ( xi , ηj ) represents the

relevance of the ith example of the training set in the deﬁnition of the j th

local model.

The locally weighted least squares estimate ˆ

βj of the local model parameter

vector βj is

βj = ( XT Qj X )−1 XT Qj y (10.1.26)

Global optimisation. The parameters of the local models are all estimated at the

same time. If the local models are assumed to be linear in the parameters, the

optimisation is a simple least-squares problem. We get the following regression

model:

j=1

ρj ( x, ηj ) xT βj = ΦΘ (10.1.27)

where Φ is a matrix [N× (n + 1)m]

Φ = 





Φ1

ΦN





(10.1.28)

268 CHAPTER 10. NONLINEAR APPROACHES

with

Φi = [ρ1 (xi , ηj )x T

i, . . . , ρ m (x i , η j )x T

i] (10.1.29)

and Θ is a matrix [(n + 1)m× 1]

Θ = 





β1

βm





(10.1.30)

The least-squares estimate ˆ

Θ returns the totality of parameters of the local models.

Note that the two approaches diﬀer both in terms of predictive accuracy and

ﬁnal interpretation of the local models. While the ﬁrst approach aims to obtain

local linear models hj somewhat representative of the local behaviour of the target

in the region described by ρj [138], the second approach disregards any qualita-

tive interpretation by pursuing only a global prediction accuracy of the modular

architecture.

10.1.9.3 Structural identiﬁcation

The structure of a BFN is characterised by many factors: the shape of the basis

functions, the number mof modules and the structure of the local models. Here,

for simplicity, we will consider a structural identiﬁcation procedure which deals

exclusively with the number mof local models.

The structural identiﬁcation procedure consists in adapting the number of mod-

ules to the complexity of the process underlying the data. According to the process

described in Section 8.8, diﬀerent BFN architectures with diﬀerent number of mod-

els are ﬁrst generated, then validated and ﬁnally selected.

Analogously to Neural Networks and Regression Trees, two are the possible

approaches to the generation of BFN architectures:

Forward: the number of local models increases from a minimum mmin to a maxi-

mum value mmax .

Backward: we start with a large number of models and we proceed gradually by

merging basis functions. The initial number must be set suﬃciently high such

that the nonlinearity can be captured accurately enough.

Once a set of BFN architectures has been generated, ﬁrst a validation measure

is used to assess the generalisation error of the diﬀerent architectures and then a

selection of the best structure is performed. An example of structural identiﬁcation

of Neuro-Fuzzy Inference Systems based on cross-validation is presented in [21].

Note that BFN structural identiﬁcation, unlike the parametric procedure de-

scribed in Section 10.1.9.2, is a non convex problems and cannot take advantage of

any linear validation technique. This is due to the fact that a BFN architecture,

even if composed of local linear modules, behaves globally as a nonlinear approxi-

mator. The resulting learning procedure is then characterised by an iterative loop

over diﬀerent model structures as illustrated in the ﬂow chart of Fig. 10.16.

10.1.10 From modular techniques to local modelling

Modular techniques are powerful engines but leave still some problems unsolved.

While these architectures have eﬃcient parametric identiﬁcation algorithms, they

are ineﬃcient in terms of structural optimisation. If the parametric identiﬁcation

takes advantage of the adoption of local linear models, the validation of the global

architecture remains a nonlinear problem which can be addressed only by compu-

tationally expensive procedures.

10.1. NONLINEAR REGRESSION 269

Figure 10.16: Flow-chart of the BFN learning procedure. The learning procedure

is made of two nested loops: the inner one (made of a linear and nonlinear step) is

the parametric identiﬁcation loop which minimises the empirical error J, the outer

one searches for the model structure which minimises the validation criterion.

270 CHAPTER 10. NONLINEAR APPROACHES

The learning problem for modular architectures is still a problem of function

estimation formulated in terms of minimisation of the empirical risk over the whole

training set. The modular conﬁguration makes the minimisation simpler but in

theoretical terms the problem appears to be at the same level of complexity as in a

generic nonlinear estimator.

Once also the constraint of global optimisation is relaxed, the divide-and-conquer

idea leads to the local modelling approach.

10.1.11 Local modelling

Local modelling is a popular nonparametric technique, which combines excellent

theoretical properties with a simple and ﬂexible learning procedure.

This section will focus on the application of local modelling to the regression

problem. The idea of local regression as a natural extension of parametric ﬁtting

arose independently at diﬀerent points in time and in diﬀerent countries in the

19th century. The early literature on smoothing by local ﬁtting focused on one

independent variable with equally spaced values. For a historical review of early

work on local regression see [46].

The modern view of smoothing by local regression has origins in the 1950's and

1960's in the kernel methods introduced in the density estimation setting . As far

as regression is concerned, the ﬁrst modern works on local regression were proposed

by Nadaraya [140] and Watson [193].

10.1.11.1 Nadaraya-Watson estimators

Let K (x, q, B ) be a nonnegative kernel function that embodies the concept of vicin-

ity. This function depends on the query point q, where the prediction of the target

value is required, and on a parameter B∈ (0,∞ ), called bandwidth , which represents

the radius of the neighbourhood. The function Ksatisﬁes two conditions:

0≤K (x, q, B )≤ 1 (10.1.31)

K( q, q, B ) = 1 (10.1.32)

For example, in the simplest one-dimensional case (dimension n= 1 of the in-

put space) both the rectangular vicinity function (also called uniform kernel) (Fig.

10.17)

K( x, q, B ) = ( 1 if kx −q k< B

0 otherwise (10.1.33)

and the soft threshold vicinity function (Fig. 10.18)

K( x, q, B ) = exp  − ( x−q) 2

B2  (10.1.34)

satisfy these requirements. Other examples of kernel functions are reported in

Appendix E.

The Nadaraya-Watson kernel regression estimator is given by

h( q ) = P N

i=1 K(x i , q, B )y i

i=1 K(x i , q, B )(10.1.35)

where N is the size of the training set. The idea of kernel estimation is simple.

Consider the case of a rectangular kernel in one dimension (n = 1). In this case,

the estimator (10.1.35) is a simple moving average with equal weights: the estimate

10.1. NONLINEAR REGRESSION 271

Figure 10.17: Hard-threshold kernel function.

Figure 10.18: Soft-threshold kernel function.

at point q is the average of observations yi corresponding to the xi 's belonging to

the window [q− B, q +B ].

If B → ∞ then the estimator tends to the average h = P N

i=1 y i

Nand thus for

mappings f ( · ) which are far from constant the bias become large.

If B is smaller than the pairwise distance between the sample points xi then the

estimator reproduces the observations h (xi ) = yi . In this extreme case, the bias

tends to zero at the cost of high variance. In general terms, by increasing B we

increase the bias of the estimator, while by reducing Bwe obtain a larger variance.

The optimal choice for B corresponds to an equal balance between bias and variance

(Section 7.7).

From a function approximation point of view, the Nadaraya-Watson estimator

is a least-squares constant approximator. Suppose we want to approximated lo-

cally the unknown function f ( ·) by a constant θ . The local weighted least-squares

estimate is

θ= arg min

i=1

wi (yi − θ )2 = P N

i=1 w i y i

i=1 w i

(10.1.36)

It follows that the kernel estimator is an example of locally weighted constant

approximator with wi =K (xi , q, B ).

The Nadaraya-Watson estimator suﬀers from a series of shortcomings: it has

large bias particularly in regions where the derivative of the regression function

f( x) or of the density π ( x ) is large. Further, it does not adapt easily to nonuniform

π( x).

An example is given in Fig. 10.19 where the Nadaraya-Watson estimator is used

to predict the value of the function f (x )=0 .9 + x2 in q = 0.5. Since most of the

272 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.19: Eﬀect of asymmetric data distribution on the Nadaraya-Watson esti-

mator: the plot reports in the input/output domain the function f = 0. 9 + x2 to be

estimated, the available points (crosses), the values of the kernel function (stars),

the value to be predicted in q = 0. 5 (dotted horizontal line) and the value predicted

by the NW estimator (solid horizontal line).

observations (crosses) are on the left of q, the estimate is biased downwards.

A more severe problem is the large bias which occurs when estimating at a

boundary region. In Fig. 10.20 we wish to estimate the value of f (x )=0 .9 + x2 at

q= 0 .5. Here the regression function has positive slope and hence the Nadaraya-

Watson estimate has substantial positive bias.

10.1.11.2 Higher order local regression

Once the weakness of the local constant approximation was recognised, a more

general local regression appeared in the late 1970's [44, 175, 114]. Work on local

regression continued throughout the 1980's and 1990's, focusing on the application

of smoothing to multidimensional problems [45].

Local regression is an attractive method both from the theoretical and the prac-

tical point of view. It adapts easily to various kinds of input distributions π(e.g.

random, ﬁxed, highly clustered or nearly uniform). See in Fig. 10.21 the local

regression estimation in q = 0. 5 for a function f (x ) = 0. 9 + x2 and an asymmetric

data conﬁguration.

Moreover, there are almost no boundary eﬀects: the bias at the boundary stays

at the same order as in the interior, without use of speciﬁc boundary kernels (com-

pare Fig. 10.20 and Fig. 10.22).

10.1.11.3 Parametric identiﬁcation in local regression

Given two variables x ∈ X ⊂ <n and y ∈ Y ⊂ < , let us consider the mapping

f:Rn →R , known only through a set of Nexamples {h xi , yi i}N

i=1 obtained as

follows:

yi = f( xi ) + wi , (10.1.37)

10.1. NONLINEAR REGRESSION 273

Figure 10.20: Eﬀect of a boundary on the Nadaraya-Watson estimator: the plot

reports in the input/output domain the function f = 0. 9 + x2 to be estimated, the

available points (crosses), the values of the kernel function (stars), the value to be

predicted in q = 0. 5 (dotted horizontal line) and the value predicted by the NW

estimator (solid horizontal line).

Figure 10.21: Local linear regression in asymmetric data conﬁguration: the plot

reports in the input/output domain the function f = 0. 9 + x2 to be estimated, the

available points(crosses), the values of the eﬀective kernel (stars), the local linear

ﬁtting, the value to be predicted in q = 0. 5 (dotted horizontal line) and the value

predicted by the local regression (solid horizontal line).

274 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.22: Local linear regression in boundary conﬁguration: the plot reports in

the input/output domain the function f = 0. 9 + x2 to be estimated, the available

points (crosses), the values of the eﬀective kernel (stars), the local linear ﬁtting, the

value to be predicted in q = 0. 5 (dotted horizontal line) and the value predicted by

the local regression (solid horizontal line).

where ∀i , wi is a random variable such that Ew [ wi ] = 0 and Ew [ wiwj ] = 0, ∀j6 = i,

and such that Ew [w m

i] = µ m (x i ), ∀m ≥ 2, where µ m (·) is the unknown m th moment

(Eq. (3.3.35)) of the distribution of wi and is deﬁned as a function of xi . In particular

for m = 2, the last of the above-mentioned properties implies that no assumption

of global constant variance (homoscedasticity) is made.

The problem of local regression can be stated as the problem of estimating the

value that the regression function f (x ) = Ey [y |x ] takes for a speciﬁc query point q,

using information pertaining only to a neighbourhood of q.

By using the Taylor's expansion truncated to the order p , a generic smooth

regression function f (· ) can be approximated by

f( x)≈

j=0

f(j) ( q)

j!( x−q)j(10.1.38)

for x in a neighbourhood of q. Given a query point q, and under the hypothesis of a

local homoscedasticity of wi , the parameter vector ˆ

βof a local linear approximation

of f (· ) in a neighbourhood of qcan be obtained by solving the locally weighted

regression (LWR)

β= arg min

i=1 ny i −x T

iβ 2 K(x i , q, B )o , (10.1.39)

where K (· ) is a kernel function, Bis the bandwidth, and a constant value 1 has been

appended to each vector xi in order to consider a constant term in the regression.

In matrix notation, the weighted least squares problem (10.1.39) can be written as

β= arg min

β(y− Xβ) T W(y− Xβ ) (10.1.40)

10.1. NONLINEAR REGRESSION 275

where X denotes the [N× (n + 1)] input matrix whose ith row is x T

i,yis a [ N×1]

vector whose ith element is yi and W is a [N× N ] diagonal matrix whose ith diagonal

element is wii =p K (xi , q, B ). From least-squares theory, the solution of the above

stated weighted least squares problem is given by the [(n + 1) × 1] vector:

β= ( XT WT W X )−1 XT WT W y = ( ZTZ )−1 ZT v= P Z T v, (10.1.41)

where Z = W X ,v = W y , and the matrix XT WT W X =ZT Z is assumed to be

non-singular so that its inverse P = (ZT Z )−1 is deﬁned.

Once obtained the local linear polynomial approximation, a prediction of f (q),

is ﬁnally given by:

ˆ yq = qT ˆ

β. (10.1.42)

10.1.11.4 Structural identiﬁcation in local regression

While the parametric identiﬁcation in a local regression problem is quite simple

and reduces to a weighted least-squares, there are several choices to be made in

terms of model structure. These are the most relevant parameters in local structure

identiﬁcation:

•the kernel function K,

•the order of the local polynomial,

•the bandwidth parameter,

•the distance function,

•the localised global structural parameters.

In the following sections, we will present in detail the importance of these structural

parameters and ﬁnally we will discuss the existing methods for tuning and selecting

them.

10.1.11.5 The kernel function

Under the assumption that the data to be analysed are generated by a continu-

ous mapping f ( ·), we want to consider positive kernel functions K ( ·,·, B ) that are

peaked at x =q and that decay smoothly to 0 as the distance between x and q

increases. Examples of diﬀerent kernel functions are reported in Appendix E.

Some considerations can be made on how relevant is the kernel shape for the ﬁnal

accuracy of the prediction. First, it is evident that a smooth weight function results

in a smoother estimate. On the other side, for hard-threshold kernels (10.1.33),

as q changes, available observations abruptly switch in and out of the smoothing

window. Second, it is relevant to have kernel functions with nonzero values on a

compact bounded support rather than simply approaching zero for |x −q | → ∞ .

This allows faster implementations, since points further from the query than the

bandwidth can be ignored with no error.

10.1.11.6 The local polynomial order

The choice of the local polynomial degree is a bias/variance trade-oﬀ. Generally

speaking, a higher degree will generally produce a less biased, but a more variable

estimate than a lower degree one.

Some asymptotic results in literature assert that good practice in local polyno-

mial regression is to adopt a polynomial order which diﬀers of an odd degree from

the order of the terms to be estimated [70]. In practice, this means that if the goal

276 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.23: Too narrow bandwidth ⇒ overﬁtting ⇒ large prediction error e..

of local polynomial regression is to estimate the value of the function in the query

point (degree zero in the Taylor expansion (10.1.38)), it is advisable to use orders

of odd degree; otherwise, if the purpose is to estimate the derivatives in the query

point it is better to ﬁt with even degrees. However, others suggest in practical

applications not to rule out any type of degree [46].

In the previous sections, we already introduced some consideration on degree

zero ﬁtting. This choice very rarely appears to be the best choice in terms of

prediction, even if it presents a strong advantage in computational terms. By using

a polynomial degree greater than zero we can typically increase the bandwidth by a

large amount without introducing intolerable bias. Despite the increased number of

parameters the ﬁnal result is smoother thanks to an increased neighbourhood size.

A degree having an integer value is generally assumed to be the only possible

choice for the local order. However, the accuracy of the prediction results to be

highly sensitive to discrete changes of the degree.

A possible alternative is polynomial mixing, proposed in global parametric ﬁtting

by Mallows [129] and in local regression by Cleveland and Loader [46]. Polynomial

mixings are polynomials of fractional degree p =m +c where m is an integer and

0< c < 1. The mixed ﬁt is a weighted average of the local polynomial ﬁts of degree

mand m+ 1 with weight 1 − cfor the former and weight cfor the latter

fp (· ) = (1 − c) fm (· ) + cfm+1 (· ) (10.1.43)

We can choose a single mixing degree for all xor we can use an adaptive method

by letting p vary with x.

10.1.11.7 The bandwidth

A natural question is how wide the local neighbourhood should be so that the local

approximation (10.1.38) holds. This is equivalent to asking how large the bandwidth

parameter should be in (10.1.33). If we take a small bandwidth B, we are able to

cope with the eventual nonlinearity of the mapping, that is, in other terms, we keep

the modelling bias small . However, since the number of data points falling in this

local neighbourhood is also small, we cannot average out the noise and the variance

of the prediction will be consequently large (Fig. 10.23).

On the other hand, if the bandwidth is too large, we could smooth excessively

the data, then introducing a large modelling bias (Fig. 10.24). In the limit case of

an inﬁnite bandwidth, for example, a local linear model turns to be a global linear

ﬁtting which, by deﬁnition, cannot take into account any type of nonlinearity.

10.1. NONLINEAR REGRESSION 277

Figure 10.24: Too large bandwidth ⇒ underﬁtting ⇒ large prediction error e.

A vast amount of literature has been devoted to the bandwidth selection prob-

lem. Various techniques for selecting smoothing parameters have been proposed

during the last decades in diﬀerent setups, mainly in kernel density estimation [112]

and kernel regression.

Two are the main strategies for the bandwidth selection:

Constant bandwidth selection. The bandwidth B is independent of the train-

ing set DN and the query point q.

Variable bandwidth selection. The bandwidth is a function B ( DN ) of the

dataset DN . For a variable bandwidth a further distinction should be made

between the local and global approach.

1. A local variable bandwidth B ( DN , q ) is not only function of the training

data DN but also changes with the query point q. An example is the

nearest neighbour bandwidth selection where the bandwidth is set to be

the distance between the query point and the k th nearest point [175].

2. A global variable bandwidth is a function B (DN ) of the data set but

is the same for all the queries. However, a further degree of distinction

should be made between the point-based case where the bandwidth B (xi )

is a function of the training point xi , and the uniform case where Bis

constant.

A constant bandwidth is easy to interpret and can be suﬃcient if the unknown

curve is not too wiggly, i.e. has a high smoothness. Such a bandwidth, however,

fails to do a good job when the unknown curve has a rather complicated structure.

To capture the complexity of such a curve a variable bandwidth is needed. A

variable bandwidth allows for diﬀerent degrees of smoothing, resulting in a possible

reduction of the bias at peaked regions and of the variance at ﬂat regions. Further,

a variable local bandwidth can adapt to the data distribution, to diﬀerent level

of noise and to changes in the smoothness of the function. Fan and Gijbels [68]

argue for point-based in favour of query-based local bandwidth selection mainly for

computational eﬃciency reasons.

10.1.11.8 The distance function

The performance of any local method depends critically on the choice of the distance

function d :Rn × Rn → R . In the following, we deﬁne some distance functions for

ordered inputs:

278 CHAPTER 10. NONLINEAR APPROACHES

Unweighted Euclidean distance

d( x, q) = v

j=1

(xj − qj )2 = q (x− q )T (x− q ) (10.1.44)

Weighted Euclidean distance

d( x, q) = q ( x− q )T MT M( x− q ) (10.1.45)

The unweighted distance is a particular case of the weighted case for Mdiag-

onal with mjj = 1.

Unweighted Lp norm (Minkowski metric)

d( x, q) = 



j=1 |x j −q j |



(10.1.46)

Weighted Lp norm It is computed through the unweighted norm d (M x, Mq).

It is important to remark that when an entire column of Mis zero, all points along

the corresponding direction get the same relevance in the distance computation.

Also, notice that once the bandwidth is selected, some terms in the matrix Mcan

be redundant parameters of the local learning procedure. The redundancy can be

eliminated by requiring the determinant of Mto be one or ﬁxing some element of

Atkeson et al. [10] distinguish between three ways of using distance functions:

Global distance function. The same distance is used at all parts of the input

space.

Query-based distance function. The distance measure is a function of the cur-

rent query point. Examples are in [174, 97, 76].

Point-based local distance functions. Each training point has an associated

distance metric [174]. This is typical of classiﬁcation problems where each

class has an associated distance metric [3, 4].

10.1.11.9 The selection of local parameters

As seen in the previous sections, there are several parameters that aﬀect the accu-

racy of the local prediction. Generally, they cannot be selected and/or optimised

in isolation as the accuracy depends on the whole set of structural choices. At the

same time, they do not all play the same role in the determination of the ﬁnal

estimation. It is a common belief in local learning literature that the bandwidth

and the distance function are the most important parameters. The shape of the

weighting function, instead, plays a secondary role.

In the following, we will mainly focus on the methods existing for bandwidth

selection. They can be classiﬁed in

Rule of thumb methods. They provide a crude bandwidth selection which in

some situations may result suﬃcient. Examples of rule of thumb is provided

in [69] and in [95].

10.1. NONLINEAR REGRESSION 279

Data-driven estimation. It is a selection procedure which estimates the gener-

alisation error directly from data. Unlike the previous approach, this method

does not rely on the asymptotic expression, but it estimates the values directly

from the ﬁnite data set. To this group belong methods like cross-validation,

Mallow's Cp , Akaike's AIC and other extensions of methods used in classical

parametric modelling.

There are several ways in which data-driven methods can be used for structural

identiﬁcation. Atkeson et al. [10] distinguish between

Global tuning. The structural parameters are tuned by optimising a data driven

assessment criterion on the whole data set. An example is the General Memory

Based Learning (GMBL) described in [135].

Query-based local tuning. The structural parameters are tuned by optimising

a data driven assessment criterion query-by-query. An example is the lazy

learning algorithm proposed by the author and colleagues in [24, 31, 30].

Point-based local tuning. A diﬀerent set of structural parameters is associated

with each point of the training set.

R implementation

A local linear algorithm for regression is implemented by the R library lazy [23].

The script lazy.R shows the prediction accuracy in the Dopler dataset for diﬀerent

number of neighbours. (Figure 10.25 and Figure 10.26).

•

10.1.11.10 Bias/variance decomposition of the local constant model

An interesting aspect of local models is that it is easy to derive an analytical ex-

pression of the bias/variance decomposition.

In the case of a constant local model the prediction in qis

h( q, αN ) = 1

i=1

y[i]

280 CHAPTER 10. NONLINEAR APPROACHES

Figure 10.25: Locally linear ﬁtting with a rectangular kernel and a bandwidth made

of 10 neighbors.

Figure 10.26: Locally linear ﬁtting with a rectangular kernel and a bandwidth made

of 228 neighbours.

10.2. NONLINEAR CLASSIFICATION 281

computed by averaging the value of y for the k closest neighbours x [i] ,i = 1, . . . , k

of q.

The bias/variance decomposition takes the form discussed in Equation (5.5.15)

that is

MSE(q ) = σ 2

w+ 1

i=1

f( x[i] )− f( q)! 2

+σ2

w/k (10.1.47)

where σ 2

wis the variance of the noise and σ 2

w/k is the variance of a sample average

estimator based on kpoints (Equation (5.5.10). Note the behaviour of the MSE

term as a function of k. By increasing k(i.e. larger neighbourhood) the ﬁrst

term is invariant, the bias is likely to increase (since farther points are potentially

uncorrelated with q) and the variance decreases.

10.2 Nonlinear classiﬁcation

In Section 9.2.1, we have shown that optimal classiﬁcation is possible only if the

quantities Prob {y = ck |x} ,k = 1, . . . , K are known. What happens if this is not

the case? Three strategies are generally used.

10.2.1 Direct estimation via regression techniques

If the classiﬁcation problem has K= 2 classes and if we denote them by y = 0 and

y= 1

E[y| x] = 1 · Prob {y = 1| x}+ 0 ·Prob {y = 0 | x} = Prob {y = 1| x}

Then a binary classiﬁcation problem can be put in the form of a regression prob-

lem where the output takes value in { 0,1} . This means that, in principle, all the

regression techniques presented so far could be used to solve a classiﬁcation task.

In practice, most of those techniques do not make any assumption about the fact

that the outcome in a classiﬁcation task should satisfy the probabilistic constrains,

e.g. 0 ≤ Prob {y = 1|x } ≤ 1. This means that only some regression algorithms (e.g.

local constant models) are commonly used for binary classiﬁcation as well.

10.2.1.1 The nearest-neighbour classiﬁer

The nearest-neighbour algorithm is an example of local modelling (Section 10.1.11)

algorithm for classiﬁcation.

Let us consider a binary { 0,1} classiﬁcation task where a a training set is avail-

able and the classiﬁcation is required for an input vector q (query point). The

classiﬁcation procedure of a k-NN classiﬁer can be summarised in the following

steps:

1. Compute the distance between the query qand the training examples accord-

ing to a predeﬁned metric.

2. Rank the observed inputs on the basis of their distance to the query.

3. Select a subset {x [1] , . . . , x[k] } of the k≥ 1 nearest neighbors. Each of these

neighbours x[k] has an associated class y[k] .

4. Compute the estimation of the conditional probability of the class 1 by con-

stant ﬁtting

ˆ p1 ( q) = P k

i=1 y [ i ]

k(10.2.48)

282 CHAPTER 10. NONLINEAR APPROACHES

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 1

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 11

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 21

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 31

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 41

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 51

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 61

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 71

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 101

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 201

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 301

202 4

0.0 0.2 0.4 0.6 0.8 1.0

Number of neighbors= 391

Figure 10.27: kNN prediction (blue line) of the conditional probability (green line)

for diﬀerent values of k . Dotted-lines represent the class-conditional densities.

or linear ﬁtting

ˆ p1 ( q) = ˆ aq +ˆ

where the parameters ˆ aand ˆ

bare locally ﬁtted by least-squares regression.

5. Return the prediction, either by majority vote or according to the conditional

probability.

It is evident that the hyperparameter kplays a key role in the trade-oﬀ between

bias and variance. Figure 10.27 illustrates the trade-oﬀ in a n = 1 and N= 400

binary classiﬁcation task where the two class-conditional distributions are Normal

with means in −1 and 3, respectively. Note that by increasing kthe prediction

proﬁle becomes smoother and smoother.

Figure 10.28 shows the trade-oﬀ in a n= 2 classiﬁcation task. Note that, though

the separating region becomes closer to the optimal for large k, an extrapolation

bias occurs in regions far from the observed examples.

It is interesting to see that the kNN classiﬁer can be justiﬁed in terms of the

Bayes theorem. Suppose that the dataset has the form DN = { (x1 , y1 ),..., (xN , yN ) }

where y ∈ {c1 , . . . , cK } and q∈Rn is the query point where we want to compute

the a-posteriori probability. Suppose that the dataset contains Nj points labeled

with the class cj , i.e.

j=1

Nj = N

Let us consider a region Raround the input x having volume V . If the volume

10.2. NONLINEAR CLASSIFICATION 283

10 5 0 5 10

Number of neighbors= 1

10 5 0 5 10

Number of neighbors= 5

10 5 0 5 10

Number of neighbors= 40

Figure 10.28: kNN class predictions in the n = 2 input space for diﬀerent values of k.

Dots represent the training points. Continuous black line is the optimal separating

hyperplane.

284 CHAPTER 10. NONLINEAR APPROACHES

is small enough, we may consider the density constant over the entire region5 . It

follows that the probability of observing a point within this volume is

P=ZR

p( z) dz ≈ p( x ) V⇒ p( x)≈ P

Given a training dataset of size N, if we observe NR example in a region Rwe can

approximate P with ˆ

P=NR

Nand consequently obtain

ˆ p( x ) = N R

NV (10.2.49)

Consider now the query point qand a neighbouring volume V containing k

points of which kj ≤ k are labeled with the class cj . From (10.2.49) we obtain the

k-NN density estimate (A.1.11) of the class-conditional probability

ˆ p( q| cj ) = k j

Nj V

and of the unconditional density

ˆ p( q ) = k

Since the class priors can be estimated by [

Prob {y = cj } =N j

Nfrom (7.3.16) it

follows

[

Prob {y =cj | q} =k j

kj= 1, . . . , K

This implies that in a binary { 0,1} case the computation (10.2.48) estimates the

conditional probability of the class 1.

10.2.2 Direct estimation via cross-entropy

The approach consists in modelling the conditional distribution Prob {y = cj | x} , j =

1, . . . , K with a set of models ˆ

Pj ( x, α), j = 1, . . . , K satisfying the constraints

Pj ( x, α) > 0 and

j=1

Pj ( x, α )=1 .

Parametric estimation boils down to the minimisation of the cross-entropy cost

function (8.6.5). Typical approaches are logistic regression and neural networks.

In logistic regression for a two-class task we have

P1 ( x, α) = exp x T α

1 + expx T α = 1

1 + exp−xT α , ˆ

P2 ( x, α) = 1

1 + expx T α (10.2.50)

where x and α are [p, 1] vectors. This implies

log ˆ

P1 ( x, α)

P2 ( x, α)= x T α

where the transformation log p

1−p is called the logit transformation and the func-

tion (10.1.2) is the logistic function. In a nonlinear classiﬁer (e.g. neural net)

log ˆ

P1 ( x, α)

P2 ( x, α)= h( x, α)

5For a discussion about the validity of this assumption in large dimensional settings, refer to

Section 12.1

10.2. NONLINEAR CLASSIFICATION 285

where h (x, α ) is the output of the learner.

In a binary case (c1 = 1, c2 = 0) the cross-entropy function (to minimise) be-

comes

J( α) = −

i=1

log ˆ

Py i ( xi , α) =

−

i=1  y i log ˆ

P1 ( xi , α) + (1 − yi ) log(1 − ˆ

P1 ( xi , α))=

i=1  −y i h(x i , α) + log(1 + exp h(xi,α) (10.2.51)

In the logistic regression case (linear h) the cost function is minimised by iteratively

reweighted least squares. For a generic h, a gradient-based iterative approach is

required.

Another formulation of the binary case is (c1 = 1, c2 = − 1) with

P( y| x) = 1

1 + exp−yh(x,α)

that satisﬁes ˆ

P( y= 1| x) = 1 − ˆ

P( y=− 1| x). In this case the classiﬁcation rule is

the sign function sign[h (x )] and the cost function to minimise is

J( α) =

i=1

log(1 + exp−yih(xi,α) ) (10.2.52)

also known as the log-loss function which is a monotone decreasing function of the

terms yi h (xi , α ) = yi hi called the margins. Minimise (10.2.52) is then equivalent

to minimise the set of training points for which yi and the prediction h (xi , α ) have

a diﬀerent sign. The decreasing nature of the function exp−yh(x,α) is such that

negative margins are much more penalised than positive ones. Note that this is not

a property of the least-squares criterion (used in regression) (y− h (x, α))2 where in

some cases a positive margin may be more penalised than a negative one. This is a

reason why regression techniques are not recommended in classiﬁcation tasks.

10.2.3 Density estimation via the Bayes theorem

Since

Prob {y=ck |x }= p(x |y=ck ) Prob {y= ck }

p( x)

an estimation of Prob {x|y = ck } allows an estimation of Prob {y = ck | x} . Several

techniques exist in literature to estimate Prob {x |y = ck } .

We will present two of them in the following section. The ﬁrst makes the as-

sumption of conditional independence to make easier the estimation. The second

relies on the construction of optimal separating hyperplanes to create convex regions

containing set of x points sharing the same class label.

10.2.3.1 Naive Bayes classiﬁer

The Naive Bayes (NB) classiﬁer has shown in some domains a performance compa-

rable to that of neural networks and decision tree learning.

Consider a classiﬁcation problem with ninputs and a random output variable y

that takes values in the set {c1 , . . . , cK } . The Bayes optimal classiﬁer should return

c∗ ( x ) = arg max

j=1 ,...,K Prob {y= c j |x}

286 CHAPTER 10. NONLINEAR APPROACHES

We can use the Bayes theorem to rewrite this expression as

c∗ ( x ) = arg max

j=1,...,K

Prob {x|y = cj } Prob {y = cj }

Prob {x}

= arg max

j=1 ,...,K Prob {x |y=c j }Prob {y = c j }

How to estimate these two terms on the basis of a ﬁnite set of data? It is easy

to estimate each of the a priori probabilities Prob {y = cj } simply by counting the

frequency with which each target class occurs in the training set. The estimation

of Prob {x|y = cj } is much harder. The NB classiﬁer is based on the simplifying

assumption that the input values are conditionally independent given the target

value (see Section 3.5.4):

Prob {x|y = cj } = Prob {x1 , . . . , xn |y = cj } =

h=1

Prob {xh |y = cj }

The NB classiﬁcation is then

cNB ( x ) = arg max

j=1 ,...,K Prob {y=c j }

h=1

Prob {xh |y =cj }

If the inputs xh are discrete variables the estimation of Prob {xh |y =cj } boils down

to the counting of the frequencies of the occurrences of the diﬀerent values of x h

for a given class cj .

Example

Obs G1 G2 G3 G

1 P.LOW P.HIGH N.HIGH P.HIGH

2 N.LOW P.HIGH P.HIGH N.HIGH

3 P.LOW P.LOW N.LOW P.LOW

4 P.HIGH P.HIGH N.HIGH P.HIGH

5 N.LOW P.HIGH N.LOW P.LOW

6 N.HIGH N.LOW P.LOW N.LOW

7 P.LOW N.LOW N.HIGH P.LOW

8 P.LOW N.HIGH N.LOW P.LOW

9 P.HIGH P.LOW P.LOW N.LOW

10 P.HIGH P.LOW P.LOW P.LOW

Let us compute the NB classiﬁcation for the query { G1=N.LOW G2= N.HIGH

G3=N.LOW Since

Prob {y = P.HIGH } = 2/ 10, Prob {y = P.LOW } = 5/10

Prob {y = N.H IGH } = 1/ 10, Prob {y = N .LOW } = 2/10

Prob {G 1 = N.LO W |y = P.HI GH } = 0/ 2, Prob {G 1 = N .LOW |y = P.LOW } = 1/5

Prob {G 1 = N.LO W |y = N.HIGH } = 1/ 1, Prob { G 1 = N .LOW |y = N.LO W } = 0/2

Prob {G 2 = N.H IGH |y = P.H IGH } = 0/ 2, Prob {G 2 = N.HI GH |y = P .LOW } = 1/5

Prob {G 2 = N.H IGH |y = N.HI GH } = 0/ 1, Prob {G 2 = N.HIGH|y = N .LOW } = 0/2

Prob {G 3 = N.LO W |y = P.HI GH } = 0/ 2, Prob {G 3 = N .LOW |y = P.LOW } = 3/5

Prob {G 3 = N.LO W |y = N.HIGH } = 0/ 1, Prob {G 3 = N .LOW |y = N.LO W } = 0/2

10.2. NONLINEAR CLASSIFICATION 287

it follows that

cNB ( x ) =

arg max

P.H,P.L,N.H,N.L{2/10 ∗0∗0∗ 0,5/10 ∗1/ 5∗ 1/ 5∗ 3/ 5, 1/10∗1∗0 ∗ 1,2/10 ∗0∗0 ∗ 0}=

=P.LOW

•

The NB classiﬁer relies on the naive (i.e. simplistic) assumption that the inputs

are independent given the target class. But why is this assumption done and when

may it be considered as realistic? There are essentially two reasons underlying the

NB approach, one of statistical nature and the second of causal nature. From a

statistical perspective the conditional independence assumption largely reduces the

capacity of the classiﬁer by reducing the number of parameters (Section 4.1). This

is a variance reduction argument which makes the algorithm eﬀective in large di-

mensional classiﬁcation tasks. However, there are classiﬁcation tasks which by their

own nature are more compliant with the NB assumptions than other. Those are

tasks where the features used to predict the class are descriptors of the phenomenon

represented by the class. Think for instance to the classiﬁcation task where a doc-

tor predicts if a patient got a ﬂu by means of symptomatic information (does she

cough? has he fever?). All those measures are correlated but they become indepen-

dent once we know the latent state. In a causal perspective (Chapter 13 ) NB makes

the assumption that the considered input features are eﬀects of a common variable

(the target class) (Figure 13.9 left). Another example where this assumption holds

is fraud detection [52, 50] where the observed features (e.g. place and amount of

transaction) are consequences of the fraudulent action and then informative about

it.

10.2.3.2 SVM for nonlinear classiﬁcation

The extension of the Support Vector (SV) approach to nonlinear classiﬁcation re-

lies on the transformation of the input variables and the possibility of eﬀectively

adapting the SVM procedure to a transformed input space.

The idea of transforming the input space by using basis functions is an intuitive

manner of extending linear techniques to a nonlinear setting.

Consider for example an input output regression problem where x ∈ X ⊂ Rn .

Let us deﬁne mnew transformed variables zj = zj (x ), j = 1, . . . , m , where zj ( ·)

is a pre-deﬁned nonlinear transformation (e.g. zj (x ) = log x1 + log x2 ). This is

equivalent to mapping the input space Xinto a new space, also known as feature

space,Z = {z= z( x ) |x ∈ X}. Note that, if m < n, this transformation boils down

to a dimensionality reduction and it is an example of feature selection (Chapter 12).

Let us now ﬁt a linear model y =P m

j=1 β m z m to the training data in the new

input space z∈Rm . By doing this, we carry out a nonlinear ﬁtting of data simply

by using a conventional linear technique.

This procedure can be adopted for every learning procedure. However, it is still

more worthy being used in a SV framework. Before discussing it, we introduce the

notion of dot-product kernel.

Deﬁnition 2.1 (Dot-product kernel) . A dot-product kernel is a function K, such

that for all xi , xj ∈ X K (xi , xj ) = h z (xi ), z (xj )i (10.2.53)

where h z1 , z2 i =z T

1z 2 stands for the inner product and z(· ) is the mapping from

the original to the feature space Z.

288 CHAPTER 10. NONLINEAR APPROACHES

Let us suppose now that we want to perform a binary classiﬁcation by SVM

in a transformed space z ∈ Z . For the sake of simplicity, we will consider a sepa-

rable case. The parametric identiﬁcation step requires the solution of a quadratic

programming problem in the space Z

max

i=1

αi − 1

i=1

k=1

αiαkyiyk z T

iz k = (10.2.54)

= max

i=1

αi − 1

i=1

k=1

αiαkyiyk hzi , zk i = (10.2.55)

= max

i=1

αi − 1

i=1

k=1

αiαkyiyk K( xi , xk ) (10.2.56)

subject to 0 =

i=1

αiyi , (10.2.57)

αi ≥0 , i = 1 , . . . , N (10.2.58)

What is interesting is that the resolution of this problem diﬀers from the linear

one (Equation (9.2.70)) by the replacement of the quantities hxi , xk i with h zi , zk i =

K( xi , xk ).

This means that whatever the feature transformation z (x ) and whatever the

dimensionality m , the SVM computation requires only the availability of the Gram

matrix in the feature space, also referred to as the kernel matrix K . What is

interesting is that once we know how to derive the kernel matrix we do not even

need to know the underlying feature transformation function z (x).

The use of a kernel function is an attractive computational short-cut. In practice,

the approach consists in deﬁning a kernel function directly, hence implicitly deﬁning

the feature space.

10.3 Is there a best learner?

A vast amount of literature in machine learning served the purpose of showing the

superiority of some learning methods over the others. To support this claim, quali-

tative considerations and tons of experimental simulations have been submitted to

the scientiﬁc community. Every machine learning researcher dreams of inventing the

most accurate algorithm, without realising that the attainment of such an objective

would necessarily mean the end of machine learning... But is there an algorithm to

be universally preferred over others in terms of prediction accuracy?

If there was a universally best learning machine, research on machine learning

would be unnecessary: we would use it all the time. (Un)fortunately, the theoretical

results on this subject are not encouraging [58]. For any number Nof observations,

there exist an input/output distribution for which the estimate of generalisation

error is arbitrarily poor. At the same time, for any learning machine L1 it exists a

data distribution and another learning machine L2 such that for all N , L2 is better

than L1 .

It can be shown that there is no learning algorithm which is inherently superior

to any other, or even to random guessing. The accuracy depends on the match

between the (unknown) target distribution and the (implicit or explicit) inductive

bias of the learner.

This (surprising ?) result has been formalised by the No Free Lunch (NFL)

theorems by D. Wolpert [197]. In his seminal work, Wolpert characterises in prob-

abilistic terms the relation between target function, dataset and hypothesis. The

10.3. IS THERE A BEST LEARNER? 289

main diﬀerence with respect to other research on generalisation is that he does not

consider the generating process as constant (e.g. fﬁxed as in the bias/variance

decomposition 7.7.36), but he supposes the existence of a probability distribution

p( f) over the target functions fand that a learning algorithm implements a prob-

ability distribution p ( h) over the hypothesis h. For instance p (h = h|DN ) denotes

the probability that a learning algorithm will return the hypothesis hgiven the

training set DN6 . Based on this formalism, he encodes the following assumptions

in a probabilistic language:

•the target distribution is completely outside the researcher's control[197] ,

•the learning algorithm designer has no knowledge about fwhen guessing a

hypothesis function.

This means that, over the input space region where we observed no training ex-

amples (oﬀ-training region), the hypothesis his conditionally independent (Sec-

tion 3.5.4) of f given the training set:

p( h| f, DN ) = p( h| DN )

which in turn is equivalent to

p( f| h, DN ) = p( f| DN )

In other terms the only information about the target process that a hypothesis may

take advantage of, is the one contained in the training set.

He then derives the generalisation error of a learning algorithm Lconditioned

on a training set DN and computed on input values which do not belong to the

training set (i.e. oﬀ-training set region) as

x6∈ DN

Ef,h [L(f , h)| DN ]( x ) =

x6∈ DN Z f,h

L(h( x ) , f ( x )) p( f, h| DN ) df dh = X

x6∈ DN Z f,h

L(h( x ) , f ( x)) p ( f | DN ) p (h|DN )dfdh

(10.3.59)

It follows that the generalisation error7depends on the alignment (or match)

between the hypothesis hreturned by Land the target fwhich is represented by

the inner-product p (f| DN )p ( h|DN ). Since the target is unknown this match may

only be assessed a posteriori: a priori there is no reason to consider a learning

algorithm better than another. For any learning algorithm which is well aligned

with the distribution p (f| DN ) in the oﬀ-training set it is possible to ﬁnd another

distribution for which the match is much worse. Equation (10.3.59) is one of the

several NFL results stating that there is no problem-independent reason to favour

one learning algorithm Lover another (not even random guessing) if

1. we are interested only in generalisation accuracy,

2. we make no a priori assumption on the target distribution,

3. we restrain to the accuracy over the oﬀ-training set region.

6Note that throughout this book we have only considered deterministic learning algorithms,

for which p (h| H ) is a Dirac function

7Note the diﬀerences between the deﬁnitions (7.7.36) and (10.3.59) of generalisation error.

In (7.7.36) fis ﬁxed and DN is random: in (10.3.59) fis random and DN is ﬁxed

290 CHAPTER 10. NONLINEAR APPROACHES

x1x2x3 yˆ y1 ˆ y2 ˆ y3 ˆ y4

0 0 0 1 1 1 1 1

0 0 1 0 0 0 0 0

0 1 0 1 1 1 1 1

0 1 1 1 1 1 1 1

1 0 0 0 0 0 0 0

1 0 1 0 0 0 0 0

1 1 0 ? 0 0 1 1

1 1 1 ? 0 1 0 1

Table 10.1: Oﬀ-training set prediction binary example

The presumed overall superiority of a learning algorithm (no matter the number of

publications or the H-factor of the author) is apparent and depends on the speciﬁc

task and underlying data generating process. The NFL theorems are then a modern

probabilistic version of the Hume skeptical argument (Section 2.4): there is no log-

ical evidence that the future will behave like the past. Any prediction or modelling

eﬀort demands (explicitly or implicitly) an assumption about the data generating

process and the resulting accuracy is strictly related to the validity of such assump-

tion. Note that such assumptions underlie also learning procedures that seem to be

general purpose and data-driven like holdout or cross-validation: for instance, an

holdout strategy makes the assumption that the relation between the training por-

tion of the dataset and the validation one is informative about the relation between

the observed dataset and future query points (in oﬀ-training regions).

A NFL example

Let us consider a classiﬁcation example from [61] where we have three binary in-

puts and one binary target which is a deterministic function of the inputs. Let us

suppose that the value of the target is known for 6 input conﬁgurations and that

we want to predict the value for the 2 remaining ones (Table ). Let us consider

4 classiﬁers which have identical behaviour for the training set yet they diﬀer in

terms of their predictions for the oﬀ-training region. Which one is the best one in

the oﬀ-training set region? May we discriminate between them on the basis of the

training behaviour, if we make no additional assumption about the input/output

relationship in the oﬀ-training set region? The No Free Lunch answer is no. If

we assume no a priori information about the conditional distribution, we have 4

equiprobable oﬀ-training behaviours. On average the four predictors have the same

accuracy. Note that each predictor could have a justiﬁcation in terms of nearest

neighbour (Table 10.2). For instance, the ﬁrst classiﬁer (in black) relies on the in-

ductive bias that the ﬁrst and the third features are the most informative about the

target, i.e. if we consider in the dataset the nearest neighbours of the oﬀ-tranining

inputs we obtain two zeros as predictions. This is not the case of the fourth (red)

which makes implicitly a diﬀerent hypothesis, (i.e. the target depends on x2 only)

and returns two ones accordingly.

•

10.4 Conclusions

A large part of machine learning research in the last forty years has been devoted to

the quest for the Holy Grail of generalisation. This chapter presented a number of

learning algorithms and their rationale. Most of those algorithms made the history

of the machine learning and were undeniably responsible for the success of the

10.4. CONCLUSIONS 291

Table 10.2: Oﬀ-training set prediction binary example: nearest-neighbour interpre-

tation of the four classiﬁers. The colours show which training points have been used

according to a nearest-neighbour strategy to return the oﬀ-training set predictions.

discipline. When they were introduced, and every time they were used afterwards,

they were shown to be competitive and often outperform other algorithms. So, how

may all this be compatible with the No Free Lunch result? First of all the NFL

does not deny that some algorithms may generalise well under some circumstances.

It simply states that there is not a single algorithm outperforming consistently all

the others. Also, NFL results assume that the oﬀ-training set is the most pertinent

measure for assessing algorithms. Last but not least, the success of statistics (and

ML) is probably indicative that the prediction tasks we are commonly confronted

with belong to not such a wide and uniform distribution but that some of them are

more probable than others.

Nevertheless, the NFL results may appear as frustrating to a young researcher

aiming to pursue a career in machine learning (and incidentally ﬁnding the Holy

Grail): this is not necessarily so if we deﬁne in a less utopian, yet more scientiﬁc

way, the mission of a data scientist. The mission of a data scientist should not

be the promotion of a speciﬁc algorithm (or family of algorithms) but acting as

a scientist through the analysis of data. This means (s)he should use his know-

how NOT to return information about the merits of his preferred algorithm BUT

about the nature of the data generating distribution (s)he is dealing with. The

outcome of a ML research activity (e.g. publication) should be additional insight

about the observed reality (or Nature) and not a contingent statement about the

temporary superiority of an algorithm. Newtons aim was to use diﬀerential calculus

to model and explain dynamics in Nature and not to promote a fancy diﬀerential

equation tool8. Consider also that every ML algorithm (also the least fashionable

and the least performing) might return some information (e.g. about the degree of

noise, nonlinearity) about the phenomenon we are observing. For instance, a low

accurate linear model tells us a lot about the lack of validity of the (embedded)

linear assumption in the observed phenomenon. In that sense wrong models might

play a relevant role as well since they might return important information about the

phenomenon under observation, notably (non)linearity, (non)stationarity, degree of

stochasticity, relevance of features and nature of noise.

8...or patent it !

292 CHAPTER 10. NONLINEAR APPROACHES

10.5 Exercises

1. Suppose you want to learn a classiﬁer for detecting spam in emails. Let the binary

variables x1 , x2 and x3 represent the occurrence of the words "Viagra", "Lottery"

and "Won", respectively, in a email.

Let the dataset of 20 emails being summarized as follows

Document x1 (Viagra) x2 (Lottery) x3 (Won) y(Class)

E1 0 0 0 NOSPAM

E2 0 1 1 SPAM

E3 0 0 1 NOSPAM

E4 0 1 1 SPAM

E5 1 0 0 SPAM

E6 1 1 1 SPAM

E7 0 0 1 NOSPAM

E8 0 1 1 SPAM

E9 0 0 0 NOSPAM

E10 0 1 1 SPAM

E11 1 0 0 NOSPAM

E12 0 1 1 SPAM

E13 0 0 0 NOSPAM

E14 0 1 1 SPAM

E15 0 0 1 NOSPAM

E16 0 1 1 SPAM

E17 1 0 0 SPAM

E18 1 1 1 SPAM

E19 0 0 1 NOSPAM

E20 0 1 1 SPAM

where

•0 stands for the case-insensitive absence of the word in the email.

•1 stands for the case-insensitive presence of the word in the email.

Let y = 1 denote a spam email and y= 0 a no-spam email.

The student should

2. Estimate on the basis of the data of exercise 1:

•Prob {x1 = 1, x2 = 1}

•Prob {y= 0 |x2 = 1, x3 = 1}

•Prob {x1 = 0 |x2 = 1}

•Prob {x3 = 1 |y= 0, x2 = 0}

•Prob {y= 0 |x1 = 0,x2 = 0,x3 = 0}

•Prob {x1 = 0 |y= 0}

•Prob {y= 0}

Solution:

•Prob {x1 = 1, x2 = 1 }= 0.1

•Prob {y= 0 |x2 = 1,x3 = 1 } = 0

•Prob {x1 = 0 |x2 = 1 }= 0 .8

•Prob {x3 = 1 |y= 0, x2 = 0 }= 0 .5

•Prob {y = 0 |x1 = 0, x2 = 0,x3 = 0 } = 1

•Prob {x1 = 0 |y = 0 } = 0.875

•Prob {y = 0 } = 0.4

10.5. EXERCISES 293

3. Answer to the following questions (Yes or No) on the basis of the data of exercise 1:

•Are x1 and x2 independent?

•Are x1 and y independent?

•Are the events x1 = 1 and x2 = 1 mutually exclusive?

Solution:

•Are x1 and x2 independent? NO

•Are x1 and y independent? NO

•Are the events x1 = 1 and x2 = 1 mutually exclusive? NO

4. Consider the following three emails

•M1: "Lowest Viagra, Cialis, Levitra price".

•M2: "From Google Promo (GOOGLEPROMOASIA) Congratulation! Your

mobile won 1 MILLION USD in the GOOGLE PROMO"

•M3: "This is to inform you on the release of the EL-GORDO SWEEPSTAKE

LOTTERY PROGRAM. Your name is attached to ticket number 025-11-464-

992-750 with serial number 2113-05 drew the lucky numbers 13-15 which con-

sequently won the lottery in the 3rd category."

Use a Naive Bayes Classiﬁer to compute for email M1 on the basis of the data of

exercise 1::

•the input x

•Prob {y= SPAM| x} Prob {x}

•Prob {y= NOSPAM| x} Prob { x}

•the email class

Solution:

•the input x = [1, 0, 0]

•Prob {y= SPAM| x} Prob {x } = 1/ 180 = 0.0055

•Prob {y= NOSPAM| x} Prob { x} = 1/40 = 0.025

•the email class: NOSP

5. Use a Naive Bayes Classiﬁer to compute for email M2 on the basis of the data of

exercise 1::

•the input x

•Prob {y= SPAM| x} Prob {x}

•Prob {y= NOSPAM| x} Prob { x}

•the email class.

Solution:

•the input x = [0, 0, 1]

•Prob {y= SPAM| x} Prob {x } = 1/18 = 0.055

•Prob {y= NOSPAM| x} Prob { x} = 7/40 = 0.175

•the email class is NOSPAM.

6. Use a Naive Bayes Classiﬁer to compute for email M3 on the basis of the data of

exercise 1:

•the input x

•Prob {y= SPAM| x} Prob {x}

294 CHAPTER 10. NONLINEAR APPROACHES

•Prob {y= NOSPAM|x } Prob { x}

•the email class

Solution:

•the input x = [0, 1, 1]

•Prob {y= SPAM| x} Prob { x} = 5/18 = 0.27

•Prob {y= NOSPAM|x } Prob { x} = 0

•the email class is SPAM

7. Consider a classiﬁcation task with two binary inputs and one binary target y ∈

{−1,+1 } where the conditional distribution is

x1x2 P( y= 1| x1 , x2 )

0 0 0.8

0 1 0.1

1 0 0.5

1 1 1

Suppose that all the input conﬁgurations have the same probability.

Let the classiﬁer be the rule:

IF x2 = 0 THEN ˆ y=−1 ELSE ˆ y= 1.

Consider a test set of size N = 10000.

For this classiﬁer compute:

•the confusion matrix,

•the precision,

•the speciﬁcity (true negative rate)

•the sensitivity (true positive rate)

Solution:

•the confusion matrix,

ˆ y=−1 ˆ y= 1

y=− 1 TN=1750 FP=2250

y= 1 FN=3250 TP=2750

•the precision TP/(TP+FP)=2750/5000=0.55

•the speciﬁcity (true negative rate) TN/(TN+FP)=1750/4000=0.4375

•the sensitivity (true positive rate) TP/(TP+FN)=2750/6000=0.458

8. Consider a classiﬁcation task with two binary inputs and one binary target y ∈

{−1,+1 } where the conditional distribution is

x1x2 P( y= 1| x1 , x2 )

0 0 0.8

0 1 0.1

1 0 0.5

1 1 1

Suppose that all the input conﬁgurations have the same probability.

Let the classiﬁer be the rule:

IF x2 = 0 THEN ˆ y=−1 ELSE ˆ y= 1.

Consider a test set of size N = 10000.

For this classiﬁer compute:

10.5. EXERCISES 295

•the confusion matrix,

•the precision,

•the speciﬁcity (true negative rate)

•the sensitivity (true positive rate)

Solution:

•the confusion matrix,

ˆ y=− 1 ˆ y= 1

y=− 1 TN=1750 FP=2250

y= 1 FN=3250 TP=2750

•the precision, TP/(TP+FP)=2750/5000=0.45

•the speciﬁcity (true negative rate) TN/(TN+FP)=1750/4000=0.4375

•the sensitivity (true positive rate) TP/(TP+FN)=2750/6000=0.458

9. Consider a regression task with input x and output y and the following training set

X Y

0 0.5

-0.3 1.2

0.2 1

0.4 0.5

0.1 0

-1 1.1

Consider the three following models:

•constant

•1NN, Nearest Neighbour with K=1

•3NN, Nearest Neighbour with K=3

Compute for the constant model

•the vector of training errors ei =yi −ˆ yi

•the vector of leave-one-out errors e −i

i=y i −ˆ y−i

•the mean-squared training error ,

•the mean-square leave-one-out error.

Compute for the 1NN model

•the vector of training errors ei = yi −ˆ yi

•the vector of leave-one-out errors e−i = yi − ˆ y−i

•the mean-squared training error ,

•the mean-squared leave-one-out error.

Compute for the 3NN model

•the vector of training errors ei = yi −ˆ yi

•the vector of leave-one-out errors e−i = yi − ˆ y−i

•the mean-squared training error ,

•the mean-squared leave-one-out error.

Solution: Constant model

296 CHAPTER 10. NONLINEAR APPROACHES

•the vector of training errors ei = yi −ˆ yi = [− 0 .2167 ,0 .4833 ,0 .2833 ,−0 .2167 ,−0 .7167 ,0 .3833]

•the vector of leave-one-out errors e−i

i=y i −ˆ y−i

i= [−0. 26, 0 .58 , 0 . 34 , − 0.26, −0 .86 ,0 . 46]

•the mean-squared training error = 0.178

•the mean-square leave-one-out error = 0.2564

1NN model:

•the vector of training errors ei =yi −ˆ yi = [000000]

•the vector of leave-one-out errors e−i =yi − ˆ y−i

i= [0. 5, 0. 7, 1,− 0 . 5, − 0.5,− 0 .1]

[0.5, 0. 7, 1,− 0 . 5, − 1, − 0. 1]

•the mean-squared training error =0

•the mean-squared leave-one-out error =0.375 or 0.5

3NN model

•the vector of training errors ei = yi −ˆ yi = [0 ,0 .6333 ,0 .5 ,0 ,−0 .5 ,0 .1667]

•the vector of leave-one-out errors e −i = yi − ˆ y−i

i= [−0. 2333, 0. 7, 0. 6667, 0, −0 .6667 ,0 .5333]

•the mean-squared training error =0.1548

•the mean-squared leave-one-out error=0.2862

10. Consider a classiﬁcation task with three binary inputs and one binary target where

the conditional distribution is

x1x2x3 P( y= 1| x1 , x2, x3 )

0 0 0 0.8

0 0 1 0.9

0 1 0 0.5

0 1 1 1

1 0 0 0.8

1 0 1 0.1

1 1 0 0.1

1 1 1 0

Suppose that all the input conﬁgurations have the same probability.

Let the classiﬁer be the rule:

IF x1 = 0 OR x2 = 0 THEN ˆ y= 1 ELSE ˆ y= 0.

Suppose we have a test set of size N = 10000.

Considering the class 1 as the positive class, for this classiﬁer compute:

•the confusion matrix,

•the precision,

•the speciﬁcity (true negative rate) and

•the sensitivity (true positive rate)

Solution:

•the confusion matrix,

x1x2x3 y = 1 ˆ y= 1 TP FP TN FN

0 0 0 1000 1250 1000 250 0 0

0 0 1 1125 1250 1125 125 0 0

0 1 0 625 1250 625 625 0 0

0 1 1 1250 1250 1250 0 0 0

1 0 0 1000 1250 1000 250 0 0

1 0 1 125 1250 125 1125 0 0

1 1 0 125 0 0 0 1125 125

1 1 1 0 0 0 0 1250 0

10.5. EXERCISES 297

ˆ y= 1 ˆ y= 0

y= 1 TP=5125 FN=125

y= 0 FP=2375 TN=2375

•the precision = 5125/(5125+2375)=0.68

•the speciﬁcity (true negative rate) = 2375/(2375+2375)=0.5

•the sensitivity (true positive rate) = 5125/(5125+125)=0.976

11. Let us consider the following classiﬁcation dataset where yis the binary target.

x1 x2 y

-4 7.0 1

-3 -2.0 1

-2 5.0 0

-1 2.5 1

1 1.0 0

2 4.0 1

3 6.0 0

4 3.0 1

5 -1.0 0

6 8.0 0

•Consider the 1st classiﬁer: IF x1 > h THEN ˆ y= 1 ELSE ˆ y= 0

Trace its ROC curve (considering 1 as the positive class)

•Consider the 2nd classiﬁer: IF x2 > k THEN ˆ y= 0 ELSE ˆ y= 1

Trace its ROC curve (considering 1 as the positive class)

•Which classiﬁer is the best one (1st/2nd)?

Solution:

•1st classiﬁer ROC curve (considering 1 as the positive class):

0.0 0.2 0.4 0.6 0.8 1.0

FPR

TPR

•2nd classiﬁer ROC curve (considering 1 as the positive class):

298 CHAPTER 10. NONLINEAR APPROACHES

0.0 0.2 0.4 0.6 0.8 1.0

0.2 0.4 0.6 0.8 1.0

FPR

TPR

•Which classiﬁer is the best one (1st/2nd)? The 2nd.

12. Consider a binary classiﬁcation task and the training set

x1 x2y

1 1 -1

2 0.5 -1

1.5 2.5 -1

3 1.5 1

2.5 3 1

4 2.5 1

Consider a linear perceptron initialised with the boundary line x2 = 2 which classiﬁes

as positive the points over the line. The student should:

1. Perform one step of gradient descent with stepsize 0.1 and compute the updated

coeﬃcients of the perceptron line with equation

β0 + β1x1 + β2 x2 = 0

2. Trace the initial boundary line (in black), the updated boundary line (in red)

and the training points.

Solution:

In the initial perceptron β0 =− 2, β1 = 0 and β2 = 1. The misclassiﬁed points are

the third and the fourth (opposite label). Since

∂R

∂β =− X

miscl

yixi = 1 .5

2.5 −  3

1. 5 =  − 1 . 5

1

and ∂R

∂β0

=−X

miscl

yi = 0

after one iteration β0 remains the same while

β t+1

βt+1

2= β t

βt

2−0. 1∗ −1. 5

1 =  0

1 +  0.15

−0.1 =  0.15

0. 9

Updated coeﬃcients of the perceptron line are then

•β0 = −2

•β1 = 0 .15

•β2 = 0 .9

10.5. EXERCISES 299

13. Consider the data set in exercise 9 and ﬁt to it a Radial Basis Function with 2 basis

functions having as parameters µ (1) =− 0. 5 and µ (2) = 0. 5. The equation of the

basis function is

ρ( x, µ) = exp−(x −µ)2

The student should

1. write in matrix notation the linear system to be solved for obtaining the weights

of the radial basis function

2. compute the weights of the radial basis function

Hint:

A= a 11 a12

a12 a22  ⇒ A −1 = 1

a11a22 −a2

12  a 22 −a 12

−a12 a11 

Solution:

1. matrix notation w = (X0X ) −1 X0 Y where







0. 779 0.105

0. 961 0.527

0. 779 0.779

0. 698 0.852

0. 613 0.914

0. 445 0.990







2. weights of the radial basis function : w = [1. 25,− 0 .27]

14. Let us consider a classiﬁcation task with 3 binary inputs and one binary output.

Suppose we collected the following training set

x1x2x3 y

0 1 0 1

0 0 1 0

0 1 0 0

0 1 1 0

1 0 0 0

1 0 1 1

1 1 0 0

0 1 1 0

1 0 1 0

1 0 0 0

1 1 0 0

0 1 1 0

1. Estimate the following quantities by using the frequency as estimator of prob-

ability

•Prob {y= 1}

•Prob {y= 1 |x1 = 0}

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0}

2. Compute the classiﬁcation returned by using the Naive Bayes Classiﬁer for the

input x1 = 0,x2 = 0,x3 = 0.

3. Suppose we test a classiﬁer for this task and that we obtain a misclassiﬁcation

error equal to 20%. Is it more accurate than a zero classiﬁer, i.e. a classiﬁer

returning always zero?

Solution: Let us note that N = 12

1. • [

Prob {y = 1}= 2/12 = 1/6

•[

Prob {y = 1|x1 = 0} = 1

300 CHAPTER 10. NONLINEAR APPROACHES

•[

Prob {y = 1|x1 = 0,x2 = 0, x3 = 0} cannot be estimated using the fre-

quency since there is no observation where x1 = 0,x2 = 0, x3 = 0

2. Since

[

Prob {y = 1|x1 = 0, x2 = 0,x3 = 0} ∝

[

Prob {x1 = 0|y = 1} [

Prob {x2 = 0|y = 1} [

Prob { x3 = 0|y = 1} [

Prob {y = 1}=

(0. 5∗ 0. 5 ∗ 0. 5∗ 1/ 6) = 0.02

and

[

Prob {y = 0|x1 = 0, x2 = 0,x3 = 0} ∝

[

Prob { x1 = 0|y = 0} [

Prob {x2 = 0|y = 0} [

Prob { x3 = 0|y = 0} [

Prob {y = 0}=

= (5/ 10 ∗ 4 / 10 ∗ 5/ 10 ∗ 5/ 6) = 0.08

the NB classiﬁcation is 0

3. A zero classiﬁer would return always the class with the highest a priori prob-

ability, that is the class 0. Its misclassiﬁcation error would be then 1/6. Since

1/ 5> 1 / 6 the classiﬁer is less accurate than the zero classiﬁer.

15. Let us consider a classiﬁcation task with 3 binary inputs and one binary output.

Suppose we collected the following training set

x1x2x3 y

0 1 0 1

0 0 1 0

0 1 0 0

0 1 1 0

1 0 0 0

1 0 1 1

1 1 0 0

0 1 1 0

1 0 1 0

1 0 0 0

1 1 0 0

0 1 1 0

1. Estimate the following quantities by using the frequency as estimator of prob-

ability

•Prob {y= 1}

•Prob {y= 1 |x1 = 0}

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0}

2. Compute the classiﬁcation returned by using the Naive Bayes Classiﬁer for the

input x1 = 0,x2 = 0,x3 = 0.

3. Suppose we test a classiﬁer for this task and that we obtain a misclassiﬁcation

error equal to 20%. Is it working better than a zero classiﬁer, i.e. a classiﬁer

ignoring the value of the inputs?

Solution: Let us note that N = 12

1. • [

Prob {y = 1}= 2/12 = 1/6

•[

Prob {y = 1|x1 = 0} = 1

•[

Prob {y = 1|x1 = 0,x2 = 0, x3 = 0} cannot be estimated using the fre-

quency since there is no observation where x1 = 0,x2 = 0, x3 = 0

10.5. EXERCISES 301

2. Since

[

Prob {y = 1|x1 = 0,x2 = 0,x3 = 0} ∝

[

Prob {x1 = 0|y = 1} [

Prob {x2 = 0|y = 1} [

Prob {x3 = 0|y = 1} [

Prob {y = 1}=

(0. 5∗ 0. 5∗ 0. 5 ∗ 1/ 6) = 0.02

and

[

Prob {y = 0|x1 = 0,x2 = 0,x3 = 0} ∝

[

Prob {x1 = 0|y = 0} [

Prob {x2 = 0|y = 0} [

Prob {x3 = 0|y = 0} [

Prob {y = 0}=

(5/ 10 ∗ 4/ 10 ∗ 5/ 10 ∗ 5/ 6) = 0.08

the NB classiﬁcation is 0

3. A zero classiﬁer would return always the class with the highest a priori prob-

ability, that is the class 0. Its misclassiﬁcation error would be then 1/6. Since

1/ 5> 1 / 6 the classiﬁer is less accurate than the zero classiﬁer.

16. Consider a regression task with input x and output y . Suppose we observe the

following training set

X Y

0 .1 1

0 0.5

-0.3 1.2

0.2 1

0.4 0.5

0.1 0

-1 1.1

and that the prediction model is constant. Compute an estimation of its mean

integrated squared error by leave-one-out.

Solution: Since the leave-one-out error is

e−i

i=y i −P N

j=1 ,j6= i y j

N−1

we can compute the vector of errors in leave-one-out

e−1

11- 0.716=0.283

e−2

20.5- 0.8= -0.3

e−3

31.2- 0.683=0.516

e−4

41- 0.716=0.283

e−5

50.5- 0.8= -0.3

e−6

60- 0.883=-0.883

e−7

71.1- 0.7=0.4

and then derive the MISE estimation

MISEloo = P N

i=1(e −i

i) 2

N= 0 .22

17. Consider a regression task with input x and output y . Suppose we observe the

following training set

X Y

0 .1 1

0 0.5

-0.3 1.2

0.3 1

0.4 0.5

0.1 0

-1 1.1

302 CHAPTER 10. NONLINEAR APPROACHES

and that the prediction model is a KNN (nearest neighbour) where K = 1 and the

distance metric is euclidean. Compute an estimation of its mean squared error by

leave-one-out.

Solution:

The leave-one-out error is

e−i

i=y i −y ∗

where y ∗

iis the value of the target associated to x∗

iand x∗

iis the nearest neighbour

of xi . Once we rank the training set according to the input value

X Y

-1 1.1

-0.3 1.2

0 0.5

0.1 1

0.1 0

0.3 1

0.4 0.5

we can compute the vector of errors in leave-one-out

e−1

11.1-1.2=-0.1

e−2

21.2- 0.5= 0.7

e−3

30.5- 1=-0.5

e−4

41- 0=1

e−5

50- 1= -1

e−6

61-0.5=0.5

e−7

70.5- 1=-0.5

and then derive the MISE estimation

MISEloo = P N

i=1(e −i

i) 2

N= 0 .464

18. Consider a regression task with input x and output y . Suppose we observe the

following training set

X Y

0.5 1

1 1

-1 1

-0.25 1

0 0.5

0.1 0

0.25 0.5

Trace the estimation of the regression function returned by a KNN (nearest neighbor)

where K = 3 on the interval [− 2, 1].

Solution: The resulting graph is piecewise constant and each piece has an ordinate

equal to the mean of three points. Once ordered the points according to the abscissa

X Y

x1 -1 1

x2 -0.25 1

x3 0 0.5

x4 0.1 0

x5 0.25 0.5

x6 0.5 1

x7 1 1

10.5. EXERCISES 303

these are the ﬁve sets of 3 points

x1 , x2, x3 ⇒ ˆ y= 2. 5/ 3 (10.5.60)

x2 , x3, x4 ⇒ ˆ y= 0 .5 (10.5.61)

x3 , x4, x5 ⇒ ˆ y= 1 /3 (10.5.62)

x4 , x5, x6 ⇒ ˆ y= 0 .5 (10.5.63)

x5 , x6, x7 ⇒ ˆ y= 2 .5 /3 (10.5.64)

The transitions from xi , xi+1 , xi+2 to xi+1 , xi+2 , xi+3 , i = 1,..., 4 occur at the x = q

points where q− xi = xi+3 − q⇒q =x i+3+x i

−2.0 −1.5 −1.0 −0.5 0.0 0.5 1.0

0.0 0.5 1.0 1.5 2.0

19. Consider a supervised learning problem, a training set of size N= 50 and a neural

network predictor with a single hidden layer. Suppose that we are able to compute

the generalisation error for diﬀerent number H of hidden nodes and we discover

that the lowest generalisation error occurs for H= 3. Suppose now that the size of

the training set increases (N= 500). For which value of Hwould you expect the

lowest generalisation error? Equal, larger or smaller than 3? Justify your answer by

reasoning on the bias/variance trade-oﬀ in graphical terms (Figure).

Solution:

According to 7.7.46 the MISE generalisation error may be decomposed as the sum

of the squared bias, the model variance and the noise variance.

In Figure 10.29 we depict the ﬁrst setting in black and the second one (i.e. increased

training set size) in red.

The relationship between the squared bias and the capacity of the model (number

H) is represented by the dashed line and the relationship between the variance

and the capacity is represented by the continuous thin line. The MISE (taking its

minimum in H = 3) is represented by the black thick line. Note that in the ﬁgure

we do not consider the noise variance since we are comparing two models for the

same regression task and then the noise variance is in this case an irrelevant additive

term.

If the training set size increases we can expect a variance reduction. This means

that the minimum of the MISE term will move to right. We should then expect that

the optimal number of hidden layers is H > 3 .

Note that additional observations have no impact on the squared bias while they

contribute to reduce the variance (red thin line). From the red thick line denoting

the MISE of the second setting, it appears that arg minH MSE (H ) moved to the

right.

20. Consider a feedforward neural network with two inputs, no hidden layer and a

logistic activation function. Suppose we want to use backpropagation to compute

the weights w1 and w2 and that a training dataset is collected. The student should

304 CHAPTER 10. NONLINEAR APPROACHES

0123456

0.0 0.5 1.0 1.5 2.0

complexity

MSE

Figure 10.29:

1. Write the equation of the mapping between x1 , x2 and y.

2. Write the two iterative backpropagation equations to compute w1 and w2 .

Solution:

1. ˆ y= g( z) = g( w1 x1 + w2x2 ) where g( z) = 1

1+e −z and g 0 (z ) = e −z

(1+e−z )2

10.5. EXERCISES 305

2. The training error is

E=PN

i=1(y i −ˆ yi )2

For j = 1,2

∂E

∂wj

=− 2

i=1

(yi − ˆ yi ) ∂ˆ yi

∂wj

where ∂ ˆ yi

∂wj

=g0 (zi )xij

where zi =w1 x1i +w2 x2i

The two backpropagation equations are then

wj ( k + 1) = wj ( k ) + η 2

i=1

(yi − ˆ yi ) g0 ( zi ) xij , j = 1 , 2

21. Consider a binary classiﬁcation problem and the following estimations of the condi-

tional probability [

Prob {y = 1| x} vs. the real value of the target.

Trace a precision recall and the AUC curve

[

Prob {y = 1| x} CLASS

0.6 1

0.5 -1

0.99 1

0.49 -1

0.1 -1

0.26 -1

0.33 1

0.15 -1

0.05 -1

Solution: Let us ﬁrst order the dataset in terms of ascending score

[

Prob {y = 1|x } CLASS

0.05 -1

0.10 -1

0.15 -1

0.26 -1

0.33 1

0.49 -1

0.50 -1

0.60 1

0.99 1

We let the threshold range over all the values of the score. For each value of the

threshold we deﬁne as positively classiﬁed the terms having a score bigger than the

threshold and negatively classiﬁed the terms having a score lower equal than the

threshold.

For instance for Thr=0.26 this is the returned classiﬁcation

[

Prob {y = 1|x } ˆ yCLASS

0.05 -1 -1

0.10 -1 -1

0.15 -1 -1

0.26 -1 -1

0.33 1 1

0.49 1 -1

0.50 1 -1

0.60 1 1

0.99 1 1

306 CHAPTER 10. NONLINEAR APPROACHES

Then we measure the quantity of TP, FP, TN and FN and FPR = F P / (T N + F P ),

T P R = T P /(T P +F N )

Threshold TP FP TN FN FPR TPR

0.05 3 5 1 0 5/6 1

0.10 3 4 2 0 2/3 1

0.15 3 3 3 0 1/2 1

0.26 3 2 4 0 1/3 1

0.33 2 2 4 1 1/3 2/3

0.49 2 1 5 1 1/6 2/3

0.50 2 0 6 1 0 2/3

0.60 1 0 6 2 0 1/3

0.99 0 0 6 3 0 0

0.0 0.2 0.4 0.6 0.8

0.0 0.2 0.4 0.6 0.8 1.0

FPR

22. Let us consider a classiﬁcation task with 3 binary inputs and one binary output.

Suppose we collected the following training set

x1x2x3 y

1 1 0 1

0 0 1 0

0 1 0 0

1 1 1 1

0 0 0 0

0 1 0 0

0 1 1 0

0 0 1 0

0 0 0 0

0 1 0 0

1 1 1 1

1. Estimate the following quantities by using the frequency as estimator of prob-

ability

•Prob {y= 1}

•Prob {y= 1 |x1 = 0}

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0}

2. Consider a Naive Bayes classiﬁer and compute its classiﬁcations if the same

dataset is used also for testing

3. Trace the ROC curve associated to the Naive Bayes classiﬁer if the same dataset

is used also for testing. (Hint: make the assumption that the denominator of

the Bayes formula is 1 for all test points)

Solution:

10.5. EXERCISES 307

•Prob {y= 1 }= 3/11

•Prob {y= 1 |x1 = 0 }= 0

•Prob {y= 1 |x1 = 0, x2 = 0, x3 = 0 }= 0

2. Note that the values of x1 are identical to the ones of y . Then Prob { x1 = A|y =¬ A} =

0. It follows that if use a Naive Bayes and the test dataset is equal to the train-

ing set all the predictions will coincide with the values of x1 . The training error

is then zero

3. Since all the predictions are correct the ROC curve is equal to 1 for all FPR

values

23. Let us consider a binary classiﬁcation task where the input x∈R2 is bivariate and

the categorical output variable ymay take two values: 0 (associated to red) and

1 (associated to green). Suppose that the a-priori probability is p (y =1)=0 .2

and that the inverse (or class-conditional) distributions are the bivariate Gaussian

distributions p (x |y = 0) = N (µ0 , Σ0 ) and p (x|y = 1) = N (µ1 , Σ1 ) where

•µ0 = [0 ,0]T

•µ1 = [1 ,1]T

and both Σ0 and Σ1 are diagonal identity matrices. The student should

1. by using the R function rmvnorm , sample a dataset of N = 1000 input/output

observations according to the conditional distribution described above,

2. visualise in a 2D graph the dataset by using the appropriate colors,

3. ﬁt a logistic classiﬁer to the dataset (see details below),

4. plot the evolution of the cost function J (α ) during the gradient-based minimi-

sation,

5. plot in the 2D graph the decision boundary.

Logistic regression estimates

P(y = 1| x) = exp x T α N

1 + expx T α N = 1

1 + exp−xT αN , ˆ

P(y = 0| x) = 1

1 + expx T α N

where

αN = arg min

αJ(α )

and

J( α) =

i=1  −y i x T

iα+ log(1 + exp x T

iα )

Note that αis the vector [α0 , α1, α2 ]T and that xi = [1, xi1 , xi2 ]T , i = 1,...,N.

The value of αN has to be computed by gradient-based minimisation of the cost

function J (α ) by performing I= 200 iterations of the update rule

α(τ) =α(τ−1) − ηdJ ( α (τ− 1))

dα , τ = 1,...,I

where α (0) = [0, 0, 0]T and η = 0.001.

Solution:

See the ﬁle Exercise5.pdf in the directory gbcode/exercises of the companion R

package (Appendix F).

24. Consider a binary classiﬁcation task where the input x∈R2 is bivariate and the

categorical output variable y may take two values: 0 (associated to red) and 1

(associated to green). Suppose that the a-priori probability is p (y =1)=0 .2 and

that the inverse (or class-conditional) distributions are

308 CHAPTER 10. NONLINEAR APPROACHES

•green/cross class : mixture of three Gaussians

p( x|y= 1) =

i=1

wi N( µ1i , Σ)

where in µ 11 = [1, 1]T , µ12 = [− 1,− 1], µ13 = [3, − 3]T , and w1 = 0. 2, w2 = 0.3.

•red/circel class: bivariate Gaussian p ( x |y= 0) = N(µ0 , Σ) where µ0 = [0, 0]T

The matrix Σ is a diagonal identity matrix.

The student should

•by using the R function rmvnorm, sample a dataset of N = 1000 input/output

observations according to the conditional distributions described above,

•visualise in a 2D graph the dataset by using the appropriate colours/marks,

•plot the ROC curves of the following classiﬁers

1. linear regression coding the two classes by 0 and 1,

2. Linear Discriminant Analysis where σ2 = 1,

3. Naive Bayes where the univariate conditional distributions are Gaussian,

4. k Nearest Neighbour with k = 3, 5, 10.

The classiﬁers should be trained and tested on the same training set.

•Choose the best classiﬁer on the basis of the ROC curves above.

No R package should be used to implement the classiﬁers.

Solution:

See the ﬁle Exercise6.pdf in the directory gbcode/exercises of the companion R

package (Appendix F).

Chapter 11

Model averaging approaches

All the techniques presented so far require a model selection procedure where dif-

ferent model structures are assessed and compared in order to attain the best repre-

sentation of the data. In model selection the winner-takes-all approach is intuitively

the approach that should work the best. However, recent results in machine learning

show that the ﬁnal accuracy can be improved not by choosing the model structure

which is expected to predict the best but by creating a model combining the output

of models with diﬀerent structures. The reason is that every hypothesis h ( ·, αN ) is

only an estimate of the real target and, like any estimate, is aﬀected by a bias and a

variance term. The theoretical results of Section 5.10 show that a variance reduction

can be obtained by combining uncorrelated estimators. This simple idea underlies

some of the most eﬀective techniques recently proposed in machine learning. This

chapter will sketch some of them.

11.1 Stacked regression

Suppose we have mdistinct predictors hj (· , αN ), j = 1, . . . , m obtained from a

given training set DN . For example, a predictor could be a linear model ﬁt on some

subset of the variables, a second one a neural network and a third one a regression

tree. The idea of averaging models is to design an average estimator

j=1

βj hj ( · , αN )

by linear combination which is expected to be more accurate than each of the

estimators taken individually.

A simple way to estimate the weights ˆ

βj is to perform a least-squares regression

of the output y on the m inputs hj (·, αN ). The training set for this regression is

then made by DN = {hi , yi }

y=









H= 









= 





h1 ( x1 , αN ) h2 ( x1 , αN ) . . . hm ( x1 , αN )

h1 ( x2 , αN ) h2 ( x2 , αN ) . . . hm ( x2 , αN )

h1 ( xN , αN ) h2 ( xN , αN ) . . . hm ( xN , αN )







where hi ,i = 1, . . . , N is a vector of mterms.

Once computed the least-squares solution ˆ

βthe combined estimator is

hcm ( x ) =

j=1

βj hj ( x, αN )

309

310 CHAPTER 11. MODEL AVERAGING APPROACHES

Despite its simplicity, the least-squares approach might produce poor results

since it does not take into account the correlation existing among the hj and induced

by the fact that all of them are estimated on the same training set DN .

Wolpert [196] presented an interesting idea, called stacked generalisation for

combining estimators without suﬀering of the correlation problem. This proposal

was translated in statistical language by Breiman who introduced the stacked re-

gression principle [36].

The idea consists in estimating the m parameters ˆ

βj by solving the following

optimisation task

β= arg min

i=1 

y i −

j=1

βj h(−i )

j(x i )



where h(−i)

j(x i ) is the leave-one-out estimate (8.8.2.3) of the jth model.

In other terms the parameters are obtained by performing a least-squares regres-

sion of the output y on the m inputs hj (· , α(−i)

N). The training set for this regression

is then made by DN = {h−

i, y i },i = 1, . . . , N

y=









H= 





h−





=







h1 ( x1 , α(−1)

N)h 2 (x 1 , α (−1)

N). . . h m (x 1 , α (−1)

h1 ( x2 , α(−2)

N)h 2 (x 2 , α (−2)

N). . . h m (x 2 , α (−2)

h1 ( xN , α(−N )

N)h 2 (x N , α (−N )

N). . . h m (x N , α (−N )







where hj ( xi , α(−i )

N) is the predicted outcome in x i of the jth model trained on

DN with the ith observation ( xi , yi ) set aside.

By using the cross-validated predictions hj (xi , α(−i )

N) stacked regression avoids

giving unfairly high weight to models with higher complexity. It was shown by

Breiman, that the performance of the stacked regressor improves when the coeﬃ-

cients ˆ

βare constrained to be non-negative. There is a close connection between

stacking and winner-takes-all model selection. If we restrict the minimisation to

weight vectors w that have one unit weight and the rest zero, this leads to the

model choice returned by the winner-takes-all based on the leave-one-out. Rather

than choose a single model, stacking combines them with estimated optimal weights.

This will often lead to better prediction, but less interpretability than the choice of

only one of the m models.

11.2 Bagging

A learning algorithm is informally called unstable if small changes in the training

data lead to signiﬁcantly diﬀerent models and relatively large changes of accuracy.

Unstable learners can have low bias but have typically high variance. Unstable

methods can have their accuracy improved by perturbing (i.e. generating multiple

versions of the predictor by perturbing the training set or learning method) and

combining. Breiman calls these techniques P&C methods.

The bagging technique is a P&C technique which aims to improve accuracy for

unstable learners by averaging over such discontinuities. The philosophy of bagging

11.2. BAGGING 311

Figure 11.1: Histogram of misclassiﬁcation rates of resampled trees: the vertical

line represents the misclassiﬁcation rate of the bagging predictor.

is to improve the accuracy by reducing the variance: since the generalisation error of

a predictor h ( ·,αN ) depends on its bias and variance, we obtain an error reduction if

we remove the variance term by replacing h (·,αN ) with ED N [h(·,αN )]. In practice,

since the knowledge of the sampling distribution of the predictor is not available, a

non-parametric estimation is required.

Consider a dataset DN and a learning procedure to build a hypothesis αN from

DN . The idea of bagging or bootstrap aggregating is to imitate the stochastic process

underlying the realisation of DN . A set of Brepeated bootstrap samples D(b)

b= 1 , . . . , B are taken from DN . A model α(b)

Nis built for each D ( b)

N. A ﬁnal

predictor is built by aggregating the B models α(b)

N. In the regression case, the

bagging predictor is

hbag ( x ) = 1

b=1

h( x, α(b)

In the classiﬁcation case a majority vote is used.

R script

The R script bagging.R shows the eﬃcacy of bagging as a remedy against overﬁt-

ting.

Consider a dataset DN = {xi , yi } ,i = 1, . . . , N of N = 100 i.i.d. normally dis-

tributed inputs x ∼ N ([0, 0, 0], I ). Suppose that y is linked to x by the input/output

relation

y= x2

1+ 4 log(|x 2 |)+5 x3 + 

where  ∼ N (0, 0 . 25) represents the noise. Let us train a single-hidden-layer neural

network with s = 25 hidden neurons on the training set (Section 10.1.1). The

prediction accuracy on the test set (N ts = 100) is \

MISEts = 70. 86. Let us apply a

bagging combination with B = 50 (R-ﬁle. The prediction accuracy on the test set

of the bagging predictor is \

MISEts = 6. 7. This shows that the bagging combination

reduces the overﬁtting of the single neural network. Below there is the histogram

of the \

MISEts accuracy of each bootstrap repetition. Figure (11.1) shows that the

bagging predictor is much better than average.

•

312 CHAPTER 11. MODEL AVERAGING APPROACHES

Tests on real and simulated datasets showed that bagging can give a substantial

gain of accuracy. The vital element is the instability of the prediction method.

If perturbing the learning set can cause signiﬁcant changes in the predictor con-

structed, then bagging can improve accuracy. On the other hand, it can slightly

degrade the performance of stable procedures. There is a cross-over point between

instability and stability at which bagging stops improving.

Bagging demands the repetition of B estimations of h (·, α(b)

N) but avoids the

use of expensive validation techniques (e.g. cross-validation). An open question,

as in bootstrap, is to decide how many bootstrap replicates to carry out. In his

experiments, Breiman suggests that B≈ 50 is a reasonable ﬁgure.

Bagging is an ideal procedure for parallel computing. Each estimation of h (· , α(b)

N),

b= 1 , . . . , B can proceed independently of the others. At the same time, bagging

is a relatively easy way to improve an existing method. It simply needs adding

1. a loop that selects the bootstrap sample and sends it to the learning machine

and

2. a back-end to perform the aggregation.

Note however that if the original learning machine has an interpretable structure

(e.g. classiﬁcation tree), this is lost for the sake of increased accuracy.

11.3 Boosting

Boosting is one of the most powerful learning ideas introduced in the last ten years.

Boosting is a general method which attempts to boost the accuracy of any given

learning algorithm. It was originally designed for classiﬁcation problems, but it

can proﬁtably be extended to regression as well. Boosting [75, 168] encompasses a

family of methods. The focus of boosting methods is to produce a series of weak

learners in order to produce a powerful combination. A weak learner is a learner

that has accuracy only slightly better than chance.

The training set used for each member of the series is chosen based on the

performance of the earlier classiﬁer(s) in the series. Examples that are incorrectly

predicted by previous classiﬁers in the series are chosen more often than examples

that were correctly predicted.

Thus Boosting attempts to produce new classiﬁers that are better able to predict

examples for which the current ensemble's performance is poor. Unlike Bagging,

the resampling of the training set is dependent on the performance of the earlier

classiﬁers. The two most important types of boosting algorithms are the Ada Boost

(Adaptive Boosting) algorithm (Freund, Schapire, 1997) and the Arcing algorithm

(Breiman, 1996).

11.3.1 The Ada Boost algorithm

Consider a binary classiﬁcation problem where the output take values in {− 1,1}.

Let DN be the training set. A classiﬁer is a predictor h (· ) which given an input

x, produces a prediction taking one of the values {− 1 ,1} . A weak classiﬁer is one

whose misclassiﬁcation error rate is only slightly better than random guessing.

The purpose of boosting is to sequentially apply the weak classiﬁcation algo-

rithm to repeatedly modiﬁed versions of the data, thereby producing a sequence of

classiﬁers hj (· ), j = 1, . . . , m . The predictions of the m weak classiﬁers are then

combined through a weighted majority vote to produce the ﬁnal prediction

hboo = sign 



j=1

αj hj ( x, αN )



11.3. BOOSTING 313

The weights αj of the diﬀerent classiﬁers are computed by the algorithm. The

idea is to give stronger inﬂuence to the more accurate classiﬁers in the sequence.

At each step, the boosting algorithm samples Ntimes from a distribution won the

training set which put a weight wi on each example (xi , yi ), i = 1, . . . , N of DN

Initially, the weights are all set to wi = 1/N so that the ﬁrst step simply trains

the classiﬁer in the standard manner. For each successive iteration j = 1, . . . , m

the probability weights are individually modiﬁed, and the classiﬁcation algorithm

is re-applied to the resampled training set.

At the generic jth step the observations that were misclassiﬁed by the classiﬁer

hj−1 (· ) trained at the previous step, have their weights wi increased, whereas the

weights are decreased for those that were classiﬁed correctly. The rationale of the

approach is that, as the iterations proceed, observations that are hard to classify

receive ever-increasing inﬂuence and the classiﬁer is forced to concentrate on them.

Note the presence in the algorithm of two types of weights: the weights αj ,j =

1, . . . , m that measure the importance of the classiﬁers and the weights wi ,i =

1, . . . , N that measure the importance of the observations.

Weak learners are added until some desired low training error has been achieved.

This is the algorithm in detail:

1. Initialise the observation weights wi = 1/N ,i = 1, . . . , N .

2. For j = 1 to m:

(a) Fit a classiﬁer hj ( ·) to the training data obtained by resampling DN

using weights wi .

(b) Compute the misclassiﬁcation error on the training set

MME(j)

emp =P N

i=1 w i I(y i 6=h j (x i ))

i=1 w i

αj = log((1 − \

MME(j)

emp)/\

MME(j)

emp)

Note that αj > 0 if \

MME(j)

emp ≤1/ 2 (otherwise we stop or we restart)

and that αj gets larger as \

MME(j)

emp gets smaller.

3. (d) For i = 1, . . . , N set

wi ← wi ( exp[− α j ] if correctly classiﬁed

exp[αj ] if incorrectly classiﬁed

(e) The weights are normalised to ensure that wi represents a true distribu-

tion.

4. Output of the weighted majority vote

hboo = sign 



j=1

αjhj ( x, αN ) 



R script

The R script boosting.R tests the performance of the Ada Boost algorithm in a

classiﬁcation task. Consider the medical dataset Pima obtained by a statistical

survey on women of Pima Indian heritage. This dataset reports the presence of

314 CHAPTER 11. MODEL AVERAGING APPROACHES

diabetes in Pima Indian women together with other clinical measures (blood pres-

sure, insulin, age,...). The classiﬁcation task is to predict the presence of diabetes

as a function of clinical measures. We consider a training set of N= 40 and a

test set of 160 points. The classiﬁer is a simple classiﬁcation tree which returns a

misclassiﬁcation rate \

MMEts = 0. 36. We use a boosting procedure with m = 15

to improve the performance of the weak classiﬁer. The misclassiﬁcation rate of the

boosted classiﬁer is \

MMEts = 0.3.

•

Boosting has its roots in a theoretical framework for studying machine learning

called the PAC learning model. Freund and Scapire proved that the empirical error

of the ﬁnal hypothesis hboo is at most

j=1 "2 r\

MME(j)

emp ∗(1 − \

MME(j)

emp)#

They showed also how to bound the generalisation error.

11.3.2 The arcing algorithm

This algorithm was proposed as a modiﬁcation of the original Ada Boost algorithms

by Breiman. It is based on the idea that the success of boosting is related to the

adaptive resampling property where increasing weight is placed on those examples

more frequently misclassiﬁed. ARCing stays for Adaptive Resampling and Combin-

ing. The complex updating equations of Ada Boost are replaced by much simpler

formulations. The ﬁnal classiﬁer is obtained by unweighted voting. This is the

ARCing algorithm in detail:

1. Initialise the observation weights wi = 1/N ,i = 1, . . . , N .

2. For j = 1 to m:

(a) Fit a classiﬁer hj to the training data obtained by resampling DN using

weights wi .

(b) Let ei the number of misclassiﬁcations of the ith example by the jclas-

siﬁers h1 , . . . , hj .

wi =1 + e 4

i=1(1 + e 4

3. The output is obtained by unweighted voting of the m classiﬁers hj .

R script

The R ﬁle arcing.R tests the performance of the ARCing algorithm in a classiﬁca-

tion task. Consider the medical dataset Breast Cancer obtained by Dr. William H.

Wolberg (physician) at the University of Wisconsin Hospital in USA. This dataset

reports the class of cancer (malignant and benign) and other properties (clump

thickness, uniformity of cell size, uniformity of cell shape, marginal adhesion,...).

The classiﬁcation task is to predict the class of breast cancer on the basis of clinical

measures. We consider a training set of size N= 400 and a test set of size 299.

The classiﬁer is a simple classiﬁcation tree which returns a misclassiﬁcation rate

MMEts = 0. 063. We use an arcing procedure with m= 15. It gives a misclassiﬁca-

tion rate \

MMEts = 0.010.

11.4. RANDOM FORESTS 315

•

Boosting is a recent and promising technique which is simple and easy to program.

Moreover, it has few parameters (e.g. max number of classiﬁers) to tune. Boosting

methods advocate a shift in the attitude of the learning-system designer: instead

of trying to design a learning algorithm which should be accurate over the entire

space, she can instead focus on ﬁnding weak algorithms that only need to be better

than random.

Furthermore, a nice property of Ada Boost is its ability to identify outliers.

11.3.3 Bagging and boosting

This section makes a short comparison of bagging and boosting techniques. First of

all, in terms of bias/variance trade-oﬀ, it is important to stress that the rationale of

bagging is to reduce variance of low bias (and then high variance) learners trained

on i.d. (identically distributed) data while boosting aims to reduce sequentially the

bias of weak-learners trained on non i.d. data.

Like bagging, boosting avoid the cost of heavy validation procedures and, like

bagging, boosting trades accuracy for interpretability. As for bagging, the main

eﬀect of boosting is to reduce variance and it works eﬀectively for high variance

classiﬁers. However, unlike bagging, boosting cannot be implemented in parallel,

since it is based on a sequential procedure.

In terms of experimental accuracy, several research works (e.g. Breiman's work)

show that boosting seems outperforms bagging. Also, a number of recent theoretical

results show that boosting is fundamentally diﬀerent from bagging [98].

Some caveats are notwithstanding worth to mention: the actual performance of

boosting on a particular problem is dependent on the data and the nature of the

weak learner. Also boosting can fail to perform well given insuﬃcient data, overly

complex weak hypothesis, and deﬁnitely too weak hypothesis.

11.4 Random Forests

Ensemble learning is eﬃcient when it combines low bias and independent estimators,

like non pruned decision trees.

Random Forests (RF) is an ensemble learning technique proposed by Breiman [38]

which combines bagging and random feature selection by using a large number of

non pruned decision trees. The rationale of RF is to reduce the variance by decor-

relating as much as possible the single trees. This is achieved in the tree-growing

process through a random selection of the input variables. In a nutshell, the algo-

rithm consists in:

1. generating by bootstrap a set of Btraining sets,

2. ﬁtting to each of them a decision tree hb ( ·, αb ), b = 1, . . . , B where the set of

variables considered for each split (Section 10.1.4.3) is a random subset of size

n0 of the original one (feature bagging),

3. storing at each split for the corresponding split variable, the improvement of

the cost function,

4. returning as the ﬁnal prediction the average of the Bpredictions

hrf ( x ) = 1

b=1

hb ( x, αb )

in a regression task and the majority vote in a classiﬁcation task,

316 CHAPTER 11. MODEL AVERAGING APPROACHES

5. returning for each variable an importance measure.

Suppose that the B trees in the forest are almost unbiased, have a comparable

variance Var [hb ] = σ2 and a mutual correlation ρ . The RF regression predictor hrf

is then almost unbiased and from (3.10.88) its variance is

Var [hrf ] = (1 −ρ2 )

B+ ρσ2

It appears then by increasing the forest size Band making the trees as uncorrelated

as possible, a Random Forest strategy reduces the resulting variance.

A rule of thumb consists of setting the size of the random subset to n0 =√ n . The

main hyperparameters of RF are the hyperparameters of single trees (e.g. depth,

max number of leaves), the number Bof trees and the size n0 of the random feature

set. Note that by reducing n0 we make the trees more decorrelated, yet we increase

the bias of each single tree (and then of the RF) by constraining its number of

features. In particular, a too small number n0 may be detrimental to accuracy in

conﬁgurations with very large nand small number of informative features.

11.4.1 Why are Random Forests successful?

Random Forests are often considered among the best "oﬀ-the-shelf" learning al-

gorithms since they do not require complex tuning to perform reasonably well on

chhallenging tasks. There are many reasons for their success [73]: (i) they use an

out-of-bag (Section 7.10.1) strategy to eﬀectively manage the bias/variance trade-oﬀ

and to assess the importance of input variables, (ii) since based on trees, they easily

cope with mixtures of numeric and categorical predictor variables, (iii) they are

resilient to input outliers and invariant under monotone input transformation, (iv)

they embed a feature ranking mechanism based on an importance measure related

to the average cost function decrease during splitting, (v) they are fast to construct

and can be made massively parallel and (vi) there exist a number of very eﬀec-

tive implementations (e.g. in the R package randomForest) and enhanced version

(notably gradient boosting trees).

11.5 Gradient boosting trees

Gradient boosting (GB) trees are an enhanced version of averaging algorithms which

rely on combining mtrees according to a forward stage-wise additive strategy [98].

The strategy consists of adding one component (e.g. a tree) at the time: after m

iterations, the resulting model is the sum of the Mindividual trees

hm ( x ) =

j=1

T( x, αj )

Given j− 1 trees, the jth tree is learned such to compensate the error between

the target and the current ensemble prediction hj−1 (x ). This means that

αj = arg min

i=1

L( yi , hj−1 ( xi ) + T ( xi , α)) (11.5.1)

where αj contains the jth tree parameters, e.g. the set of disjoint regions and the

local model holding in each region. Note that in the forward stage-wise philosophy,

no adjustment of the previously added trees is considered.

11.6. CONCLUSION 317

It can shown that, for a regression task with a squared error loss function L , the

solution αj corresponds to the regression tree that best predicts the residuals

ri = yi − hj−1 ( xi ) , i = 1 , . . . , N

Gradient based versions exists for other diﬀerentiable loss criteria and for clas-

siﬁcation tasks. Also weighted versions of (11.5.1) exist

(αj , wj ) = arg min

α,w

i=1

L( yi , hj−1 ( xi ) + wT ( xi , α))

where the contribution αj of each new tree is properly tuned.

A stochastic version of gradient boosting has been proposed in [77] where at

each iteration only a subsample of the training set is used to train the new tree.

Though gradient-boosting algorithms are considered ones of the most promising

in complex learning tasks, it is recommended to remember that their accuracy

depends, like all learning algorithms, on a number of hyperparameters, notably the

size of the constituent trees, the number mof iterations, contribution wj of each

tree, loss function degree and subsample size.

11.6 Conclusion

The averaging of ensembles of estimators relies on the counterintuitive principle

that combining predictors is (most of the time) more convenient than selecting

(what seems to be) the best. This principle is (probably together with the idea of

regularisation) one of the most genial and eﬀective ideas proposed by researchers in

Machine Learning1. Most state-of-the-art learning strategies do owe a considerable

part of their success to the integration of the combination principle. Such principle

is so powerful that some authors suggest nowadays not to include combination in

the assessment of learning strategies (e.g. in new publications) given the risk that

the only visible beneﬁcial eﬀect is the one due to the combination.

The fact that this idea might appear counterintuitive sheds light on the stochas-

tic nature of the learning problem and the importance of taking a stochastic per-

spective to really grasp the problem of learning and generalising from a ﬁnite set of

observations.

11.7 Exercises

1. Verify by Monte Carlo simulation the relations (5.10.31) and 5.10.30 concerning the

combination of two unbiased estimators.

Hint: deﬁne an estimation task (e.g. estimate the expected value of a random

variable) and choose two unbiased estimators.

1...and a note of distinction should be here deﬁnitely attributed to the seminal work of re-

searchers like Jerome H. Friedman and Leo Breiman.

318 CHAPTER 11. MODEL AVERAGING APPROACHES

Chapter 12

Feature selection

In many challenging learning tasks, the number of inputs (or features) may be

extremely high: this is the case of bioinformatics [167] where the number of variables

(typically markers of biological activity at diﬀerent functional levels) may go up to

hundreds of thousands. The race to high-throughput measurement techniques in

many domains allows us to easily foresee that this number could grow by several

orders of magnitude.

Using such a large number of features in learning may negatively aﬀect general-

isation performance, especially in the presence of irrelevant or redundant features.

Nevertheless, traditional supervised learning algorithms techniques have been de-

signed for supervised tasks where the ratio between the input dimension and the

training size is small, and most inputs (or features) are informative. As a conse-

quence, their accuracy may rapidly degrade when used in tasks with few observa-

tions and a huge number of inputs.

At the same time, it is common to make the assumption that data are sparse

or possess an intrinsic low dimensional structure. This means that most input

dimensions are correlated, only a few of them contain information or equivalently

that most dimensions are irrelevant for the learning task.

For this reason, learning pipelines include more and more a feature selection

phase aiming to select a small subset of informative (or relevant) features to capture

most of the signal and avoid variance and instability issues during learning. In that

sense, feature selection can be seen as an instance of model selection problem where

the alternative models do not diﬀer in terms of functional representation but in

terms of the used subset of inputs.

Example

This example illustrates the impact of the number of features on the model variance

in a learning task with a comparable number of features and observations. Let us

consider a linear regression dependency

y=β0 +β1 x1 +β2 x2 +β3 x3 +β4 x4 + w

where Var [w ]=0 .5, β0 = 0 . 5, β1 =− 0 . 5, β2 = 0.5, β3 =− 0 .5, β4 = 0 . 5. Suppose

we collect a dataset of N= 20 input/output observations where the input set

(n = 8) contains, together with the four variables x1 ,...,x4 , a set of 4 irrelevant

variables x5 ,..., x8 .

Let us consider a set of linear regression models with an increasing number of

features, ranging from zero (constant model) to 8.

The script bv linfs.R illustrates the impact of the number of features on the

average bias (estimated by Monte Carlo) and the average variance (both analytical

319

320 CHAPTER 12. FEATURE SELECTION

Figure 12.1: Trade-oﬀ bias/variance for diﬀerent number of features. Bias and

variance are averaged over the set of Ninputs.

and Monte Carlo estimated) of the predictors. Figure 12.1 shows that the larger is

the number of features, the higher is the prediction variance. Note that the analyt-

ical form of the variance of a linear model prediction is presented in Section 9.1.14.

The bias has the opposite trend, reaching zero once the 4 inputs x 1,...,x4 are

included in the regression model. Overall, the more variables are considered, the

more bias is reduced at the cost of an increased variance. If a variable has no pre-

dictive value (e.g. it belongs to the set x5 ,...,x8 ), considering it merely increases

the variance with no beneﬁt in terms of bias reduction. In general, if the addition

of a variable has a small impact on bias then the increase in prediction variance

may exceed the beneﬁt from bias reduction [132]. The role of a feature selection

technique should be to detect those variables and remove them from the input set.

•

The beneﬁts of feature selection have been thoroughly discussed in literature [89,

91]:

•facilitating data visualisation and data understanding,

•reducing the measurement and storage requirements,

•reducing training and utilisation times of the ﬁnal model,

•defying the curse of dimensionality to improve prediction performance.

At the same time, feature selection implies additional time for learning since it

introduces an additional layer to the search in the model hypothesis space.

12.1 Curse of dimensionality

Feature selection addresses what is known in several scientiﬁc domains as the curse

of dimensionality. This term, coined by R E Bellman, refers to all computational

problems related to large dimensional modelling tasks.

The main issue in supervised learning is that the sparsity of data increases

exponentially with the dimension n. This can be illustrated by several arguments.

Let us consider a ndimensional space and a unit volume around a query point

xq ∈Rn (Figure 12.2) [98]. Let V < 1 be the volume of a neighbourhood hypercube

of edge d . It follows that dn =V and d =V 1/n . Figure 12.3 illustrates the link

between neighbourhood volume Vand edge size dfor diﬀerent values of n . It

12.1. CURSE OF DIMENSIONALITY 321

n=1 d=1/2 V=1/2

n= 2 d=1/2 V=1/4

n=3 d=1/2 V=1/8

Figure 12.2: Locality and dimensionality of the input space for diﬀerent values of

n: unit volume (in black) around a query point (circle) containing a neighbourhood

(in red) of volume Vand edge d.

322 CHAPTER 12. FEATURE SELECTION

Figure 12.3: Neighbourhood volume vs. edge size for diﬀerent values of n.

appears that for a given neighbourhood volume V, the edge length increases by

increasing n while for a given edge length d, the neighbourhood volume decreases

by increasing n . For instance if V = 0. 5 we have d = 0. 7,0.87,0. 98 for n = 2, 5,50;

if V = 0. 1 we have d = 0. 3, 0.63,0. 95 for n = 2, 5, 50. This means that for n = 50

we need to have an edge length which is 95% of the unit length if we want to barely

cover 10% of the total volume.

Let us now assess the impact of dimensionality on the accuracy of a local learn-

ing algorithm (e.g. knearest neighbour) by considering the relation between the

training set size N, the input dimension n and the number of neighbours k . If

the N points are uniformly distributed in the unit volume around the query point,

the number of neighbours kin the neighbourhood Vamounts to roughly k = NV .

Given the value of N and k (and consequently the local volume V) the edge dof the

neighbourhood increases with the dimension nand converges rapidly to one (Fig-

ure 12.4). This implies that if we use a kNN (nearest neighbour) learner for two

supervised learning tasks with same Nbut diﬀerent n, the degree of locality of the

learner (represented by the length of d) is the smaller the larger is n . Analogously

if N and 0 < d < 1 are ﬁxed, the number k = Ndn of neighbours in Vdecreases by

increasing n . In other terms, as nincreases the amount of local data goes to zero

(Figure 12.5) or equivalently all data sets are sparse for large n.

Let us now consider the case where k > 0 and 0 < d < 1 (degree of locality) are

ﬁxed and Nmay be adjusted (e.g. by observing more points). Since

N= k/dn

we need to exponentially grow the size of the training set Nto guarantee a constant

kfor increasing n. Suppose that k= 10, d= 0 .1 and N= 100 for n= 1. If we

want to preserve the same number kof neighbours for increasing n then N has to

grow according to the following law

N= k/dn =10

(1/10)n = 10 n+1

For instance we need to observe N = 106 observations for n = 5 if we want the same

degree of locality we had for n= 1. This implies that given two supervised learning

tasks (one with n = 1 and the other with n >> 1), the second should be trained

with a number Nof a much higher order of magnitude (Figure 12.6) to guarantee

the same degree of locality of the n = 1 conﬁguration.

Another interesting result about the impact of dimensionality on data distribu-

tion is the following: given N observations uniformly distributed in a ndimensional

unit ball centred at the origin, the median of the distance from the origin to the

closest data point is (1 − 1/ 2 1/N ) 1/n (Figure 12.7).

12.1. CURSE OF DIMENSIONALITY 323

Figure 12.4: Neighbourhood edge size vs. dimension n(for ﬁxed N and k)

Figure 12.5: Number of neighbours K vs. dimension n for ﬁxed N and d

324 CHAPTER 12. FEATURE SELECTION

Figure 12.6: Number of training examples required to preserve the same kind of

locality obtained for n = 1 with k= 10 and d= 0.1

020406080100

0.0 0.2 0.4 0.6 0.8

N=1M points

median distance

Figure 12.7: Median nearest neighbour as a function of nfor very large N.

12.2. APPROACHES TO FEATURE SELECTION 325

All those considerations should sound like a warning for those willing to extend

local learning approaches to large dimensional settings where familiar notions of

distance and closeness lose their meaning and relevance. Large dimensionality in-

duces high sparseness with negative impact on predictive accuracy as shown by the

bias/variance decomposition in (10.1.47). For a ﬁxed Nand by increasing nthe

algorithm is more and more exposed to one of those two low generalisation con-

ﬁgurations: i) too small k, i.e. too few points are close to the query points (with

negative impact in terms of variance) or ii) too large d implying that the nearest

neighbours are not suﬃciently close the query point (with negative impact on bias).

Though, from a bias/variance perspective, the curse of dimensionality is partic-

ularly harmful for local learning strategies, the other learning strategies should not

be considered immune either. A too large n/N ratio implies an overparametriza-

tion of the learned hypothesis and a consequent increase of the variance term in the

generalisation error which is hardly compensated by the related bias reduction. For

this reason, the adoption of a feature selection step is more and more common in

modern machine learning pipelines.

12.2 Approaches to feature selection

There are three main approaches to feature selection:

•Filter methods: they are preprocessing methods. They attempt to assess

the merits of features from the data, ignoring the eﬀects of the selected feature

subset on the learning algorithm's performance. Examples are methods that

select variables by ranking them through compression techniques (like PCA

or clustering) or computing correlation with the output.

•Wrapper methods: these methods assess subsets of variables according to

their usefulness to a given predictor. The method searches a good subset using

the learning algorithm itself as part of the evaluation function. The problem

boils down to a problem of stochastic state-space search. Examples are the

stepwise methods proposed in linear regression analysis (notably the leaps

subset selection algorithm available in R [15]).

•Embedded methods: they perform variable selection as part of the learn-

ing procedure and are usually speciﬁc to given learning machines. Examples

are classiﬁcation trees, random forests, and methods based on regularisation

techniques (e.g. lasso)

Note that, in practice, hybrid strategies combining the three approaches above

are often considered as well. For instance in the case of a huge dimensional task

(e.g. n > 1000K as in epigenetics) it would make sense to ﬁrst reduce the size of

features to a more reasonable size (e.g. some thousands or hundreds of features) by

ﬁltering and then use some search approaches within this smaller space.

12.3 Filter methods

Filter methods are commonly used in very large dimensional tasks (e.g. n > 2000)

for the following reasons: they easily scale to very high-dimensional datasets, they

are quick because computationally simple, and they are independent of the classi-

ﬁcation algorithm. Also, since feature selection needs to be performed only once,

they can be integrated into validation pipelines comparing several classiﬁers.

However, they are not perfect. Filter methods, by deﬁnition, ignore any interac-

tion with the classiﬁer and are often univariate or low-variate. The relevance of each

326 CHAPTER 12. FEATURE SELECTION

Figure 12.8: Two ﬁrst principal components for a n= 2 dimensional Gaussian

distribution.

feature is assessed separately, thereby ignoring feature dependencies. This may be

detrimental in case of complex multivariate dependencies.

12.3.1 Principal component analysis

Principal component analysis (PCA) is one of the oldest and most popular pre-

processing methods to perform dimensionality reduction. It returns a set of linear

combinations of the original features so as to retain most of their variance and their

information. Those combinations may be used as compressed (or latent) versions

of the original features and used to perform learning in a lower dimensional space.

The method consists of projecting the data from the original orthogonal space

Xinto a lower-dimensional space Z , in an unsupervised manner, maximising the

variance and minimising the loss due to the projection. The new space is orthog-

onal (as the original) and its axes, called principal components, are speciﬁc linear

combinations of the original ones.

The ﬁrst principal component (i.e. the axis z1 in Figure 12.8) is the axis

along which the projected data have the greatest variation. Its direction a∗ =

[a ∗

1, . . . , a ∗

n]∈R n is obtained by maximising the variance of

z=a1 x·1 + · ·· + an x ·n = aT x

a linear combination of the original features. It can be shown that a∗ is also the

eigenvector of the covariance matrix Var [x] associated to the largest eigenvalue [56].

The procedure for ﬁnding the other principal components is based on the same

principle of variance maximisation. The second principal component (i.e. the axis

z2 in Figure 12.8) is the axis, orthogonal to the ﬁrst, along which the projected data

have the largest variation, and so forth.

12.3. FILTER METHODS 327

12.3.1.1 PCA: the algorithm

Consider the training input matrix Xhaving size [N, n]. The PCA consists of the

following steps:

1. the matrix Xis normalised and transformed to a matrix ˜

Xsuch that each

column ˜

X[ , j], j = 1, . . . , n, has null mean and unit variance1,

2. the Singular Value Decomposition (SVD) [83] (Appendix B.5.10) of ˜

Xis com-

puted ˜

X= UDV T

where U is a [N, N ] matrix with orthonormal columns, Dis a [N, n] rectan-

gular diagonal matrix with diagonal singular values d1 ≥d2 ≥ ··· ≥ dn ≥ 0,

dj = p λj with λj eigenvalue of XT X and V is a [n, n] matrix whose or-

thonormal columns are the eigenvectors of XT X ,

3. the matrix ˜

Xis replaced by the linear transformation

Z=˜

XV = UD (12.3.1)

whose columns (also called eigen-features) are a linear combination of the

original features and the related variances are sorted in a decreasing order,

4. a truncated version of Zmade of the ﬁrst h < n columns (associated to the h

largest singular values) is returned.

But how do we select the convenient number hof eigen-features? In the litera-

ture, three main strategies are considered:

1. ﬁx a threshold αon the proportion of variance to be explained by the principal

components, e.g. choose h such that

λ1 +· ·· + λh

j=1 λ j ≥α

where λj is the jth largest eigenvalue and P h

j=1 λ j is the amount of variance

retained by the ﬁrst h components,

2. plot the decreasing values of λj as a function of j(scree plot) and choose the

value of h corresponding to a knee in the curve,

3. select the value of h as if it was a hyperparameter, e.g. by cross-validation.

The outcome of PCA is a rotated, compressed and lower dimension version of

the original input set { x1 ,...,xn } made of h < n orthogonal features {z1 ,...,zh },

sorted by decreasing variance. In that sense, PCA can be considered as a linear auto-

encoder where the encoding step is performed by (12.3.1) and the reconstruction

of the coded data to the original space is obtained by ˜

X= ZV T . It can also

be shown [56] that the PCA implements an optimal linear auto-encoder since it

minimises the average reconstruction error

i=1 kx i −V T V x i k 2 (12.3.2)

which amounts, for hcomponents, to P n

j= h+1 λ j /N .

1A R dataframe may be easily normalised by using the R command scale.

328 CHAPTER 12. FEATURE SELECTION

Figure 12.9: A separable n= 2 dimensional binary classiﬁcation task reduced to a

non separable one because of PCA dimensionality reduction.

PCA works in a completely unsupervised manner since the entire algorithm is

independent of the target y. Though such unsupervised nature reduces the risk

of overﬁtting, in some cases, it may cause a deterioration of the generalisation

accuracy since there is no reason that principal components be associated with y.

For instance, in the classiﬁcation example of Figure 12.9, the choice of the ﬁrst PCA

component would reduce the accuracy of the classiﬁer instead of increasing it. In

order to account both for input variation and correlation with the target, supervised

versions of PCA exist, like principal component regression or partial least squares.

Another limitation of PCA is that it does not return a subset but a weighted

average of the original features (eigen-feature). In some cases, e.g. in bioinformatics

gene selection, PCA is then not recommended since it may hinder the interpretabil-

ity of the resulting model.

R script

The scripts pca.R and pca3D.R illustrate the PCA decomposition in the n = 2

and n = 3 case for Gaussian distributed data and compute the reconstruction

error (12.3.2).

The script pca uns.R illustrates the limits of PCA due to its unsupervised na-

ture. Consider a binary classiﬁcation task with n= 2 and a separating boundary

between the two classes which is directed as the ﬁrst component. In this case a

dimensional reduction is rather detrimental to the ﬁnal accuracy since it transforms

the separable n= 2 problem into a non separable n= 1 problem (Figure 12.9).

•

PCA is an example of linear dimensionality reduction. In the machine learning

literature, however, there are several examples of nonlinear versions of PCA: among

the most important we mention the kernel-based version of PCA (KPCA) and

(deep) neural auto-encoders (Section 10.1.2).

12.3. FILTER METHODS 329

12.3.2 Clustering

Clustering, also known as unsupervised learning, is presented in Appendix A. Here

we will discuss how it plays a role in dimensionality reduction by determining groups

of features or observations with similar patterns (e.g. patterns of gene expressions

in microarray data).

The use of a clustering method for feature selection requires the deﬁnition of a

distance function between variables and the deﬁnition of a distance between clusters.

The two most common methods are

•Nearest-neighbour clustering: the number of clusters is set by the user,

then each variable is assigned to a cluster at the end of an iterative procedure.

Examples are Self Organizing Maps (SOM) and K-means.

•Agglomerative clustering: it is a bottom-up method where clusters are ini-

tially empty and sequentially ﬁlled with variables. An example is hierarchical

clustering (R command hclust) which starts by considering all the variables

as belonging to separate clusters. Next, it joins pairs of similar features in the

same cluster and then it proceeds hierarchically by merging the closest pairs

of clusters. The algorithm requires a measure of dissimilarity between sets of

features and a linkage criterion that quantiﬁes the set dissimilarity as a func-

tion of the set elements pairwise distances. The visual output of hierarchical

clustering is a dendrogram, a tree diagram used to illustrate the arrangement

of the clusters. Figure 12.10 illustrates the dendrogram returned by a clus-

tering of features in a bioinformatics task. Note that the dendrogram returns

diﬀerent clusters of features (and a diﬀerent number of clusters) at diﬀerent

heights. The choice of the optimal height cut is typically done by means of a

cross-validation strategy [126].

Clustering and PCA are both unsupervised dimensionality reduction techniques,

which are commonly used in several domains (notably bioinformatics). However, the

main advantage of clustering resides in the higher interpretability of the outcome.

Unlike the PCA linear weighting, the grouping of the original features is much more

informative and may return useful insights to the domain expert (e.g. about the

interaction of a group of genes in a pathology [92]).

12.3.3 Ranking methods

Unlike PCA and clustering, ranking methods are supervised ﬁlters since they take

into account the relation between inputs and target yto proceed with the selection.

Ranking methods consist of three steps: i) they ﬁrst assess the importance (or

relevance) of each variable for the output by using a univariate measure, ii) they

rank them in decreasing order of relevance and iii) select the top k variables.

Relevance measures commonly used in assessing a feature are:

•the Pearson linear correlation (the larger, the more relevant);

•in case of binary classiﬁcation tasks, the p-value of hypothesis tests like t-test

or Wilcoxon (the lower, the more relevant).

•mutual information (Section 3.8) (the larger the more relevant).

Ranking methods are fast (complexity O (n )), and their output is intuitive and

easy to understand. At the same time, they disregard redundancies and higher-

order interactions between variables. Two typical situations where ranking does

not perform well are complementary and highly redundant conﬁgurations. In the

complementary case, two input features are very low informative about the target,

330 CHAPTER 12. FEATURE SELECTION

Figure 12.10: Dendrogram.

yet they are very informative if taken together (see the XOR conﬁguration later).

Because of their low univariate relevance, ranking methods will rank them low and

consequently discard them. Otherwise, two variables could be both highly relevant

about the target but very similar (or identical). In this redundant case, both will

be ranked very high and selected, despite their evident redundancy.

Feature selection in a gene expression dataset

A well-known high-dimensional classiﬁcation task is gene expression classiﬁcation

in bioinformatics, where the variables correspond to genomic features (e.g. gene

probes), the observations to patients and the targets are biological phenotypes (e.g.

cancer grade). Because of the growing capabilities of sequencing technology, the

number of genomic features is typically much larger than patient cohorts' size.

In the script featsel.R we analyze the microarray dataset from [84]. This dataset

contains the genome expressions of n= 7129 genes for N = 72 patients, and V = 11

related phenotype variables. The expression matrix Xand the phenotype vector

Yare contained in the dataset data(golub). The script studies the dependency

between the gene expressions and the binary phenotype ALL.AML indicating the

leukaemia type: lymphoblastic leukaemia (ALL) or acute myeloid leukaemia (AML).

Relevant features are selected by correlation ranking and the misclassiﬁcation errors

are computed for diﬀerent sizes of the feature set.

•

12.4 Wrapping methods

Wrapper methods combine a search in the space of possible feature subsets with

an assessment phase relying on a learner and a validation (often cross-validation)

12.4. WRAPPING METHODS 331

technique. Unlike ﬁlter methods, wrappers take into consideration the interaction

between features, and this in a supervised manner. Unfortunately, this implies a

much higher computational cost, especially in the case of expensive training phases.

Also, the dependance of the ﬁnal result on the learner choice could be considered as

anuisance factor confounding the impact of the feature set on the ﬁnal accuracy2.

In other terms, the issue is: was the feature set returned by the wrapper because it

was good in general or only for that speciﬁc learner (e.g. a neural network)?

The wrapper search can be seen as a search in a space W = { 0, 1}n where a

generic vector w∈ W is such that

w[ j] = ( 0 if the input jdoes NOT belong to the set of features

1 if the input jbelongs to the set of features

Wrappers look for the optimal vector w∗ ∈ { 0, 1}n such that

w∗ = arg min

w∈ W

MISEw(12.4.3)

where \

MISEw is the estimation of the generalisation error of the model based on

the set of variables encoded by w. Since in real-settings the actual generalisation

error is not directly observable, the computation of \

MISEw requires the deﬁnition

of a learner and of a validation strategy.

Note that the number of vectors in Wis equal to 2n, that it doubles for each

new feature and that for moderately large n (e.g. n > 20), the exhaustive search

is no more aﬀordable. For this reason, wrappers typically rely on heuristic search

strategies.

12.4.1 Wrapping search strategies

Three greedy strategies are commonly used to avoid the exponential complexity

O(2n ) of the exhaustive approach:

•Forward selection: the procedure starts with no variables and progressively

incorporates features. The ﬁrst selected input is the one that returns the

lowest generalisation error. The second input selected is the one that, together

with the ﬁrst, has the lowest error, and so on, until no further improvement is

made or the required number of features is attained. An example of forward

selection is implemented in the R script fs wrap.R.

•Backward selection: it works in the opposite direction of the forward ap-

proach by progressively removing features from the original feature set. The

procedure starts by learning a model using all the nvariables and, therefore,

requires at least N > n. Then the impact of dropping one feature at a time

from the current subset is assessed. The feature which is actually removed

is the one that yields the lowest generalisation error after deletion. In other

terms, it is the one whose absence causes the lowest increase (or highest de-

crease) of the generalisation error. The procedure iterates until the desired

number of features is attained.

•Stepwise selection: it combines the previous two techniques by testing for

each set of variables, ﬁrst the removal of features belonging to the set, then

the addition of variables not in the set.

2This is the reason why a blocking factor approach to control the variability due to the learner

algorithm and improve the robustness of the solution has been proposed in [28].

332 CHAPTER 12. FEATURE SELECTION

It can be shown that the forward and the backward strategies have a O (n2 ) time

complexity in the case of n steps: since the i th step (i = 0, . . . , n − 1) requires

n− iassessments to select (or remove) the (i + 1)th feature, the computational

complexity for n steps amounts to P n−1

i=0 (n− i) = n(n+1)

Nevertheless, since such complexity cannot be aﬀordable either in case of very

large n , it is common usage to reduce ﬁrst the number of features by using a fast

ﬁlter method (e.g. ranking) and then apply a wrapper strategy on the remaining

number of features. Another trick consists of limiting the maximum size of the

feature set, then reducing the computational cost.

12.4.2 The Cover and van Campenhout theorem

The rationale of forward and backward greedy heuristics is that an optimal set of

size k should contain the optimal set of size k− 1. Though this seems intuitive, in

the general case there is no reason why this relation should hold. A formal result

in that sense is provided by the Cover and van Campenhout theorem [58], which

contains a negative result about the aim of wrapper search techniques to ﬁnd the

optimal subset by local procedures.

Let us consider a learning problem and denote by R∗ (w ) the lowest functional

risk (7.2.6) for the subset of variables w . Cover and van Campenhout proved that

the only generally valid (i.e. which holds for all data distributions) monotonic

relation linking feature size and generalisation is :

w2 ⊂ w1 ⇒ R∗ ( w1 )≤ R∗ ( w2 ) (12.4.4)

i.e. by adding variables we reduce the minimal risk3.

Given n features, any ordering of the 2nsubsets which is consistent with the

above constraint is indeed possible. This means for any possible ordering, there

exists a distribution of the data that is compatible with that. If the three variables

optimal set is

{x ·1 ,x ·3 , , x ·13 }

there is no guarantee that the best set of four variables is a superset of w1 (as it is

assumed in forward selection). According to this theorem, there exists a distribution

for which the best set of 4 features could well be

{x·2 ,x·6 ,x·16 , x ·23 }

since this is not in contradiction with the constraint (12.4.4). In other words, the

Cover and van Campenhout theorem states that there are data distributions for

which forward/backward strategies could be arbitrarily bad.

12.5 Embedded methods

They are typically less computationally intensive than wrapper methods but are

speciﬁc to a learning machine. Well-known examples are classiﬁcation trees, Ran-

dom Forests (Section 11.4), Naive Bayes (Section 10.2.3.1), shrinkage methods and

kernels.

3Note that this relation refers to the optimal model that could be learned with the input subset

wand that the notion of lowest functional risk does not take into consideration the model family

nor the ﬁnite-size setting. In other terms, this inequality refers only to the bias and not the

variance component of the generalisation error. So in practice though in theory R∗ (w1 )≤R∗ (w2 )

it could happen that GN (w1 )≥GN (w2 ) where GN is the generalisation error of the model learned

with N observations

12.5. EMBEDDED METHODS 333

12.5.1 Shrinkage methods

Shrinkage is a technique to improve a least-squares estimator by regularisation

and consists of reducing the model variance by adding constraints on the value

of coeﬃcients. In what follows, we present two shrinkage approaches that penalise

the least-squares solutions having a large number of coeﬃcients with values diﬀerent

from zero. The rationale is that only those variables, whose impact on the empirical

risk is considerable, deserve a coeﬃcient diﬀerent from zero and should appear in

the ﬁtted model. Shrinkage is an implicit (and more continuous) embedded manner

of doing feature selection since only a subset of variables contributes to the ﬁnal

predictor.

12.5.1.1 Ridge regression

Ridge regression is an example of shrinkage method applied to least squares regres-

sion

βr = arg min

i=1

(yi − x T

ib) 2 +λ

j=1

j}=

= arg min

b(Y− Xb ) T (Y− Xb) + λb T b

where λ > 0 is a complexity parameter that controls the amount of shrinkage:

the larger the value of λ, the greater the amount of shrinkage. Note that if λ = 0

the approach boils down to a conventional unconstrained least-squares.

An equivalent formulation of the ridge problem is

βr = arg min

i=1

(yi − x T

ib) 2 ,

subject to

j=1

j≤L

where there is a one-to-one correspondence between the parameter λ and L[98].

It can be shown that the ridge regression solution is

βr = ( XT X+ λIp ) −1 XT Y(12.5.5)

where Ip is the [p, p ] identity matrix (p =n + 1) and it is typically recommended

that the Xcolumns are normalised (zero mean and unit variance) [132]. In algebraic

terms, a positive λensures that the matrix to be inverted be symmetric and strictly

positive deﬁnite.

If n >> N it is recommended to take advantage of the SVD decomposition (B.5.11)

to avoid the inversion of a too large matrix [100]. If we set X = U DV T then we

obtain from (12.5.5) and (B.9.15)

βr = ( V DU T U DV T + λIp )−1 V DU T Y= V ( RT R+ λIN ) −1 RT Y

where R = UD is a [N, N ] matrix and IN is the [N, N ] identity matrix.

In general, ridge regression is beneﬁcial in numerical, statistical and inter-

pretability terms. From a numerical perspective, it is able to deal with rank deﬁcient

matrices X and reduces the ill-conditioning of the matrix XT X . From a statistical

perspective, it reduces the variance of the least-squares solution ˆ

βr (Section (9.1.14))

at the cost of a slight bias increase. Given the predominance of the variance term

in high-dimensional tasks, ridge regression enables a reduction of the generalisation

error. Last but not least, pushing the absolute value of many coeﬃcients to zero, it

allows the identiﬁcation of a small (then interpretable) number of input features.

334 CHAPTER 12. FEATURE SELECTION

12.5.1.2 Lasso

Another well-known shrinkage method is lasso which estimates the linear parame-

ters by

βr = arg min

i=1

(yi − x T

ib) 2 ,(12.5.6)

subject to

j=1 |b j | ≤ L(12.5.7)

If on one hand the 1-norm penalty of the lasso approach allows a stronger constraint

on the coeﬃcients, on the other hand it makes the solution nonlinear and demands

the adoption of a quadratic programming algorithm (details in Appendix B.8).

To formulate the problem (12.5.6) in the form (B.8.12) with linear constraints,

we may write the bj terms as the sum of two non-negative numbers

bj =b+

j−b −

j=|b j |+b j

2−|bj | − b j

The function to optimize becomes

J( b) = bT XT Xb − 2 YT Xb = ( b+ −b− )T XTX ( b+ −b− )− 2 YT X( b+ −b− ) =

b + b −  X T X−XT X

−XT X X T X  b +

b−  +  −2YT X 2YT X b +

b−  (12.5.8)

with the constraints







1 1 . . . 1

−1 0 . . . 0

0− 1. . . 0

0 0 . . . − 1





 b+

b−  ≤ 





···







where the left-hand matrix is [2p + 1, 2 p ]. The ﬁrst line of the inequality is (12.5.7)

since P p

j=1(b +

j+b −

j) = P p

j=1 |b j | ≤ L.

Note that if L > P p

j=1 | ˆ

βj |the lasso returns the common least-squares solution.

The penalty factor Lis typically set by having recourse to cross-validation strategies.

Though the diﬀerence between ridge regression and lasso might seem negligible,

the use of a 1-norm penalty instead of a 2-norm has a sensible impact on the number

of ﬁnal coeﬃcients which are set to zero. Figure 12.11 visualises this in a bivariate

case: ˆ

βdenotes the least-squares solution which would be returned by both methods

if λ = 0. Since λ > 0, the minimisation combines the empirical risk function (whose

contour lines are the ellipsoids around ˆ

β) and the regularisation term (whose contour

lines are around the origin). Note that the only diﬀerence between the two ﬁgures

is the shape of the regularisation contour lines (related to the used norm). The

minimisation solution is a bivariate vector which lies somewhat (depending on the

λvalue) at the intersection of an empirical risk contour line and a regularisation

one. The ﬁgure shows that this intersection in the lasso case tends to be closer

to the axis β1 = 0, this meaning that the ﬁrst estimated coeﬃcient is set to zero.

Because of the circular shape of regularisation contours, this is much less probable

in the ridge regression case.

R script

The R script lasso.R implements the quadratic programming minimisation in (12.5.8)

by using the R library quadprog. The script applies the lasso strategy to a regression

12.5. EMBEDDED METHODS 335

Figure 12.11: Ridge regression vs lasso [98].

task where the number of features nis comparable to the number of observations N

and only a small number of features is relevant. The results show the impact of the

constraint L on the empirical risk and the evolution of the lasso solution moving

towards one of the axis. In particular the smaller L, the less importance is given to

minimise J , the larger the empirical risk and the smaller is the number of estimated

parameters diﬀerent from zero.

•

The shrinkage approach has been very successful in recent years and several

variants of the methods mentioned above exist in literature: some adopt diﬀerent

penalty norms, some combine diﬀerent norms (e.g. Elastic-net) and some combine

shrinkage with greedy search (e.g. Least Angle Regression).

12.5.2 Kernel methods

Many learning algorithms, such the perceptron, support vector machine (SVM) and

PCA, process data in a linear manner through inner products (Section B.2). Those

techniques are exposed to two main limitations: the linear nature of the model and

the curse of dimensionality for large n.

Kernel methods [171] adapt those techniques by relying on the combination of

two smart ideas: i) address large dimension nproblems by solving a dual problem

in a space of dimension Nii) generalise the notion of inner product by adopting a

user-speciﬁed kernel function, i.e., a similarity function over pairs of data points.

Kernel functions operate in a high-dimensional, implicit feature space without

computing the coordinates of the data in that space. This allows to take advantage

of high nonlinear dimensional representations without actually having to work in

the high dimensional space.

336 CHAPTER 12. FEATURE SELECTION

Figure 12.12: Implicit transformation of the problem to a high-dimensional space.

12.5.3 Dual ridge regression

We introduced the dual formulation of the linear least-squares problem in Sec-

tion 9.1.18. Consider now a ridge regression problem (Section 12.5.1.1) with pa-

rameter λ ∈ <+ . The conventional least-squares solution is the [n, 1] parameter

vector

β= ( X0 X+ λIn ) −1 X0 y

where In is the identity matrix of size n. Since from (B.9.15)

(X0 X + λIn )−1 X0 =X0 (XX0 + λIN )−1

where IN is the identity matrix of size N, the dual formulation is

β= X0 ( XX0 +λIN )−1 y= X0 α

where

α= ( K+ λIN )−1 y

is the [N, 1] vector of dual variables and K = XX0 is the Kernel or Gram [N , N]

matrix. Note that all the information required to compute α is this matrix of inner

products.

The prediction for a test [Nts , n ] dataset Xts is

ˆ yts = Xts ˆ

β= Xts X0 α= Kts ( K+ λIN ) −1 y

where Kts is a [N ts , N ] matrix with k j,i = hxj , xi i, j = 1, . . . , Nts , , i = 1, . . . , N .

This derivation allows transforming a ndimensional linear task into a Ndimen-

sional one. This is of course very relevant if n >> N. However, the model remains

linear. What about non-linear models?

12.5.4 Kernel function

Suppose to apply the nonlinear transformation Φ : x ∈ <n → Φ(x ) ∈ <M to

the inputs of the ridge regression problem discussed above (Figure 12.12). The

prediction for an input xwould now be

ˆ y= y0 ( K+ λIN )−1 k

where

Ki,j = hΦ( xi ) , Φ( xj ) i , ki =h Φ( xi ) , Φ( x )i

12.6. SIMILARITY MATRIX AND NON NUMERIC DATA 337

The rational of kernel methods is that those inner products can be computed

eﬃciently without explicitly computing the mapping Φ thanks to a kernel func-

tion [171]. A kernel function is a function κ that for all x, z ∈ X satisﬁes

κ( x, z) = h Φ( x ) , Φ( z )i

where Φ is a mapping from Xto a feature space F. For instance

κ( x, z) = h x, z i2 = h Φ( x ) , Φ( z )i

where

Φ : x = (x1 , x2 )→ Φ(x ) = (x 2

1, x 2

2,√ 2x 1 x 2 )∈F

Kernels de-couple the speciﬁcation of the algorithm from the speciﬁcation of the

feature space since they provide a way to compute dot products in some feature

space without even knowing what this space the function Φ are.

For instance

κ( x, z) = (1 + xT z )2

corresponds to a transformation to M= 6 dimensional space

Φ(x1 , x2 ) = (1, x2

1, x 2

2,√ 2x 1 ,√ 2x 2 ,√ 2x 1 x 2 )

A Gaussian kernel κ ( x, z ) = exp−γk x −z k2 corresponds to a transformation to an

inﬁnite-dimensional space.

Theoretically, a Gram matrix must be positive semi-deﬁnite (PSD). Empirically,

for machine learning heuristics, choices of a function κthat do not satisfy PSD

condition may still perform reasonably if κat least approximates the intuitive idea

of similarity.

The general idea of transposing a low-dimensional method to a nonlinear high-

dimensional setting by using a dual formulation is generally referred to as kernel

trick : given any algorithm that can be expressed solely in terms of dot products,

the kernel trick allows us to construct diﬀerent nonlinear versions of it.

Kernel methods are together with deep learning and random forests among the

most successful methods in the history of machine learning. We decided to present

them in this section for their powerful strategy in dealing with settings with high

dimension and low number of observations. Their strength can however turn into

a weakness if we aim to scale the approach to very large N . At the same time, as

for all the other methods presented in this book, their generalisation accuracy is

strictly dependent on the adequate choice of the related hyperparameters. In the

case of kernel methods the most important hyperparameters are the regularisation

term λ , the analytical form of the kernel function and the related parameters.

12.6 Similarity matrix and non numeric data

In the previous sections, we have considered feature selection techniques for conven-

tional supervised tasks where data are numeric and represented in a conventional

tabular form DN . What about non-conventional tasks where the training set is not

a data table but a set of items? Examples of items could be music tracks, texts,

images, web sites or graphs. Often, in those cases, we are not able (or conﬁdent)

in encoding each item as a numeric vector of size n. Nevertheless, we could be con-

ﬁdent in deﬁning a similarity score between pairs of items. For instance, we may

use musical genre to measure the similarity between tracks or user access statistics

to obtain the similarity between web sites.

As a result, we may encode the item set as a similarity matrix S of size [N, N ]

which becomes an alternative way of representing the dataset.

338 CHAPTER 12. FEATURE SELECTION

A symmetric factorisation of a symmetric [N, N ] matrix

S≈ F F T (12.6.9)

is an approximation of the similarity matrix where Fis a [N, K] matrix. The

matrix F may be used as an approximate Kdimensional numeric representation of

the non-numeric item-set.

Note that the positive deﬁnitiveness of S is a necessary and suﬃcient condition

for having an exact factorisation, i.e. an identity in (12.6.9). This is guaranteed

in the numeric case where Sis the covariance matrix and the pairwise similarity is

computed by dot product. In the generic non-numeric case, techniques to repair the

positive deﬁnitiveness of Smay be adopted. An alternative is the use of optimisation

techniques to obtain Fas the solution of the minimisation task

F= arg min

UkS− UU T k 2

Another limitation of the factorisation approach is that it is hardly scalable for

very large N . For such cases sampling based solution have been proposed in [2].

12.7 Averaging and feature selection

The role of averaging methods in supervised learning has been discussed in the

previous chapter. Averaging may play a crucial role also in dealing with large

dimensionality. Instead of choosing one particular feature selection method, and

accepting its outcome as the ﬁnal subset, diﬀerent feature selection methods can

be combined using ensemble approaches. Since there is not an optimal feature

selection technique and due to the possible existence of more than one subset of

features that ﬁts the data equally well, model combination approaches have been

adapted to improve the robustness and stability of ﬁnal, discriminative methods.

Ensemble techniques typically rely on averaging the outcome of multiple models

learned with diﬀerent feature subsets. A well-known technique is the random sub-

space method [102], also known as feature bagging, which combines a set of learners

trained on random subsets of features.

12.8 Feature selection from an information-theoretic

perspective

So far, we focused on algorithmic methods to return a subset of relevant features,

without making any formal deﬁnition of relevance. In this section, we formalise the

notion of feature relevance by using concepts of information theory, like entropy,

mutual information and conditional information from Sections 3.8 and 3.8.1.

12.8.1 Relevance, redundancy and interaction

This section deﬁnes in information-theoretic terms what is a relevant variable in a

supervised learning task where Xis a set of ninput variables and yis the target.

These deﬁnitions are obtained by interpreting in information-theoretic terms the

deﬁnitions made by [117].

Deﬁnition 8.1 (Strong relevance) . A variable xi ∈ X is strongly relevant to the

target yif

I(X−i ;y ) < I (X ;y )

where X −i is the set obtained by removing the variable xi from X.

12.8. INFORMATION-THEORETIC PERSPECTIVE 339

In other words, a variable is strongly relevant if it carries some information about

ythat no other variable can carry. Strong relevance indicates that the feature is

always necessary for an optimal subset.

Deﬁnition 8.2 (Weak relevance).A variable is weakly relevant to the target yif

it is not strongly relevant and

∃S ⊆X −i :I ( S; y) < I ( {xi , S };y )

In other words, a variable is weakly relevant when it exists a certain context S

in which it carries information about the target. Weak relevance suggests that the

feature is not always necessary but may become necessary at certain conditions.

This deﬁnition makes clear that for some variables (typically the majority) the

relevance is not absolute but more a context-based notion. In a large variate setting,

those features are the hardest to deal with since their importance depends on the

other selected ones.

Deﬁnition 8.3 (Irrelevance) . A variable is irrelevant if it is neither strongly or

weakly relevant.

Irrelevance indicates that the feature is not necessary at all. This is deﬁnitely

the easiest case in feature selection. Irrelevant variables should be simply discarded.

Example

Consider a learning problem where n = 4, x2 =− x3 +w 2

y=( 1 + w,x1 + x2 >0

0,else

where w and w2 are noise terms. Which variables are strongly, weakly relevant and

irrelevant?

•

Deﬁnition 8.4 (Markov blanket).Let us consider a set X of n r.v.s., a target

variable y and a subset My ⊂ X . The subset My is said to be a Markov blanket of

y, y/ ∈My iﬀ

I(y ;X −(My ) | My ) = 0

The following theorem can be shown [183, 150]:

Theorem 8.5 (Total conditioning). If the distribution has a perfect map in a DAG

(Section 4.3.2.1) then

x∈ My ⇔I ( x; y| X −(x,y ) )> 0

This theorem proves that, under speciﬁc assumptions about the distribution

discussed in Section 4.3.2.1, the Markov blanket of a target yis composed of the

set of all the strongly relevant variables in X.

Another useful notion to reason about the information of a subset of variables

is the notion of interaction.

Deﬁnition 8.6 (Interaction) . Given three r.v.s. x1 ,x2 and y we deﬁne the inter-

action between these three variables as

I(x1 ;y )− I(x1 ;y |x2 )

340 CHAPTER 12. FEATURE SELECTION

Figure 12.13: XOR classiﬁcation separable task with two inputs and one binary class

taking two values (stars and rounds). Two variables x1 and x2 are complementary:

they bring alone no information but they bring the maximal information about y

when considered together.

The interaction term satisﬁes the following relation:

I(x1 ;y )− I(x1 ;y |x2 ) = I(x1 ;x 2) − I(x1 ;x2 | y ) = I(x2 ;y )− I(x2 ;y |x1 )

In what follows, we show that it is possible to decompose the joint information

of two variables in the sum of the two univariate terms and the interaction. From

the chain rule (3.8.82)

I(x2 ;y| x1 ) + I(x1 ;y ) = I(x1 ;y |x2 ) + I(x2 ;y )

we have

I(x2 ;y| x1 ) = I(x2 ;y )− I(x1 ;y ) + I(x1 ;y| x2 )

By summing I (x1 ;y ) to both sides, from (3.8.82) it follows that the joint information

of two variables about a target ycan be decomposed as follows:

I({ x1 ,x2 }; y) = I(x1 ;y ) + I(x2 ;y )− [I(x1 ;y )− I(x1 ;y |x2 )]

| {z }

interaction

=I (x1 ;y ) + I (x2 ;y )− [I (x1 ;x2 )−I (x1 ;x2 | y )]

| {z }

interaction

(12.8.10)

What emerges is that the joint information of two variables is not necessarily equal,

greater or smaller than the sum of the two individual information terms. All de-

pends on the interaction term: if the interaction term is negative, the two variables

are complementary, or in other terms, they jointly bring a higher information than

the sum of the univariate terms. This is typically the case of the XOR exam-

ple illustrated in Figure 12.13 [89]. In this case I (x1 ;y ) = 0, I (x2 ;y ) = 0 but

I({x1 ,x2 } ;y ) >0 and maximal. When they are redundant, the resulting joint

information is lower than the sum I ( x1 ;y ) + I (x2 ;y ).

Since (12.8.10) holds also when x1 and/or x2 are sets of variables, this result

sheds an interesting light about the non-monotonic nature of feature selection [199].

12.8. INFORMATION-THEORETIC PERSPECTIVE 341

12.8.2 Information-theoretic ﬁlters

In terms of mutual information the feature selection problem can be formulated as

follows. Given an output target yand a set of input variables X ={ x1 ,...,xn } ,

the optimal subset of dvariables is the solution of the optimisation problem

X∗ = arg max

XS ⊂X,| XS |=dI(X S ;y) (12.8.11)

Thanks to the chain rule (3.8.82), this maximisation task can be tackled by

adopting an incremental approach (e.g. forward approach).

Let X = {xi }, i = 1, . . . , n the whole set of variables and XS the set of s variables

selected after ssteps. The choice of the (s + 1)th variable x(s+1) ∈ X− XS can be

done by solving

x(s+1) = arg max

xk ∈ X − X S

I({XS ,xk } ;y ) (12.8.12)

This is known as the maximal dependency problem and requires at each step multi-

variate estimation of the mutual information term I ({XS , xk } ;y ). Such estimation

is often inaccurate in large variate settings (i.e. large nand large s) because of

ill-conditioning and high variance issues.

In literature several ﬁlter approaches have been proposed to solve the optimi-

sation (12.8.12) by approximating the multivariate term I ({XS , xk } ;y ) with low

variate approximations. These approximations are necessarily biased, yet much less

prone to variance than their multivariate counterparts.

We mention here two of the most used information theoretic ﬁlters:

•CMIM [74]: since according to the ﬁrst (chain-rule) formulation

arg max

xk ∈ X − X S

I({XS ,xk } ;y ) = arg max

xk ∈ X− XS

I(xk ;y |XS )

this ﬁlter adopts the low-variate approximation

I(xk ;y| XS )≈ min

xj ∈ XS

I(xk ;y| xj )

•mRMR (minimum Redundancy Maximal Relevance) [152]: the mRMR method

approximates at the (s + 1)th step I ({XS , xk } ;y ) with

I(xk ;y )− 1

xi ∈X S

I(xi ;xk )

where s is the number of features in XS . The method implements a forward

selection which selects at the (s + 1)th step

x(s +1) = arg max

xk ∈ X− XS "I(x k ;y)− 1

xi ∈ XS

I(xi ; xk )#

that is a variable which has both high relevance I (xk ;y ) and low average

redundancy with the set XS .

12.8.3 Information-theoretic notions and generalisation

Most of this book has dealt with generalisation error to assess, compare and select

prediction models and in this chapter, we presented feature selection as an instance

of model selection. Nonetheless, this last part of the chapter has been mainly

referring to information-theoretic notions for performing feature selection. It is

342 CHAPTER 12. FEATURE SELECTION

then important to provide some elucidation on how information-theoretic notions

relate to generalisation error.

Given a set Xof input features and a target feature y, the quantity I ( X ;y ) is

not directly observable and has to be estimated before use. Since

I(X ;y ) = H(y )− H(y| X ),

maximising I (X ;y ) in (12.8.11) is equivalent to minimise H (y| X ). The term

H(y| X ) is the entropy (or uncertainty) of yonce the value of the input set X

is given. In the Normal case, this term is proportional to the conditional variance

(Equation (3.5.59)). It follows that ﬁnding the set of inputs Xwhich minimises

H(y| X ) boils down to ﬁnd the set of features that attains the lowest generalisation

error (7.2.6).

In real-world settings, since the conditional entropy H (y| X ) is not observable,

it may be approximated by the generalisation error, e.g. by MISE in the regres-

sion case. Hopefully, the link between feature selection, generalisation error and

information-theory becomes clear: ﬁnding the set that maximises the mutual in-

formation in (12.8.11) boils down to ﬁnd the set that minimises the estimated

generalisation error (12.4.3).

12.9 Assessment of feature selection

Most of the discussed techniques aim to ﬁnd the best subset of features by per-

forming a large number of comparisons and selections. This additional search layer

increases inevitably the space of possible models and the variance of the resulting

one. Despite the use of validation procedures, low misclassiﬁcation or low predic-

tion errors may be found only due to chance. As stated in [132], given a suﬃciently

exhaustive search, some apparent pattern can always be found, even if all predictors

have come from a random number generator. This is due to the fact that, as a

consequence of the search process, the set of features is dependent on the data used

to train the model, introducing then what is called selection bias4 [132].

A bad (and dangerous) practice is using the same set of observations to select

the feature set and assess the accuracy of the classiﬁer. Even if cross-validation

is used to assess the accuracy of the classiﬁer, this will return an overoptimistic

assessment of the generalisation error (Figure 12.14). Cross-validation has to be

used to assess the entire learning process which is composed of both a feature

selection and a classiﬁcation step. This means that for each fold, both feature

selection and classiﬁcation has to be performed before testing on the observations

set aside. Keeping feature selection out of cross-validation will return an assessment

which will be as much biased as the number of observations is small.

If cross-validation cannot be carried out (e.g. because of too small training size),

then the use of external validation sets is strongly recommended.

If no additional data are available, an alternative consists of comparing the gen-

eralisation accuracy returned by cross-validation on the original data with the one

obtained by re-running the learning procedure with randomised datasets. This is

inspired by the method of permutation testing described in Section 6.6. The proce-

dure consists of repeating the feature selection and the cross-validation assessment

several times by using a randomised dataset instead of the original one. For in-

stance, a randomised dataset may be obtained by reshuﬄing the output vector, a

permuting operation that artiﬁcially removes the dependency between the inputs

and the output. After a number of repetitions with randomised datasets, we obtain

4For a formal justiﬁcation of selection bias, look at Appendix C.11. A causal perspective on

selection bias is in Section 13.7.4

12.10. CONCLUSION 343

Figure 12.14: Selection bias associated to feature selection: the internal leave-one-

out is an overoptimistic estimator of the test generalisation error.

the null distribution of the accuracy in case of no dependency between inputs and

output. If the accuracy associated with the original data is not signiﬁcantly better

than the one obtained with randomised data, we are overﬁtting the data. For in-

stance, let us consider a large-variate classiﬁcation task where the cross-validated

misclassiﬁcation error after feature selection is 5%. If we repeat the same learning

procedure 100 times with randomised datasets and we obtain a signiﬁcant (e.g. 10)

number of times a misclassiﬁcation error smaller or equal than 5%, this is a sign of

potential overﬁtting.

Making a robust assessment of a feature selection outcome has a striking im-

portance today because we are more and more confronted with tasks characterised

by a very large feature to sample ratio (e.g. in bioinformatics [8]), where a bad

assessment procedure can give too optimistic (overﬁtted) results.

Example

The script fselbias.R illustrates the problem of selection bias in the case of in-

tensive search during feature selection. Consider a linear input/output regression

dependency with n= 54 inputs (of which only 4 relevant and others irrelevant) and

a dataset of size N= 20. Let us perform a forward search based on internal leave-

one-out. Figure 12.14 shows the evolution of the internal leave-one-out MSE and a

more reliable estimation of the generalisation MSE based on an independent test set

(5000 i.i.d. examples from the same input/output process). It appears that, as the

feature set size increases, the internal leave-one-out error returns a very optimistic

estimation of the generalisation error. Therefore, the internal leave-one-out error is

unable to detect that the optimal size of the input set (i.e. the number of strongly

relevant variables) is equal to four.

•

12.10 Conclusion

Nowadays, feature selection is an essential component of a real-world learning

pipeline. This chapter discussed how the problem is typically addressed as a stochas-

tic optimisation task in a combinatorial state-space where the assessment of each

solution and the search strategy are key elements. Most heuristic approaches rely

on a monotonic assumption, stating that the best subset of size kis always con-

tained in the best subset of size > k. The theorem in Section 12.4.2 and the notions

344 CHAPTER 12. FEATURE SELECTION

of interaction discussed in Section 12.8.1 show that this assumption is simplistic.

Variables that are almost non-informative alone may become extremely informative

together since the relevance of a feature is context-based. This is made formal by

notions of graphical models (e.g. d-separation or Markov blanket) which unveil the

conditional nature of dependency (Chapter 4). Our opinion is that the best way

of conceiving feature selection is not by black-box optimisation but by reasoning

on the conditional structure of the distribution underlying the data. The ﬁnal aim

should be, as much as possible, to shed light on the context-based role of each fea-

ture. In recent years there have been many discussions about the interpretability

of data-driven models though it is not always made clear what is the most valuable

information for the human user. We deem that in a large variate task the most use-

ful outcome should be an interpretable description of features, returning for each of

them a context-based degree of relevance. Accuracy is only a proxy of information:

the real information is in the structure.

12.11 Exercises

1. Consider the dataset

x1x2x3 y

1 1 0 1

0 0 1 0

0 1 0 0

0 1 1 0

1 1 0 0

1 0 1 1

1 0 0 1

0 1 1 0

1 0 1 0

1 0 0 0

1 1 0 0

0 1 1 0

Rank the input features in a decreasing order of relevance by using the correlation

ρxy =ˆ σxy

ˆ σx ˆ σy

as measure of relevance.

Solution: Since ρx 1 y = 0. 488, ρx 2 y =− 0. 293, ρ x 3 y =− 0. 192, the ranking is

x1 , x2 ,x3 .

2. Consider a regression task with two inputs x1 ,x2 and output y. Suppose we observe

the following training set

X1X2Y

-0.2 0 .1 1

0.1 0 0.5

1 -0.3 1.2

0.1 0.2 1

-0.4 0.4 0.5

0.1 0.1 0

1 -1 1.1

1. Fit a multivariate linear model with β0 = 0 to the dataset.

2. Compute the mean squared training error.

3. Suppose you use a correlation-based ranking strategy for ranking the features.

What would be the top ranked variable?

12.11. EXERCISES 345

Hint:

A= a 11 a12

a12 a22  ⇒ A −1 = 1

a11a22 −a2

12  a 22 −a 12

−a12 a11 

Solution:

1. XT X = 2. 23 −1. 45

−1. 45 1. 31 

(XT X )−1 =  1. 599 1.77

1. 77 2. 72

XT Y= 2 .05

−0.96

β= ( XTX)−1 XT Y= 1 .58

1. 016

2. e =Y− Xβ =







1. 21

0. 34

−0. 08

0. 64

0. 73

−0. 26

0. 54







It follows that the Mean Squared training error

amounts to 0.41.

3. Since

ρX 1 Y =PN

i=1(X i1−µ 1 )(Y i −µ Y )

qP N

i=1(X i1−µ 1) 2 (Y i−µ Y) 2

= 0.53

and

ρX 2 Y =PN

i=1(X i2−µ 2 )(Y i −µ Y )

qPN

i=1(X i2−µ 2) 2 (Y i−µ Y ) 2

=− 0.48

where µ1 = 0. 24, µ2 = − 0. 07, µY = 0. 75, X1 is the top ranked variable.

3. The .Rdata ﬁle bonus4.Rdata in the directory gbcode/exercises of the companion

R package contains a regression dataset with N = 200 observations, n= 50 input

features (in the matrix X) and one target variable (vector Y).

Knowing that there are 3 strongly relevant variables and 2 weakly relevant variables,

the student has to deﬁne and implement a strategy to ﬁnd them.

No existing feature selection code has to be used. However, the student may use

libraries to implement supervised learning algorithms.

The student code should

•return the position of the 3 strongly relevant variables and 2 weakly relevant

variables,

•discuss what strategy could have been used if the number of strongly and

weakly variables was not known in advance.

Solution:

See the ﬁle Exercise4.pdf in the directory gbcode/exercises of the companion R

package (Appendix F).

346 CHAPTER 12. FEATURE SELECTION

Chapter 13

From prediction to causal

knowledge

We live in an uncertain and large dimensional world where it is getting easier and

easier to observe and collect large amounts of data but not necessarily control their

generation. This setting is called observational (Section 8.3) and refers to situations

in which we cannot interfere or intervene in the process of generating and capturing

the data. In this setting, machine learning is a powerful tool to detect dependencies

(e.g. correlations) between variables (Chapter 12). Figure 13.1 shows two highly

dependent variables: in such observational setting we may easily predict one variable

given the other, but what can we deduce about their causal relation, i.e. what will

happen to one variable once we manipulate the other? (Un)fortunately not much,

as shown by the label of the variables [190]1.

An accurate predictive model, built from observational data, might tell us very

little about what would happen in case of input manipulation. Most machine learn-

ing algorithms are not conceived to estimate causal eﬀects but to predict observed

outcomes. Now, many of the most crucial questions in science are not merely pre-

dictive but causal. For example, what is the eﬃcacy of a given drug in a given

population? What fraction of deaths from a given disease could have been avoided

by a given treatment or policy? What was the cause of death of a given individual

in a speciﬁc incident? What is the impact of education on criminality?

Answering those questions, and then taking the correct decision, requires being

able to make predictions under manipulations or potential experiments (and not

only observations). Is this possible if, for practical or ethical reasons, the experi-

ments or manipulations cannot be done? Have observational data-driven approaches

1You can ﬁnd several other funny examples of spurious correlation in the website http://

tylervigen.com/spurious-correlations

Figure 13.1: A signiﬁcant correlation between two causally unrelated variables.

347

348 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.2: Pearl's causal ladder [149]

any hope of untangling causality from association? This will be the topic of this

chapter.

13.1 About the notion of cause

The notion and importance of causal reasoning has been largely debated by philoso-

phers:

•Men are never satisﬁed until they know the why of a thing (Aristotle).

•I would rather discover one cause than gain the kingdom of Persia (Democri-

tus).

•A thing cannot occur without a cause that produces it (Laplace).

•Felix, qui potuit rerum cognoscere causas (Virgil).

At the same time, for many philosophers and statisticians, causation is a suspi-

cious metaphysical concept that should be avoided when discussing science and/or

making statistics. This suspicion derives from the Hume empiricist tradition (Sec-

tion 2.4). Empiricism always tried to interpret science without having recourse to

unobservable and hidden entities. In that perspective, causation is a hidden and

unobservable connection between things.

The philosopher John Stuart Mill (1843) was one of the ﬁrst to make explicit

the properties of causal relationships. According to his deﬁnition, event A can be

deﬁned to be a cause of event B if

1. A is repeatedly associated with B (concomitant variation)

2. A has to be present each time the eﬀect B occurs (necessity ).

3. B occurs regularly when A is introduced (suﬃciency )

13.2. CAUSALITY AND DEPENDENCIES 349

Although this deﬁnition is valuable from a historical perspective, it has a number

of limitations: it refers only to categorical aspects (no intensity), it assumes deter-

minism (no variability), and it relates to a univariate setting only (no context).

Also, pretending that logical induction can formalise causality is wishful think-

ing. Consider the following example from [147]. In logics if "A→ B " and "B→ C "

then "A→ C ". But "if the sprinkler is on then the ground is wet" and "if the

ground is wet then it rained" does not imply that "if the sprinkler is on then it

rained."

Now, interesting tasks are multivariate and stochastic. For this reason, we will

explore a more advanced formalism of causality (proposed by J. Pearl2[146]) which

relies on (and extends) probabilistic reasoning. We will show that such formalism

i) eliminates any aspect of vagueness from the deﬁnition of causality, ii) makes

this notion falsiﬁable and iii) pinpoints the crucial role of causal modelling in data

understanding and decision making.

The debate on the scientiﬁc role of causality is related to the debate about

the aim of science: is it more about prediction or explanation? This book has

largely covered the issue of prediction without having recourse to explanation or

interpretability aspects. In the following section, we will show that uncovering

mechanisms from data is a way of explaining something and returning human in-

terpretable and valuable information. As visualised in the J. Pearl causal ladder

(Figure 13.2) [149], causal information can be considered as the ultimate (and prob-

ably the most precious) outcome of knowledge discovery from data.

13.2 Causality and dependencies

A dependence between an input x and an output ydoes not always imply the

existence of a causal relationship between them. For instance, dependence may

occur when a latent phenomenon (confounder) causes two eﬀects. In this case, a

statistically signiﬁcant, yet non-causal, relation between eﬀects is established. For

instance, in the example in Figure 13.1, the latent confounder is the time: both

(causally independent) variables change in the same manner with time and then

happen to be statistically correlated.

In general, it would be erroneous and fallacious to deduce a causality from the

existence of a statistical dependency only: "correlation does not imply causation"

as statisticians are used saying. To better illustrate this notion, let us consider the

following example.

The Caca-Cola study

The Caca-Cola marketing department aims to show that, unlike what feared by

most parents, the famous refreshing drink is so healthy for its drinkers to improve

their sports performances. To support this idea, the department funds a statistical

study on the relationship between the amount of Caca-Cola litres drunk per day

and the time spent (in seconds) by a drinker to run the 100 meters. Here it is the

dataset collected by statisticians:

2He was awarded the 2011 Turing Award, known as the Nobel Prize of computing

350 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.3: Sport performance improves with the amount of Caca-Cola drinking.

Is this a real causal eﬀect?

Liters per day Seconds

1.00 11.9

1.09 12.8

0.20 14.3

0.18 15.3

1.02 12.2

1.04 12.5

1.06 12.6

0.00 16.3

1.08 12.7

0.18 17.7

0.50 14.0

0.17 17.6

illustrated by Figure 13.3 which plots the performance (in seconds) as a function of

the number of drunk litres per day.

The Caca-Cola marketing department is excited. Caca-Cola seems to have mag-

niﬁcent eﬀects on the sprinter performance, as illustrated by the signiﬁcant corre-

lation between amount of litres and running time of Figure 13.3. The CEO of the

company triumphally extrapolates on the basis of a sophisticated machine learning

tool that any human being fed with more than 3 litres per day could easily beat

the world's record. In front of such enthusiasm, the League of Parents is skeptical

and ask for a simple elucidation: did the Caca-Cola statisticians record the age of

the sprinter? In front of the growing public opinion pressure, Caca-Cola is forced

to publish the complete dataset.

13.2. CAUSALITY AND DEPENDENCIES 351

Figure 13.4: Performance deteriorates as Caca-Cola drinking increases for young

athletes.

Age Liters per day Seconds

17 1.00 11.9

19 1.09 12.8

49 0.20 14.3

59 0.18 15.3

21 1.02 12.2

19 1.04 12.5

17 1.06 12.6

62 0.00 16.3

21 1.08 12.7

61 0.18 17.7

30 0.50 14.0

65 0.17 17.6

At last truth can triumph! Caca-Cola statisticians had hidden the real cause

of good (or bad performance): the age of the athletes! Since youngsters tend to

drink more Caca-Cola as well as having better sport performance, the ﬁrst causal

relationships between drunk litres and performance was fallacious. On the contrary

a more detailed analysis on a homogenous group of young athletes show that Caca-

Cola tend to deteriorate the performances. Figure 13.4 plots the performance (in

seconds) as a function of drunk litres per day exclusively for the subgroup of young

(less than 30) athletes. Note that the Caca-Cola marketing department was not

wrong in claiming the existence of a signiﬁcative relationship between Caca-Cola

and performance. On the contrary they were deﬁnitely wrong when they claimed

the existence of a cause-eﬀect relationship between these two variables.

•

13.2.1 Simpson's paradox

The Caca-Cola example is a typical instance of Simpson's paradox: an association

between a pair of variables can consistently be inverted in each subpopulation of a

population when the population is partitioned, and conversely, associations in each

subpopulation can be inverted when data are aggregated.

352 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Note that in the Simpson's paradox there is nothing paradoxical from the stand-

point of arithmetic and is simply due to the close connections between proportions,

percentages and probabilities. It is possible in fact to ﬁnd eight integers such that

a/b < A/B and c/d < C/D but

(a + c ) /(b +d )> (A +C )/ (B +D )

For instance 1/ 5< 2/ 8 and 6/ 8< 4/ 5, yet 7/ 13 > 6/13.

Let us consider now the experimental outcome of a medical experiment where a

treatment T has been administered to patients and the binary outcome Yhas been

recorded. Note that G = 0 stands for females, G= 1 for males, Y= 1 stands for

recovery and T = 1 for treatment administration. Suppose that the distribution of

treatment among genders is represented by the table below:

T=0 T=1

G=0 5 8 P (T = 1|G = 0) = 8/13

G=1 8 5 P (T = 1|G = 1) = 5/13

P( G= 1| T= 0) = 8 /13 P( G= 1| T= 1) = 5 /13

It follows that the conditional distribution of recovery conditional on gender and

treatment is given by:

T=0 T=1

G=0 P (Y = 1|T = 0, G = 0) = 1/ 5P (Y = 1|T = 1, G = 0) = 2/ 8P (Y = 1|G = 0) = 3/13

G=1 P (Y = 1|T = 0, G = 1) = 6/ 8P (Y = 1|T = 1, G = 1) = 4/ 5P (Y = 1|G = 1) = 10/13

P( Y= 1| T= 0) = 7 /13 P( Y= 1| T= 1) = 6 /13

We obtain then the following probabilistic inequalities:

P( Y= 1| T= 0 , G = 0) = 1 /5 < P ( Y = 1| T= 1 , G = 0) = 2 /8

P( Y= 1| T= 0 , G = 1) = 6 /8 < P ( Y = 1| T= 1 , G = 1) = 4 /5

P( Y= 1| T= 0) = 7 /13 > P ( Y= 1| T= 1) = 6 /13

If we interpret the probabilistic relationships in causal terms, it appears that the

treatment is eﬀective for both the female subpopulation (P (Y = 1|T = 1, G = 0) >

P( Y= 1| T= 0 , G = 0)) and the male population, yet this is not the case for the

entire population (P (Y = 1|T = 0) > P (Y = 1|T = 1)).

Though all this is perfectly compliant with arithmetic (and statistics) it poses

problems from a decision-making perspective: should we base a decision on the

statistics from the aggregate population or from the partitioned subpopulations?

Does it make sense to have a treatment which is eﬀective both for men and women

but not for the union of them? In probabilistic terms, should we trust the marginal

or the conditional distribution? The answer of Pearl is that the answer cannot be

found in probability but has to be found in causal reasoning. This means that we

need a piece of missing information to answer the question and this information is

in the causal graph. In other words, translating a probabilistic relationship into a

causal relationship is the "original sin" in the Simpson paradox.

Figure 13.5 and 13.6 show two alternative causal graphs and, for the moment,

it is suﬃcient for the reader to know that a direct arrow from a node to another

represents a direct causal action. Knowing which causal graph corresponds to our

medical setting will allow solving the paradox [148].

Looking at both ﬁgures, it appears evident that in our case, the correct causal

graph is Figure 13.5: in fact, it is not conceivable that a treatment causes a person

gender (as in Figure 13.6), while it is possible that both a treatment and its eﬃcacy

depend on the gender. This means that in our case the treatment is beneﬁcial

and that the only meaningful probabilistic relation is the unconditional one P (Y=

1|T = 0) > P (Y = 1|T = 1).

We will show afterwards that for a graph like Figure 13.5, the causal dependency

of the treatment T on Y is correctly represented by conditioning on gender.

13.3. CAUSAL VS ASSOCIATIONAL KNOWLEDGE 353

Figure 13.5: Causal graph where Gis a direct cause of both T and Y.

Figure 13.6: Causal graph where Tis a direct cause of G.

The paradox derived from the fact that gender and treatment have an impact

on recovery and that gender has an impact on treatment as well. Females recover

much less (in probabilistic terms P (Y = 1|G = 0) < P (Y = 1|G = 1)) and take

more drugs (P (T = 1|G = 0) > P (T = 1|G = 1)). Using the non-conditioned

relation would be erroneous and counterintuitive.

Now let us keep ﬁxed all the observed numbers but let us change the semantics

of variables. Suppose that Gstands for some beneﬁcial mechanism P (Y = 1|G =

1) > P (Y = 1|G = 0) whose functioning is aﬀected by the treatment P (G = 1|T =

1) < P (G = 1|T = 0). Which causal graph do you think is more adequate? Would

you condition on G or not, to measure the causal eﬀect of T on Y?

13.3 Causal vs associational knowledge

Human common sense knowledge is about how things work in the world. Such

knowledge is (i) eﬀective since it gives humans the ability to intervene in the world

and change it and (ii) causal because it is about the mechanisms that bring from

causes to eﬀects. Here mechanism stands for an input/output relationship where

the choice of inputs determines the outputs, but the reverse does not hold.

The goal of many sciences is to formalise such common sense and understand the

mechanism by which variables came to take on the values they have, and to predict

what the values of those variables would be if the naturally occurring mechanisms

354 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

were subject to manipulations. It follows that the questions that motivate most

studies in the health, social and behavioural sciences are not associational but causal

in nature.

Causal analysis aims to infer not only beliefs or probabilities under static con-

ditions but also the dynamics of beliefs under changing conditions, for example,

changes induced by treatments or external interventions. However, according to

Pearl [146], associational studies deserved more interest than causal studies for the

following reasons:

•associational assumptions, even untested, are testable in principle, given suﬃ-

ciently large samples and suﬃciently ﬁne measurements. Causal assumptions,

in contrast, cannot be veriﬁed even in principle unless one resorts to experi-

mental control.

•associational assumptions can be expressed in the familiar language of prob-

ability calculus, and then they assumed an aura of scholarship and scientiﬁc

respectability. Causal assumptions, until recently, were deprived of that and

became suspect of informal, anecdotal or metaphysical thinking.

In order to address the lack of an adequate probabilistic formalism to represent

the notion of manipulation, Pearl introduced the do() operator, which allows us to

distinguish between the conventional observational notion of statistical dependency

(quantiﬁed in terms of conditional probability) and the interventional notion of

causal dependency.

Let us ﬁrst remind that a (discrete) random variable yis said to be dependent

(Section 3.5.2) on a variable xif the distribution of yis diﬀerent from the marginal

one when we observe the value x = x

Prob {y |x =x } 6 = Prob {y }

The property of dependency is symmetric , i.e. if y is dependent on x , then xis

dependent on yas well, i.e.

Prob {x|y =y } 6 = Prob {x}

The concept of causality describes a process where the control (not simply the

observation) of one event changes the likelihood of the occurrence of another event.

Deﬁnition 3.1 (Cause) . A variable x is a cause of a variable yif the distribution

of y is diﬀerent from the marginal one when we set the value x = x

Prob {y | do (x =x )} 6 = Prob {y }

In other terms, xis a cause of yif we can change y by manipulating x but not

the other way round. Unlike dependency, causality is asymmetric, i.e.

Prob {x|do (y =y )} = Prob {x}

It is important to summarise then the diﬀerences between conditioning on in-

tervention and conditioning on observation.

Intervention is formalised by the do () operator and corresponds to set (or bet-

ter manipulate) a variable to a speciﬁc value. This manipulation may change the

probabilistic model and the nature of the dependencies between variables (e.g. the

stochastic dependency between a cause and an eﬀect is lost once we manipulate the

eﬀect).

Conditioning is formalised by the conditional probability operator and corre-

sponds to observe a speciﬁc value of a variable. By observing, we change nothing

to the causal mechanism: we simply narrow our focus to a set of observations. In

plain words, the world does not change, our perception does.

13.4. THE TWO MAIN PROBLEMS IN CAUSALITY 355

13.4 The two main problems in causality

The literature on causality addresses two main problems: (i) the estimation of causal

eﬀects and (ii) the causal discovery from observational data. In the ﬁrst problem,

the causal mechanism is known or guessed, but the goal is to assess causal eﬀects

from an observational or experimental study. For instance, in a medical study, we

could be confronted with two groups of units (exposed/non-exposed), and we would

like to quantify the causal eﬀect of the exposure (smoking versus not smoking). In

the second problem, we have no (or very limited) information about the causal

mechanism, and the goal is to reconstruct the causal mechanism from data. For

instance, in bioinformatics, this problem is encountered every time we want to infer

a transcriptional network from observed genomics data.

In what follows, we will show how the ﬁrst problem has been addressed both by

the potential outcomes and the graphical modelling approaches. Then we will dis-

cuss the issue of causal discovery, notably thanks to structure learning in graphical

models.

13.5 Causality and potential outcomes

Causality is about estimating the eﬀect of an action (or manipulation, treatment,

intervention), applied to a unit (e.g. a person, a physical object, a ﬁrm), over a

speciﬁc outcome. Although a unit has been (at a particular point in time) exposed

to a particular action (treatment/regime), in principle, it could have been exposed

to an alternative action at the very same point in time. This concept has been

formalised by Rubin [164] with the notion of potential outcome, denoting the random

distribution of an outcome variable which can be observed only if the action is

performed. In a binary treatment, for each realised action, there is a potential

outcome associated with the alternative action which could not be observed. The

eﬀect of an action is then related to both potential outcomes (what happened and

what could have happened).

Let y (x) be read as the outcome y under treatment x . If x can take two values,

the potential outcomes y (1) (y (0) ) are two outcome variables (distinct from y) which

denote the distribution of the outcome if the (binary) action xhad (not) been done:

y(0) ∼Prob { y| do( x = 0)}, y(1) ∼Prob { y| do( x = 1)}

Note that the deﬁnition of those outcomes is irrespective of which action actually

occurred. The link between y (observation), y (0) and y (1) is

y=( y (1) if x= 1

y(0) if x= 0

Since in the reality we can only observe either y (1) or y (0) , the observed yi

satisﬁes the following relation:

yi =xi y (1)

i+ (1 −x i )y (0)

i=y (0)

i+ (y (1)

i−y (0)

i)x i .(13.5.1)

Potential outcomes and observations

Let us consider the notion of potential outcome in the context of the following

observed dataset:

356 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

unit treatment x observed yy (0) y (1)

1 1 y1 ? y1

2 0 y2y2 ?

i 1 yi ?y i

N 0 yNyN ?

For each unit in the population, the observed outcome is called the factual

outcome yF , while the other is called the counterfactual yCF . Note that sometimes

the factual outcome is drawn from y (0) and sometimes from y (1), according to the

performed action. This deﬁnition makes explicit that the observed random variable

yis only a lumped version of the potential outcomes.

•

Beyond treatment and outcome, the third major actor of the causal inference

problem are covariates , denoted by z . For instance, in a medical setting, they could

be measures related to individual medical history or family background information.

The key characteristic of these covariates is that they are a priori known to be

unaﬀected by the treatment assignment. We will see later that covariates may help

to improve the accuracy of the causal eﬀect estimation by deﬁning conﬁgurations

or groups of interest which might play a role in the assignment mechanism.

13.5.1 Causal eﬀect

The causal eﬀect is deﬁned as the comparison of potential outcomes, for the same

unit, at the same moment in time. For a given unit (or individual) iand a binary

treatment, the unit level causal eﬀect (or individualised treatment eﬀect) is

y(1)

i−y (0)

The average causal eﬀect over the population is

τ= E[y (1) ]− E[y(0) ] .(13.5.2)

The deﬁnition of causal eﬀect depends on the potential outcome but not on the

actual observed outcome: nevertheless, in practice, if we wish to estimate it, we will

have to use observed outcomes over multiple units and diﬀerent times.

For the sake of estimation it is then crucial to know (or at least make assumptions

about) why some actions were made and others not: this is known as the assignment

mechanism and quantiﬁes the statistical dependency between treatment, outcome

and covariates.

Example

Let us consider a heterogeneous population of patients in terms of ages. Let us

suppose that the older the patient, the higher is the death risk y. Let us give a

treatment (e.g. drug or hospitalisation) to the oldest patients only and compute

the average risk of treated patients vs untreated ones. Since the a priori risk of old

people was already high, after the treatment their average risk (though decreased

thanks to treatment) remains higher than the risk of non treated persons. In other

terms, though over the entire population

E[y (1) ] < E [y (0) ]

13.5. CAUSALITY AND POTENTIAL OUTCOMES 357

y(1)

y(0)

E[y| x=1]=E[y(1)| x=1]

E[y| x=0]=E[y(0)| x=0]

E[y(0) | x=1]

E[y(1) | x=0]

causal eﬀect

bias

observed eﬀect

Low risk

Age

Treatment

No treatment

Factual

y(0)i

y(1)i

High risk

Figure 13.7: Observed vs. causal eﬀect. Left: y (0) distribution (untreated popu-

lation). Right: y (1) distribution (treated population). Black dots denote the set

observed patients: half of them are young and untreated (left) and half of them are

aged and treated (right).

i.e. the individual causal eﬀect is constant and negative (risk reduction), we observe

(Figure 13.7)

E[y| x = 1] > E[y |x = 0]

i.e. a positive treatment eﬀect (risk increase).

It follows that by looking only at the observed data, we would make the wrong

conclusion that the treatment is harmful. Note that this is analogous to consider

a surgeon with many deceased patients during the operations less talented than a

surgeon with few deceased patients: if we ignore the severity of the patients state,

the diﬀerent death rate could be misleading since the ﬁrst (smarter) surgeon could

be the one dealing (because more talented) with all diﬃcult cases.

•

13.5.2 Estimation of causal eﬀect

We can never measure a causal eﬀect (for instance (13.5.2)) directly since the coun-

terfactual outcome is missing by deﬁnition; such missing data problem is called the

fundamental problem of causal inference.

In the absence of counterfactual data, can observed data be used to estimate

causal treatment eﬀects for a given covariate? This is the aim of the causal analysis

which targets the quantity

E[y(1) | x = 1] − E[y (0) |x = 1] = E[y (1) −y (0) |x = 1]

representing the average diﬀerence between the risk of the treated and what would

have happened to them (counterfactual) had they not been treated.

358 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

It can be shown that

E[y |x = 1] − E[y| x = 0]

| {z }

observed

=E [y (1) |x = 1] −E [y (0) |x = 0] =

E[y(1) |x = 1] − E[y (0) |x = 1]

| {z }

avg causal eﬀect on the treated

+E [y(0) |x = 1] −E [y (0) |x = 0]

| {z }

selection bias

(13.5.3)

The selection bias term shows that the observed causal eﬀect may be biased

if the distribution of the potential outcomes is not independent of the assignment

strategy ((y (0) ,y (1) ) 6⊥⊥ x ). This is also known as a non-exchangeable conﬁguration.

We invite the reader to retrieve the terms of (13.5.3) in Figure 13.7.

In order to draw valid causal inferences, we need to make assumptions about

the assignment mechanism , that is the distribution

P(x = 1|z = z, y (0), y(1) ) (13.5.4)

describing the probability of a unit of receiving a treatment given the distribution

of the covariates and of the potential outcomes of the problem.

13.5.3 Assignment mechanisms assumptions

An assignment mechanism is

•individualistic if the treatment of a unit does not depend on other units. For

instance, sequential assignments are not allowed in this case.

•probabilistic if the probability of assignment (13.5.4) is strictly between zero

and one. Each unit has a non null probability of being assigned both treat-

ments (level 0 and 1).

•uncounfounded if it is conditionally independent of the potential outcomes,

i.e. P (x = 1|z = z, y (0), y(1) ) = P (x = 1|z =z ) = e (z ).

An individualistic, probabilistic and unconfounded assignment is also referred

to as a strongly ignorable treatment assignment. A randomised experiment is an

assignment mechanism that has a known probabilistic functional form that is con-

trolled by the researcher. In an observational study, the functional form of the

assignment mechanism is unknown.

13.5.4 About unconfoundness

The main obstacle to unbiased causal reasoning is confounding which occurs when

the assignment of the treatment is not independent of the potential outcomes, e.g.

because of omitted or non observable variables related to both the treatment and

the outcome. Unconfoundness (also known as ignorability ) states that potential

outcomes are independent of the observed treatment conditional on confounding

covariates

(y (0) ,y (1) ) ⊥⊥ x |z. (13.5.5)

In plain words, for individuals with the same z, knowing the treatment brings no

information about y (0) or y (1) or, equivalently, the treatment does not depend on

the causal type. For instance for individuals with the same age, the treatment is

not given to patients who are supposed to react better and knowing that a patient

got a treatment does not provide any information about the success odds.

13.5. CAUSALITY AND POTENTIAL OUTCOMES 359

Note that the lack of dependence between the treatment and the outcome does

not imply that the assignment is ignorable

y⊥⊥ x 6⇒ (y (0) ,y (1) ) ⊥⊥ x

At the same time, an ignorable treatment but with a treatment eﬀect diﬀerent

from zero does not imply independence between the treatment and the outcome.

(y (0) ,y (1) ) ⊥⊥ x 6⇒ y ⊥⊥ x

The correct interpretation of unconfoundness is that for a given covariate z,

knowing which treatment has been given to a unit (e.g. observing x) does not

provide any additional information on the distribution p (y |do (x )). This is not the

case of the example of Figure 13.7 where by knowing the treatment we know the

age and then the death risk of the patient.

On the other way round, we have a non ignorable assignment mechanism if

the treatment is assigned on the basis of unmeasured characteristics of the unit.

Naively, we could then expect that to ensure plausibility of such assumption, we

should condition on as much pre-treatment information as possible. However, one

can never prove that the treatment assignment process in an observational study is

ignorable since it is always possible that the choice of treatment depends on relevant

yet latent information.

13.5.5 Randomised designs

A randomised experiment is a probabilistic assignment mechanism whose form is

known and controlled by the researcher. The aim of randomisation is to make the

treatment independent of its other causes, thereby destroying possible confounding

settings, due either to observed or unobserved variables. In case of binary treatment,

a random assignment process guarantees that the resulting treatment and control

groups are identical on any other characteristic (than the treatment) inﬂuencing

the outcome.

Though it is the "gold standard" procedure for causal reasoning, in practice

there are many concerns about its adoption, ranging from ethical concerns to the

representativeness of of the generated population with respect to the observed pop-

ulation.

The most common types of randomised designs are:

•Bernoulli trials: they assign a treatment by tossing a fair coin for each unit.

Though this is the most intuitive design, it is exposed to a high risk of un-

helpful assignments.

•Completely randomised experiment: a classical randomised experiment where

each unit has the same assignment probability (i.e. the number of treated units

is ﬁxed), and this probability is diﬀerent from 0.5. In this design, the risk is

that the treatment assignment neglects covariate eﬀects.

•Stratiﬁed (or conditionally) randomised experiments: they partition the pop-

ulation of units in blocks (strata), e.g. according to covariates (for instance,

age).

•Paired randomised experiments: two units (chosen on the basis of covariates)

per block, one is treated and the other not.

360 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

13.5.5.1 Estimation of the treatment eﬀect

Suppose we want to estimate the average treatment eﬀect (13.5.2) in a completely

randomised experiment where the treatment xis binary. In a population of Nunits,

this is given by

τ=P N

i=1(y (1)

i−y (0)

Unfortunately, this quantity cannot be computed directly since for all i one of the

two terms is missing. Let us then consider an estimator

τ=PN

i=1(ˆyi1 −ˆyi0 )

N(13.5.6)

Such estimator can be implemented if the terms ˆ yare observable (i.e. dependent

on observations yi ) and is unbiased if

E[ˆ

τ] = τ(13.5.7)

Since from (13.5.1) yi =xi y (1)

i+ (1 −x i )y (0)

i, we put

ˆyi1 = wi xi y (1)

i,ˆyi0 =wi (1 − xi ) y (0)

i(13.5.8)

In order to guarantee the unbiasedness (13.5.7), the following constraints have

to be satisﬁed

i=1

ˆyi1 ] =

i=1

y(1)

i, E[

i=1

ˆyi0 ] =

i=1

y(0)

Note that the ﬁrst constraint, because of the unconfoundness (13.5.5) and (13.5.8),

boils down to

i=1

ˆyi1 ] = E[

i=1

wi xi y(1)

i] =

i=1

wi E [xi ] y (1)

i=1

y(1)

and analogously the second.

Since the treatment xis binary, E [ xi ] = P ( xi = 1). To ensure (13.5.7) we

must weigh each treated example with a weight equal to 1/P (xi = 1) and each

untreated example with 1/P (xi = 0). Since in a completely randomised experiment

N1 + N0 = N we have

P(xi = 1) = N1 /N, P ( xi = 0) = N0 /N

and from (13.5.6) and (13.5.8) we obtain

ˆ τ=P N

i=1(ˆ yi1 −ˆ yi0 )

N= P N

i=1(x i y i1/(N 1 /N )− (1 −x i )y i0 /(N 0 /N))

=P N 1

j=1 y j1

N1 − P N 0

j=1 y j0

Note that this boils down to increase the weight of the observed treated cases

by a factor equal to N/N1 . In general, the lower the probability of an observation,

the higher should be its weight in the estimator of the treatment eﬀect.

This derivation shows that in a randomised case, it is possible to obtain an

unbiased estimation of the causal eﬀect by a proper weighting of the observations

and without access to counterfactual information.

13.5. CAUSALITY AND POTENTIAL OUTCOMES 361

13.5.5.2 Stratiﬁed (or conditionally) randomised experiments

This class of randomisation experiments is particularly important since it can be

used as a template to deal with observational experiments. In a stratiﬁed randomi-

sation setting, the population is ﬁrst partitioned into blocks or strata so that the

units within each block are similar in terms of covariates expected to be predictive of

potential outcomes. The simplest case is when we stratify the population according

to gender (two blocks) to estimate the eﬀect of a medical treatment.

Within each block, a completely randomised experiment is performed, and the

relative size of treatment groups is the same over strata. Analogously to the ran-

domised case, the estimation of the treatment eﬀect is obtained by the weighted

average of the treatment eﬀect over each stratum, with a weight proportional to the

population in the stratum.

13.5.6 Observational study

Unlike a randomisation experiment, in an observational study, the functional form

of the assignment mechanism is (at least partially) unknown. This means that we

could have a risk of outcome systematic diﬀerences between the treatment groups,

which are not due to treatment exposure (Figure 13.7). The diﬀerences between

the groups might be due to confounding variables (e.g., age, genetic susceptibility

to cancer) rather than treatments themselves.

The rationale of the potential outcome approach is to make the observational

setting as conform as possible to the experimental one in order to make causal

inference possible. Since an investigator cannot assign the treatment in nonexper-

imental studies, s(he) must rely on the remaining degree of freedom: the selection

of subjects. It is then necessary to make the assumption that the observational

study is a conditionally randomised experiment, or equivalently that for a given set

of covariate the unconfoundness hypothesis (13.5.5) (i.e. assignment independent

from potential outcomes given covariates) holds. Without such assumption (and

in the absence of alternatives) we would have no idea of how to use observational

values for causal inference.

If uncounfoundness holds, within a subpopulation deﬁned by the same covariates

(e.g. females or males), the diﬀerence in the distributions of the observed outcomes

(i.e. treated and untreated) is an unbiased estimator of the treatment eﬀect since

both treated and control units are assumed to be random samples from this subpop-

ulation. This makes irrelevant the fact that we do not know a priori the assignment

mechanism.

Unconfoundness is a formal version of the principle stating that you should

compare "like with like". As Rubin said: "it would make little sense to compare

disease rates in well-educated non-smokers and poorly educated smokers".

Note, however, that, unfortunately, such assumption is hardly testable in prac-

tice. No information in the observed data can tell us whether such assumption holds

or not. At the same time, the validity of a causal estimation based on observational

data relies on the hypothesis that we are conditioning on the right covariates, i.e.

the ones that make treatment and potential outcomes independent. Once condi-

tioned on those covariates, it is expected that the remaining degree of variability

related to the treatment choice (though not random like in randomisation studies)

will be unrelated to potential outcomes.

In practice, the choice of those variables is not easy, notably when the number of

confounding variables is large, or most of them are continuous. Literature abounds

in terms of generic recommendations (e.g. variables that are aﬀected by the treat-

ment like intermediate outcomes should not be included), but no formal procedure

is proposed.

362 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

13.5.7 Strategies for estimation in observational studies

If the unconfoundness assumption is granted, the causal eﬀect estimation may pro-

ceed according to four strategies [163]

1. Model-based imputation: it imputes the missing potential outcomes by build-

ing and using a model to predict what would have happened to a unit had

been exposed to the alternative treatment.

2. Weighting

3. Blocking

4. Matching methods: they impose the same distribution of the control and case

population with respect to some confounding factors.

The ﬁrst method requires a model of the potential outcomes, the other three can

be implemented before seeing any outcome data. We will brieﬂy discuss here only

matching which refers to a variety of procedures that restrict and reorganise the

original sample for statistical analysis. Matching is a way of discarding observations

so that the remaining data show good balance and overlap. The simplest form is

one-to-one matching where the data points are divided into pairs: each pair contains

both a treatment and a control unit, and the two units are as similar as possible

on the pre-treatment variables. For instance, if the ith unit is associated with zi ,

we match it to an unit whose set of covariates zj is as close as possible to zi . The

diﬀerence of the observed outcomes associated to those matched units is supposed to

be an accurate estimate of the causal eﬀect. In multivariate settings, the similarity

is typically measured by metrics like the Mahalanobis distance (9.2.64). Note,

however, that, though the notion of similarity is intuitive, its adoption in a high

variate case is non trivial as discussed in Section 12.1.

In order to avoid the issue of specifying which variables to use for matching,

a well-known alternative is the adoption of propensity score modelling [86]. This

approach relies on classiﬁcation techniques (e.g. logistic regression) to estimate

from data the propensity score Prob {x = 1|z } , that is a compact way to summarise

high-variate covariates in a single value. Note that the target of a propensity score

classiﬁcation model is not the outcome ybut the treatment value x . However,

the aim of this approach is not to predict the treatment but deﬁne a metric of

the observations in the space of covariates: observations with similar estimated

propensity scores have similar proﬁles in terms of the covariates z and then should

be candidates for matching.

13.6 From potential outcomes to graphical models

Potential outcomes approach played a major historical role in the history of causality

in sciences. Its main merit was to deﬁne under which circumstances it is possible

to transpose methods from the randomisation setting to the observational one.

However, the opacity of the notion of unconfoundness in observational setting is

probably the most relevant Achilles' heel of this approach since no procedural way

exists to assess the validity of such condition, in particular in large variate problems.

According to J. Pearl [146], most investigators have diﬃculty in understanding

what ignorability means and tend to assume that it is automatically satisﬁed or

that it is likely to be satisﬁed the larger is the number of covariates. This led to the

naive (somewhat dangerous) illusion that adding more covariates causes no harm.

The lack of a formal algorithmic way to deﬁne the conditioning variables to assess

causal eﬀects is the main reason to introduce the Pearl work on causal graphical

methods.

13.7. CAUSAL BAYESIAN NETWORK 363

We introduced in Chapter 4 the notion of graphical models, notably DAGs,

stressing their eﬃcient representation of large variate distributions and the bijec-

tive mapping between graphical properties (d-separation) and probabilistic prop-

erties (conditional independence). What is also relevant here is their role from

a causal perspective. DAGs is a formalism that encodes and visualises the link

between (conditional) statistical associations and causal mechanisms. As such, it

allows the prediction of the impact on the probability distribution of manipulations

and interventions. Last but not least, it provides visual support to recognise and

avoid common mistakes in causal reasoning. Overall, DAGs are a very convenient

formalism to represent the following properties of causal relationships:

1. Transitivity: if event A causes event B and event B causes event C, then it

must also be true that A causes C.

2. Locality : if A causes C only through the eﬀect of an intermediate B, then

the causal inﬂuence is blocked once the event B is kept ﬁxed.

3. Irreﬂectiveness : an event cannot cause itself.

4. Asymmetry : if A is a cause of B (i.e. one sees changes of B when changing

A), then B cannot be a cause of A (i.e. one cannot expect changes of A when

changing B). Note that this does not exclude temporal feedback loops.

13.7 Causal Bayesian network

A Causal Bayesian Network (CBN) is a graphical model where the notion of edge

has a speciﬁc causal meaning and is then semantically richer than in conventional

Bayesian Networks where it merely represents a probabilistic dependence.

Deﬁnition 7.1 (Causal BN) . A BN is causal if for each edge xi →xj , the variable

xi is one of the direct causes of xj (Deﬁnition 3.1).

Unlike graphical models which are used to encode an (in)dependence structure,

causal graphical models support a stronger interpretation, i.e. the manipulation of

xi induces a change of distribution of xj , whatever the value of all other variables.

This implies that, thanks to the notion of d-separation (Section 4.3.1), it is possible

to associate testable conditional independence patterns to speciﬁc causal patterns.

For instance the pattern x1 ⊥⊥ x2 ,x1 6⊥⊥ x2 |y is associated to the common eﬀect

causal pattern in Figure 13.8 where conditioning on a collider (or a descendant)

makes the two causes conditionally dependent. Other interesting patterns are en-

coded in the DAG of Figure 13.9. On the left we have a variable ywhich is a

common cause of x1 and x2 where x1 6⊥⊥ x2 , x1 ⊥⊥ x2 |y . On the right we have a

causal chain pattern where x2 6⊥⊥ y, x2 ⊥⊥ y |x 1

Causal patterns: examples

We may visualise those dependency patterns and their sensitivity to conditioning

by running the scripts collid.R and chain.R . For instance the leftmost plot in

Figure 13.10 represents the pattern in Figure 13.8: variables x1 and x2 are inde-

pendent (black dots) but they become dependent when we restrict the dataset by

conditioning (red dots).

•

364 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.8: Common eﬀect causal pattern.

Figure 13.9: Left: common cause pattern. Right: causal chain pattern.

420 2 4

COLLIDER (y=x1+x2): conditioned on y=0

1.0 0.5 0.0 0.5 1.0

FORK (x1=y+e1, x2=y^3+e2): conditioned on y=0

210 1 2

420 2 4

CHAIN (y=x1+e1, x2=2y+e2): conditioned on y=0

Figure 13.10: Visualisation of three causal patterns in terms of dependency (black

dots) and conditional dependency (red dots).

13.7. CAUSAL BAYESIAN NETWORK 365

Figure 13.11: Causal Bayesian Network.

13.7.1 Causal networks and Structural Causal Models

It may be shown that for every DAG characterised by a distribution, Pthere ex-

ists a set of equations, called Structural Causal Model (SCM), that generates a

distribution identical to P.

Deﬁnition 7.2 (Structural causal model) . A SCM consists of n equations of the

form

xi = fi (πi , wi ), i = 1, . . . , n (13.7.9)

where πi stands for the set of variables judged to be the immediate causes of x i

and wi represent the noise due to omitted factors.

A structural model is a set of n equations fi . If the noise terms wi are jointly

independent, the model is called Markovian. This implies that no noise term (un-

observed variable) inﬂuences more than one observed variable. This assumption is

also known as causal suﬃciency that means that we observe all relevant variables.

Note that the symbol = in (13.7.9) should be read like an assignment symbol (e.g.

the symbol := in programming languages) rather than like an algebraic equality.

The SCM correspondent to the CBN is Figure 13.11 is

x= wx

y= wy

z=fz ( x, y, wz )

k=fk ( z, wk )

v=fv ( y, wv )

w=fw ( v, ww )

We see that each edge in the DAG is associated with a function in the SCM.

13.7.2 Pre and post-intervention distributions

The essence of causal reasoning by CBNs is that the observational (pre-interventional)

and the post-interventional distributions are not necessarily the same, yet they are

somewhat related, and their relation is made explicit by the causal graph. A causal

graph is not only a model of the (in)dependences in an observational setting, but

also indicates how those (in)dependences change as a consequence of experiments

and interventions.

366 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

x=x

P(y | x =x) P(y | x=do(x))

Figure 13.12: DAG surgery: on the left (right) the BN associated to the pre(post)-

intervention distribution.

Consider for instance two diﬀerent causal conﬁgurations: x1 → y→ x2 and

x1 ← y→ x2 . Suppose we are interested in the post-intervention distribution

of y once we act on (i.e. we manipulate) x1 . The post intervention distribution

in the ﬁrst conﬁguration is given by the conditional distribution p (y| x1 ,x2 ): the

post intervention distribution in the second conﬁguration is the marginal density

p(y |x2 ). In this second conﬁguration, intervening on x1 removed the probabilistic

dependency between x1 and y (i.e. after intervention the value of x1 provides

no more information about y). At the same time, the observation of x2 is still

informative about y.

If we consider instead the post-intervention distribution of x2 under an inter-

vention on y , this is the same (and equal to p (x2 |y )) in both causal conﬁgurations.

It is important then in a general setting to understand how to move from pre-

intervention (observational) data to a post-intervention setting. The major advan-

tage of DAGs is that organising causal knowledge in a graphical manner allows

predicting the eﬀect of external interventions with a minimum of extra information.

Interventions can be modelled by removing links or equivalently adding conditional

independence relationships. This makes explicit the diﬀerence between observing

and doing : observing does not change the causal structure, while a do() action

induces a "surgery" change in the topology of the DAG, or equivalently it modiﬁes

a set of functions in the SCM (Figure 13.12).

In particular we deﬁne atomic intervention (or manipulation) the intervention

where a variable xi is forced to take a value xi (denoted by do ( xi =xi )). The

atomic intervention pulls xi out of the SCM functional mechanism xi = fi (. . . ) and

places it under the inﬂuence of a new mechanism that sets xi = xi , while keeping

all the other mechanisms unperturbed. This amounts to removing the equation

xi =fi (. . . ) in the corresponding SCM and replacing it with the equation xi =xi .

In plain words, the eﬀect of manipulations is to disconnect the manipulated variables

from their natural causes.

13.7.3 Causal eﬀect estimation and identiﬁcation

Post-intervention distribution is required to estimate the causal (treatment) eﬀect

P(y = y|do( x= x0 )) − P(y = y|do( x= x"))

13.7. CAUSAL BAYESIAN NETWORK 367

Unfortunately, the post-intervention distribution is not observable. So it is essential

trying to answer the following question: "Can the controlled (post-intervention)

distribution be estimated from data governed by the pre-intervention distribution?".

This is the problem of identiﬁcation.

A fundamental theorem in causal analysis states that such identiﬁcation is fea-

sible whenever the model is Markovian, i.e. the graph is acyclic (i.e., containing no

directed cycles), and all the error terms are jointly independent.

Theorem 7.3 (Adjustment for direct causes [146]) . Let πi the set of direct causes

of xi and let y be any other variable disjoint of πi ∪xi . The post-intervention

distribution of yis

P(y = y|do( xi )) = X

πi

P(y = y| xi , πi ) P( πi ) (13.7.10)

where P (y =y |xi , πi ) and P (πi ) are pre-intervention distributions.

The essence of this theorem is that it transforms a causal statement (left-hand

term of (13.7.10)), i.e. a statement that cannot be directly estimated from observa-

tions, into a probabilistic statement (right-hand term), i.e. a statement that can be

estimated from observational data. The post-intervention conditional distribution

is then obtained by conditioning P (y =y |xi ) further on the parents of xi and then

averaging the results where the weights are given by the prior probability of πi . In

the case of multiple interventions, this general rule holds:

Theorem 7.4 (Truncated factorisation). For any Markovian model, the distribu-

tion generated by an intervention do( XS = X0 ) on a set XS of endogenous variables

is given by the truncated factorisation

P(x1 ,x2 ,..., xk |do( XS = X0 )) = Y

i|i6∈S

P(xi | πi )|X S =X0

where πi are the parents of xi and P( xi |πi ) are pre-intervention distributions.

We have to remove from the factorisation all factors associated with the inter-

vened variables (members of XS ).

Example

Let us consider the problem of computing the post-intervention distribution in two

diﬀerent (yet related) trivariate causal conﬁgurations. In Figure 13.13 we have a

causal conﬁguration3 where z is confounding the eﬀect of xon the outcome y.

The pre-intervention distribution P(associated to the left-side of Figure 13.13)

P(y = y|x= x) = X

P(y = y|x= x, z= z) P(z = z|x= x) (13.7.11)

The post-intervention distribution P0 (associated to the right-side of Figure 13.13)

and obtained by removing all the edges pointing towards xis

P(y = y|do( x)) = P0 (y = y|x= x) =

P0 (y = y|x= x, z= z ) P0 (z = z|x= x) = X

P0 (y = y|x= x, z= z ) P0 (z = z)

P(y = y|x= x, z= z ) P(z = z)6 = P(y = y|x= x) (13.7.12)

3Check the analogy with Figure 13.5.

368 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

x y

P: pre-intervention P': post-intervention

Figure 13.13: Confounding conﬁguration.

x y

P: pre-intervention P': post-intervention

x y

Figure 13.14: Intermediary conﬁguration.

This is in agreement with Theorem 7.3 and shows the diﬀerence between conditional

distribution P (y =y |x =x ) and causal intervention P (y =y |do (x)).

Let us consider now the causal conﬁguration4 where z plays the role of inter-

mediate between x and y (Figure 13.14). Since the pre- and the post-intervention

DAGs coincide, so the pre and the post-intervention distribution do as well.

P(y = y|do( x)) = P(y = y|x= x)

This means that in order to measure the causal eﬀect of x and y in the conﬁguration

of Figure 13.14, conditioning is not required. Note that the diﬀerence between

those two conﬁgurations is i) only due to the diﬀerent underlying causal structures

and ii) could not be detected by relying on conditional independence tests at the

observational level (see the Simpson paradox in Section 13.2.1).

•

13.7.3.1 Backdoor criterion

We discussed in Section 13.6 that the unconfoundness assumption, essential for es-

timating causal eﬀects in an observational study, is non-testable in the potential

outcomes approach. Theorem 7.3 shows that indeed we need a causal graph to per-

form causal reasoning. But what about the most general case, e.g. non observable

direct parents (Figure 13.15)? How to select a set of variables (also called suﬃ-

cient set ) such that conditioning on it, we can estimate the causal eﬀect with no

bias? The backdoor criterion [146] is a graphical criterion which allows to ﬁnd the

suﬃcient set.

Deﬁnition 7.5. A set of variables Z is admissible (or "suﬃcient") for adjustment

if two conditions hold:

4Check the analogy with Figure 13.6.

13.7. CAUSAL BAYESIAN NETWORK 369

Figure 13.15: Unobserved parent conﬁguration: zis a latent variable. [146]

Figure 13.16: What is the causal eﬀect of x on y?

1. No element of Zis a descendant of the treatment x.

2. The elements of Z block (Deﬁnition 3.5) all "backdoor" paths from x to y,

namely all paths that end with an arrow pointing to x.

Given a suﬃcient set Z, the average causal eﬀect of x(boolean) on y (boolean)

Prob {y = 1|do (x = 1)} − Prob {y = 1| do (x = 0)}=

[Prob {y = 1|x = 1,Z =Z } − Prob {y = 1|x = 0,Z =Z } ]P (Z =z )

In plain words, the suﬃcient set takes the role of the parents in (13.7.10). By

conditioning on this set, the observational distribution may be used to estimate

causal eﬀects.

Exercise

Compute the backdoor set for measuring the causal eﬀect of x on y ,w on y in the

causal structure of Figure 13.16.

•

The rationale of the backdoor criterion is that the total association between

treatment and eﬀect is a composition of the true causal eﬀect and the non-causal

association that is generated by the backdoor paths. The backdoor paths in the

diagram carry spurious associations from x to y , while the paths along the arrows

from x to y carry causative associations. The causal eﬀect of x on the outcome yis

identiﬁable if all spurious paths are blocked, and no new spurious paths are created.

Blocking the backdoor paths (by conditioning on Z) ensures that the observed

association between x and y is the causal eﬀect of x on y . This criterion can be

applied systematically to diagrams of any size and shape, avoiding the ambiguity

related to unconfoundness seen in the potential-response framework.

370 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.17: Application of rule R2.

13.7.3.2 Beyond suﬃcient set: do-calculus

Adjusting for suﬃcient covariates is only one of many methods that permits us

to estimate causal eﬀects in nonexperimental studies. Pearl (1995a) has presented

examples in which such a set of variables does not exist (or is not observable)

and where the causal eﬀect can nevertheless be estimated consistently. This led to

the frontdoor criterion (intermediary variable)5. All those results were masterfully

regrouped by J. Pearl in a set of rules which constitute the foundations of the

calculus of intervention.

A causal eﬀect of x on y is identiﬁable when the expression containing the

do( x=x ) operator can be transformed in an expression containing conventional

probabilistic statements. Let G ¯

x(G x) the graph obtaining by deleting from Gall

arrows pointing to (emerging from) x

1. P (y |do (x), z, w ) = P (y |do (x), w ) if (y ⊥⊥ z |w, x )G ¯

2. P (y |do (x), do(z ) , w ) = P (y |do (x), z, w ) if (y ⊥⊥ z |w, x)G ¯

3. P (y |do (x), do(z ) , w ) = P (y |do (x ) , w ) if (y ⊥⊥ z |w, x )G ¯

xz( w) where z (w ) is

the subset of z that are not ancestors of any wnode in G ¯

The rule R1 formalises the notion of surgery, while the rule R2 formalises the

notion of backdoor since it may be used to transform an action into an observation.

An example of the adoption of rule R2 is in Figure 13.17 where P (y| do(z), w ) =

P( y| z, w) since (y ⊥⊥ z |w )G z (as shown by d-separation in the lower graph Gz ).

R3 may be used to remove a do() operator, e.g.

P( y|do( z)) = P( y)

if (y ⊥⊥ z )G z i.e. a cause y is not aﬀected by the manipulation of the descendent z.

13.7.4 Selection bias

Another important insight of CBN on the risks of associative data analysis is related

to the notion of selection bias. Selection bias refers to any association created as

5For more details on a genial interpretation of the smoke/cancer causal relation in terms of

front-door criterion, we refer to [146]

13.7. CAUSAL BAYESIAN NETWORK 371

Figure 13.18: Selection bias: conditioning on a descendant of the outcome. [163]

Figure 13.19: Selection bias: spurious association due to selection. [163]

a result of the process by which individuals are selected into the analysis. In more

general terms, it refers to biases that arise from conditioning on a common eﬀect of

two variables, one of which is either the treatment or a cause of treatment, and the

other is either the outcome or a cause of the outcome. Selection bias in observational

studies may be due to the design (e.g. enrolment criteria) or unintended. A typical

case of bias is due to censoring (informative censoring) or to the restriction of

the study to volunteering persons (self-selection bias). Note also that the risk of

selection bias is not limited to the observational setting. In fact, randomisation

protects against confounding but not against selection bias when the selection occurs

after the randomisation.

Below we will present several examples of selection biases, taken from [163],

which illustrate the risk of spurious associations due to implicit conditioning. In

Figure 13.18 we see an example of selection bias due to conditioning on a descendant

y∗ (e.g. by thresholding) of the outcome y. The relation between xand uis

unconfounded, but the choice of limiting the training set to outcome values which

are smaller than a threshold may induce selection bias.

Figure 13.19 represents a causal graph where xstands for college education, i

for the presence of impaired memory (precursor of Alzeihmer) in memory and y

denotes the Alzeihmer pathology. sis a variable denoting the examples which were

included in the study (0 stands for not selected). Suppose we pool two datasets:

persons with college education (x= 1) and persons with impaired memory (i = 1).

The variable sis then the logical OR of x and i . In the resulting dataset, the

patients with x= 0 will have necessarily i= 1 and then y= 1. As a consequence of

the related selection bias, a negative association between education and Alzheimer

will appear, in spite of their independence made visible by d-separation.

The examples above illustrate two major beneﬁts of a causal representation of

the mechanism underlying data: DAGs visualise how causal relationships translate

into associations and provide a formal tool to detect biased situations (confounding,

selection bias). All this however requires the capability of drawing (or inferring) a

diagram that adequately describes contextually plausible associations.

372 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Another famous example of selection bias occurred in relation to the Challenger

Space Shuttle disaster in 1986 [94]. The night before the launch, an emergency

meeting took place to understand whether it was dangerous to proceed with the

launch given the exceptional low temperature. The committee concluded that no

relationship existed between temperature and past accidents. Unfortunately, a pos-

teriori it emerged that such conclusion was due to the fact that only data related

to past accidents (and then biased) were taken into consideration. If data related

to launches with no damage would have been included in the analysis as well, the

relation would have been easily detected and the tragedy probably avoided6.

13.8 Counterfactual

A counterfactual question that now you, patient reader of this book, could legiti-

mately ask yourself is "How much would I have been happier if I had decided NOT

to read such a book ?" The reasoning in order to answer such question should rely

1. your current state due to the choice of reading this book,

2. the causal model linking the decision and your happiness state (e.g. repre-

sented in Figure 13.20)

Counterfactual reasoning is probably one of the most important characteristics of

human intelligence and refers to the human ability to imagine unseen scenarios and

predict unobserved situations. This is the reason why Pearl [149] put counterfactuals

at the uppermost level of the reasoning skills about causality (Figure 13.2). While

do-operators P (y =y| do (x = 1)) model the average consequence of an intervention

(do(x = 1)) on a population, counterfactuals P (y(x=1) = 1|e ) model the impact

of an unseen intervention (do (x = 1)) on a segment of the population deﬁned by

some evidence e. Counterfactual reasoning combines the observed evidence with

the eﬀect of an intervention in three steps:

1. Abduction: use the evidence eto infer information about the pre-intervention

state (e.g. your motivation this morning or the quality of this book),

2. Intervention: modify the structural model to keep into account the action,

3. Prediction: combine the pre-intervention distribution with the action to pre-

dict what would have happened.

In order to link the factual and the counterfactual worlds, Pearl proposes the

adoption of a twin network merging the distribution of the observational world and

the distribution of the world which was manipulated according to the counterfactual

action (e.g. in the case of our reader, decide not to open this book). Such a twin

network (Figure 13.21) is a handy way to use the DAG formalism to model and

answer counterfactual questions.

Counterfactual example

Consider the talented doctor House, often urged to operate desperate cases, and

the success rate of his operations. Let x= 1 denote the fact of being operated

by doctor House, and y = 1 the death of a patient. Consider the three following

diﬀerences:

1. P (y = 1|x = 1) −P (y = 1|x = 0)

6https://bookdown.org/egarpor/PM-UC3M/glm- challenger.html

13.8. COUNTERFACTUAL 373

Happiness

this morning

Happiness

Now

Decision

Quality of

lecture

Motivation

Figure 13.20: Causal model of the reader satisfaction.

Happiness

Before

Happiness

Decision

Quality of

lecture

Motivation

Happiness (d=0)

Decision=0

Figure 13.21: Twin counterfactual model of the reader satisfaction.

374 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

2. P (y = 1|do (x = 1)) −P (y = 1| do (x = 0)) = P (y (1) = 1) −P (y (0) = 1)

3. P (y (1) = 1|x = 0) −P (y (0) = 1|x = 0)

The ﬁrst quantity is positive, not because the doctor House is a bad doctor, but

because he is used dealing with desperate situations. The second diﬀerence is neg-

ative since being operated by dr. House reduces the death risk. The third quantity

is counterfactual and states what would have happened to a patient who has not

been operated on by dr. House if this would have indeed happened. This quantity

should be more negative than the second one since the observation x = 0 provides

additional evidence that the case was not desperate.

•

Counterfactual reasoning in law

It is interesting to see how law directives (e.g. the ones against discrimination) are

often written in counterfactual language: The central question in any employment-

discrimination case is whether the employer would have taken the same action had

the employee been of a diﬀerent race (age, sex, religion, national origin, etc.) and

everything else had been the same. In counterfactual notation, this boils down to

estimate the following quantity

P(y(x=1) = 1|x = 0 ,y= 0)

where y = 0 where y denote the hiring and xhis race. The quantity above returns

the probability that the refused person (observation y= 0) with race x = 0 would

have been hired if the perceived race would have been diﬀerent.

•

13.9 Causal structure identiﬁcation

The estimation of causal eﬀects requires the availability of a causal graph. Even

when experimental interventions are possible, performing the large amount of ex-

periments that would be required to discover causal relationships between tens or

thousands of variables is not practical. Causal discovery aims to return plausible

explanations for observable associations in terms of DAGs. The rationale is that

some causal relationships can be tested without doing experimentation. This means

that some causal dependencies can be inferred from non-temporal statistical data

if one makes certain simplifying assumptions about the underlying process of data

generation.

•Causal Markov assumption (Deﬁnition 3.1): the causal interpretation is that

all dependencies (or associations) between variables are due to causal relations.

Note that is an oversimpliﬁcation since dependencies between variables could

be generated in non causal ways as well (see selection bias in Section 13.7.4).

•Faithfulness: all independencies found in the distribution are due to d-separations

in the graph or equivalently a d-connection implies a dependency (Section 4.3.2.1).

In causal terms, this means that all causal pathways should induce a depen-

dency, though this is not always the case (e.g. two causal pathways could

cancel out each other).

•Stability: the set of independencies of the associated distribution depends

only on the structure of the graph and not on the parametrisation. Unstable

independencies (i.e. dependencies that disappear with a parameter change)

are unlikely to occur in the data, so all the independencies are structural.

13.9. CAUSAL STRUCTURE IDENTIFICATION 375

•Causal suﬃciency: A set of variables Xis causally suﬃcient if for every pair

of variables x1 ,x2 ∈ X , every common direct cause of x1 and x2 is also a

member of X. If this is not the case, there are latent variables.

There are two main families of causal structure identiﬁcation: score-based and

constraint-based algorithms. Score-based algorithms search, within a set of can-

didate structures, the one which optimises some cost function. Commonly used

score functions are the z-score of hypothesis test (e.g. under the assumption of

Gaussian linear dependencies), maximum-likelihood or information-theoretic score

(e.g. BIC score). Such algorithms transform the problem of structure identiﬁcation

into a problem of optimisation. Though a number of state-of-the-art algorithms

may be used to address such optimisation, the large size of the search space makes

such an approach unfeasible for a moderate number of variables. Constraint-based

algorithms are more commonly used and discussed in the following sections.

13.9.1 Constraint-based approaches

We have extensively discussed in Chapter 4 the relation between topological proper-

ties and conditional independence in DAGs. The rationale of the constraint-based

approach is to use conditional independence relations as constraints during the

learning of the DAG structure from data. The idea is to derive from data a number

of testable implications (notably by estimating conditional independence) and use

them to disambiguate causal conﬁgurations (e.g. directionality) as much as possible.

The resulting algorithm iteratively looks for a DAG compliant with the statistical

constraints and consists of 2 main steps:

1. Skeleton discovery compliant with conditional independence patterns.

2. Orientation based on v-structures and acyclic constraints.

The ﬁnal goal is to discover the class of Markov equivalent DAGs (Deﬁnition 3.10)

which is consistent with the available dataset.

13.9.1.1 Normal conditional independence test

In order to speed up the conditional independence tests, constraint-based algorithms

often assume a Normal distribution. Consider a multivariate normal vector X , such

that xi ,xj ∈ X , XS ⊂ X and s is the dimension of XS . Then

ρx i x j |XS = 0 ⇔ I(xi ,xj | XS )=0

The sample partial correlation ˆ ρx i x j |XS can be computed by regression, inversion

of the covariance matrix or recursively (Section 3.8.3). To test the null hypothesis

ˆ ρx i x j |XS = 0 the Fisher's z-transform

Z=1

2log  1 + ˆ ρx i x j |XS

1− ˆ ρx i x j |XS 

is typically considered. The null hypothesis is rejected with the signiﬁcance level α

(false positive rate) if

Z√ N− s−3 >Φ−1 (1 − α/2)

where Φ(·) is the cumulative distribution of a standard Normal variable (Sec-

tion 3.4.2).

376 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Let H the empty graph over X// forward strategy

for each pair ( xi , xj ) in X do

Search XS such that I ( xi , xj | XS ) = 0 // conditional independence

if no XS exists then

connect (xi ,xj ) in H

else

Si,j =X S

end if

end for

Figure 13.22: IC algorithm: Si,j stores the set of variables that make xi and x j

conditionally independent).

Let H the complete undirected graph over X// backward strategy

for each pair ( xi , xj ) in H do

for XS ⊆ (V\ ( xi ,xj )) do

if I( xi , xj | XS )=0 then

Delete edge (i, j) from H

Si,j =X S

break

end if

end for

Figure 13.23: SGS algorithm.

13.9.1.2 Skeleton discovery

The ﬁrst example of constraint-based algorithm is the IC algorithm (Figure 13.22)

which adopts a forward and exhaustive approach. It starts with the empty graph

and for each pair (xi ,xj ) of nodes it adds a connecting edge if no conditioning set

makes them d-separated, otherwise it stores the separating set in the variable Si,j .

The number of tests in the worst case is bounded by

n

2 2 n−2

where n is the number of variables.

Given the high computational cost of this approach, some IC variants start with

the complete undirected graph (backward) or limit the conditioning size. An ex-

ample is the SGS algorithm (Figure 13.23) which requires, however, an exponential

search. In order to bound the computational complexity, the PC (Peter and Clark)

algorithm [172] (Figure 13.24) sets a maximum size Lof the conditioning size. For

a given L , the number of tests is bounded polynomially by

2 n

2 L−1

i=0 n−2

i ≤3n2 (n− 2)L

13.9.1.3 Dealing with immoralities in the skeleton

A potential immorality in the skeleton is a triplet of variables such that xi −xk − x j

but no edge exists between xi and xj (i.e I ( xi ;xj |xk )> 0 or equivalently xk / ∈Si,j )

It is then possible to extend the skeleton retrieved in the previous section by

marking the immoralities, orienting the edges of the associated collider and then

13.9. CAUSAL STRUCTURE IDENTIFICATION 377

Let H the complete undirected graph over X

for l = 0 to L do

for adjacent xi , xj in H do

for XS ⊆(N ( xi )∪N ( xj )) \ ( xi ∪ xj ) such that | XS |=l do

if I ( xi ,xj | XS ) = 0 then

Delete edge (i, j) from H

Si,j =X S

break

end if

end for

l= l+ 1

end for

Figure 13.24: PC algorithm: N ( xi ) denotes the set of neighbors of the node xi and

Lthe maximum conditioning set size.

Let H be the skeleton

for each non adjacent pair ( xi ,xj ) with a common neighbor xk in H do

if xk / ∈Si,j then

Add arrowheads pointing to xk (v-structure)

end if

end for

Orient as many undirected edges as possible by avoiding 1) new v-structures

and 2) directed cycles

Figure 13.25: Edge orientation strategy.

obtaining a Partially DAG. This is the ﬁrst part of the edge orientation strategy in

Figure 13.25. Once the immoralities are identiﬁed, we have identiﬁed the equiva-

lence class of the resulting PDAG.

The second part of the edge orientation strategy (Figure 13.25) relies on the

following rules:

•R1: Orient j −kinto j →kwhenever there is an arrow i→ j and i, k are not

adjacent.

•R2: Orient i −jinto i →jwhenever i →k→j

•R3: Orient i −jinto i →jwhenever i −k →jand i −l →jand k, l are not

adjacent.

•R4: Orient i −jinto i →jwhenever i −k →land k →l→jand k, l are

not adjacent.

PC example

The diagrams in Figure 13.26B-E illustrate the of steps of a PC algorithm re-

constructing the ground truth DAG in Figure 13.26A [80]. Step B relies on the

independence x ⊥⊥ y . Step C is due to the relation x ⊥⊥ w |z, y ⊥⊥ w |z . In D, the

algorithm creates a collider from triplet x− z− y since no x− y exists. Step E

implements the R1 orientation rule.

•

378 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.26: PC algorithm [80]. Diagram in A denotes the real DAG. The dia-

grams B to F represent the inferred structure at diﬀerent PC steps. The ﬁrst step

corresponds to the complete undirected graph. The last two steps are performed

by edge orientation.

13.9.1.4 Limitations

Constraint-based algorithms are well-known structure identiﬁcation algorithms and

have been intensively used in a lot of practical domains. Nevertheless, they suﬀer

from some limitations:

•I-equivalence classes: two graphs that are I-equivalent (Deﬁnition 3.8) cannot

be distinguished by constraint-based approaches without resorting to manip-

ulative experimentation or temporal information.

•Conditional independence: the use of conditional independence tests requires

an assumption on the dependence (e.g. linear as in Section 13.9.1.1 or nonlin-

ear). In the nonlinear case, it may be particularly expensive to have recourse

to conditional tests.

•Finite-sample error propagation: the sequential nature of the constraint-based

algorithms suﬀers from an error propagation (false positive) which is hard to

monitor and control.

•Asymptotic results of correctness: if the conditional independence decisions

are correct in the large sample limit, the PC algorithm is guaranteed to con-

verge to the true Markov Equivalence Class in the large sample limit, assuming

i.i.d. samples and the Markov, Faithfulness, Suﬃciency assumptions.

•Curse of dimensionality: the exponential complexity nature of the algorithm

(Figure 13.27) makes those algorithms inadequate in large dimensional set-

tings (e.g. bioinformatics).

13.10 Beyond conditional independence

The importance of causal reasoning in large dimensional settings (notably bioinfor-

matics) paved the way for developing alternative strategies for performing causal

inference from data. We will limit here to sketch two main approaches: the use

13.10. BEYOND CONDITIONAL INDEPENDENCE 379



                

0 200 400 600 800 1000

1e+06 1e+20 1e+34

number of varia bles n

number of CI tests (log scale)

d=15

d=10

d=5

Figure 13.27: Constraint-based algorithms complexity: upper bound on the number

of conditional tests as a function of the problem dimension nand conditioning size

of causal feature selection strategies and the adoption of data-driven techniques to

deal with indistinguishable situations.

13.10.1 Causality and feature selection

Feature selection and causal inference are related by the notion of Markov blanket

(MB) [182]. A Markov Blanket (deﬁnition 8.4) is the smallest set of strongly relevant

variables [118], i.e. variables containing information about the target which cannot

be obtained from any other variable (deﬁnition 8.1). A MB of a target contains

direct causes (parents), direct eﬀects (children) and spouses (nodes that share a

child with the target). Feature selection techniques, being able to discriminate

between causes and eﬀects, may then play a major role in causal modelling.

Tsamardinos et al. [182] proposed several algorithms to identify the Markov

Blanket of a target variable. Pellet and Elisseef [151] proposed an algorithm in

two steps: the ﬁrst builds an approximate structure of the causal graph using a

feature selection algorithm, and the second improves it by local adjustments and

orientation.

Most existing algorithms of causal feature selection [88] decompose feature selec-

tion and causality and rely on conditional independence test to orient arcs and detect

causal relationships. Nevertheless, not many approaches address large feature-to-

sample ratio settings. The author proposed a causal ﬁlter in [33] which integrates

into the cost function the relevance component and a causal component and ad-

dresses the issues of large feature-to-sample ratio settings. The algorithm, called

mIMR, is a causal extension of the mRMR algorithm (Section 12.8.2) and was

successfully applied to bioinformatics applications [32].

13.10.2 Beyond observational equivalence

Approaches like Bayesian network or mIMR rely on notions of (conditional) in-

dependence and faithfulness to detect causal patterns in the data. They cannot

deal with indistinguishable (or equivalent) conﬁgurations (Section 4.3.3) like the

two-variables setting and the completely connected triplet conﬁguration.

However, in what follows, we will see that indistinguishability does not prevent

380 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

Figure 13.28: An example of Markov Blanket in a Causal Bayesian Network. [90]

the existence of statistical algorithms able to reduce the uncertainty about the causal

pattern. Note that this is a recent result in contrast with the common belief that

it was impossible to learn a causal graph with two variables (as stated for instance

by Wasserman in [192]). Indeed this impossibility is limited to the Gaussian case

where the two variables are linked by a linear relation: in this case the joint Gaussian

distribution is fully determined by the mean and covariance matrix and there is no

way to reconstruct directionality form the distribution. However, asymmetry in the

dependence structure is a fundamental property that distinguishes causation from

association. It follows that, under some speciﬁc constraints (e.g. non Gaussian noise

or nonlinear relationships), some asymmetric features of the distribution (beyond

(in)dependence relations) might be informative about the causal structure.

13.10.2.1 Learning directionality in bivariate associations

A bivariate association between two variables x and y is necessarily due to one of

those reasons:

1. a causal inﬂuence going from x to y,

2. a causal inﬂuence going from y to x,

3. a (possibly unobserved) common cause (confounder) of x and y (note that

also time can play the role of confounder),

4. a (possibly unobserved) common eﬀect of x and y (inducing selection bias).

or any of their combinations.

In recent years several approaches addressed the two variable settings (e.g.

ANM [104] and IGCI [54]) by using asymmetric statistical properties (e.g. due

to non Gaussian noise or nonlinear mapping) to detect causal patterns. An addi-

tive noise model (ANM) dependence between two variables x and y satisﬁes the

13.10. BEYOND CONDITIONAL INDEPENDENCE 381

Figure 13.29: Illustration of asymmetric eﬀects due to the existence of a causal

dependency x→ y [104]

following relations:

y=f ( x) + w, w⊥⊥ x

where the noise term is independent from the input x. Consider a bivariate training

set related to two random variables x and y and suppose that either x→ y or y→ x

holds. The algorithm proposed in [104] is based on the idea that if x→ y is additive,

the independence between cause and noise does not hold for y→ x . The algorithm

steps are:

1. Regress both y on x and x on y.

2. Compute the residuals wy and wx for the two regressions.

3. Compute two independence tests (e.g. HSIC): wy ⊥⊥ x and wx ⊥⊥ y .

4. Compare the two independence test statistics ˆ

Cx→y and ˆ

Cy→x to determine

whether x→ y is more probable than y→ x or vice-versa.

A further important step in this direction has been made by the ChaLearn cause-

eﬀect pair challenge [87] where participants should learn from data the answer to

questions like Does altitude cause temperature or vice-versa? 7 . Hundreds of pairs

of real variables with known causal relationships from several domains (chemistry,

climatology, ecology, economy, engineering, epidemiology, genomics, medicine) were

made available to competitors for training their models. Real data were intermixed

with controls (pairs of independent variables and pairs of variables that are depen-

dent but not causally related) and semi-artiﬁcial cause-eﬀect pairs (real variables

mixed in various ways to produce a given outcome). The good accuracy obtained

by several competitors showed that learning strategies can address with success

(or at least signiﬁcantly better than random) indistinguishable conﬁgurations. This

competition opened the way to a recent research direction which poses causal infer-

ence as the problem of learning to classify probability distributions [154]. In those

approaches causal inference is typically based on two steps:

1. a featurisation of the observed dataset,

7See YouTube video "CauseEﬀectPairs" by Isabelle Guyon at https://www.youtube.com/

watch?v=pgoZ5lmRRvE.

382 CHAPTER 13. FROM PREDICTION TO CAUSAL KNOWLEDGE

2. the training of a binary classiﬁer to distinguish between causal directions.

Existing approaches diﬀer mainly in the featurisation step: for instance, [127] pro-

posed an approach based on kernel mean embeddings.

The author proposed instead a machine learning approach based on mutual in-

formation in [34, 35], called D2C. Given two variables, the D2C approach infers from

a number of asymmetric statistical features of the n -variate distribution the proba-

bility of the existence of a directed causal link. Causal inference is then addressed

as a supervised learning task where: i) inputs are asymmetric features describing

the probabilistic dependency and ii) output is a class denoting the existence of the

causal link.

Once suﬃcient training data are made available, conventional feature selection

algorithms and classiﬁers can be used to return a prediction. The rationale of those

approaches is that, though "correlation does not imply causation", it happens that

"causation creates dependence" (if faithfulness holds). In other terms, causality

leaves footprints (e.g. asymmetric descriptors) in the statistical distribution that

can be reused to reduce the uncertainty about the existence (or directionality) of a

causal relationship.

13.11 Concluding remarks

A ﬁnal chapter on causal inference in an introductory machine learning book could

appear as an unnecessary burden for an already exhausted reader. In fact, the

author deems that conventional machine learning books tend to create in the reader

a feeling of overconﬁdence about the power of predictive models. On the contrary,

in many crucial real-life applications, the real question at stake is causal (e.g. will

the lockdown policy have an impact on epidemics?) and not associative. Only an

understanding of causal relationships can support a reliable prediction of how a

system will behave once subject to intervention.

It is then important to stress the diﬀerence between the notion of probability

conditional on observation (Prob {y|x =x }) and probability conditional on ma-

nipulation (Prob {y | do (x ) = x } ). Those two quantities are diﬀerent and confuse

them may have disastrous consequences (e.g. overconﬁdence) in terms of decision

making. Though machine learning provides many solutions to the algorithmic esti-

mation of Prob {y|x =x} , the practitioner should be encouraged to think whether

Prob {y|x =x} is the quantity (s)he is really interested in. For instance, though a

trader would be more than happy in improving the estimation of Prob {y |x =x }

(where x stands for today stock price and ystands for tomorrow's), this quantity

would be of little use for an economist aiming to predict the impact of a Tobin-like

tax on the markets8.

Notions like the Simpson paradox, confounding, selection bias, latent variables,

counterfactuals should sound like a "forewarned is forearmed" message to machine

learning practitioners (and over-optimistic deep-learning evangelists :-). Accurate

prediction (whatever the depth of your network) does not imply accurate under-

standing either good decision making. So, the old data mining adage "We are

drowning in data and starving for knowledge" should instead read 'We are drown-

ing in associations and starving for causality".

8For a thorough analysis of the diﬀerence between predicting modelling and causal predictive

modelling, we refer the reader to [173]

Chapter 14

Conclusions

We have come to the end, almost. We will take a few words to remind you that

machine learning is not perfect and to cover a bit of ethical considerations. Then

we will conclude with some take-home messages and ﬁnal recommendations.

14.1 About ML limitations

From the dawn of the AI discipline, machine learning has been considered a key

component of autonomous intelligent agents. In recent years, though a full-ﬂedged

artiﬁcial intelligence does not seem within reach yet, machine learning found great

success thanks to its data-driven and assumption-free nature.

This book insisted on the fact that no modelling eﬀort may be completely

assumption-free. Assumptions (though often implicit and hard-coded in the al-

gorithms) are everywhere in the learning process, from problem formulation to data

collection, model generation and assessment! When such assumptions happen to

match with reality, the resulting method is successful: if it is not the case, the result

may be disappointing (see NFL theorem).

Another misjudgment about machine learning is to consider it as a reliable proxy

of human learning. Machine learning owes its success to the generic and eﬀective way

of transforming a learning problem into a (stochastic) optimisation one. Machines

do not think or learn like us: if this is the key to their success, this makes them

fragile, too. A large part of human rational decision making and understanding

cannot be reduced to the optimisation of a cost function.

This has been put into evidence by a recent trend taking a critical attitude

about the machine learning approach (e.g. the i.i.d. assumption) and its limits.

For instance, research on adversarial learning aims to show that the limited training

and validation set may induce very optimistic expectations about the generalisation

of learners. Recent research showed that automatic learners, which appear to be

accurate emulators of human knowledge (e.g. in terms of classiﬁcation accuracy),

may be easily fooled once required to work in speciﬁc situations. A well-known

example (Figure 14.1) shows that deep learning classiﬁers, able to reach an almost

100% accuracy rate in recognising animal images, may return pitiful predictions,

when confronted with properly tweaked inputs [67]. Though this seems anecdotal,

such vulnerabilities in learning machines could be very dangerous in safety-critical

settings (e.g. self-driving cars).

Another interesting research direction is the study of the robustness of automat-

ically learned model in settings which are not identical to the one used for training,

e.g. because of nonstationarity and concept drift. This is particularly critical in

health problems where models returning high-quality predictions for a speciﬁc co-

383

384 CHAPTER 14. CONCLUSIONS

Figure 14.1: Bad generalisation in front of adversarial examples

hort (e.g. in a given hospital) miserably fail when tested on diﬀerent patients (e.g.

from another hospital). How to transfer learned models to close settings is then

another hot topic in recent learning research. In this context, causal interpretation

of data generation could play an important role in reducing the risk of drift and

increasing the model stability.

14.2 A bit of ethics

Last but not least, a word of ethics should not hurt in a book for computer scien-

tists. "Data-driven does not necessarily mean "objective. Machine learning models

predict what they have been trained to predict and their forecasts are only as good

as the data used for their training. In that sense, machine learning can reinforce hu-

man prejudices if trained on biased data sets derived from human decisions. Feeding

learners with biased data can have dangerous consequences [141]. In 2016 Twitter

chatbot Tay began uttering racist statements after a single interaction day. The

predictive justice software COMPAS, deciding whether a suspect should be incar-

cerated before trial or not, has been accused of being racially biased by an NGO.

In 2015, Google Photos identiﬁed two African American people as "gorillas".

Every ML practitioner should be aware that even models developed with the

best of intentions may exhibit discriminatory biases, perpetuate inequality, or per-

form less well for historically disadvantaged groups 1. Recent eﬀorts in modelling

and introducing fairness in machine learning (most of the time based on causal

considerations) are then more than welcome.

At the same time, the problem goes beyond the scientiﬁc and technical realm

and involves human responsibility. Automating a task is a responsible decision-

making act which implies encoding (implicitly or explicitly) ethical priorities in an

autonomous agent. A self-driving car that decides to brake (or not to brake) is

somewhat trading the cost of human life vs. the cost of an over-conservative action.

The choice of entrusting to machines tasks that could have an impact on human

security or human sensibility should never exempt humans (from the programmer

to the decision maker) from legal and moral responsibilities of probable errors.

To conclude, the ethical dilemma of ML may be summarized by the contrapo-

sition of the two citations at the beginning of this book: on the one hand, any

machine learning endeavour harbours the ambition (or the illusion) of catching the

1see also the presentation https://tinyurl.com/y4ld3ohz

14.3. TAKE-HOME NOTIONS 385

essence of reality with numbers or quantitative tools. On the other hand, "not

everything that counts" (notably ethics) can be counted or easily translated into

numerical terms.

14.3 Take-home notions

Quoting Einstein "the supreme goal of all theory is to make the irreducible basic

elements as simple and as few as possible without having to surrender the adequate

representation of a single datum of experience". This sentence catches probably

the primary take-home concept in machine learning: trade-oﬀ, notably the trade-

oﬀ between bias and variance, underﬁtting and overﬁtting, parametric and non-

parametric, false positive and false negative, type I and type II error,...(please add).

Other bulk notions the author would like you to remember (or revise) are:

•information theory: it is a powerful language to talk about stochastic depen-

dence,

•estimators: do not forget they are random variables with their own sampling

distribution, and, even if very (very) good, they may be wrong sometimes2,

•conditional probability: supervised learning is all about estimating it,

•conditional (in)dependence and its non-monotonicity: mastering the complex-

ity (and beauty) of high dimensionality goes that way.

14.4 Recommendations

We would like then to end this manuscript not by selling you a unique and superior

way of proceeding in front of data but by proposing some golden rules for anyone

who would like to adventure in the world of statistical modelling and data analysis:

•However complex is your learning algorithm (either adaptive or deep or pre-

ferred by GAFA), do not forget it is an estimator, and as such, it makes

assumptions (often implicitly). Each approach has its own assumptions! Be

aware of them before using one.

•Simpler things ﬁrst! According to Wasserman [192], using fancy tools like

neural nets,...without understanding basic statistics is like doing brain surgery

before knowing how to use a band-aid.

•Reality is probably almost always nonlinear but a massive amount of (theo-

retical and algorithmic) results exists only for linear methods.

•Expert knowledge MATTERS... But data too :-)

•It is better to be conﬁdent with a number of alternative techniques (preferably

linear and nonlinear) and use them in parallel on the same task.

•Resampling and combining are at the forefront of the data analysis techniques.

Do not forget to test them when you have a data analysis problem.

•Do not be religious about learning/modelling techniques. The best learning

algorithm does NOT exist.

2...even the divine Roberto Baggio missed a penalty in the 1994 FIFA World Cup Final against

Brazil :-(

386 CHAPTER 14. CONCLUSIONS

•Statistical dependency does not imply causality, though it may shed some

light on it.

and the best motto for a machine learner:

Once you stop learning, you start dying (Albert Einstein).

Appendix A

Unsupervised learning

A.1 Probability density estimation

The probability density estimation is the problem of inferring a probability density function

pz , given a ﬁnite number of data points { z1 , z2 ,...,z N}drawn from that density function.

We distinguish three alternative approaches to density estimation:

Parametric. This approach assumes a parametric model of the unknown density prob-

ability. The parameters are estimated by ﬁtting the parametric function to the

observed dataset. This approach has been extensively discussed in Chapter 5.

Nonparametric. This approach does not assume any a priori form of the density model.

The form of the density is entirely determined by the data and the number of

parameters grows with the size of the dataset.

Semi-parametric. In this approach the number of parameter is not ﬁxed a priori but is

independent of the size of the dataset.

A.1.1 Nonparametric density estimation

The term nonparametric is used to describe probability density functions whose functional

form is not speciﬁed in advance, but is dependent on data [162, 144].

Let us consider a random variable z with density probability pz (z ) and a region R

deﬁned on the zspace. The probability that a value zdrawn according to pz (z ) falls

inside Ris

PR = prob{z ∈R } =Z R

pz ( z) dz (A.1.1)

Let us deﬁne with k the random variable which represents the number of points which

falls within R , after we have drawn N points from pz (z ) independently.

From (C.1.1) we have that its probability distribution is

pk ( k) = N!

k!( N− k)! P k

R(1 − P R) ( N−k) (A.1.2)

Moreover, the random variable k /N satisﬁes

E[k /n] = PR(A.1.3)

and

Var[k /N ] = E [(k /N − PR )2 ] = P R (1 − P R )

N(A.1.4)

Since according to (A.1.4), the variance of k /N converges to zero as N → ∞, it is reason-

able to expect that the fraction k/N return a good estimate of the probability PR

PR ∼

N(A.1.5)

387

388 APPENDIX A. UNSUPERVISED LEARNING

At the same time, if we assume that pz (z ) is continuous and does not vary appreciably

over R , we can approximate PRwith:

PR =Z R

p( z) dz ∼

=p( z) V(A.1.6)

with V volume of R . From (A.1.5) and (A.1.6) it follows that for values of z inside R

p( z)∼

NV (A.1.7)

In order for (A.1.5) to hold it is required to have a large R . This implies a sharply peaked

pz ( z). In order for (A.1.6) to hold it is required to have a small R. This ensures a pz ( z)

constant in R . These are two clashing requirements. We deduce that it is necessary to

ﬁnd an optimal trade-oﬀ for R in order to guarantee a reliable estimation of pz (z ). This

issue is common to all nonparametric approaches to density estimation.

In particular, we will introduce two of them

Kernel-based. This approach ﬁxes R and searches for the optimal number of points k.

k-Nearest Neighbor (k-NN). This approach ﬁxes the value for kand searches for the

optimal R.

The two approaches are discussed in detail in the following sections.

A.1.1.1 Kernel-based methods

Consider a random vector zof dimension [n× 1] and suppose we take an hypercube region

Rwith sides of length Bcentered on the point z. The volume of Ris

V= Bn

Let us now deﬁne a kernel function (or Parzen window) K (u ) as

K( u) =  1 if | u j | <1/ 2j = 1 ,...,n

0 else (A.1.8)

where uj is the j th component of the vector u . It follows that the quantity

K z−zi

B

is equal to unity if zi is inside the hypercube centered at z with side B.

Therefore, given a set of N points, the number of points falling inside Ris given by

i=1

K z−zi

B (A.1.9)

From (A.1.7) and (A.1.9) it is possible to deﬁne the kernel-based estimate of the prob-

ability density for the kernel (A.1.8) as

ˆ p( z) = P N

i=1 K z− z i

B

NBn (A.1.10)

Note that the estimate (A.1.10) is discontinuous over the z-space. In order to smooth

it we may choose alternative kernel functions, as the Gaussian kernel. The kernel-based

method is a traditional approach to density estimation. However, two are the most relevant

shortcomings of this approach:

1. it returns a biased estimator [25],

2. it requires the memorisation of the whole set of observations. As a consequence the

estimation is very slow if there is an high number of data.

A.1. PROBABILITY DENSITY ESTIMATION 389

A.1.1.2 k-Nearest Neighbors methods

Consider an hyper sphere Rcentered at a point z, and let us grow it until it contains

a number of kpoints. Using Eq. (A.1.7) we can derive the k-Nearest Neighbor (k-NN)

density estimate

ˆ pz ( z) = k

NV (A.1.11)

where k is the value of a ﬁxed a priori parameter, Nis the number of available observations

and V is the volume of the hyper sphere.

Like kernel-based methods, k-NN is a state-of-the-art technique in density estimation.

However it features two main shortcomings

1. the quantity (A.1.11) is not properly a probability, since its integral over the whole

Z space is not equal to one but diverges

2. as in the kernel method, it requires the storage of the whole dataset.

A.1.2 Semi-parametric density estimation

In semi-parametric techniques the size of the model does not grow with the size of the data

but with the complexity of the problem. As a consequence, the procedure for deﬁning the

structure of the model is more complex than in the approaches previously seen.

A.1.2.1 Mixture models

The unknown density function is represented as a linear superposition of mbasis functions.

The distribution is called called mixture model and has the form

pz ( z) =

j=1

p( z| j) π( j) (A.1.12)

where m is a parameter of the model and typically m << N . The coeﬃcients π (j ) are

called mixing coeﬃcients and satisfy the following constraints

j=1

π( j) = 1 0 ≤ π( j)≤ 1 (A.1.13)

The quantity π (j ) is typically interpreted as the prior probability that a data point be

generated by the j th component of the mixture. According to Bayes' theorem, the corre-

sponding posterior probabilities is

p( j| z) = p(z| j )π (j)

p( z)= p(z| j )π (j)

j=1 p(z| j )π(j )(A.1.14)

Given a data point z , the quantity p (j| z ) represents the probability that the component j

had been responsible for generating z.

An important property of mixture models is that they can approximate any continuous

density with arbitrary accuracy provided the model has a suﬃcient number of components

and provided the parameters of the model are tuned correctly.

Let us consider a Gaussian mixture model with components p (z| j )∼N (µj , σ2

j) and

suppose that a set of Nobservations is available. Once ﬁxed the number of basis functions,

the parameters to be estimated from data are the mixing coeﬃcients π (j ), and the terms

µj and σj .

The procedure of maximum likelihood estimation of a mixture model is not simple,

due to existence of local minima and singular solutions. Standard nonlinear optimisation

techniques can be employed, once the gradients of the log-likelihood with respect to the

parameters is given. However, there exist algorithms which avoid the complexity of a

nonlinear estimation procedure. One of them is the EM algorithm, which will be introduced

in the following section.

390 APPENDIX A. UNSUPERVISED LEARNING

A.1.2.2 The EM algorithm

The expectation-maximisation or EM algorithm [57] is a simple and practical method for

estimating the mixture parameters avoiding complex nonlinear optimisation algorithm.

The assumption of the EM algorithm is that the available dataset is incomplete. This

incompleteness can either be due to some missing measurements or because some imaginary

data are introduced to simplify the mathematical form of the likelihood function.

The second situation is assumed to hold in the case of mixture models. The goal of

the EM algorithm is then to maximize the likelihood of the parameters of a mixture model

assuming that some data is missing in the available dataset.

The algorithm has an iterative form in which each iteration consists of two steps: an

expectation calculation (E step) and a maximisation (the M step). It has been shown in

literature that the iteration of EM estimates converge to a local maximum of the likelihood

of the incomplete data.

Assume that there exists a statistical model of our dataset DN and that it is parametrized

by a real vector θ. Assume also that further data, denoted by Ξ, exist but are not observ-

able. The quantity ∆N is used to denote the whole dataset, containing both the observed

and unobserved data, and is usually referred to as the complete data.

Let us denote by lcomp (θ ) the log likelihood of the parameter θgiven the complete data.

This is a random variable because the values of Ξ are not known. Hence, it is possible for a

given value θ (τ) of the parameter vector to compute the expected value of lcomp (θ (τ) ). This

gives a deterministic function of the current value of the parameter, denoted by Q (θ (τ) ),

that can be considered as an approximation to the real value of l , called the incomplete

likelihood. The maximisation step is expected to ﬁnd the parameter value θ (τ +1) which

maximize Q . The EM procedure in detail is the following:

1. Make an initial estimate θ (0) of the parameter vector.

2. The log likelihood l comp (θ (τ) ) of the parameters θ (τ) with respect to the complete

data ∆N is calculated. This is a random function of the unknown dataset Ξ.

3. The E-step: the expectation Q (θ (τ) ) of l comp (θ (τ) ) is calculated.

4. The M-step: a new estimate of the parameters is found by the maximisation

θ(τ +1) = arg max

θQ(θ ) (A.1.15)

The theoretical justiﬁcation comes from the following result proved in [57]: for a sequence

θ(τ) generated by the EM algorithm it is always true that for the incomplete likelihood

l( θ(τ+1) )≥ l( θ(τ) ) (A.1.16)

Hence the EM algorithm is guaranteed to converge to a local maximum of the incomplete

likelihood.

A.1.2.3 The EM algorithm for the mixture model

In the mixture model estimation problem the problem of determining the parameters (i.e.

the mixing coeﬃcients and the parameters of the density p (z| j ) in Eq. (A.1.12)) would be

straightforward if we knew which component jwas responsible for generating each data

point in the dataset. We therefore consider a hypothetical complete dataset in which each

data point is labeled by the component which generated it. Thus, for each point zi we

introduce m indicator random variables ζ ij ,j = 1,...,m , such that

ζij =( 1 if z i is generated by the j th basis

0 otherwise (A.1.17)

Let ∆Nbe the extension of the dataset DN , i.e. it represents the complete dataset,

including the unobservable ζij . The probability distribution for each (zi , ζij ) is either zero

or p (zi | j ). If we let ζi represent the set {ζi1 ,ζi2 ,..., ζ im }then

pζ i ( ζi ) = π( j0 ) where j0 is such that ζij 0 = 1 (A.1.18)

A.1. PROBABILITY DENSITY ESTIMATION 391

p( zi , ζi ) = p( ζi ) p( zi | j0 ) = π( j0 ) p( zi | j0 ) =

j=1

[π (j)p (zi |j )]ζ ij (A.1.19)

Thus the complete log likelihood is given by

lcomp (θ ) = ln Lcomp (θ ) = ln

i=1

j=1

[π (j)p (zi |j )]ζ ij (A.1.20)

i=1

j=1

[π (j )p(zi |j )]ζ ij (A.1.21)

i=1

j=1

ζij {ln π (j ) + ln p (zi |j) } (A.1.22)

where the vector θincludes the mixing coeﬃcients and the parameters of the density p (z| j )

in Eq. (A.1.12).

Introducing the terms ζ ij the logarithm can be brought inside the summation term.

The cost of this algebraic simpliﬁcation is that we do not know the values of the ζ ij for

the training data. At this point the EM algorithm can be used. For a value θ (τ) of the

parameters the E-step is carried out:

Q( θ (τ) ) = E[ lcomp ( θ (τ) ) = E [

i=1

j=1

ζij {ln π (j ) + ln p (zi |j) } ] (A.1.23)

i=1

j=1

E[ζij ]{ ln π( j) + ln p( zi | j)} (A.1.24)

Since

E[ζ ij ] = P( ζij = 1| zi ) = p(zi | ζij ) P(ζij )

p( zi )= p(zi | j )π (j )

p( zi )= p(j| zi ) (A.1.25)

from Eq. (A.1.14) and (A.1.18) we have

Q( θ (τ) ) =

i=1

j=1

p( j| zi ){ ln π( j) + ln p( zi | j)} (A.1.26)

The M-step maximizes Qwith respect to the whole set of parameters θbut it is known

that this can be done individually for each parameter, if we consider a Gaussian mixture

model

p( z| j) = 1

(2πσj2 ) n/2 exp  − (z− µj ) 2

2σj2  (A.1.27)

In this case we have:

Q( θ (τ) ) =

i=1

j=1

p( j| zi ){ ln π( j) + ln p( zi | j)}(A.1.28)

i=1

j=1

p( j| zi ) ln π( j)− nln σj (τ) −( z i − µ j (τ) ) 2

2(σj (τ) )2  + constant (A.1.29)

We can now perform the maximisation (A.1.15). For the parameters µj and σj the max-

imisation is straightforward:

µ(τ+1)

j=P N

i=1 p(j| z i )z i

i=1 p(j| z i )(A.1.30)

σ (τ+1)

j 2 =1

nPN

i=1 p(j| z i )(z i −µ ( τ+1)

j) 2

i=1 p(j| z i )(A.1.31)

392 APPENDIX A. UNSUPERVISED LEARNING

For the mixing parameters the procedure is more complex [25] and returns:

π( j)(τ +1) =1

i=1

p( j| zi ) (A.1.32)

where p (j| zi ) is computed as in (A.1.25).

A.2 K-means clustering

The K-means algorithm partitions a collection of N vectors xi ,i = 1,...,N , into Kgroups

Gk , k = 1 ,...,K , and ﬁnds a cluster center in each group such that a cost function of

dissimilarity (or distance) measure is minimized. When the Euclidean distance is chosen

as the dissimilarity measure between a vector x in the k th group and the corresponding

cluster center ck , the cost function can be deﬁned by

k=1

Jk =

k=1 X

x∈ Gk

d( x, ck ) (A.2.33)

where Jk is the cost function within group k and d is a generic distance function

d( x, ck ) = ( x− ck )T M( x− ck ) (A.2.34)

where M is the distance matrix. The partitioned groups are typically deﬁned by a mem-

bership [K× N ] matrix U, where the element u ki is 1 if the i th data point xi belongs to

group k , and 0 otherwise. The matrix Usatisﬁes the following conditions:

k=1

uki = 1 ∀ i= 1 ,...,N

k=1

i=1

uki = N

(A.2.35)

Once the cluster centers ck are ﬁxed, the terms u ki which minimize Eq. (A.2.33) are:

uki =( 1 if d(xi , ck )≤ d( xi , cj ), for each j 6= k

0 otherwise (A.2.36)

This means that xi belongs to the group k if ck is the closest center among all centers.

Once the terms u ki are ﬁxed, the optimal center ck that minimizes Eq. (A.2.33) is the

mean of all vectors in the kth group:

ck =1

|Gk |X

x∈ Gk

x(A.2.37)

where | Gk | is the size of Gk .

The K-means algorithm determines iteratively the cluster centers ck and the member-

ship matrix Uusing the following procedure:

1. Initialize the cluster centers ck , typically by randomly selecting K points among all

data points.

2. Evaluate the membership matrix U through Eq. (A.2.36).

3. Compute the cost (A.2.33). If it is below a certain tolerance value or if the improve-

ment is not signiﬁcant, stop and return the centers and the groups.

4. Update the cluster centers according to Eq. (A.2.37). Go to step 2.

Some ﬁnal remarks should be made on the K− means algorithm. As many other clustering

algorithms, this technique is iterative and no guarantee of convergence to an optimum

solution can be found. Also, the ﬁnal performance is quite sensitive to the initial position

of the cluster centers and to the number Kof clusters, typically ﬁxed a priori by the

designer.

Appendix B

Linear algebra notions

Linear algebra, the science of vector spaces, plays a major role in machine learning, where

data are represented in a vectorial form. Though the readers are supposed to have attended

numerical analysis classes, here we will remind some basic notions of linear algebra. For a

more extensive presentation of linear algebra and its links with machine learning, we refer

the reader to recent references like [2, 56].

B.1 Rank of a matrix

Let us consider a [N, n] matrix. Many deﬁnitions exist for the rank of a matrix: here we

limit to consider the rank of Xas the maximal number of linearly independent columns

of X . Since the rank of a [N, n ] matrix is at most min{ N , n} , a matrix is full-rank if its

rank is min{ N, n} . A matrix which is not full-rank is also called rank-deﬁcient.

B.2 Inner product

In linear algebra, the dot product, also known as the scalar or inner product, is an operation

which takes two vectors over the real numbers R and returns a real-valued scalar quantity.

It is the standard inner product of the orthonormal Euclidean space. The dot product of

two [n, 1] vectors x = [x1 , x2 ,...,x n] Tand y = [y 1, y2,...,y n] Tis deﬁned as:

hx, y i=

j=1

xj yj = xT y (B.2.1)

The dot product underlies the deﬁnition of the following quantities:

•the Euclidean norm of a vector x:

kx k= p h x, xi, (B.2.2)

also known as the L2 norm,

•the Euclidean distance of two [n, 1] vectors x1 and x2

kx1 −x2 k=p h x1 −x2 , x1 −x2 i, (B.2.3)

•the angle ω between two vectors x1 and x2 which satisﬁes the relation

−1 ≤hx1 x2 i

kx1 kkx2 k = cos(ω)≤ 1 , (B.2.4)

•the projection of a vector x1 onto a direction x2

πx 2 ( x1 ) = h x 1 , x 2 i

kx2 k2 x 2 = x 2 x T

kx2 k x 1 (B.2.5)

where the [n, n ] matrix P = x2 x T

2is called the projection matrix.

393

394 APPENDIX B. LINEAR ALGEBRA NOTIONS

In more qualitative terms, the notion of inner product allows the introduction of a

similarity score between vectors. In this sense, the least similar vectors are two orthogonal

vectors, i.e. two vectors x and y such that h x, yi = 0 and ω = π/2.

Note also that the following relation holds:

xxT y= h x, yix

B.3 Diagonalisation

A [N, N ] matrix X is diagonalisable if it exists an invertible matrix Psuch that

X= P DP −1 . (B.3.6)

A symmetric matrix can always be diagonalised and the diagonal entries of Dare its

eigenvalue.

B.4 QR decomposition

Let us consider a [N, n] matrix X with N≥ n and n linearly independent columns. By

Gram-Schmidt orthogonalisation [2] it is possible to write

X= QR (B.4.7)

where Q is a [N, n] matrix with n orthonormal columns qj (i.e. q T

jq j= 1 and q T

jq k= 0 if

j6= k) and Ris a [ n, n] upper-triangular matrix. Since QT Q= In the pseudo-inverse of

Xcan be written as

X† = ( XTX)−1 XT = ( RT QTQR) −1 RT QT = R−1 QT (B.4.8)

If X is rank-deﬁcient (i.e. only n0 < n < N columns of X are linearly independent) it

is possible to perform the generalised QR decomposition

X= QR

where Q is [N, n0 ] and R is a [n0 , n ] rectangular upper-triangular matrix with n0 < n . Since

Ris of full row rank, the matrix RRT is invertible and the pseudo-inverse of Xcan be

written as

X† = RT ( RRT ) −1 QT (B.4.9)

also known as the Moore-Penrose pseudo-inverse.

B.5 Singular Value Decomposition

Let us consider a [N, n] matrix X with N≥ n : such matrix can always be factorised into

the product of three matrices

X= UDV T (B.5.10)

where U is a [N, N ] matrix with orthonormal columns (i.e. UT U =IN ), D is a [N , n]

diagonal matrix whose diagonal entries d ii ≥ 0 are called the singular values and V is a

[n, n ] matrix with orthonormal columns.

It can be shown that the N columns of U (also called the left singular vectors) are the

Neigenvectors of the [ N, N ] symmetric matrix XX T and the n columns of V (also called

the right singular vectors) are the neigenvectors of the [n, n] symmetric matrix XTX.

The non-zero singular values are the square-roots of the non-zero eigenvalues of XT X and

of the non-zero eigenvalues of XX T . This is made evident by the link between SVD and

diagonalisation of XT X :

XT X= ( UDV T )T( U DV T ) = V DT UT UDV T = V DT DV T

B.6. CHAIN RULES OF DIFFERENTIAL CALCULUS 395

The SVD of a matrix X of rank r can be written also as:

j=1

dii ui vT

where uj is the j th column of U and vj is the j th column of V.

If in the decomposition above we stop at the order r0 < r , we obtain a low-rank

approximation of X:

X0 =

j=1

dii ui v T

Another common SVD decomposition is the economy (or reduced) SVD:

X= UDV T (B.5.11)

where k = min{ N, n} ,U is [N , k] with orthonormal columns, Dis a square [k, k] matrix

and V is a [n, k] matrix with orthonormal columns.

SVD plays an important role in determining the ill-conditioning of a square matrix,

i.e. how much the matrix is close to be singular. The condition number of a matrix is the

ratio of its largest singular value to its smallest singular value. The larger this number

(which is ≥ 1) the larger is the ill-conditioning of the matrix.

Note also that if Xis a symmetric matrix, the SVD decomposition returns the diago-

nalisation (B.3.6).

B.6 Chain rules of diﬀerential calculus

Let J be the scalar function of α∈ R:

J( α) = f( g( h( α)))

where f, g , h :R→ R are scalar functions. Then the univariate chain rule is

dα = dJ

dα

Let us consider the function J :R→ R

J= f( g1 ( α) , g2 ( α) ,...,g n(α))

between α∈R and the scalar J , where gj :R→ R, j = 1,...,n . Then the multivariate

chain rule returns the scalar gradient

dα =

j=1

∂J

∂gj  dg j

dα 

Let J :Rn →Rm be the mapping between an input vector α∈Rn and an output

vector of size m . The associated Jacobian matrix is the [m, n ] matrix

∇α J=h ∂J (α )

∂α1

∂J ( α )

∂α2 . . . , ∂J (α)

∂αn i= 





∂J1 ( α )

∂α1

∂J1 ( α )

∂α2 . . . ∂J 1 ( α )

∂αn

∂Jm ( α )

∂α1

∂Jm ( α )

∂α2 . . . ∂J m ( α )

∂αn







In the most generic case, suppose that J =Rn → Rm ,α∈Rn , and

J= Fk (Fk−1 ( . . . F1 ( α )))

where Fi :Rn i →Rn i+1 , n1 =n and nk+1 = m . Then the vectored chain rule [2] is

∂J

∂α

|{z}

[m,n]

=∂Fk

∂Fk−1

| {z }

[m,nk ]

∂Fk−1

∂Fk−2

| {z }

[nk ,nk−1 ]

. . . ∂F 1

∂α

|{z}

[n2 ,n]

396 APPENDIX B. LINEAR ALGEBRA NOTIONS

B.7 Quadratic norm

Consider the quadratic norm

J( x) = k Ax + bk 2

where J :Rn → R ,A is a [N, n] matrix, xis a [n, 1] vector and bis a [N, 1] vector. It can

be written in the matrix form

J( x) = xTAT Ax + 2 bT Ax + bT b

The ﬁrst derivative of J with respect to xis the [n, 1] vector

∂J (x)

∂x = 2 A T ( Ax +b )

and the second derivative is the [n, n] matrix

∂2 J( x)

∂x∂xT = 2 A T A

B.8 Quadratic programming

Quadratic programming is the resolution procedure of continuous optimisation problems

with a squared objective function, for instance

b∗ = J( b) = arg min

bb T Db

where b is a [n, 1] vector and Dis a [n, n] matrix. For instance if n= 2 and Ais a

diagonal matrix, J (b ) = b 2

1+b 2

2and has a single global minimum in [0, 0]. If the solution

is submitted to no constraints, the problem is called unconstrained. If D is a positive

(negative) semideﬁnite matrix the function J is convex (concave).

In machine learning the most common quadratic programming task is strictly convex

since it derives from the least-squares formulation where D is a deﬁnite positive matrix.

The general form of an unconstrained strictly convex quadratic objective function is

b∗ = J( b) = arg min

bb T Db −d T b+k = arg min

bb T Db − d T b(B.8.12)

where D is deﬁnite positive, dis a [n, 1] vector and kis a scalar (which has no impact on

the minimisation problem).

The constrained version has a set of linear inequality constraints in the form

AT b≥ b 0

where A is a [n, c] matrix deﬁning the cconstraints under which we want to minimise the

Jfunction.

The R package quadprog provides the implementation solve.QP of a method to solve

a strictly convex constrained quadratic programming task.

B.9 The matrix inversion formula

Let us consider the four matrices F , G ,H and K and the matrix F + GHK . Assume that

the inverses of the matrices F ,G and (F + GHK ) exist. Then

(F +GHK )−1 =F−1 −F−1 G H−1 +K F −1 G −1 KF −1 (B.9.13)

Consider the case where F is a [n× n ] square nonsingular matrix, G =z where z is a

[n× 1] vector, K =zT and H = 1. Then the formula simpliﬁes to

(F + zz T ) −1 =F −1 −F −1 zz T F −1

1 + zT F z

where the denominator in the right hand term is a scalar.

B.9. THE MATRIX INVERSION FORMULA 397

If X and Z are two [N, p] matrices, from (B.9.13) it can be shown the push-through

identity [2]

XT ( IN + ZX T ) −1 = ( Ip + XT Z)−1 XT (B.9.14)

Then for any [N, p ] matrix Xand scalar λ > 0

XT ( λIN +XX T ) −1 = (λIp + XTX)−1 XT (B.9.15)

398 APPENDIX B. LINEAR ALGEBRA NOTIONS

Appendix C

Probability and statistical

notions

C.1 Common univariate discrete probability func-

tions

C.1.1 The Bernoulli trial

ABernoulli trial is a random experiment with two possible outcomes, often called "suc-

cess" and "failure". The probability of success is denoted by pand the probability of

failure by (1 −p ). A Bernoulli random variable z is a binary discrete r.v. associated with

the Bernoulli trial. It takes z= 0 with probability (1 −p ) and z= 1 with probability p.

The probability function of zcan be written in the form

Prob {z =z} =Pz (z ) = pz (1 −p )1−z , z = 0,1

Note that E [z ] = p and Var [z ] = p (1 −p ).

C.1.2 The Binomial probability function

Abinomial random variable represents the number of successes z in a ﬁxed number N of

independent Bernoulli trials with the same probability p of success for each trial. A typical

example is the number z of heads in N tosses of a coin.

The probability function of z∼ Bin(N, p ) is given by

Prob {z =z} =Pz (z ) = N

z! p z (1 −p)N−z , z = 0 , 1 ,...,N (C.1.1)

The mean of the probability function is µ = Np . Note that:

•the Bernoulli probability function is a special case (N= 1) of the binomial function,

•for small p, the probability of having at least 1 success in Ntrials is proportional to

N, as long as Np is small,

•if z1 ∼Bin(N1 , p ) and z2 ∼Bin(N1 , p ) are independent then z1 +z2 ∼Bin(N1 +

N2 , p)

The Binomial distribution returns then the probability of zsuccesses out of N draws

with replacement. The probability of z successes out of N draws without replacement

from a population of size P that contains k terms associated to success is returned by the

hypergeometric distribution:

Prob {z =z} =  k

z P−k

N− z

P

N.(C.1.2)

399

400 APPENDIX C. PROBABILISTIC NOTIONS

C.2 Common univariate continuous distributions

C.2.1 Uniform distribution

A random variable zis said to be uniformly distributed on the interval (a, b) (written as

z∼ U ( a, b)) if its probability density function is given by

p( z) = ( 1

b− aif a < z < b

0,otherwise

It can be shown that the skewness of a continuous random variable which is uniformly

distributed is equal to 0.

Exercise

Show that the variance of U ( a, b) is equal to 1

12 (b− a) 2 .

•

C.2.2 The chi-squared distribution

It describes the distribution of squared normal r.v.s. An r.v. z has a χ 2

Ndistribution if

z= x2

1+· ·· +x 2

where N∈N and x1 ,x2 ,. . . ,xN are i.i.d. standard normal random variables N (0, 1). The

distribution is called a chi-squared distribution with Ndegrees of freedom. Note also that

•The probability distribution is a gamma distribution with parameters ( 1

2N, 1

2).

•E[z ] = Nand Var [z ] = 2 N.

The χ 2

Ndensity and distribution function for N= 10 are plotted in Figure C.1 (R script

chisq.R in the package gbcode).

C.2.3 Student's t-distribution

It describes the distribution of the ratio of normal and χsquared r.v.s. If x ∼ N (0, 1) and

y∼χ2

Nare independent then the Student's t -distribution with N degrees of freedom is

the distribution of the r.v.

z= x

py /N (C.2.3)

We denote this with z ∼ TN . Note that E [z ] = 0 and V ar [z ] = N/ (N− 2) if N > 2.

The Student density and distribution function for N= 10 are plotted in Figure C.2

by means of the script stu.R in the package gbcode .

C.2. COMMON UNIVARIATE CONTINUOUS DISTRIBUTIONS 401

Figure C.1: χ 2

Nprobability distribution (N = 10)

Figure C.2: Student probability distribution (N = 10)

402 APPENDIX C. PROBABILISTIC NOTIONS

Figure C.3: F probability distribution (N = 10)

C.2.4 F-distribution

It describes the distribution of the ratio of χ squared r.v.s. Let x∼ χ2

Mand y∼χ2

Nbe

two independent r.v.. An r.v. z has a F-distribution with M and N degrees of freedom

(written as z∼F M,N ) if

z= x/M

y/N (C.2.4)

Note that if z∼F M,N then 1/z∼ F N,M , while if z ∼ TN then z2 ∼F1,N . The F-density

and distribution function are plotted in Figure C.3 by means of the script f.R in the

package gbcode .

C.3 Common statistical hypothesis tests

C.3.1 χ2 -test: single sample and two-sided

Consider a random sample from N ( µ, σ2 ) with µ known. Let

H: σ2 = σ2

0;¯

H: σ2 6= σ2

Let c

SS = P i ( zi − µ )2 . From Section 5.7 it follows that if H is true then c

SS/σ2

0∼χ 2

(Section C.2.2 )

The level α χ2 -test rejects H if c

SS/σ2

0< a 1or c

SS/σ2

0> a 2where

Prob ( c

σ2

< a1 ) + Prob ( c

σ2

> a2 ) =α

A slight modiﬁcation is necessary if µis unknown. In this case you must replace µ

with ˆ µin the quantity c

SS and use a χ 2

N−1 distribution.

C.3.2 t-test: two samples, two sided

Consider two r.v.s x ∼ N (µ1 , σ2 ) and y ∼ N (µ2 , σ 2 ) with the same variance. Let D x

N←x

and D y

M←ytwo independent sets of samples of size N and M , respectively..

We want to test H :µ1 =µ2 against ¯

H: µ1 6= µ2 .

Let

ˆ µx =PN

i=1 x i

N, c

SS x =

i=1

(xi − ˆ µx )2 ,ˆ µy =PM

i=1 y i

M, c

SS y =

i=1

(yi − ˆ µy )2

It can be shown that if His true then the statistic

t(DN ) = ˆ

µx − ˆ

µy

r 1

M+ 1

N c

SSx + c

SSy

M+ N−2∼ T M+N −2

C.4. TRANSFORMATION OF RANDOM VARIABLES AND VECTORS 403

It follows that the test of size α rejects Hif

|t( DN ) |> tα/2,M+ N −2

C.3.3 F-test: two samples, two sided

Consider a random sample { x1 ,...,x M} ← x ∼ N (µ 1, σ 2

1) and a random sample {y 1,...,y N} ←

y∼ N (µ2 , σ 2

2) with µ 1and µ 2unknown. Suppose we want to test

H: σ2

1=σ 2

2;¯

H: σ2

16=σ 2

Let us consider the statistic

f=ˆ

σ2

SS1 /( M− 1)

SS2 /( N− 1) ∼ σ 2

1χ 2

M−1/(M− 1)

σ2

2χ 2

N−1/(N− 1) = σ 2

σ2

FM−1 ,N−1

It can be shown that if His true, the ratio f has a F-distribution FM−1 ,N −1 (Section

C.2.4) The F-test rejects H if the ratio fis large, i.e. f > Fα,M −1 ,N −1 where

Prob {f > Fα,M−1 ,N −1 } = α

if f∼FM−1 ,N−1 .

C.4 Transformation of random variables and vec-

tors

Theorem 4.1 (Jensen's inequality) . Let x be a continuous r.v. and f a convex function.

Then, E[ f( x)] ≥f (E [ x ]) while if f is concave then E[ f( x)] ≤f (E [ x ])

Given a [n× 1] constant vector aand a random vector z of dimension [n× 1] with

expected value E [ z ] = µ and covariance matrix [z ] = Σ, then

E[ aT z] = aT µ, Var h aT zi = aT Σ a

Also if z ∼ N ( µ, Σ) then aT z ∼ N (aT µ, aT Σ a)

Given a [n× n ] constant matrix Aand a random vector zof dimension [n× 1] with

expected value E [ z ] = µ and covariance matrix Var[z ] = Σ, then

E[ Az] = Aµ, Var [Az ] = A Σ AT

R script

The relation above may be used to sample a [n, 1] random vector xwith covariance

Var [x ] = Σ, starting from the sampling of a [n, 1] random vector z with Var [z ] = In . If we

factorize Σ = AAT , then Var [x] = Var [Az ] = AInAT = Σ. In the script chol2cor.R , we

ﬁrst deﬁne the symmetric matrix Σ, then we sample N times the z vector in the dataset

DN and we multiply DN by A : it is possible to verify numerically that this is equivalent

to sample N times a vector xwith covariance Σ.

•

Theorem 4.2. Given a random vector z of dimension [n× 1] with expected value E[ z] = µ

and covariance matrix Var[ z] = σ2 I , for a generic matrix A of dimension [n× n ] the

following relation holds

E[zT Az] = σ2 tr( A) + µT Aµ (C.4.5)

where tr( A) is the trace of matrix A.

404 APPENDIX C. PROBABILISTIC NOTIONS

C.5 Correlation and covariance matrices

Given n r.v.s z1 ,...,zn the correlation matrix C is a symmetric positive-semideﬁnite [n, n]

matrix whose (i, j ) entry is the correlation coeﬃcient ρ (zi ,zj ) (Equation (3.6.68)).

The following relation exists between the covariance Σ of the nvariables and the

correlation matrix:

C= (diag(Σ))−1/ 2 Σ (diag(Σ))−1/ 2 ,Σ = 





σ1 . . . . . . 0

0σ2 . . . 0

0. . . .. . 0

0. . . . . . σn





C 





σ1 . . . . . . 0

0σ2 . . . 0

0. . . . . . 0

0. . . . . . σn







By using the formula above, the script corcov.R shows how it is possible to generate

a set of examples with predeﬁned pairwise correlation ¯ ρ.

C.6 Convergence of random variables

Let { zN } ,N = 1, 2,..., be a sequence of random variables and let z be another random

variable. Let FN ( · ) denote the distribution function of zN and Fz the distribution of z.

We introduce the following deﬁnitions:

Deﬁnition 6.1 (Convergence in probability). We say that

lim

N→∞z N =zin probability (C.6.6)

and we note zN

→zif for each ε > 0

lim

N→∞P{|z N −z| ≥ ε } = 0 (C.6.7)

Deﬁnition 6.2 (Convergence with probability one).We say that

lim

N→∞z N =zwith probability one (or almost surely) (C.6.8)

and we note zN

a.s.

→zif

P{ ω: lim

N→∞z N (ω ) = z (ω )} = 1 (C.6.9)

Deﬁnition 6.3 (Convergence in Lp ) . For a ﬁxed number p≥ 1 we say that

lim

N→∞z N =zin L p (C.6.10)

lim

N→∞E[|z N −z| p ] = 0 (C.6.11)

The following theorems hold:

Theorem 6.4. Convergence in Lp implies convergence in probability.

Theorem 6.5. Convergence with probability one implies convergence in probability.

Note however that convergence in probability does not imply convergence in L2 .

Deﬁnition 6.6 (Convergence in distribution) . The sequence zN converges in distribution

to z and we note z N

→zif

lim

N→∞F N (z) = F( z) (C.6.12)

for all z for which F is continuous.

It can be shown that

Theorem 6.7. Convergence with probability implies convergence in distribution.

Note however that convergence in distribution does not imply convergence in proba-

bility.

As a summary

a.s.

→zimplies zN

→z

C.7. THE CENTRAL LIMIT THEOREM 405

C.6.1 Example

Let z ∼ U (1, 2) and θ= 0. Consider the two estimators (stochastic processes) ˆ

θ(1)

Nand

θ(2)

Nfor N→ ∞ where

θ1

N= exp −zN, ˆ

θ2

N=( exp − N with probability 1 −1/N

1 with probability 1/N

For the ﬁrst estimator, all the tra jectories converge to θ(strongly consistent). For the

second process, the trajectory which does not converge has a probability decreasing to

zero for N → ∞ (weakly consistent).

C.7 The central limit theorem

Theorem 7.1. Assume that z1 , z2 ,. . . ,zN are i.i.d. random variables, discrete or contin-

uous, each having the same probability distribution with ﬁnite mean µ and ﬁnite variance

σ2 . As N→ ∞, the standardised random variable

(¯

z−µ) √ N

which is identical to ( S N − Nµ)

√Nσ

converges in distribution (Deﬁnition 6.6) to a r.v. having the standardized normal distri-

bution N (0,1).

This theorem, which holds regardless of the common distribution of zi , justiﬁes the

importance of the normal distribution, since many r.v. of interest are either sums or

averages. Think for example to the commute time of the example in Section 3.2 which can

be considered as the combined eﬀect of several causes.

An illustration of the theorem by simulation is obtained by running the R script

central.R.

C.8 The Chebyshev's inequality

Let z be a generic random variable, discrete or continuous, having a mean µand a variance

σ2 . The Chebyshev's inequality states that for any positive constant d

Prob {|z −µ | ≥ d} ≤ σ 2

d2 (C.8.13)

An illustration of the Chebyshev's inequality by simulation can be found in the R

script cheby.R.

Note that if we put zequal to the quantity in (3.10.89), then from (3.10.90) and

(C.8.13) we ﬁnd

Prob {| ¯

z−µ | ≥ d } ≤ σ 2

Nd2 (C.8.14)

i.e. the weak law of large numbers (Section 3.1.5). This law states that the average of a

large sample converges in probability to the mean of the distribution.

C.9 Empirical distribution properties

Let (5.2.2) be the empirical distribution of z obtained from a dataset DN . Note that, being

DN a random vector, the function ˆ

Fz ( ·) is random, too. The following two properties

(unbiasedness and consistency) are valid:

406 APPENDIX C. PROBABILISTIC NOTIONS

Theorem 9.1. For any ﬁxed z

ED N [ˆ

Fz (z )] = Fz (z ) (C.9.15)

Var h ˆ

Fz (z)i =F z (z)(1 − Fz (z ))

N(C.9.16)

Theorem 9.2 (Glivenko-Cantelli theorem).

sup

−∞<z< ∞| ˆ

Fz (z )−Fz (z)|N→∞

→0almost surely (C.9.17)

where the deﬁnition of almost sure convergence is in Appendix (Def. 6.2).

The two theoretical results can be simulated by running the R scripts cumdis 2.R and

cumdis 1.R, respectively.

C.10 Useful relations

Some relations

E[(z− µ)2 ] = σ2 = E[z2 − 2 µz+ µ2 ] = E[z2 ]− 2 µE[ z] + µ 2

=E [z2 ]− 2µµ +µ2 =E [z2 ]−µ2

For N = 2

E[(z1 + z2 )2 ] = E[z2

1] + E [z 2

2] + 2E[z 1z2]

= 2E [ z2 ] + 2µ 2

= 4µ2 + 2σ 2

For N = 3

E[(z1 +z2 +z3 )2 ] = E[z2

1] + E[z 2

2] + E [z 2

3] + 2E[z 1z2] + 2E[z 1z3] + 2E [z 2z3]

= 3E [z2 ] + 6µ2 = 9µ2 + 3σ 2

In general for N i.i.d. zi ,E [(z1 +z2 + · ·· + zN )2 ] = N2 µ2 + Nσ2 .

C.11 Minimum of expectation vs. expectation of

minimum

Theorem 11.1. Let us consider M random variables zm , m = 1 ,...,M . Then

E[min

mz m ]≤min

mE[z m ]

Proof. For each m deﬁne the r.v. xm =zm − minm zm . Now E [xm ]≥ 0 since zm ≥

minm zm . Then E [ zm ]−E [minm zm ]≥ 0. It follows that

∀m, E[min

mz m ]≤E[z m ]

then

E[min

mz m ]≤min

mE[z m ]

The diﬀerence minm E [ zm ]−E [minm zm ] quantiﬁes then the selection bias that occurs

in a selection (e.g. by minimisation) process that relies on observed data in a random

setting.

C.12. TAYLOR EXPANSION OF FUNCTION 407

C.12 Taylor expansion of function

Let J (· ) be a function with pdimensional argument of the form α = [α1 ,...,α p]. The

Taylor expansion of the function J (· ) about ¯ αcan be written as follows

J( α) = J(¯ α) +

j=1

(αi − ¯ αi ) ∂ J (α )

∂αj   α= ¯ α+

i=1

j=1

(αi − ¯ αi )(αj − ¯ αj )

∂2 J( α)

∂αi∂αj   α= ¯ α+. . .

which can be written in vector-form as follows:

J( α)≈ J(¯ α) + ( α−¯ α)T ∇ J( ¯ α) + ( α−¯ α)T H( ¯ α)( α−¯ α)

where ∇J (α ) is the gradient vector and H (α )=[ Hij ] is the Hessian square matrix [p, p]

of all second-order derivatives

Hij = ∂ 2 J(α )

∂αi∂αj

C.13 Proof of Eq. (7.5.28)

Since the cost is quadratic, the input uniform density is π ( x ) = 1

4in the interval [−2, 2],

the regression function is x3 and the noise is i.i.d. with unit variance, from Eq. (7.2.3) we

obtain

R( α) = Z X,Y

C( y, αx) pf ( y| x) π( x) dy dx (C.13.18)

=Z 2

x=− 2Z Y

(y− αx )2 pf (y| x )π(x)dx dy (C.13.19)

=Z 2

x=− 2Z W

(x3 +w −αx )2 pw (w )1

4dx dw (C.13.20)

4[ Z W

pw ( w) dw Z 2

x=−2

(x3 − αx )2 dx + (C.13.21)

+Z 2

x=−2

dx Z W

w2 pw ( w) dw +Z W Z 2

x=−2

2w (x3 − αx )pw (w )dwdx ] (C.13.22)

4Z2

−2

(x3 − αx )2 dx + 4σ 2

w(C.13.23)

4Z 2

−2

(x3 −αx )2 dx +σ 2

w(C.13.24)

C.14 Biasedness of the quadratic empirical risk

Consider a regression framework where y =f (x ) + w , with E [w ] = 0 and Var [w ] = σ 2

and hN an estimation of fobtained by minimising the empirical risk in a dataset DN ∼y .

According to the derivation in [64], let us consider the quantity

gN (x) = E D N ,y [(y− h ( x, α(DN )))2 ] = ED N [Ey [(y− hN )2 ]]

where hN stands for h ( x, α (DN )). Since

(y− hN )2 = (y− f +f −hN )2 = (y− f )2 + (f− hN )2 + 2(y− f )(f− hN )

we obtain

(y− f )2 + (f− hN )2 = (y− hN )2 + 2(y− f )(hN − f )

408 APPENDIX C. PROBABILISTIC NOTIONS

Note that since Ey [y ] = f , for a given h N

Ey [(y− hN )2 ] = Ey [(y− f)2 + ( f− hN )2 + 2(y− f )( f− hN )] =

=Ey [(y−f )2 + (f− hN )2 ]

Since Ey [(y−f )2 ] = E D N [(y−f )2 ] and

(y− h )2 = (y− f +f −h )2 = (y− f )2 + (f− h )2 − 2(y− f )(h− f )

it follows

(y− f )2 + (f− h )2 = (y− h )2 + 2(y− f )(h− f )

and

gN (x) = E D N [ Ey [(y− hN )2 ]] =

=ED N [Ey [(y−f )2 ] + (f− hN )2 ] = E D N [(y−f )2 + (f−hN )2 ] =

=ED N [(y− hN )2 + 2(y−f )(hN −f )]

By averaging the quantity gN ( x) over the X domain we obtain

MISE = ED N ,y,x [(y− h (x, α (DN )))2 ]] =

=ED N ,x [(y− hN )2 ] + 2ED N ,x [(y−f )(hN −f )] = ED N [ \

MISEemp ] + 2Cov[ hN , y ]

where Cov[hN , y ] = ED N ,x [(y−f )(hN −f )] and \

MISEemp is the quantity (7.2.7) for a

quadratic error loss. This means we have to add a covariance penalty term to the apparent

error \

MISEemp in order to have an unbiased estimate of MISE.

Suppose that hN is a linear estimator, i.e.

hN =S y

where S is known as the smoother matrix. Note that in least-square regression Sis the

Hat matrix H =X (XTX ) −1 XT . In the linear case, since HT = H

NCov[hN ,y] = ED N [(Y− F)T( HY− F)] =

=ED N [YT H Y− YT F− FT H Y− YT HT F ] = E D N [YT H (Y−F )] =

=σ2 tr(H ) + FT HF − FT H F =σ2 tr(H ) = σ2 tr((XTX ) −1 XTX ) = σ2 p

where tr(H) is the trace of the matrix H ,Y is a random vector of size [N, 1] and Fis

the vector of the N regression function values f ( xi ). It follows that Cov[hN , y ] = σ2 p/N

and then the Cp formula (8.8.33). Note that the trace of His also known as the eﬀective

number of parameters.

Appendix D

Plug-in estimators

This appendix contains the expression of the plug-in estimators of some interesting pa-

rameters:

•Skewness of a random variable z: given a dataset DN ={z1 ,...,z N}the plug-in

estimate of the skewness (3.3.36) is

ˆ γ=PN

i=1(z i −ˆ µ)3

Nˆ σ3 (D.0.1)

where ˆ µand ˆ σare deﬁned in (5.3.4) and (5.3.5), respectively.

•Kurtosis of a random variable z : given a dataset DN = {z1 ,...,z N}the plug-in

estimate of the kurtosis (3.3.37) is

ˆ κ=P N

i=1(z i −ˆ µ)4

Nˆ σ4 (D.0.2)

where ˆ µand ˆ σare deﬁned in (5.3.4) and (5.3.5), respectively.

•Correlation of two random variables x and y : given a dataset DN = {hx1 , y1 i,..., hxN , yN i}

where xi ∈R , yi ∈R the plug-in estimate of the correlation (3.6.68) is

ˆ ρ=PN

i=1(x i −ˆ µx )(yi − ˆ µy )

ˆ σx ˆ σy

(D.0.3)

where ˆ µx (ˆ µy ) and ˆ σ2

x(ˆ σ2

y) denote the sample mean and sample variance of x( y ).

•Covariance matrix of a n-dimensional random vector z: given a dataset DN =

{z1 ,...,z N}where z i= [z i1,...,z in ] T is a [n, 1] vector, the plug-in estimator of the

covariance matrix (3.7.72) is the [n, n] matrix

Σ = P N

i=1(z i −ˆ µ)( zi −ˆ µ)T

N−1(D.0.4)

whose jk entry is

Σjk = P N

i=1(z ij −ˆ µj )( zik −ˆ µk )T

N−1

and ˆ µis the [n, 1] vector

ˆ µ=P N

i=1 z i

and

ˆ µj =P N

i=1 z ij

Note that (D.0.4) can be also written in matrix form

Σ = P N

i=1(Z−1 N ˆ µT )T ( Z−1N ˆ µT )

N−1

where Z is a [N, n] matrix whose i th row is z T

iand 1 Nis a [N, 1] vector of ones.

409

410 APPENDIX D. PLUG-IN ESTIMATORS

•Correlation matrix of a n-dimensional random vector z: the correlation matrix is a

symmetric [n, n] matrix whose jk entry is the correlation between the scalar random

variables zj and zk . Given a dataset DN ={z1 ,...,z N}where z i= [z i1,...,z in ] T

is a [n, 1] vector, the plug-in estimator can be written as the covariance1

P = ˜

ZT Z

of the scaled matrix ˜

Z= CZD −1

where

C= IN −1 N 1 T

is the centering [N, N ] matrix, IN is the diagonal unit matrix, 1Nis a [N, 1] vector

of ones and

D= diag(ˆ σ1 ,...,ˆ σn )

is a diagonal [n, n] scaling matrix where ˆ σ2

jis the sample variance of z j.

The diagonal entries of ˆ

P are all 1. The jk entry (j 6 =k ) of the matrix ˆ

P can be also

obtained by applying (D.0.3) to the j th and k th column of Z.

1see also http://users.stat.umn.edu/~helwig/notes/datamat-Notes.pdf

Appendix E

Kernel functions

A kernel function K is a nonnegative function

K:Rn × Rn ×R+ →R +

where the ﬁrst argument is a n-dimensional input, the second argument is typically called

the center and the third argument is called width or bandwidth. Once a distance function

between the input and the center

d:Rn × Rn →R+ (E.0.1)

is deﬁned, the kernel function can be expressed as a function

K:R+ ×R+ →R+ (E.0.2)

of the distance dand the bandwidth parameter. The maximum value of a kernel function

is located at zero distance and the function decays smoothly as the distance increases.

Here you have some examples of kernel functions.

Inverse distance:

K( d, B) = 1

(d/B )p (E.0.3)

This function goes to inﬁnity as the distance approaches zero.

Corrected inverse distance:

K( d, B) = 1

1 + (d/B )p (E.0.4)

Gaussian kernel:

K( d, B) = exp  − d 2

B2  (E.0.5)

Exponential kernel:

K( d, B) = exp  −    d

B   (E.0.6)

Quadratic or Epanechnikov kernel:

K( d, B) = ( 1− d

2if |d |< B

0 otherwise (E.0.7)

Tricube kernel:

K( d, B) = 



 1−  d

B 3 3 if |d |< B

0 otherwise

(E.0.8)

Uniform kernel:

K( d, B) = ( 1 if |d | < B

0 otherwise (E.0.9)

Triangular kernel:

K( d, B) = ( 1−  d

B if |d |< B

0 otherwise (E.0.10)

411

412 APPENDIX E. KERNEL FUNCTIONS

Appendix F

Companion R package

Several scripts are used in the main text to illustrate statistical and machine learning

notions. All the scripts have been implemented in R and are contained in the R package

gbcode.

To install the R package gbcode containing all the scripts mentioned in the text you

should run the following R instructions in the R console.

> library(devtools)

> install_github("gbonte/gbcode")

> require(gbcode)

Once installed, all the scripts will be available in the root directory of the package. In

order to retrieve the directory containing the gbcode package you should type

> system.file(package = "gbcode")

To change the directory to the one containing scripts

> setwd(find.package("gbcode"))

If you wish to run a script mentioned in the main text (e.g. the script freq.R) without

changing the local directory you should run

> source(system.file("scripts","freq.R",package = "gbcode"))

If you wish to edit a script mentioned in the main text (e.g. the script freq.R) without

changing the local directory you should run

> edit(file=system.file("scripts","freq.R",package = "gbcode"))

If you wish to execute a Shiny dashboard (e.g. leastsquares.R) you should run

> library(shiny)

> source(system.file("shiny","leastsquares.R",package = "gbcode"))

413

414 APPENDIX F. COMPANION R PACKAGE

Appendix G

Companion R Shiny

dashboards

Several Shiny dashboards are used in the main text to illustrate statistical and machine

learning notions. All the Shiny dashboards are contained in the directory shiny of the R

package gbcode and require the installation of the library shiny. To run a Shiny dashboard

(e.g. condpro.R ) you should ﬁrst move to their directory by

> setwd(paste(find.package("gbcode"),"shiny",sep="/"))

and then run

> runApp("condpro.R")

The Shiny dashboards are also active under Shinyapps1. To run a Shiny dashboard named

NAME.R go to https://gbonte.shinyapps.io/NAME. For instance, to run the Shiny dash-

board condpro.R go to:

https://gbonte.shinyapps.io/condpro

G.1 List of Shiny dashboards

•mcarlo.R: visualisation by means of Monte Carlo simulation.of

1. transformation of a r.v.

2. result of operation on two r.v.s

3. central limit theorem

4. result of linear combination of two independent r.v.s

•condpro.R: visualisation of conditional probability vs. marginal probability in the

bivariate gaussian case and in the regression function case.

•estimation.R: visualisation of diﬀerent problems of estimation:

1. estimation of mean and variance of a univariate normal r.v.: bias/variance

visualisation

2. estimation of mean and variance of a univariate uniform r.v.: bias/variance

visualisation

3. estimation of conﬁdence interval of the mean of a univariate normal r.v.

4. maximum-likelihood estimation of the mean of a univariate normal r.v.: visual-

isation of the log-likelihood function together with the value of the maximum-

likelihood estimator

1https://www.shinyapps.io

415

416 APPENDIX G. COMPANION R SHINY DASHBOARDS

5. maximum-likelihood estimation of the mean and the variance of a univariate

normal r.v.: visualisation of the bivariate log-likelihood function together with

the value of the maximum-likelihood estimator

6. estimation of mean and covariance of a bivariate normal r.v.: bias/variance

visualisation

7. least-squares estimation of the parameters of a linear target function: visu-

alisation of bias/variance of the predicted conditional expectation and of the

parameter estimators

8. least-squares estimation of the parameters of a nonlinear target function: vi-

sualisation of bias/variance of the predicted conditional expectation

•bootstrap.R: study of the accuracy of the bootstrap estimation of the sampling

distribution, estimator variance and estimator bias. The dashboard considers the

case of sample average for which it is known that bias is null and variance is inversely

proportional to N (Section 5.5.3). The dashboard shows that the bootstrap returns

an accurate estimation of bias and variance of sample average.

•leastsquares.R: visualisation of the minimisation of the empirical risk with gradient-

based iteration. 3 dashboards:

1. linear least-squares: visualisation of the convex empirical risk function and

position of the estimation as gradient-based iteration proceeds

2. NNet least-squares: visualisation of the estimated regression function (single

layer, 3 hidden nodes NNET) and the associated empirical risk as gradient-

based iteration proceeds

3. KNN cross-validation: illustration of the points used for test as cross-validation

proceeds in the case of a KNN regressor (variable number of neighbours).

•regression.R: visualisation of the model selection trade-oﬀ in regression by show-

ing the impact of diﬀerent kinds of hyper-parameters (degree of polynomial model,

number of neighbours in locally constant and locally linear ﬁtting, number of trees

in Random Forest) on the bias, variance and generalisation error.

•classif.R: visualisation of diﬀerent classiﬁcation notions in 4 dashboards:

1. Univariate: visualise the relation between posterior probability and class con-

ditional densities in a univariate binary classiﬁcation task

2. Linear discriminant: visualise the relation between bivariate class conditional

densities and linear discriminant

3. Perceptron: visualise the evolution of the perceptron hyperplane during the

gradient based minimisation of the hyperplane misclassiﬁcation and the SVM

hyperplane

4. Assessment: visualise the relation between ROC curve, PR curve, confusion

matrix and classiﬁer threshold in a univariate binary classiﬁcation task.

•classif2.R: visualisation of direct and inverse conditional distributions in the uni-

modal and bimodal case.

Bibliography

[1] Mart´ın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig

Citro, Greg S. Corrado, Andy Davis, Jeﬀrey Dean, Matthieu Devin, Sanjay Ghe-

mawat, Ian Goodfellow, Andrew Harp, Geoﬀrey Irving, Michael Isard, Yangqing Jia,

Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man´e,

Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon

Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Van-

houcke, Vijay Vasudevan, Fernanda Vi´egas, Oriol Vinyals, Pete Warden, Martin

Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-

scale machine learning on heterogeneous systems, 2015. Software available from

tensorﬂow.org.

[2] Charu C. Aggarwal. Linear Algebra and Optimization for Machine Learning - A

Textbook. Springer, 2020.

[3] D. W. Aha. Incremental, instance-based learning of independent and graded concept

descriptions. In Sixth International Machine Learning Workshop , pages 387–391, San

Mateo, CA, 1989. Morgan Kaufmann.

[4] D. W. Aha. A Study of Instance-Based Algorithms for Supervised Learning Tasks:

Mathematical, Empirical and Psychological Observations. PhD thesis, University of

California, Irvine, Department of Information and Computer Science, 1990.

[5] D. W. Aha. Editorial of special issue on lazy learning. Artiﬁcial Intelligence Review,

11(1–5):1–6, 1997.

[6] H. Akaike. Fitting autoregressive models for prediction. Annals of the Institute of

Statistical Mechanics, 21:243–247, 1969.

[7] D. M. Allen. The relationship between variable and data augmentation and a method

of prediction. Technometrics, 16:125–127, 1974.

[8] Christophe Ambroise and Geoﬀrey J. McLachlan. Selection bias in gene extraction

on the basis of microarray gene-expression data. PNAS , 99(10):6562–6566, 2002.

[9] B. D. O. Anderson and M. Deistler. Identiﬁability in dynamic errors-in-variables

models. Journal of Time Series Analysis , 5:1–13, 1984.

[10] C. G. Atkeson, A. W. Moore, and S. Schaal. Locally weighted learning. Artiﬁcial

Intelligence Review, 11(1–5):11–73, 1997.

[11] R. Babuska. Fuzzy Modeling and Identiﬁcation. PhD thesis, Technische Universiteit

Delft, 1996.

[12] R. Babuska and H. B. Verbruggen. Fuzzy set methods for local modelling and

identiﬁcation. In R. Murray-Smith and T. A. Johansen, editors, Multiple Model

Approaches to Modeling and Control, pages 75–100. Taylor and Francis, 1997.

[13] D. Barber. Bayesian reasoning and machine learning. Cambridge University Press,

2012.

[14] A. R. Barron. Predicted squared error: a criterion for automatic model selection. In

S. J. Farlow, editor, Self-Organizing Methods in Modeling, volume 54, pages 87–103,

New York, 1984. Marcel Dekker.

[15] Thomas Lumley based on Fortran code by Alan Miller. leaps: Regression Subset

Selection, 2020. R package version 3.1.

417

418 BIBLIOGRAPHY

[16] W. G. Baxt. Improving the accuracy of an artiﬁcial neural network using multiple

diﬀerently trained networks. Neural Computation , 4:772–780, 1992.

[17] Atilim Gunes Baydin, Barak A. Pearlmutter, Alexey Andreyevich Radul, and Jef-

frey Mark Siskind. Automatic diﬀerentiation in machine learning: a survey, 2015.

cite arxiv:1502.05767Comment: 43 pages, 5 ﬁgures.

[18] M. G. Bello. Enhanced training algorithms, and integrated training/architecture se-

lection for multilayer perceptron networks. IEEE Transactions on Neural Networks,

3(6):864–875, 1992.

[19] H. N. Bensusan. Automatic bias learning: an inquiry into the inductive basis of

induction. PhD thesis, University of Sussex, 1999.

[20] H. Bersini and G. Bontempi. Fuzzy models viewed as multi-expert networks. In

IFSA '97 (7th International Fuzzy Systems Association World Congress, Prague),

pages 354–359, Prague, 1997. Academia.

[21] H. Bersini and G. Bontempi. Now comes the time to defuzzify the neuro-fuzzy

models. Fuzzy Sets and Systems , 90(2):161–170, 1997.

[22] H. Bersini, G. Bontempi, and C. Decaestecker. Comparing rbf and fuzzy inference

systems on theoretical and practical basis. In F. Fogelman-Soulie' and P. Gallinari,

editors, ICANN '95,International Conference on Artiﬁcial Neural Networks, pages

169–174, 1995.

[23] M. Birattari and G. Bontempi. The lazy package for r. lazy learning for local regres-

sion. Technical Report 38, IRIDIA ULB, 2003.

[24] M. Birattari, G. Bontempi, and H. Bersini. Lazy learning meets the recursive least-

squares algorithm. In M. S. Kearns, S. A. Solla, and D. A. Cohn, editors, NIPS 11,

pages 375–381, Cambridge, 1999. MIT Press.

[25] C. M. Bishop. Neural Networks for Statistical Pattern Recognition. Oxford University

Press, Oxford, UK, 1994.

[26] S. Bittanti. Model Identiﬁcation and Data Analysis. Wiley, 2019.

[27] Joseph K. Blitzstein and Jessica Hwang. Introduction to Probability Second Edition.

2019.

[28] G Bontempi. A blocking strategy to improve gene selection for classiﬁcation of gene

expression data. Computational Biology and Bioinformatics, IEEE/ACM Transac-

tions on, 4(2):293–300, 2007.

[29] G. Bontempi and H. Bersini. Identiﬁcation of a sensor model with hybrid neuro-fuzzy

methods. In A. B. Bulsari and S. Kallio, editors, Neural Networks in Engineering

systems (Proceedings of the 1997 International Conference on Engineering Applica-

tions of Neural Networks (EANN '97), Stockolm, Sweden), pages 325–328, 1997.

[30] G. Bontempi, M. Birattari, and H. Bersini. Lazy learning for modeling and control

design. International Journal of Control , 72(7/8):643–658, 1999.

[31] G. Bontempi, M. Birattari, and H. Bersini. A model selection approach for local

learning. Artiﬁcial Intelligence Communications , 121(1), 2000.

[32] G. Bontempi, B. Haibe-Kains, C. Desmedt, C. Sotiriou, and J. Quackenbush.

Multiple-input multiple-output causal strategies for gene selection. BMC bioinfor-

matics, 12(1):458, 2011.

[33] G. Bontempi and P.E. Meyer. Causal ﬁlter selection in microarray data. In Proceed-

ing of the ICML2010 conference, 2010.

[34] G. Bontempi, C. Olsen, and M. Flauder. D2C: Predicting Causal Direction from

Dependency Features, 2014. R package version 1.1.

[35] Gianluca Bontempi and Maxime Flauder. From dependency to causality: A machine

learning approach. Journal of Machine Learning Research, 16:2437–2457, 2015.

[36] L. Breiman. Stacked regressions. Machine Learning, 24(1):49–64, 1996.

[37] L. Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classiﬁcation and

Regression Trees. Wadsworth International Group, Belmont, CA, 1984.

BIBLIOGRAPHY 419

[38] Leo Breiman. Random forests. Machine Learning, 45(1):5–32, 2001.

[39] D. Broomhead and D. Lowe. Multivariable functional interpolation and adaptive

networks. Complex Systems , 2:321–355, 1988.

[40] Gavin C. Cawley. Over-ﬁtting in model selection and its avoidance. In Jaakko

Hollmn, Frank Klawonn, and Allan Tucker, editors, IDA, volume 7619 of Lecture

Notes in Computer Science, page 1. Springer, 2012.

[41] A. Chalmers. What is this thing called science? (new and extended) . Open University

Press, 2012.

[42] V. Cherkassky and F. Mulier. Learning from Data: Concepts, Theory, and Methods.

Wiley, New York, 1998.

[43] F. Chollet and J.J. Allaire. Deep Learning with R. Manning, 2018.

[44] W. S. Cleveland. Robust locally weighted regression and smoothing scatterplots.

Journal of the American Statistical Association, 74:829–836, 1979.

[45] W. S. Cleveland and S. J. Devlin. Locally weighted regression: an approach to

regression analysis by local ﬁtting. Journal of American Statistical Association,

83:596–610, 1988.

[46] W. S. Cleveland and C. Loader. Smoothing by local regression: Principles and

methods. Computational Statistics , 11, 1995.

[47] T. Cover and P. Hart. Nearest neighbor pattern classiﬁcation. Proc. IEEE Trans.

Inform. Theory, pages 21–27, 1967.

[48] P. Craven and G. Wahba. Smoothing noisy data with spline functions: Estimat-

ing the correct degree of smoothing by the method of generalized cross-validation.

Numer. Math., 31:377–403, 1979.

[49] G. Cybenko. Just-in-time learning and estimation. In S. Bittanti and G. Picci,

editors, Identiﬁcation, Adaptation, Learning. The Science of Learning Models from

data, NATO ASI Series, pages 423–434. Springer, 1996.

[50] Andrea Dal Pozzolo, Giacomo Boracchi, Olivier Caelen, Cesare Alippi, and Gianluca

Bontempi. Credit card fraud detection: a realistic modeling and a novel learning

strategy. IEEE transactions on neural networks and learning systems, 29(8):3784–

3797, 2017.

[51] Andrea Dal Pozzolo, Olivier Caelen, and Gianluca Bontempi. When is undersam-

pling eﬀective in unbalanced classiﬁcation tasks? In Joint European Conference on

Machine Learning and Knowledge Discovery in Databases, pages 200–215. Springer,

Cham, 2015.

[52] Andrea Dal Pozzolo, Olivier Caelen, Yann-Ael Le Borgne, Serge Waterschoot, and

Gianluca Bontempi. Learned lessons in credit card fraud detection from a practi-

tioner perspective. Expert systems with applications, 41(10):4915–4928, 2014.

[53] Peter Dalgaard. Introductory statistics with R. Springer, 2002.

[54] P. Daniusis, D. Janzing, J. Mooij, J. Zscheischler, B. Steudel, K. Zhang, and

B. Schlkopf. Inferring deterministic causal relations. In Proceedings of the 26th Con-

ference on Uncertainty in Artiﬁcial Intelligence (UAI-2010), pages 143–150, 2010.

[55] A. Dean and D. Voss. Design and Analysis of Experiments. Springer Verlag, New

York, NY, USA, 1999.

[56] Marc Peter Deisenroth, A. Aldo Faisal, and Cheng Soon Ong. Mathematics for

Machine Learning. Cambridge University Press, 2020.

[57] A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete

data via the em algorithm. Journal of the Royal Statistical Society, B, 39(1):1–38,

1977.

[58] L. Devroye, L. Gy¨orﬁ, and G. Lugosi. A Probabilistic Theory of Pattern Recognition.

Springer Verlag, 1996.

[59] N. R. Draper and H. Smith. Applied Regression Analysis. John Wiley and Sons,

New York, 1981.

420 BIBLIOGRAPHY

[60] R. O. Duda and P. E. Hart. Pattern Classiﬁcation and Scene Analysis. Wiley, 1976.

[61] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern classiﬁcation. John Wiley and

sons, 2001.

[62] B. Efron. The estimation of prediction error: Covariance penalties and cross-

validation. Annals of Statistics , pages 1–26, 1979.

[63] B. Efron. The Jacknife, the Bootstrap and Other Resampling Plans. SIAM, 1982.

Monograph 38.

[64] B. Efron. Bootstrap methods: Another look at the jacknife. JASA , pages 619–642,

2004.

[65] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman and Hall,

New York, NY, 1993.

[66] B. Efron and R. J. Tibshirani. Cross-validation and the bootstrap: estimating the

error rate of a prediction rule. Technical report, Stanford University, 1995.

[67] Gamaleldin F. Elsayed, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey

Kurakin, Ian J. Goodfellow, and Jascha Sohl-Dickstein. Adversarial examples that

fool both computer vision and time-limited humans. In Samy Bengio, Hanna M. Wal-

lach, Hugo Larochelle, Kristen Grauman, Nicol Cesa-Bianchi, and Roman Garnett,

editors, NeurIPS , pages 3914–3924, 2018.

[68] J. Fan and I. Gijbels. Variable bandwidth and local linear regression smoothers. The

Annals of Statistics, 20(4):2008–2036, 1992.

[69] J. Fan and I. Gijbels. Adaptive order polynomial ﬁtting: bandwidth robustiﬁcation

and bias reduction. J. Comp. Graph. Statist. , 4:213–227, 1995.

[70] J. Fan and I. Gijbels. Local Polynomial Modelling and Its Applications. Chapman

and Hall, 1996.

[71] U. Fayyad, G. Piatetsky-Shapiro, and P. Smyth. The KDD process for extracting

useful knowledge from volumes of data. Communications of the ACM, 39(11):27–34,

November 1996.

[72] V. Fedorov. Theory of Optimal Experiments. Academic Press, 1972.

[73] Manuel Fern´andez-Delgado, Eva Cernadas, Sen´en Barro, and Dinani Amorim. Do

we need hundreds of classiﬁers to solve real world classiﬁcation problems? Journal

of Machine Learning Research, 15(1):3133–3181, January 2014.

[74] F. Fleuret. Fast binary feature selection with conditional mutual information. Jour-

nal of Machine Learning Research, 5:1531–1555, 2004.

[75] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learn-

ing and an application to boosting. Journal of Computer and System Sciences,

55(1):119–139, 1997.

[76] J. H. Friedman. Flexible metric nearest neighbor classiﬁcation. Technical report,

Stanford University, 1994.

[77] Jerome H. Friedman. Stochastic gradient boosting. Comput. Stat. Data Anal.,

38(4):367–378, February 2002.

[78] A. Gelman. Bayesian Data Analysis . Chapman and Hall, 2004.

[79] Z. Ghahramani and M. I. Jordan. Supervised learning from incomplete data via an

em approach. In J. D. Cowan, G. T. Tesauro, and J. Alspector, editors, Advances in

Neural Information Processing Systems, volume 6, pages 120–127, San Mateo, CA,

1994. Morgan Kaufmann.

[80] Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods

based on graphical models. Frontiers in Genetics, 10:524, 2019.

[81] P. Godfrey-Smith. Theory and reality: an introduction to the philosophy of science.

The University of Chicago Press, 2003.

[82] D. E. Goldberg. Genetic Algorithms in Search, Optimization and Machine Learning.

Addison-Wesley, Reading, MA, 1989.

BIBLIOGRAPHY 421

[83] G.H. Golub and C.F. Van Loan. Matrix computations . Johns Hopkins University

Press, 1996.

[84] T. R. Golub, D. K. Slonin, P. Tamayo, C. Huard, and M. Gaasenbeek. Molecu-

lar clssiﬁcation of cancer: Class discovery and class prediction by gene expression

monitoring. Science , 286:531–537, 1999.

[85] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press,

2016. http://www.deeplearningbook.org.

[86] S. Guo and M.W. Fraser. Propensity Score Analysis: Statistical Methods and Appli-

cations. SAGE, 2014.

[87] I. Guyon. Results and analysis of the 2013 ChaLearn cause-eﬀect pair challenge.

JMLR Workshop and Conference Proceedings, 2014.

[88] I. Guyon, C. Aliferis, and A. Elisseeﬀ. Computational Methods of Feature Selection,

chapter Causal Feature Selection, pages 63–86. Chapman and Hall, 2007.

[89] I. Guyon and A. Elisseeﬀ. An introduction to variable and feature selection. Journal

of Machine Learning Research, 3:1157–1182, 2003.

[90] I. Guyon and A. Elisseeﬀ. An introduction to variable and feature selection. Journal

of Machine Learning Research, 3:1157–1182, 2003.

[91] Isabelle Guyon, Steve Gunn, Masoud Nikravesh, and Lotﬁ A. Zadeh. Feature Ex-

traction: Foundations and Applications. Springer-Verlag New York, Inc., 2006.

[92] B. Haibe-Kains, C. Desmedt, S. Loi, M. Delorenzi, C. Sotiriou, and G. Bontempi.

Computational Intelligence in Clinical Oncology: Lessons Learned from an Analysis

of a Clinical Study, pages 237–268. Springer Berlin Heidelberg, Berlin, Heidelberg,

2008.

[93] D. J. Hand. Discrimination and classiﬁcation. John Wiley, New York, 1981.

[94] D.J. Hand. Statistics: a very short introduction, volume 196. Oxford University

Press, USA, 2008.

[95] W. Hardle and J. S. Marron. Fast and simple scatterplot smoothing. Comp. Statist.

Data Anal., 20:1–17, 1995.

[96] T. Hastie and R. Tibshirani. Generalized Additive Models. Chapman and Hall,

London, UK, 1990.

[97] T. Hastie and R. Tibshirani. Discriminant adaptive nearest neighbor classiﬁcation.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(6):607–615,

1996.

[98] T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning.

Springer, 2001.

[99] T. Hastie, R. Tibshirani, and J. Friedman. The elements of statistical learning (2nd

edition). Springer, 2009.

[100] Trevor Hastie and Robert Tibshirani. Eﬃcient quadratic regularization for expres-

sion arrays. Biostatistics (Oxford, England), 5(3):329–40, Jul 2004.

[101] J. S. U. Hjorth. Computer Intensive Statistical Methods. Chapman and Hall, 1994.

[102] Tin Kam Ho. The random subspace method for constructing decision forests. IEEE

Trans. Pattern Anal. Mach. Intell., 20(8):832–844, 1998.

[103] W. Hoeﬀding. Probability inequalities for sums of bounded random variables. Jour-

nal of American Statistical Association, 58:13–30, 1963.

[104] PO Hoyer, D. Janzing, J. Mooij, J. Peters, and B. Scholkopf. Nonlinear causal

discovery with additive noise models. In Advances in Neural Information Processing

Systems, pages 689–696, 2009.

[105] P. J. Huber. Robust Statistics . Wiley, New York, 1981.

[106] P. Hurley. A Concise Introduction to Logic. CENGAGE Learning Custom Publish-

ing, 2011.

422 BIBLIOGRAPHY

[107] A. K. Jain, R. C. Dubes, and C. Chen. Bootstrap techniques for error estimation.

IEEE Transactions on Pattern Analysis and Machine Intelligence, 5:628–633, 1987.

[108] J.-S. R. Jang. Anﬁs: Adaptive-network-based fuzzy inference systems. IEEE Trans-

actions on Fuzzy Systems, 23(3):665–685, 1993.

[109] J. S. R. Jang, C. T. Sun, and E. Mizutani. Neuro-Fuzzy and Soft Computing . Matlab

Curriculum Series. Prentice Hall, 1997.

[110] E.T. Jaynes. Probability theory : the logic of science. Cambridge University Press,

2003.

[111] T. A. Johansen and B. A. Foss. Constructing narmax models using armax models.

International Journal of Control, 58:1125–1153, 1993.

[112] M. C. Jones, J. S. Marron, and S. J. Sheather. A brief survey of bandwidth selection

for density estimation. Journal of American Statistical Association, 90, 1995.

[113] M. I. Jordan and T. J. Sejnowski, editors. Graphical models: foundations of neural

computation. The MIT Press, 2001.

[114] V. Y. Katkovnik. Linear and nonlinear methods of nonparametric regression analysis.

Soviet Automatic Control, 5:25–34, 1979.

[115] S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by simulated annealing.

Science, 220(4598):671–680, 1983.

[116] R. Kohavi. A study of cross-validation and bootstrap for accuracy estima-

tion and model selection. In Proceedings of IJCAI-95, 1995. available at

http://robotics.stanford.edu/users/ronnyk/ronnyk-bib.html.

[117] R. Kohavi and G. H. John. Wrappers for feature subset selection. Artiﬁcial Intelli-

gence, 97(1-2):273–324, 1997.

[118] R. Kohavi and G. H. John. Wrappers for feature subset selection. Artiﬁcial Intelli-

gence, 97(1-2):273–324, 1997.

[119] D. Koller and N. Friedman. Probabilistic graphical models. The MIT Press, 2009.

[120] A. N. Kolmogorov. Foundations of Probability. Berlin, 1933.

[121] J. Kolodner. Case-Based Reasoning. Morgan Kaufmann, 1993.

[122] Yann LeCun, Yoshua Bengio, and Geoﬀrey Hinton. Deep learning. Nature,

521(7553):436–444, 2015.

[123] Raoul LePage and Lynne Billard. Exploring the limits of bootstrap. John Wiley &

Sons, 1992.

[124] R. J.A. Little and D. B. Rubin. Statistical analysis with missing data. Wiley, 2002.

[125] L. Ljung. System identiﬁcation: Theory for the User . Prentice-Hall, Englewood

Cliﬀs, NJ, 1987.

[126] Sherene Loi, Benjamin Haibe-Kains, Christine Desmedt, Pratyaksha Wirapati,

Fran¸coise Lallemand, Andrew M Tutt, Cheryl Gillet, Paul Ellis, Kenneth Ryder,

James F Reid, Gianluca Bontempi, et al. Predicting prognosis using molecular

proﬁling in estrogen receptor-positive breast cancer treated with tamoxifen. BMC

genomics, 9(1):1–12, 2008.

[127] D. Lopez-Paz. From Dependence to Causation. PhD thesis, Cambridge University,

2016.

[128] D. G. Luenberger. Linear and Nonlinear Programming. Addison Wesley, Reading,

MA, 1984.

[129] C. Mallows. Discussion of a paper of beaton and tukey. Technometrics, 16:187–188,

1974.

[130] C. L. Mallows. Some comments on cp . Technometrics, 15:661, 1973.

[131] O. Maron and A. Moore. The racing algorithm: Model selection for lazy learners.

Artiﬁcial Intelligence Review, 11(1–5):193–225, 1997.

[132] A. Miller. Subset Selection in Regression (2nd ed.). Chapman and Hall, 2002.

BIBLIOGRAPHY 423

[133] J. Moody. The eﬀective number of parameters: An analysis of generalization and

regularization in nonlinear learning systems. In J. Moody, Hanson, and Lippmann,

editors, Advances in Neural Information Processing Systems, volume 4, pages 847–

854, Palo Alto, 1992. Morgan Kaufmann.

[134] J. Moody and C. J. Darken. Fast learning in networks of locally-tuned processing

units. Neural Computation , 1(2):281–294, 1989.

[135] A. W. Moore, D. J. Hill, and M. P. Johnson. An empirical investigation of brute force

to choose features, smoothers and function approximators. In S. Janson, S. Judd, and

T. Petsche, editors, Computational Learning Theory and Natural Learning Systems,

volume 3. MIT Press, Cambridge, MA, 1992.

[136] K. P. Murphy. An introduction to graphical models. Technical report, 2001.

[137] R. Murray-Smith. A local model network approach to nonlinear modelling. PhD

thesis, Department of Computer Science, University of Strathclyde, Strathclyde,

UK, 1994.

[138] R. Murray-Smith and T. A. Johansen. Local learning in local model networks.

In R. Murray-Smith and T. A. Johansen, editors, Multiple Model Approaches to

Modeling and Control, chapter 7, pages 185–210. Taylor and Francis, 1997.

[139] R. H. Myers. Classical and Modern Regression with Applications. PWS-KENT

Publishing Company, Boston, MA, second edition, 1994.

[140] E. Nadaraya. On estimating regression. Theory of Prob. and Appl. , 9:141–142, 1964.

[141] Cathy O'Neil. Weapons of Math Destruction. Crown, New York, 2016.

[142] A. Papoulis. Probability, Random Variables, and Stochastic Processes. McGraw Hill,

1991.

[143] Simon Parsons and Anthony Hunter. A Review of Uncertainty Handling Formalisms,

pages 8–37. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998.

[144] E. Parzen. On estimation of a probability density function and mode. Annals of

Mathematical Statistics, 33:1065–1076, 1962.

[145] Y. Pawitan. In all likelihood: statistical modelling and inference using likelihood.

Oxford Science, 2001.

[146] J. Pearl. Causality. Cambridge University Press, 2000.

[147] Judea Pearl. Probabilistic reasoning in intelligent systems : networks of plausible

inference. Morgan Kaufmann, San Francisco, Calif., 2009. Example for Explaning

away.

[148] Judea Pearl. Comment: Understanding simpson's paradox. The American Statisti-

cian, 68:8–13, 2014.

[149] Judea Pearl and Dana Mackenzie. The Book of Why. Basic Books, New York, 2018.

[150] Jean-Philippe Pellet and Andr Elisseeﬀ. Using markov blankets for causal structure

learning. J. Mach. Learn. Res., 9:1295–1342, 2008.

[151] J.P. Pellet and A. Elisseeﬀ. Using markov blankets for causal structure learning.

Journal of Machine Learning Research, 9:1295–1342, 2008.

[152] H. Peng, F. Long, and C. Ding. Feature selection based on mutual information:

criteria of max-dependency, max-relevance, and min-redundancy. IEEE Transactions

on Pattern Analysis and Machine Intelligence, 27, 2005.

[153] M. P. Perrone and L. N. Cooper. When networks disagree: Ensemble methods for

hybrid neural networks. In R. J. Mammone, editor, Artiﬁcial Neural Networks for

Speech and Vision, pages 126–142. Chapman and Hall, 1993.

[154] Jonas Peters, Dominik Janzing, and Bernhard Sch¨olkopf. Elements of Causal Infer-

ence: Foundations and Learning Algorithms. Adaptive Computation and Machine

Learning. MIT Press, Cambridge, MA, 2017.

[155] D. Plaut, S. Nowlan, and G. E. Hinton. Experiments on learning by back propaga-

tion. Technical Report CMU-CS-86-126, Department of Computer Science, Carnegie

Mellon University, Pittsburgh, PA, 1986.

424 BIBLIOGRAPHY

[156] M. J. D. Powell. Algorithms for Approximation , chapter Radial Basis Functions

for multivariable interpolation: a review, pages 143–167. Clarendon Press, Oxford,

1987.

[157] W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical

Recipes in C: The Art of Scientiﬁc Computing. Cambridge University Press, 1992.

Second ed.

[158] J. R. Quinlan. Induction of decision trees. Machine Learning, 1:81–106, 1986.

[159] J. R. Quinlan. Simplyfying decision trees. International Journal of Man-Machine

Studies, 27:221–234, 1987.

[160] R Development Core Team. R: A language and environment for statistical computing.

R Foundation for Statistical Computing, Vienna, Austria, 2004. ISBN 3-900051-07-0.

[161] J. Rissanen. Modeling by shortest data description. Automatica, 14:465–471, 1978.

[162] M. Rosenblatt. Remarks on some nonparametric estimates of a density function.

Annals of Mathematical Statistics, 27:832–837, 1956.

[163] K. J. Rothman, S. Greenland, and Timothy L. Lash. Modern Epidemiology . Lippin-

cott Williams & Wilkins, Philadelphia, PA, 3rd edition, 2008.

[164] D. B. Rubin. Inference and missing data (with discussion). Biometrika , 63:581–592,

1976.

[165] D. E. Rumelhart, G. E. Hinton, and R. K. Williams. Learning representations by

backpropagating errors. Nature, 323(9):533–536, 1986.

[166] S. Russel and Peter Norvig. Artiﬁcial Intelligence: a modern approach. Pearson,

2016.

[167] Y. Saeys, I. Inza, and P. Larranaga. A review of feature selection techniques in

bioinformatics. Bioinformatics , 23:2507–2517, 2007.

[168] R. E. Schapire. Nonlinear Estimation and Classiﬁcation, chapter The boosting ap-

proach to machine learning: An overview. Springer,.

[169] L. Schneps and C. Colmez. Math on Trial: How Numbers Get Used and Abused in

the Courtroom. EBL ebooks online. Basic Books, 2013.

[170] D. W. Scott. Multivariate density estimation . Wiley, New York, 1992.

[171] John Shawe-Taylor and Nello Cristianini. Kernel Methods for Pattern Analysis.

Cambridge University Press, illustrated edition edition, 2004.

[172] P. Spirtes, C. Glymour, and R. Scheines. Causation, Prediction and Search. Springer

Verlag, Berlin, 2000.

[173] P. Spirtes and K. Zhangl. Causal discovery and inference: concepts and recent

methodological advances. Applied Informatics , 3, 2016.

[174] C. Stanﬁll and D. Waltz. Toward memory-based reasoning. Communications of the

ACM, 29(12):1213–1228, 1987.

[175] C. Stone. Consistent nonparametric regression. The Annals of Statistics , 5:595–645,

1977.

[176] M. Stone. Cross-validatory choice and assessment of statistical predictions. Journal

of the Royal Statistical Society B, 36(1):111–147, 1974.

[177] M. Stone. An asymptotic equivalence of choice of models by cross-validation and

akaike's criterion. Journal of Royal Statistical Society, Series B, 39:44–47, 1977.

[178] T. Takagi and M. Sugeno. Fuzzy identiﬁcation of systems and its applications

to modeling and control. IEEE Transactions on Systems, Man, and Cybernetics,

15(1):116–132, 1985.

[179] M. Taniguchi and V. Tresp. Averaging regularized estimators. Neural Computation,

(9), 1997.

[180] H. Tijms. Understanding probability. Cambridge, 2004.

[181] V. Tresp. Handbook for neural network signal processing, chapter Committee ma-

chines. CRC Press, 2001.

BIBLIOGRAPHY 425

[182] I. Tsamardinos and C. Aliferis. Towards principled feature selection: Relevancy. In

Proceedings of the 9th International Workshop on Artiﬁcial Intelligence and Statis-

tics, 2003.

[183] I Tsamardinos and CF Aliferis. Towards Principled Feature Selection: Relevancy,

Filters and Wrappers. In Ninth International Workshop on Artiﬁcial Intelligence

and Statistics, AISTAT, 2003.

[184] B. van Fraassen. The Scientiﬁc Image. Oxford University Press, 1980.

[185] V. N. Vapnik. Principles of risk minimization for learning theory. In Advances in

Neural Information Processing Systems, volume 4, Denver, CO, 1992.

[186] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, NY,

1995.

[187] V. N. Vapnik. Statistical Learning Theory. Springer, 1998.

[188] V. N. Vapnik and A. J. Chervonenkis. The necessary and suﬃcient conditions for

consistency of the method of empirical risk. Pattern Recognition and Image Analysis,

1(3):284–305, 1991.

[189] W. N. Venables and D. M. Dmith. An Introduction to R. Network Theory, 2002.

[190] Tyler Vigen. Spurious correlations. Hachette Book, 2015.

[191] T. P. Vogl, J. K. Mangis, A. K. Rigler, W. T. Zink, and D. L. Alkon. Accelerating

the convergence of the back-propagation method. Biological Cybernetics, 59:257–263,

1988.

[192] L. Wasserman. All of statistics . Springer, 2004.

[193] G. Watson. Smooth regression analysis. Sankhya, Series, A(26):359–372, 1969.

[194] S. M. Weiss and C. A. Kulikowski. Computer Systems that learn. Morgan Kaufmann,

San Mateo, California, 1991.

[195] B. Widrow and M.E. Hoﬀ. Adaptive switching circuits. In WESCON Convention

Record Part IV, 1960.

[196] D. H. Wolpert. Stacked generalization. Technical Report LA-UR-90-3460, Los

Alamos, NM, 1990. He proposes a more sophisticated voting scheme where the

a second level function output performs the ﬁnal estimation of the true class. I don't

think this is going to referenced in the ﬁnal thesis.

[197] D. H. Wolpert. The lack of a priori distinctions between learning algorithms. Neural

Computation, 8:1341–1390, 1996.

[198] D. H. Wolpert and R. Kohavi. Bias plus variance decomposition for zero-one loss

functions. In Proceedings of the 13th International Conference on Machine Learning,

pages 275–283, 1996.

[199] Zenglin Xu, Rong Jin, Jieping Ye, Michael R. Lyu, and Irwin King. Non-monotonic

feature selection. In Andrea Pohoreckyj Danyluk, Lon Bottou, and Michael L.

Littman, editors, ICML , volume 382 of ACM International Conference Proceeding

Series, pages 1145–1152. ACM, 2009.

[200] L. A. Zadeh. Fuzzy sets. Information and Control, 8:338–353, 1965.

... La définition d'un modèle de pendule dépend de plusieurs facteurs, comme discuté en Section 1.2. 1. On pourrait envisager un modèle pour calculer la période T du pendule pour une longueur fixée à l'avance ou au contraire établir un modèle pour déterminer quelle longueur l * du fil détermine une période T * . ...

... Afin de pouvoir étudier un système dynamique il est nécessaire d'introduire, comme dans l'exemple précédent, une troisième grandeur, appelée état, qui jouit des propriétés suivantes : 1. elle résume l'ensemble de l'information sur le passé et le présent du système (mémoire). ...

... Définition 2. 6. Un état x (2) est accessible à l'instant t 2 à partir d'un état x (1) s'il existe un instant t 1 et une fonction d'entrée u(·) ∈ Ω telle que ϕ(t 2 , t 1 , x (1) , u(·)) = x (2) Parfois il est intéressant de considérer des systèmes où chaque état x (2) à l'instant t est accessible à partir d'un quelconque état x (1) . ...

Ce syllabus est destiné, en premier lieu, aux étudiants de Bachelier en Sciences Informatiques de l'Université Libre de Bruxelles. Le cours de Modélisation et Simulation vise à fournir une présentation des fondements théoriques de la modélisation des systèmes dynamiques et des outils pour la simulation. La motivation pour le cours naquit d'un constat: nonobstant la réputation mondiale (notamment un prix Nobel et plusieurs prix Francqui) de l'ULB dans le domaine de la modélisation de systèmes dynamiques, aucun cours sur ce sujet n'était dispensé aux élèves informaticiens. Ceci était d'autant plus surprenant si on considère i) que des grandes avancées ont été rendues possible dans la modélisation par l'utilisation de l'ordinateur (il suffit penser aux fractales et à l'étude des phénomènes chaotiques) et ii) que les progrès des méthodes numériques et l'augmentation des performances des ordinateurs permettent aujourd'hui de simuler et prédire le comportement de systèmes de plus en plus complexes. Aussi plusieurs domaines industriels utilisent couramment la simulation numérique afin de raccourcir le cycle de conception et développement de nouveaux produits. La matière enseignée se répartit sur 9 sujets principaux: l'introduction à la modélisation et à la simulation, l'introduction aux systèmes dynamiques, les systèmes à états discrets et à temps discret, les systèmes dynamiques à temps continu, les systèmes linéaires continus, les systèmes non-linéaires continus, les systèmes à temps discret, la simulation Monte-Carlo et la simulation à événements discrets. On trouvera dans l'annexe des rappels d'équations différentielles et de probabilité. Le côté pratique de la modélisation et de la simulation n'est pas négligé et chaque chapitre introduit des applications à des problèmes concrets. Ce manuel ne vise ni à l'originalité, ni à être complet. Il n'a d'autre but que de fournir un supplément à l'étudiant qui suit régulièrement le cours. De nombreuses références à des ouvrages publiés sont faites tout au long de la présentation. En particulier, la présentation des fondements des systèmes dynamiques est inspirée du livre du Pr. Sergio Rinaldi (Politecnico di Milano, Italie) at de son cours de Teoria dei sistemi que j'ai eu l'honneur de suivre dans le lointain 1989. Et, afin de stimuler l'étudiant à entreprendre la découverte de la complexité qui se cache derrière la dynamique d'un système apparemment simple, voici une citation extraite de Gleick: Tout irait bien mieux, non seulement en recherche, mais aussi dans le monde quotidien de la politique et de l'économie, si davantage de gens prenaient conscience du fait que les systèmes élémentaires ne possèdent pas nécessairement des propriétés dynamiques simples.

... Our research considered the implementation of a streaming algorithm capable of performing analysis on a univariate time series as new data becomes available. To make this possible, recursive least squares techniques can be applied to an autoregressive (AR) model [5]. The AR model is commonly used for time series forecasting. ...

... We can then apply recursive least squares to allow the model to work on a data stream [5]. The regression coefficients can be updated as a new sample ( +1 , +1 ) becomes available. ...

... Nonlinear regression is based on developing predictive models, which combine basic functions, such as polynomial, sigmoid, and spline [13]. The polynomial regression is one of the simplest approaches, and it aims at fitting a model by using curves of order n > 2 (quadratic, cubic, etc. ), while the spline approach aims at producing a piece-wise model in which each model is trained with only the value lying in a specified interval. ...

... It performs this through an unsupervised process that projects the data from the original space to a lower dimensional one, where the axis, which are called Principal Components (PC), of this new space are computed by combining the original variables. The first PC is oriented over the direction with the maximum variance of data [13]. This mathematically corresponds to find the vector a = [a 1 , . . . ...

The large-scale deployment of pervasive sensors and decentralized computing in modern smart grids is expected to exponentially increase the volume of data exchanged by power system applications. In this context, the research for scalable, and flexible methodologies aimed at supporting rapid decisions in a data rich, but information limited environment represents a relevant issue to address. To this aim, this paper outlines the potential role of Knowledge Discovery from massive Datasets in smart grid computing, presenting the most recent activities developed in this field by the Task Force on "En-abling Paradigms for High-Performance Computing in Wide Area Monitoring Protective and Control Systems" of the IEEE PSOPE Technologies and Innovation Subcommittee.

... The choice of the optimal number of neighbors k will be performed through automatic leave-one-out selection as described in [5]. Our implementation of the kNN models is based on the R package gbcode [7]. ...

In finance, volatility is defined as a measure of variation of a trading price series over time. As volatility is a latent variable, several measures, named proxies, have been proposed in the literature to represent such quantity. The purpose of our work is twofold. On one hand, we aim to perform a statistical assessment of the relationships among the most used proxies in the volatility literature. On the other hand, while the majority of the reviewed studies in the literature focuses on a uni-variate time series model (NAR), using a single proxy, we propose here a NARX model, combining two proxies to predict one of them, showing that it is possible to improve the prediction of the future value of some proxies by using the information provided by the others. Our results , employing artificial neural networks (ANN), k-Nearest Neighbours (kNN) and support vector regression (SVR), show that the supplementary information carried by the additional proxy could be used to reduce the forecasting error of the aforementioned methods. We conclude by explaining how we wish to further investigate such relationship.

... The coefficient of determination R 2 is in the range from zero to one, the closer the coefficient to one, the better the results. However, this criterion is not appropriate for the comparison of candidate models because overfitting increases artificially its value (Bontempi & Ben Taieb, 2011), as well as the number of features. Finally, this measure is useful to check how well the independent variables explain the variability of the dependent one. ...

The design of experiments and the validation of the results achieved with them are vital in any research study. This paper focuses on the use of different Machine Learning approaches for regression tasks in the field of Computational Intelligence and especially on a correct comparison between the different results provided for different methods, as those techniques are complex systems that require further study to be fully understood. A methodology commonly accepted in Computational intelligence is implemented in an R package called RRegrs. This package includes ten simple and complex regression models to carry out predictive modeling using Machine Learning and well-known regression algorithms. The framework for experimental design presented herein is evaluated and validated against RRegrs. Our results are different for three out of five state-of-the-art simple datasets and it can be stated that the selection of the best model according to our proposal is statistically significant and relevant. It is of relevance to use a statistical approach to indicate whether the differences are statistically significant using this kind of algorithms. Furthermore, our results with three real complex datasets report different best models than with the previously published methodology. Our final goal is to provide a complete methodology for the use of different steps in order to compare the results obtained in Computational Intelligence problems, as well as from other fields, such as for bioinformatics, cheminformatics, etc., given that our proposal is open and modifiable.

La cerveza es un bien de alto consumo en la actualidad que se caracteriza, en países latinos, por su alta concentración en la producción nacional. Estas cerveceras han creado niveles elevados de lealtad en los consumidores, pero carecen de características diferenciadoras debido a la baja diversidad de productos de este tipo. En los últimos años se ha observado un auge de las microcervecerías, las cuales en menos de diez años han crecido exponencialmente haciéndose fuerte en diferentes lugares como bares, pubs, cafés y mostrando aspectos de fortalecimiento ante un mercado dominado por una única empresa. Frente a esto, los consumidores han respondido generando patrones en los que, para diferentes ocasiones o eventos, se inclinan por cierto tipo de cerveza (artesanal o comercial). El estudio apunta a mostrar el panorama actual de la cerveza artesanal en Colombia y generar bases suficientes para enlazar tendencias actuales de la cerveza artesanal y cómo esto influye en el sentimiento y motivaciones del consumidor. La investigación pretende destacar en un campo que no es muy conocido: las microcervecerías en la producción de cerveza artesanal, y obtener una respuesta satisfactoria desde el punto de vista del consumidor. Los resultados obtenidos son ideales para entender el comportamiento del consumidor durante el consumo de cerveza artesanal.

Dalila Hattab

The aim of this paper is to investigate the effect of volatility surges during the COVID-19 pandemic crisis on long-term investment trading rules. These trading rules are derived from stock return forecasting based on a Multiple Step Ahead Direct Strategy, and built on the combination of machine learning models and the Autoregressive Fractionally Integrated Moving Average (ARFIMA) model. ARFIMA has the feature to account for the long memory and structural change in conditional variance process. The machine learning models considered are a particular Neural Network model (MLP), K-Nearest Neighbors (KNN) and Support Vector Regression (SVR). The trading performances of the produced models are evaluated in terms of economical metrics reflecting profitability and risk like: Annualized Return, Sharpe Ratio and Profit Ratio. The hybrid model performances are compared to the simple machine learning models and to the classical ARMA-GARCH model using a Volatility Proxy as external regressor. When applying these long-term investment trading rules to the CAC40 index, from May 2016 to May 2020, the finding is that both MLP-based and hybrid ARFIMA-MLP-based trading models show higher performances with a Sharpe Ratio close to 2 and a Profit Ratio around 40% despite the COVID-19 crisis.

Lectures and exercise classes of the ULB/VUB Big Data course (Scalable Analytics part). The GitHub archive contains the lectures and the practical classes on how to implement from scratch different ML algorithms (ordinary least squares, gradient descent, k-means, alternating least squares), using Python NumPy, and how to then make these implementations scalable using Map/Reduce and Spark. Code on: https://github.com/Yannael/BigDataAnalytics_INFOH515 Slides of the course on: https://github.com/Yannael/BigDataAnalytics_INFOH515/tree/master/Analytics_Course_Slides

Clark Glymour
Kun Zhang
Peter Spirtes

A fundamental task in various disciplines of science, including biology, is to find underlying causal relations and make use of them. Causal relations can be seen if interventions are properly applied; however, in many cases they are difficult or even impossible to conduct. It is then necessary to discover causal relations by analyzing statistical properties of purely observational data, which is known as causal discovery or causal structure search. This paper aims to give a introduction to and a brief review of the computational methods for causal discovery that were developed in the past three decades, including constraint-based and score-based methods and those based on functional causal models, supplemented by some illustrations and applications.

Gamaleldin F. Elsayed
Shreya Shankar
Brian Cheung
Jascha Sohl-Dickstein

Machine learning models are vulnerable to adversarial examples: small changes to images can cause computer vision models to make mistakes such as identifying a school bus as an ostrich. However, it is still an open question whether humans are prone to similar mistakes. Here, we create the first adversarial examples designed to fool humans, by leveraging recent techniques that transfer adversarial examples from computer vision models with known parameters and architecture to other models with unknown parameters and architecture, and by modifying models to more closely match the initial processing of the human visual system. We find that adversarial examples that strongly transfer across computer vision models influence the classifications made by time-limited human observers.

Detecting frauds in credit card transactions is perhaps one of the best testbeds for computational intelligence algorithms. In fact, this problem involves a number of relevant challenges, namely: concept drift (customers' habits evolve and fraudsters change their strategies over time), class imbalance (genuine transactions far outnumber frauds), and verification latency (only a small set of transactions are timely checked by investigators). However, the vast majority of learning algorithms that have been proposed for fraud detection rely on assumptions that hardly hold in a real-world fraud-detection system (FDS). This lack of realism concerns two main aspects: 1) the way and timing with which supervised information is provided and 2) the measures used to assess fraud-detection performance. This paper has three major contributions. First, we propose, with the help of our industrial partner, a formalization of the fraud-detection problem that realistically describes the operating conditions of FDSs that everyday analyze massive streams of credit card transactions. We also illustrate the most appropriate performance measures to be used for fraud-detection purposes. Second, we design and assess a novel learning strategy that effectively addresses class imbalance, concept drift, and verification latency. Third, in our experiments, we demonstrate the impact of class unbalance and concept drift in a real-world data stream containing more than 75 million transactions, authorized over a time window of three years.

K.J. Hastings

Updated to conform to Mathematica® 7.0, Introduction to Probability with Mathematica®, Second Edition continues to show students how to easily create simulations from templates and solve problems using Mathematica. It provides a real understanding of probabilistic modeling and the analysis of data and encourages the application of these ideas to practical problems. The accompanying CD-ROM offers instructors the option of creating class notes, demonstrations, and projects. New to the Second Edition • Expanded section on Markov chains that includes a study of absorbing chains • New sections on order statistics, transformations of multivariate normal random variables, and Brownian motion • More example data of the normal distribution • More attention on conditional expectation, which has become significant in financial mathematics • Additional problems from Actuarial Exam P • New appendix that gives a basic introduction to Mathematica • New examples, exercises, and data sets, particularly on the bivariate normal distribution • New visualization and animation features from Mathematica 7.0 • Updated Mathematica notebooks on the CD-ROM (Go to Downloads/Updates tab for link to CD files.) • After covering topics in discrete probability, the text presents a fairly standard treatment of common discrete distributions. It then transitions to continuous probability and continuous distributions, including normal, bivariate normal, gamma, and chi-square distributions. The author goes on to examine the history of probability, the laws of large numbers, and the central limit theorem. The final chapter explores stochastic processes and applications, ideal for students in operations research and finance.

T.J. Hastie
R.J. Tibshirani

This book describes an array of power tools for data analysis that are based on nonparametric regression and smoothing techniques. These methods relax the linear assumption of many standard models and allow analysts to uncover structure in the data that might otherwise have been missed. While McCullagh and Nelder's Generalized Linear Models shows how to extend the usual linear methodology to cover analysis of a range of data types, Generalized Additive Models enhances this methodology even further by incorporating the flexibility of nonparametric regression. Clear prose, exercises in each chapter, and case studies enhance this popular text.

Leo Breiman
Jerome H. Friedman
Richard A. Olshen
Charles J. Stone

The methodology used to construct tree structured rules is the focus of this monograph. Unlike many other statistical procedures, which moved from pencil and paper to calculators, this text's use of trees was unthinkable before computers. Both the practical and theoretical sides have been developed in the authors' study of tree methods. Classification and Regression Trees reflects these two sides, covering the use of trees as a data analysis method, and in a more mathematical framework, proving some of their fundamental properties.

Posted by: rosierosiemezzinnie0269567.blogspot.com

Source: https://www.researchgate.net/publication/242692234_Statistical_foundations_of_machine_learning_2nd_edition_handbook