ENSEMBLE LEARNING:Bagging and Boosting

Présentation au sujet: "ENSEMBLE LEARNING:Bagging and Boosting"— Transcription de la présentation:

1 ● Habiba Abderrahim ● Fouad Oueslati ● Nasredin Znaidi ● Zeineb Mrabet Team Members 1

2 ENSEMBLE LEARNING: Bagging and Boosting

3 OUTLINE BAGGING What, Where and When ? BAGGING Vs BOOSTING Stacking What, Where and When ? INTRODUCTION Ensemble Learning Techniques 01 02 04 What, Where and When ? BOOSTING 03 Conclusion 05 3

4 INTRODUCTION 01 4

5 Supervised Learning ENSEMBLE LEARNING ? Methods, Techniques in Machine Learning Combining several models to make one very reliable model Improving the accuracy of results HOW? GOAL? WHAT? Base Learners ● Algorithmes ● Hyperparameters ● Training set 5

6 Why Not a Single Model ? 6

7 Model Common problem Bias under-fitting over- fitting ML MODEL Variance BiasVariance Ensemble methods try to overcome bias-variance tradeoff by combining the results of multiple weak learners. These weak learners result in very strong robust model. 7

8 Main Types of Ensemble Methods 8

9 Bagging Bootstrap aggregation 02 9

10 Bootstrap: Resampling Method oxyclass 1001+ 1110+ 1211- Original DatasetN=3 oxyclass 1001+ 1211- 1001+ Boostraped dataset oxyclass 1110+ 1001+ 1211- Boostraped dataset Sampling with replacement Randomly selecting data and allowing for duplicates ! ● Bootstrap dataset is different from the original dataset. ● Feeding different Train Set To the same Model or different Model. ● Sampling can be : Row-wise column-wise combination of both. 10

11 out-of-bag sample 11

12 Bootstrap This procedure of using the bootstrap method can be summarized as follows: 1. Choose a number of bootstrap samples to perform 2. Choose a sample size N’ <= N 3. For each bootstrap sample ● Draw a sample with replacement with the chosen size ● Fit a model on the data sample ● Estimate the skill of the model on the out-of- bag sample. 12

13 Bootstrap Aggregation Summarizing The steps: 1. Bootstrapping 2. feeding data to various models, each model runs independently. 3. taking all the outputs, counting the majority of votes for classification or averaging them for regression problems. Bagging : Bootstrap + Aggregation 13

14 Majority Voting vs. Soft Voting Result Aggregation 14

15 Forest = Large Number of Trees Random Forest 15

16 Feature Randomness Bagging + 16

17 Bagging Algorithms Demo Code 02 17

18 18

19 19

20 BOOSTING 03 20

21 ●Improved classifier performance ●Selection of data points where the classifier is wrong (false predictions) ●A variety of Bagging method Training a bunch of individual models in a sequential way. Each individual model learns from mistakes made by the previous model. Bagging and Boosting ? Boosting 21

22 Adaboost 22

23 Adaboost: Process 1-Initially, Adaboost selects a training subset randomly 2-It iteratively trains the AdaBoost machine learning model by selecting the training set based on the accurate prediction of the last training. 3-It assigns the higher weight to wrong classified observations so that in the next iteration, these observations will get the high probability for classification. 4-Also, It assigns the weight to the trained classifier in each iteration according to the accuracy of the classifier(The more accurate classifier will get high weight) 5-This process iterate until the complete training data fits without any error or until reached to the specified maximum number of estimators. The AdaBoost makes a new prediction by adding up the weight (of each tree) multiply the prediction (of each tree). Obviously, the tree with higher weight will have more power of influence the final decision. 23

24 Adaboost: Example1 x1x2actualweightweighted actual 2310.1 221 461 430.1-0.1 410.1-0.1 5710.1 530.1-0.1 6510.1 860.1-0.1 820.1-0.1 Initially, we distribute weights in uniform distribution. We set weights of all instances to 1/n n total number of instances. 24

25 Adaboost :Example2 ●We will place predictions as a column ●We check the equality of actual and prediction in loss column. It will be 0 if the prediction is correct, will be 1 if the prediction is incorrect. Iteration : 1 25

26 Adaboost:Example3 Sum of weight = 0.3 alpha = ln[(1-epsilon)/epsilon] / 2 = ln[(1 – 0.3)/0.3] / 2= 0.42 We’ll use alpha to update weights in the next round. w i+1 = w i * math.exp(-alpha * actual * prediction): i refers to instance number. Sum of weights must be equal to 1. We have to normalize weight values. Dividing each weight value to sum of weights column enables normalization. End Iteration : 1 ! 26

27 Adaboost: Example4 Prediction of the 1st instance will be : 0.42 x 1 + 0.65 x (-1) + 0.38 x 1 + 1.1 x 1 = 1.25 And we will apply sign function : Sign(0.25) = +1 true which is correctly classified. Iteration : N 27

28 Gradient Boosting Gradient boosting is a sequential model in which the error is reduced with the help of gradient descent and makes a model in a form of a base model. 28

29 Gradient Boosting ● A large and growing list of data scientists globally that are actively contributing to XGBoost open source development. ● Cloud integration that supports AWS, Azure, Yarn clusters, and other ecosystemsYarn clusters ● A library that was built from the ground up to be efficient, flexible, and portable 29

30 03 Boosting Algorithms Demo code 30

31 31

32 Stacking 04 32

33 Stacking is one of the popular ensemble modeling techniques in machine learning. Various weak learners are ensembled in a parallel manner in such a way that by combining them with Meta learners, we can perform better predictions for the future.

34 Stacking Algorithms 04 34

35 Staking 35

36 Conclusion 05 36

37 Bagging Vs Boosting Similarities : ➔ Both use voting ➔ Both sample to generate multiple datasets randomly ➔ Both ensemble techniques deduce N learners from solitary learner ➔ Both help lowering the model variance and generating stable models 37

38 Bagging Vs Boosting BaggingBoosting Individual models are built separatelyEach new model in influenced by the performance of the previous ones Parallel ensemble learning methodSequential ensemble learning method Gives equal weights to each modelThe models’ weights is based on their result New subsets of data used for training contain observations that the previous model misclassified Uses randomly generated training data subsets Differences 38

39 Two heads are better than one 39

40 The ideas, concepts and diagrams are taken from the following websites: ●https://quantdare.com/what-is-the-difference-between-bagging-and-boosting/https://quantdare.com/what-is-the-difference-between-bagging-and-boosting/ ●https://medium.com/swlh/difference-between-bagging-and-boosting- f996253acd22https://medium.com/swlh/difference-between-bagging-and-boosting- f996253acd22 ●https://www.geeksforgeeks.org/comparison-b-w-bagging-and-boosting-data- mining/https://www.geeksforgeeks.org/comparison-b-w-bagging-and-boosting-data- mining/ ●https://hub.packtpub.com/ensemble-methods-optimize-machine-learning- models/https://hub.packtpub.com/ensemble-methods-optimize-machine-learning- models/ ●https://towardsdatascience.com/decision-tree-ensembles-bagging-and- boosting-266a8ba60fd9https://towardsdatascience.com/decision-tree-ensembles-bagging-and- boosting-266a8ba60fd9 ●https://dropsofai.com/bagging-boosting-and-stacking-in-machine-learning/https://dropsofai.com/bagging-boosting-and-stacking-in-machine-learning/ ●https://machinelearningmastery.com/bagging-ensemble-with-python/ [References] 40

41 Thanks! Do you have any questions? 41