Up until now we have learned the basics of decision trees. In this lesson we will learn some more advanced algorithms which improve upon the basic decision tree algorithm and are some of the most commonly used algorithms in industry today. There are two extensions of the decision tree algorithm we will learn about, random forests and gradient boosted trees.

Random Forests

Random forests are an extension of the decision tree algorithm that use a collection of decision trees to make predictions. The idea is that each decision tree in the collection will make a prediction and the final prediction will be the average of all of the predictions. This is called ensemble learning and is a common technique in machine learning. The idea is that the average of many predictions will be more accurate than any single prediction.

Let's see how this works in practice. We will use the same iris dataset we have been using in the previous lessons. We will use the RandomForestRegressor class in the scikit-learn's ensemble library to create a random forest model. We will use the same iris dataset as before, including the other petal and sepal measures as features. We will also use the same predict function to make predictions. The main difference then is that we will use the RandomForestClassifier class instead of the DecisionTreeClassifier class.

• Lines 2 & 3: Prepare the same data we used in the advanced linear regression lesson.
• Line: Create the random forest model. We need to specify the number of trees in the forest using the n_estimators parameter. We will use 5 trees, and since there is some randomness to this model we also give specify the random state so that we will consistently get the same results when we run this code.
• Line 9: We fit the model.
• Lines 12 & 13: We make predictions using this model and calculate the mean squared error.

We see that the mean squared error is quite good at 0.07 which beats even the advanced linear regression model we created before, which was 0.1. Let's take a look at the inner workings of this model now. As before we will use the plot_tree function to visualize the tree. However, since this is an ensemble of trees, we need to access each individual tree separately, and we will do that using the estimators_ attribute of the model and display the results for the first tree in the ensemble. Feel free to change the index to see the other trees, you will notice they split the data in different ways.

As we have seen, the random forest model makes many trees, that's where the forest part of the name comes from. The random part of the name comes from the fact that each tree is trained on a random subset of the data. This is called bootstrap aggregating (bagging for short) and is a common technique in machine learning. Once all of the trees are fit, then they are combined by averaging the predictions of each tree to make the final prediction, this is the ensemble learning.

Gradient Boosted Trees

Gradient boosted trees are another extension of the decision tree algorithm. Gradient boosted trees are also an ensemble learning algorithm, but instead of using many trees, gradient boosted trees use a single tree. The idea is that the tree is trained in a series of steps, and at each step the tree is trained to correct the mistakes of the previous step. This is called boosting and is another common technique in machine learning. Let's take a look at this method in code.

• Line 6: We specify there to be 5 steps in the boosting process, with each tree having a depth of 3, and use the same random state as before.
• All other lines: Are the same as before.

The mean squared error here is 1.14, which is the worst of all the models, however the gradient boosted tree model is more often the most accurate of all the models used in practice, the reason for why it underperformed here is because this model really needs more trees, depth, and data to perform well.

As before, feel free to increase the increment to see the other trees in the ensemble. Usually gradient boosted trees are more complex than the trees in the random forest model. This is because the trees are trained to correct the mistakes of the previous tree, so each tree is more complex than the previous tree. This is the boosting part of the name. We don't see this here since we limited the depth to 3.