Gradient Boosting Models

What is Gradient Boosting?

Gradient boosting builds an ensemble of trees, where each new tree tries to fix the mistakes of the previous ones. It can make very accurate models for both regression and classification.

Common uses:

• Predicting categories or numbers with complex data
• Winning machine learning competitions

Why Use Gradient Boosting?

• Very accurate
• Works for both regression and classification
• Can handle many types of data

How Does Gradient Boosting Work?

• Builds trees one after another
• Each tree tries to correct the errors of the previous trees
• Combines all trees for the final prediction

Key Parameters in scikit-learn

Step-by-Step Example: Gradient Boosting for Regression

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error
import numpy as np

# Example data
y = np.array([1, 2, 3, 4, 5])
X = np.arange(5).reshape(-1, 1)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

# Create and fit Gradient Boosting
gb = GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=0)
gb.fit(X_train, y_train)

# Predict and evaluate
preds = gb.predict(X_test)
mse = mean_squared_error(y_test, preds)
print('Predictions:', preds)
print('MSE:', mse)

Explanation:

• GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, max_depth=3): Builds 100 trees, each correcting the last.
• fit(X_train, y_train): Trains the model.
• predict(X_test): Makes predictions.

How is Gradient Boosting Different from Random Forests?

• Trees are built one after another, not all at once
• Each tree focuses on fixing previous mistakes
• Can be more accurate, but also more sensitive to parameters

Visualizing the Gradient Boosting Process

Differences Between Major Gradient Boosting Libraries

Summary:

• Use sklearn's GradientBoostingRegressor for learning and small projects.
• Use XGBoost or LightGBM for large datasets and competitions.
• Use CatBoost if you have lots of categorical features.
• Use HistGradientBoosting for fast, scalable boosting in sklearn.

See the individual pages for each library for detailed examples and explanations!