Artificial intelligence

Regression (Supervised Learning)

Statistical method used to model the relationship between a dependent (target or outcome) variable and one or more independent (predictors or features) variables

Predicts the continuous output variables based on the independent input variable

Goal is to understand how changes in the independent variables are associated with changes in the dependent variable and to make predictions about the dependent variable based on known values of the independent variables

• Application:
• Sales forecasting
• Satisfaction analysis
• Price estimation
• Employment income

• Regression models:
• Linear Regression
• Polynomial Regression
• Ridge Regression (L2 Regularization)
• Lasso Regression (L1 Regularization)
• Elastic Net Regression
• Support Vector Regression (SVR)
• Decision Tree Regression
• Random Forest Regression
• Gradient Boosting Regression
• XGBoost Regression
• K-Nearest Neighbors (KNN) Regression
• Bayesian Regression
• Principal Component Regression (PCR)
• Partial Least Squares Regression (PLSR)
• Quantile Regression

Assessing Performance

• Cross-Validation

Statistical method used in machine learning and data science to assess the performance of a model by splitting the data into multiple subsets (folds) and training/testing the model on different combinations of these subsets.

The purpose is to provide a more accurate and robust estimate of the model’s performance on unseen data, helping to prevent overfitting and improve generalization

• K-Fold Cross-Validation
• dataset is randomly split into k equally sized folds or subsets
• model is trained on k−1 folds and tested on the remaining fold, rotating through all folds so that each fold serves as a test set once
• final performance score is the average of the scores from each fold, providing a reliable estimate of model accuracy
• Leave-One-Out Cross-Validation (LOOCV)
• k equals the number of data points
• model is trained on all data points except one, and this process is repeated for each data point in the dataset
• provides an unbiased performance estimate but is computationally intensive for large datasets
• Stratified Cross-Validation
• useful for imbalanced datasets, as it maintains the same proportion of classes (for classification problems) in each fold as in the full dataset, ensuring consistent class distribution across training and validation sets
• Nested Cross-Validation
• used when hyperparameter tuning is involved. It has two levels of cross-validation: the outer loop estimates model performance, while the inner loop is used for hyperparameter tuning, ensuring an unbiased evaluation of tuned models

from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LinearRegression
from sklearn.datasets import make_regression

# Sample data
X, y = make_regression(n_samples=100, n_features=2, noise=0.1)

# Model
model = LinearRegression()

# KFold Cross-Validation
kfold = KFold(n_splits=5)
results = cross_val_score(model, X, y, cv=kfold, scoring='neg_mean_squared_error')

# Average MSE across folds
average_mse = -results.mean()
print("Average MSE:", average_mse)
                    

Copy Code

model_selection.KFold K-Fold cross-validator creates number of k-fold splits, allowing cross validation

• n_splits - number of folds, must be at least 2

model_selection.cross_val_score evaluates model's score through cross validation

• estimator - the object to use to fit the data
• X, y - the data to fit, the target variable to try to predict in the case of supervised learning
• cv - determines the cross-validation splitting strategy (None, int, CV splitter)
• scoring - a str or a scorer callable object / function with signature scorer (estimator, X, y) which should return only a single value

model_selection.cross_val_predict produces the out-of-bag prediction for each row

model_selection.GridSearchCV scans over parameters to select the best hyperparameter set with the best out-of-sample score

Regularization

Technique used to prevent overfitting by penalizing high-valued coefficients (reduces parameters and shrinks the model)

Specially useful in complex models like high-degree polynomial regression, deep neural networks, and decision trees, where the risk of overfitting is high

Modifies the cost function used to train the model by adding a penalty term, typically multiplied by a hyperparameter λ (regularization strength)

Regularization techniques have an analytical, a geometric, and a probabilistic interpretation

• Ridge (L2 Regularization)

\[ \text{L2 penalty} = \lambda \sum_{i=1}^{n} w_i^2 \]

• penalizes the size magnitude of the regression coefficients by adding a squad term
• complexity penalty λ is applied proportionally to squared coefficient values
• enforces the coefficients to be lower, but not 0
• minimizes irrelevant features and does not remove them
• faster to train
• this imposes bias on the model, but also reduces variance
• we can select the best regularization strength lambda via cross-validation
• it’s a best practice to scale features (i.e. using StandardScaler) so penalties aren’t impacted by variable scale
• LASSO (L1 Regularization) - Least Absolute Shrinkage and Selection Operator

\[ \text{L1 penalty} = \lambda \sum_{i=1}^{n} |w_i| \]

• penalizes the absolute value of the coefficients
• complexity penalty λ is proportional to the absolute value of coefficients
• sets irrelevant features to 0
• finds features you don't need
• LASSO is more likely than Ridge to perform feature selection, for a fixed λ LASSO is more likely to result in coefficients being set to zero
• LASSO’s feature selection property yields an interpretability advantage, but may underperform if the target truly depends on many of the features
• Elastic Net (L1+L2 Regularization)

\[ \text{Elastic Net penalty} = \lambda_1 \sum_{i=1}^{n} |w_i| + \lambda_2 \sum_{i=1}^{n} w_i^2 \]

• combines both L1 and L2 regularization, balancing between feature selection (L1) and feature shrinkage (L2)
• penalizes the size magnitude of the regression and absolute value of the coefficients
• sets irrelevant features to 0 and enforces the coefficients to be lower
• introduces a new parameter α (alpha) that determines a weighted average of L1 and L2 penalties
• Dropout Regularization (for Neural Networks)
• temporarily “drops out” random neurons during each training step by setting their outputs to zero with a specified probability
• prevents any single neuron from becoming too important, encouraging the network to learn redundant representations and reducing overfitting
• Early Stopping
• training is stopped as soon as the model’s performance on a validation set begins to deteriorate (prevents the model from learning noise in the training data)

Evaluation

• Mean Absolute Error (MAE) - Computes the average absolute differences between the predicted and actual values. It provides a measure of the average magnitude of errors. Treats all errors equally by taking their absolute values.

\[ MAE = \frac{1}{n} \sum_{i=1}^{n} \left| y_i - \hat{y}_i \right| \]

• \[n: \text{Total number of observations}\]

• \[y_i: \text{Actual value for the}\] \[i^{th} \text{ data point}\]

• \[\hat{y}_i: \text{Predicted value for the}\] \[i^{th} \text{ data point}\]

• \[\left| y_i - \hat{y}_i \right|: \text{Absolute error for the}\] \[i^{th} \text{ prediction}\]

• Properties:
• Can take any non-negative value
• A perfect model will have an MAE of 0, meaning no error between actual and predicted values
• Gives equal importance to small and large errors, making it less sensitive to outliers
• Retains the same unit as the dependent variable, providing a direct interpretation of the average error
• Advantages/Disadvantages:
• A: Robust to Outliers - it doesn’t square the errors
• A: Simple to Understand and Calculate
• D: Not Differentiable at Zero
• D: Insensitive to Large Errors - may not penalize large errors
• When to use:
• When Outliers are Present - If the dataset contains outliers or noisy data
• When an easily interpretable error metric is needed in the same unit as the target variable
• Often used in time series forecasting to evaluate how close predictions are to the observed values
• How it works

Actual (y)	Predicted (y_hat)	Error (y-y_hat)	Absolute Error
3.0	2.5	0.5	0.5
5.0	4.5	0.5	0.5
2.0	2.5	-0.5	0.5
7.0	6.0	1.0	1.0

• Calculate Errors: For each data point, find the difference between the actual value and the predicted value
• Take the Absolute Value: Ignore whether the error is positive or negative by taking the absolute value of each error
• Average the Absolute Errors: Sum all the absolute errors and divide by the number of observations
• 0.5 + 0.5 + 0.5 + 1.0 = 2.5 / 4 = 0.625 - on average, the model’s predictions deviate from the actual values by 0.625 units
• Mean Squared Error (MSE) - Squares the differences between predicted and actual values before averaging, emphasizing larger errors. It captures how well a model’s predictions align with the true outcomes, with more weight given to larger errors due to squaring.

\[ MSE = \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2 \]

• \[n: \text{Total number of observations}\]

• \[y_i: \text{Actual value for the}\] \[i^{th} \text{ data point}\]

• \[\hat{y}_i: \text{Predicted value for the}\] \[i^{th} \text{ data point}\]

• \[\left( y_i - \hat{y}_i \right)^2: \text{Squared error for the}\] \[i^{th} \text{ prediction}\]

• Properties:
• Always non-negative
• A perfect model has an MSE of 0, indicating no difference between actual and predicted values
• Penalizes large deviations more than small ones, making it sensitive to outliers
• Expressed in the square of the unit of the target variable
• Advantages/Disadvantages:
• A: Popular in Machine Learning - used extensively in model evaluation and optimization
• A: Mathematically Convenient - differentiable, which makes it suitable for gradient-based optimization techniques to minimize error during training
• D: Sensitive to Outliers - large errors have a significant impact on MSE, making it less robust to datasets with noisy data or outliers
• D: Squared units of MSE can make it harder to interpret
• When to use:
• Training Machine Learning Models - common loss function for many algorithms
• Evaluating Models - provides insight into the magnitude of the average squared error
• When Large Errors are Critical
• How it works

Actual (y)	Predicted (y_hat)	Error (y-y_hat)	Squared Error
3.0	2.5	0.5	0.25
5.0	4.5	0.5	0.25
2.0	2.5	-0.5	0.25
7.0	6.0	1.0	1.00

• Calculate Errors: Subtract the predicted value from the actual value for each observation
• Square the Errors: Squaring ensures that all errors are positive and emphasizes larger errors more than smaller ones
• Average the Squared Errors: Sum the squared errors and divide by the total number of observations
• 0.25 + 0.25 + 0.25 + 1.00 = 1.75 / 4 = 0.4375
• Example in Python

from sklearn.metrics import mean_squared_error
import numpy as np

# Example usage
y_true = np.array([3.0, -0.5, 2.0, 7.0])
y_pred = np.array([2.5, 0.0, 2.1, 7.8])

# Calculate MSE with sklearn
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)

# Define function for MSE
def mse(y_true, y_pred):
    return np.mean((y_true - y_pred) ** 2)

# Calculate MSE
error = mse(y_true, y_pred)
print("Mean Squared Error:", error)
                        

Copy Code

metrics.mean_squared_error(y_true, y_pred) - mean squared error regression loss

• y_true: ground truth (correct) target values
• y_pred: estimated target values
• Root Mean Squared Error (RMSE) - The square root of Mean Squared Error (MSE) shares the same unit as the target variable, enhancing its interpretability

\[ RMSE = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2 } \]

• \[n: \text{Total number of observations}\]

• \[y_i: \text{Actual value for the}\] \[i^{th} \text{ data point}\]

• \[\hat{y}_i: \text{Predicted value for the}\] \[i^{th} \text{ data point}\]

• Properties:
• Can take any non-negative value
• A perfect model would have an RMSE of 0, meaning there is no difference between actual and predicted values
• Gives higher weight to large errors because of the squaring step, this makes it sensitive to outliers
• Retains the same unit of measurement as the dependent variable
• Advantages/Disadvantages:
• A: Easy Interpretation - same units as the target variable
• A: Effective for Penalizing Large Errors - particularly useful when large errors are undesirable and need to be minimized
• D: Sensitive to Outliers - single large error can disproportionately increase the RMSE
• D: Not Robust - if the data contains noise or outliers, RMSE may not be the best performance measure
• D: Comparison Limitation - it is difficult to compare RMSE across different datasets with varying scales
• When to use:
• Regression Models
• Forecasting - helps in understanding how much the forecast deviates from the observed values
• Model Selection - often used to compare models, the one with the lowest RMSE is considered better
• How it works

Actual (y)	Predicted (y_hat)	Error (y-y_hat)	Squared Error
3.0	2.5	0.5	0.25
5.0	4.5	0.5	0.25
2.0	2.5	-0.5	0.25
7.0	6.0	1.0	1.00

• Calculate Errors: For each data point, find the difference between the actual value and the predicted value
• Square the Errors: Ensures that negative errors don’t cancel out positive ones and gives more weight to larger errors
• Average the Squared Errors: Compute the mean of these squared errors
• Take the Square Root: The square root brings the result back to the original unit of measurement
• 0.25 + 0.25 + 0.25 + 1.00 = 1.75 / 4 = 0.4375 ^ 0.5 = 0.661
• Example in Python

import numpy as np
from sklearn.metrics import mean_squared_error

# Example usage
y_true = np.array([3.0, -0.5, 2.0, 7.0])
y_pred = np.array([2.5, 0.0, 2.1, 7.8])

# Calculate RMSE with sklearn
rmse = np.sqrt(mean_squared_error(y_true, y_pred))
print("Root Mean Squared Error:", rmse)

# Define function for RMSE
def rmse(y_true, y_pred):
    return np.sqrt(np.mean((y_true - y_pred) ** 2))

# Calculate RMSE
error = rmse(y_true, y_pred)
print("Root Mean Squared Error:", error)
                        

Copy Code
• R-squared (R2) - Quantifies the predictability of the dependent variable based on the independent variables, ranging from 0 to 1, with 1 denoting flawless predictions. The coefficient of determination represents statistical measure that indicates the proportion of variance in the dependent (target) variable that is explained by the independent (predictor) variables in a regression model.

\[ R^2 = 1 - \frac{SS_{res}}{SS_{tot}} \]

• \(SS_{res} = \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2\) : Residual Sum of Squares (the sum of squared errors between actual and predicted values).

• \(SS_{tot} = \sum_{i=1}^{n} \left( y_i - \bar{y} \right)^2\) : Total Sum of Squares (the total variance of the actual values from their mean).

• \[y_i: \text{Actual value for the}\] \[i^{th} \text{ data point}\]

• \[\hat{y}_i: \text{Predicted value for the}\] \[i^{th} \text{ data point}\]

• \[\bar{y}: \text{Mean of the actual values}\]

• Interpretation:
• R² = 1: model explains all the variance in the target variable (perfect fit)
• R² = 0: model explains none of the variance, meaning predictions are no better than simply using the mean of the actual values
• R² < 0: indicates that the model performs worse than a simple mean-based model. It can happen if the predictions are far from the actual values
• Advantages/Disadvantages:
• A: Easy Interpretation - provides a clear percentage of how much variation in the target variable is explained by the model
• A: Model Comparison - helps compare multiple models, model with a higher R² is generally better at explaining variance in the target variable
• A: Widely Used - tandard measure in regression analysis and is useful for determining model quality
• D: Does Not Measure Predictive Accuracy - high R² does not guarantee that the model will make accurate predictions for new data
• D: Sensitive to the Number of Predictors - adding more predictors can artificially inflate R²
• D: Only Works with Linear Relationships - may be misleading if the relationship between variables is non-linear
• How it works

Actual (y)	Predicted (y_hat)	Mean of Actual ()
3.0	2.5	4.25
5.0	4.5	4.25
2.0	2.5	4.25
7.0	6.0	4.25

• Calculate Total Variance: measures how much the actual values vary from the mean. It reflects the overall variability in the data
• Calculate Residual Variance: measures how far the predictions are from the actual values (prediction errors)
• Compute R²: compares the proportion of unexplained variance (residual variance) to the total variance. If the residual variance is small compared to the total variance, R² will be close to 1, indicating that the model fits the data well.
• SStot: (3.0 - 4.25) ^ 2 + (5.0 - 4.25) ^ 2 + (2.0 - 4.25) ^ 2 + (7.0 - 4.25) ^ 2 = 17.0
• SSres: (3.0 - 2.5) ^ 2 + (5.0 - 4.5) ^ 2 + (2.0 - 2.5) ^ 2 + (7.0 - 6.0) ^ 2 = 2.75
• R²: 1 - (2.75 / 17.0) = 0.838 - meaning the model explains 83.8% of the variance in the target variable
• Example in Python

import numpy as np
from sklearn.metrics import r2_score

# Example true and predicted values
y_true = [3.0, -0.5, 2.0, 7.0]
y_pred = [2.5, 0.0, 2.1, 7.8]

# Calculate R-squared
r_squared = r2_score(y_true, y_pred)
print("R-squared:", r_squared)

# Convert to numpy arrays
y_true = np.array(y_true)
y_pred = np.array(y_pred)

# Calculate the mean of the true values
y_mean = np.mean(y_true)

# Calculate the total sum of squares (TSS) and residual sum of squares (RSS)
ss_total = np.sum((y_true - y_mean) ** 2)
ss_residual = np.sum((y_true - y_pred) ** 2)

# Calculate R-squared
r_squared = 1 - (ss_residual / ss_total)
print("R-squared:", r_squared)
                        

Copy Code

metrics.r2_score(y_true, y_pred) - R2 (coefficient of determination) regression score function (best possible score is 1.0)

• y_true: ground truth (correct) target values
• y_pred: estimated target values
• Adjusted R-squared - An adjustment of R-squared that penalizes the addition of unnecessary predictors, providing a better measure of model complexity

\[ R^2_{adj} = 1 - \left( \frac{SS_{res} / (n - k - 1)}{SS_{tot} / (n - 1)} \right) \]

• \(SS_{res} = \sum_{i=1}^{n} \left( y_i - \hat{y}_i \right)^2\) : Residual Sum of Squares.

• \(SS_{tot} = \sum_{i=1}^{n} \left( y_i - \bar{y} \right)^2\) : Total Sum of Squares.

• \[n: \text{Total number of observations}\]

• \[k: \text{Total number of predictors}\]

• Interpretation:
• Adjusted R² will be smaller than R² if additional predictors do not significantly improve the model
• Example in Python

import numpy as np
from sklearn.metrics import r2_score

# Example true and predicted values
y_true = [3.0, -0.5, 2.0, 7.0]
y_pred = [2.5, 0.0, 2.1, 7.8]

# Calculate R-squared
r_squared = r2_score(y_true, y_pred)
print("R-squared:", r_squared)

# Number of observations and predictors
n = len(y_true)  # number of data points
p = 1  # number of predictors (set to 1 for simplicity, adjust for more features)

# Calculate Adjusted R-squared
r_squared_adjusted = 1 - ((1 - r_squared) * (n - 1) / (n - p - 1))
print("Adjusted R-squared:", r_squared_adjusted)
                        

Copy Code

Linear Regression (simple / multiple)

Model the relationship between a dependent (response/target) continuous variable and one or more independent (predictors/features) variables by fitting a linear equation to observed data

Simple Linear Regression involves a single independent variable (relationship between this variable and the target is represented as a straight line)

Multiple Linear Regression extends simple linear regression by including multiple predictors

Used when relationship between the dependent and independent variables is linear, dataset is small to medium-sized and does not have too many complex features, outliers are minimal and the assumptions of linearity and homoscedasticity are reasonable

• Fitting a Linear Regression model (cost function???)

Ordinary Least Squares (OLS) minimizes the sum of squared residuals (errors). The sum of squared differences between observed values and predicted values

\( \text{Minimize } \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 \)

• \[y_i\] : actual value

• \[\hat{y}_i\] : predicted value

• Assumptions
• Linearity: Relationship between predictors and the target variable is linear
• Independence: Observations are independent of each other
• Homoscedasticity: Constant variance of errors across values of the independent variables
• Normality of Errors: Residuals (differences between observed and predicted values) are normally distributed
• Applications
• Finance: Predicting stock prices or return on investment based on market indicators
• Healthcare: Estimating health outcomes based on risk factors
• Economics: Modeling economic indicators like GDP growth or unemployment rates
• Marketing: Predicting sales based on advertising expenditure or customer demographics
• Limitations
• Assumes Linearity: Poor performance if the relationship is non-linear
• Sensitive to Outliers: Outliers can disproportionately affect the model
• Assumes Homoscedasticity: If the variance of errors varies, the model’s predictions are likely biased
• Multicollinearity: High correlation between independent variables can lead to unreliable estimates
• Example in Python

import numpy as np
import pandas as pd

from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score

df = pd.DataFrame(data)

# Define predictor and response variable
X = df[['Advertising']]
y = df['Sales']

# Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and fit the model
model = LinearRegression()
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)

# Evaluation
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print("Mean Squared Error:", mse)
print("R-squared:", r2)
                    

Copy Code

linear_model.LinearRegression - ordinary least squares Linear Regression

• coef_ - estimated coefficients for the linear regression problem
• intercept_ - independent term in the linear model
• fit(X, y) - fit linear model, X: training data; y: target values
• predict(X) - predict using the linear model
• score(X, y) - return the coefficient of determination R2 of the prediction

Polynomial Regression (simple / multiple)

Type of regression analysis in which the relationship between the independent variable (predictor) and the dependent variable (response) is modeled as an n-th degree polynomial

Polynomial regression is capable of capturing non-linear relationships by fitting a curved line to the data

As the degree n increases, the curve becomes more flexible, allowing the model to capture more complex patterns in the data

Used when relationship between variables is non-linear, but still smooth (can be captured by a polynomial curve)

• Fitting a Polynomial Regression model (cost function)

Ordinary Least Squares (OLS) minimizes the sum of squared residuals (errors). The sum of squared differences between observed values and predicted values

\( \text{Minimize } \sum_{i=1}^{n} \left( y_i - \left( \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \cdots + \beta_n x_i^n \right) \right)^2 \)

• \[y_i\] : actual value

• \[\hat{y}_i\] : predicted value

• Steps
• Choose the Degree of the Polynomial: The degree n determines the complexity of the curve
• Transform the Independent Variable: The original variable x is expanded to include higher powers (x, x^2, x^3...)
• Fit the Model Using Linear Regression: Treat the transformed terms as new features and apply ordinary least squares (OLS) regression
• Evaluate and Refine: Check the model’s performance, possibly adjusting the polynomial degree to balance bias and variance
• Choosing the Degree
• Underfitting: When the degree is too low, the model might not capture the underlying trend in the data, leading to high bias
• Overfitting: When the degree is too high, the model might capture noise or random fluctuations in the data rather than the underlying pattern, leading to high variance
• Optimal Degree: Cross-validation or evaluating metrics on test data can help determine the optimal polynomial degree, balancing bias and variance (bias-variance tradeoff)
• Assumptions
• Linearity of Parameters: Although the model itself is non-linear, it is linear in terms of the parameters (coefficients)
• Independence of Errors: Observations are independent of each other
• Homoscedasticity: Constant variance of errors across values of the independent variables
• Normality of Errors: Residuals (differences between observed and predicted values) are normally distributed
• Applications
• Economics: Modeling complex relationships between economic indicators
• Environmental Science: Modeling growth patterns, such as population growth over time
• Engineering: Estimating stress-strain relationships in material science
• Marketing: Understanding the impact of advertising spending on sales, where effects are non-linear
• Advantages
• Captures Non-Linear Relationships: By introducing polynomial terms, the model can represent curved patterns in the data
• Flexibility: Polynomial regression is more flexible than simple linear regression, making it suitable for data with a non-linear trend
• Easy to Implement: Can be implemented using simple linear regression tools, with the only modification being the transformation of the input variables
• Limitations
• Overfitting: High-degree polynomials can overfit the training data, leading to poor generalization to new data
• Sensitive to Outliers: Polynomial regression is particularly sensitive to outliers, which can skew the polynomial curve
• Extrapolation Challenges: Predictions outside the range of the data can be highly unreliable, as polynomial regression is often unstable in these regions
• Complexity Increases Quickly: As the polynomial degree increases, the model complexity increases, making it harder to interpret
• Example in Python

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
from sklearn.metrics import mean_squared_error

# Sample data
x = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9]).reshape(-1, 1)
y = np.array([1, 4, 9, 16, 25, 36, 49, 64, 81])

# Transform the data to include polynomial terms (e.g., x, x^2)
poly = PolynomialFeatures(degree=2)  # Change degree as needed
x_poly = poly.fit_transform(x)

# Fit the polynomial regression model
model = LinearRegression()
model.fit(x_poly, y)

# Predict values
y_pred = model.predict(x_poly)

# Evaluation
mse = mean_squared_error(y, y_pred)
print("Mean Squared Error:", mse)

# Plotting the results
plt.scatter(x, y, color='blue', label='Actual data')
plt.plot(x, y_pred, color='red', label='Polynomial regression fit')
plt.xlabel('X')
plt.ylabel('Y')
plt.legend()
plt.show()
                    

Copy Code

preprocessing.PolynomialFeatures - generate polynomial and interaction features

• degree - int specifies the maximal degree of the polynomial features, or tuple for (min_degree, max_degree)
• fit_transform - fit to data, then transform it (X: input samples; y: target values)

Ridge Regression (L2 Regularization)

Linear regression technique that includes a regularization term to mitigate overfitting, especially when working with multicollinear data

Penalty is added to the linear regression objective function based on the squared magnitude of the coefficients, effectively “shrinking” coefficients towards zero but never allowing them to be exactly zero

Used for scenarios with multicollinearity among features, high-dimensional data where overfitting is a concern, situations where interpretability is less critical than predictive performance

• How Ridge Regression Works
• reduces the model’s variance by shrinking the coefficient estimates, making the model less sensitive to small fluctuations in the data
• increasing 𝜆 introduces more bias, the goal is to find a balance that minimizes overall prediction error
• when 𝜆 = 0 Ridge Regression is equivalent to ordinary least squares regression since no penalty is applied
• as 𝜆 approaches infinity the model shrinks all coefficients towards zero, resulting in a model that only predicts the mean of the target variable
• the optimal value for 𝜆 is determined through cross-validation
• in datasets where features are correlated (multicollinear), Ridge Regression reduces the variance caused by multicollinearity by penalizing large coefficients
• Ridge does not perform feature selection, so all features remain in the model with adjusted coefficients
• Applications
• Credit Risk Scoring: Predicting the probability of loan default with many features (e.g., customer demographics, financial history)
• Marketing Campaign Response: Estimating customer response to an email campaign when some features (e.g., past purchases) are correlated
• Biological Research: Identifying significant genes for a disease from thousands of genetic features (Lasso helps with feature selection)
• Advantages
• Reduces Overfitting: The L2 penalty stabilizes the model by controlling high-variance coefficients
• Works with High-Dimensional Data: Performs well when there are more features than observations
• Closed-Form Solution: For simple cases, Ridge Regression has a closed-form solution, making it computationally efficient
• Disadvantages
• Does Not Perform Feature Selection: Unlike Lasso Regression, Ridge does not eliminate features, which can lead to less interpretable models
• Sensitivity to Feature Scaling: Like many distance-based techniques, Ridge Regression is sensitive to the scale of features, so standardization or normalization is typically required
• Example in Python

from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate or load data
X = np.random.rand(100, 3)  # example features
y = X @ np.array([3, 5, 2]) + np.random.normal(0, 1, 100)  # example target with added noise

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Scale the features (important for regularized models)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Fit the Ridge Regression model
ridge = Ridge(alpha=1.0)  # alpha is the regularization parameter (lambda)
ridge.fit(X_train_scaled, y_train)

# Predict and evaluate the model
y_pred = ridge.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)
print("Model Coefficients:", ridge.coef_)
                    

Copy Code

linear_model.Ridge - linear least squares with l2 regularization

• alpha - constant that multiplies the L2 term, controlling regularization strength
• coef_ - weight vector(s)
• intercept_ - independent term in decision function
• fit(X, y) - fit Ridge regression model, X: training data; y: target values
• predict(X) - predict using the linear model
• score(X, y) - return the coefficient of determination R2 of the prediction

Lasso Regression (L1 Regularization) - Least Absolute Shrinkage and Selection Operator

Regression technique that introduces a penalty equal to the absolute value of the magnitude of coefficients

Uses an L1 penalty (the sum of absolute values of coefficients) that results in feature selection as it drives some coefficients to zero, effectively removing less important features from the model

Used when feature selection is required (high-dimensional datasets with many irrelevant features), sparse solutions are desirable (only a few significant features should have non-zero coefficients), data overfitting is a concern

• How lasso Regression Works
• can reduce some coefficients to exactly zero that is particularly helpful in sparse datasets where only a subset of features influences the target variable
• By tuning 𝜆 you can control the number of features retained, balancing interpretability with predictive performance
• increasing 𝜆 introduces more bias, as it forces the model to generalize by shrinking coefficients
• Lowering 𝜆 allows more variance in the coefficients, but this can lead to overfitting if there is high noise in the data
• If the data has highly correlated features, Lasso may arbitrarily select only one of them, ignoring the others
• Applications
• Credit Risk Scoring: Predicting the probability of loan default with many features (e.g., customer demographics, financial history)
• Marketing Campaign Response: Estimating customer response to an email campaign when some features (e.g., past purchases) are correlated
• Biological Research: Identifying significant genes for a disease from thousands of genetic features (Lasso helps with feature selection)
• Advantages
• Feature Selection and Sparsity: Lasso automatically performs feature selection, producing simpler and more interpretable models
• Prevents Overfitting: The regularization term reduces the risk of overfitting by penalizing complex models
• Useful for High-Dimensional Data: Particularly helpful in scenarios with many irrelevant features
• Disadvantages
• Feature Exclusion with Correlated Data: Lasso can exclude correlated features, leading to biased feature importance
• Sensitivity to Feature Scaling: Lasso requires standardized or normalized data to perform effectively
• Example in Python

from sklearn.linear_model import Lasso
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate or load sample data
X = np.random.rand(100, 3)  # example features
y = X @ np.array([3, 5, 0]) + np.random.normal(0, 1, 100)  # target with sparse signal

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Scale the features (important for regularized models)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Fit the Lasso Regression model
lasso = Lasso(alpha=1.0)  # alpha is the regularization parameter (lambda)
lasso.fit(X_train_scaled, y_train)

# Predict and evaluate the model
y_pred = lasso.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)
print("Model Coefficients:", lasso.coef_)
                    

Copy Code

linear_model.Lasso - linear model trained with L1 prior as regularizer

• alpha - constant that multiplies the L1 term, controlling regularization strength
• coef_ - weight vector(s)
• intercept_ - independent term in decision function
• fit(X, y) - fit Lasso regression model, X: training data; y: target values
• predict(X) - predict using the linear model
• score(X, y) - return the coefficient of determination R2 of the prediction

K-Nearest Neighbors (KNN) Regression

Non-parametric, instance-based algorithm that predicts the value of a target variable by averaging the values of the 𝑘-closest data points (neighbors) to a given input point

Doesn’t assume any underlying distribution of the data or a linear relationship between variables (flexible choice, especially for non-linear datasets)

Used when the data has local patterns and you need a non-parametric model, works well with small datasets but struggles with large datasets

• Key Concepts
• relies on measuring the distance between data points to identify neighbors, commonly using the Euclidean distance
• Manhattan, Minkowski, or Mahalanobis can also be used depending on the nature of the data and dimensionality
• parameter 𝑘 represents the number of closest points used to make predictions, smaller 𝑘 values make the model sensitive to noise (high variance), while larger 𝑘 values lead to smoother predictions but can underfit (high bias)
• typically 𝑘 is chosen through cross-validation, where the optimal value minimizes the prediction error on a validation set
• for a given input x KNN regression identifies the 𝑘-nearest neighbors, retrieves their target values, and averages (makes KNN regression robust to outliers, as each prediction is influenced by multiple data points) them to produce the prediction
• in weighted KNN neighbors closer to the query point have a higher influence on the prediction (particularly useful when neighbors vary significantly in their distance from the query point)
• Applications
• House Rent Estimation: Estimating rent based on nearby properties with similar characteristics
• User Rating Prediction: Predicting product ratings by aggregating the ratings of similar users
• Restaurant Recommendation: Predicting restaurant ratings for a user based on their similarity to other users' preferences
• Advantages
• Simplicity: KNN regression is straightforward to understand and implement
• Non-Parametric: It doesn’t assume a specific form for the data distribution, which makes it adaptable to different patterns and useful for non-linear data
• Interpretability: Each prediction can be explained by the nearest neighbors, which can provide insights into the relationships within the data
• Disadvantages
• Computationally Intensive: KNN regression requires calculating the distance from the query point to all other points, making it slow for large datasets
• Sensitivity to Feature Scaling: Since KNN relies on distance calculations, feature scaling (e.g., normalization or standardization) is crucial for ensuring all features contribute equally to the distance metric
• High Variance: The model’s performance depends heavily on the choice of 𝑘 and can fluctuate with small changes in the training data
• Example in Python

from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate or load sample data
X = np.arange(1, 21).reshape(-1, 1)  # example feature (e.g., month)
y = np.random.normal(50, 10, 20)     # example target (e.g., sales)

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Scale the features
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Create and fit the KNN Regressor model
knn = KNeighborsRegressor(n_neighbors=3, weights='distance')  # using weighted KNN
knn.fit(X_train_scaled, y_train)

# Predict and evaluate the model
y_pred = knn.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)
                    

Copy Code

neighbors.KNeighborsRegressor - regression based on k-nearest neighbors

• n_neighbors - number of neighbors to use by default for kneighbors queries
• weights - weight function used in prediction (uniform, distance or callable)
• coef_ - weight vector(s)
• intercept_ - independent term in decision function
• fit(X, y) - fit k-nearest neighbors regression model, X: training data; y: target values
• predict(X) - predict the target for the provided data
• score(X, y) - return the coefficient of determination R2 of the prediction

Support Vector Regression (SVR)

Type of Support Vector Machine (SVM) adapted for regression tasks, where the goal is to predict continuous values rather than classify data points

Attempts to find a function that fits the data within a specified margin of error (epsilon-insensitive margin), where the model disregards errors that fall within a distance 𝜖 of the true values

• Key Concepts
• margin 𝜖 is set around the predicted line, errors within this margin are not penalized making it “insensitive” to small deviations, only points outside this margin contribute to the model’s error calculation
• goal of SVR is to find a function that minimizes a combined objective: maximizing the margin while keeping errors outside the epsilon margin small
• can handle non-linear relationships by using the kernel trick, allowing it to project data into higher-dimensional spaces where linear separability (or linear regression) is easier to achieve
• support vectors are the data points that lie outside the epsilon margin, influencing the position and orientation of the regression line (only these points affect the SVR model)
• regularization parameter 𝐶 controls the trade-off between maximizing the margin and minimizing the error beyond 𝜖 (high 𝐶 assigns a larger penalty to errors leading to overfitting)
• Applications
• House Rent Estimation: Estimating rent based on nearby properties with similar characteristics
• User Rating Prediction: Predicting product ratings by aggregating the ratings of similar users
• Restaurant Recommendation: Predicting restaurant ratings for a user based on their similarity to other users' preferences
• Advantages
• Robust to Outliers: SVR is robust to outliers because of its epsilon-insensitive margin, which can ignore minor deviations within the 𝜖-margin
• Effective in High-Dimensional Spaces: SVR performs well in high-dimensional data, making it suitable for complex and sparse datasets
• Non-Linear Capabilities: With kernels, SVR can capture non-linear patterns effectively, offering flexibility across different types of data
• Example in Python

from sklearn.svm import SVR
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
import numpy as np

# Generate or load data
# X = features, y = target
X = np.arange(1, 21).reshape(-1, 1)  # example feature (e.g., months)
y = np.random.normal(50, 10, 20)     # example target (e.g., sales)

# Split data
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Standardize data (SVR often performs better with scaled data)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Fit SVR model with RBF kernel
svr = SVR(kernel='rbf', C=1.0, epsilon=0.1)
svr.fit(X_train_scaled, y_train)

# Predict and evaluate
y_pred = svr.predict(X_test_scaled)
mse = mean_squared_error(y_test, y_pred)
print("Mean Squared Error:", mse)
                    

Copy Code

sklearn.svm.SVR - epsilon-support vector regression

• kernel - specifies the kernel type to be used in the algorithm (linear, poly, rbf, sigmoid, precomputed)
• C - regularization parameter
• epsilon - specifies the epsilon-tube within which no penalty is associated in the training loss function with points predicted within a distance epsilon from the actual value
• fit(X, y) - fit the SVM model according to the given training data, X: training vectors; y: target values
• predict(X) - perform regression on samples in X
• score(X, y) - return the coefficient of determination R2 of the prediction

Logistic Regression

statistical and machine learning method commonly used for binary classification problems, where the goal is to predict the probability of a binary outcome (e.g., yes/no, true/false, 0/1) based on one or more predictor variables

logistic regression is fundamentally a classification algorithm that applies a logistic (or sigmoid) function to estimate probabilities

• Key Concepts
• logistic function, or sigmoid function maps any real-valued number to a range between 0 and 1, which can then be interpreted as a probability
• 𝑧 is a linear combination of input features, where 𝑤 represents the model’s coefficients and 𝑥 the input features
• sigmoid function transforms this linear output to a probability score, making it suitable for binary classification
• outputs a probability score between 0 and 1, a threshold of 0.5 is used to classify the output: values greater than or equal to 0.5 are classified as one class (often labeled as 1)
• optimized using a loss function known as log loss or binary cross-entropy, which measures the difference between the predicted probability and the actual label
• logistic regression coefficients are estimated using maximum likelihood estimation, adjusts the weights so that the predicted probabilities align as closely as possible with the observed labels
• assumes a linear relationship between the independent variables and the log-odds of the outcome
• each observation should be independent of others, as logistic regression does not handle correlated data well
• highly correlated features can affect the model's stability, so feature selection or dimensionality reduction techniques (e.g., PCA) are often applied beforehand
• Advantages
• Simple and Interpretable: Logistic regression is easy to interpret and provides straightforward probabilistic outputs
• Efficient for Binary Classification: It is computationally efficient and performs well on linearly separable datasets
• Works Well with Small Datasets: Logistic regression can perform well with relatively small datasets and is less prone to overfitting compared to more complex models when regularization is used
• Disadvantages
• Linear Decision Boundary: Logistic regression works best for linear decision boundaries. For more complex boundaries, it may underperform unless features are transformed or nonlinear techniques are used
• Sensitive to Outliers: Logistic regression can be sensitive to outliers, particularly if regularization is not applied
• Limited to Binary Classification: Standard logistic regression is inherently binary, although it can be extended to multiclass classification using techniques such as One-vs-Rest (OvR) and Softmax Regression
• Example in Python

from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, confusion_matrix

# Generate or load example data
X = [[1.1], [2.5], [3.3], [4.5], [5.1], [6.2], [7.4]]  # example features
y = [0, 0, 1, 1, 0, 1, 1]  # binary target

# Split data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Initialize and train the model
model = LogisticRegression()
model.fit(X_train, y_train)

# Make predictions
y_pred = model.predict(X_test)

# Evaluate the model
accuracy = accuracy_score(y_test, y_pred)
conf_matrix = confusion_matrix(y_test, y_pred)
print("Accuracy:", accuracy)
print("Confusion Matrix:\n", conf_matrix)
                    

Copy Code