Machine Learning Maverick – My Path of Learning

In this article, I will describe the next step (Evaluation) for the Machine Learning Workflow. In the previous article, How do I work with data using a Machine Learning Model? I described three of the six steps.

Evaluation of a Machine Learning Model is often comparing predicted labels to the truth labels. Evaluating the results of a Machine Learning Model is as important as building one.

Machine Learning Workflow

1. Defining the problem
   • Predict whether a person has heart disease or not
2. The data
   • Medical records are available for free – Kaggle: Your Machine Learning and Data Science Community
3. Assessment / Evaluation
   • Remaining step – described in this article
4. Characteristics / Features
   • Remaining step – NOT covered
5. Modeling
   • Classification Model (Supervised ML) using RandomForestClassifier from scikit-learn
6. Examination / Experiments
   • Remaining step – NOT covered

Covered steps

These steps were covered:

• Defining the problem – What problem are we trying to solve?
• The data – What kind of data do we have?
• Modeling – What kind of model should we use?

Evaluation step

Assessment/Evaluation – What is the definition of success?

Why evaluating the ML Model is so important? Having in mind the heart disease prediction for a patient. We would like to know how well the model predicted heart disease based on the data we provided.

Evaluation metrics

There are evaluation metrics for different types of Machine Learning Models. I will focus on Supervised ML Models: Classification and Regression.

1. Classification evaluation metrics:
   • Accuracy (default for score() method)
   • Classification Report
   • Confusion Matrix
   • ROC Curve
   • AUC
2. Regression evaluation metrics:
   • R² Score
   • MAE
   • MSE

---

Here is a little recap of the previous code where I worked with data to create a Machine Learning Model. After the recap, I will work with the remaining three steps.

# Importing the tools
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# Get the data with medical records from Kaggle.com
# Load the .data file into a DataFrame
heart_disease_ch = pd.read_csv('data/heart-disease_kaggle')

# X - training input samples, features
X = heart_disease.drop("target", axis=1)

# y - training input labels, the desired result, the target value
y = heart_disease["target"]

# Import 'train_test_split()' function
# "Split arrays or matrices into random train and test subsets."
from sklearn.model_selection import train_test_split

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Setup random seed - to have the same results, me and you
np.random.seed(42)

# Import the RandomForestClassifier estimator class
from sklearn.ensemble import RandomForestClassifier

# Instantiate RandomForestClassifier to create a Machine Learning Model
model = RandomForestClassifier()

# 'fit()' - Build a forest of trees from the training set (X, y).
model.fit(X_train, y_train)

# 'predict()' - Predict class for X.
y_preds = model.predict(X_test)

The above code brought us to the point where we have a Machine Learning Model (model = RandomForestClassifier()) and we used it to predict the desired outcome (y_preds = model.predict(X_test)).

---

The questions arise:

• How can we know if the model performed well?
• How can we measure the performance?
• How can we tune the model to perform better?

Model Evaluation

There are many different ways to evaluate the Machine Learning Model. It’s good to know what options we have. In most cases, we will use selected evaluation metrics for a specific problem. In this section, I will only describe evaluation metrics for the Classification, one type of Machine Learning.

Scoring the Model – score() method

The easiest way to measure model performance is to run the model.score() method on the model itself, as a result, we will get values between 0 and 1. It shows how likely it is to predict the right label. How likely is it to predict the heart disease?

# 'score()' - Return the mean accuracy on the given test data and labels.
model.score(X_test, y_test)

Where 1 is the perfect score, the model is 100% accurate and works outstandingly. When we get a score of 1, there is something wrong with the model or we didn’t split data into three sets: train, validation, and test. Maybe we run the scoring method on the training data, which the model “knows” very well.

Classification Report – the main classification metrics

A classification report is a textual summary of various evaluation metrics for a classification model, typically generated using the predicted labels (y_preds) and the true labels (y_test) of a test dataset. It provides a detailed breakdown of the performance of the classifier for each class in the dataset.

# 'classification_report()' - Build a text report showing the main classification metrics.

from sklearn.metrics import classification_report
model.score(X_test, y_test)

	precision	recall	f1-score	support
0	0.60	0.68	0.64	22
1	0.81	0.74	0.77	39
accuracy			0.72	61
macro avg	0.70	0.71	0.71	61
weighted avg	0.73	0.72	0.72	61

A typical classification report includes the following metrics for each class:

• Precision: The ratio of true positive predictions to the total number of positive predictions, indicating the accuracy of positive predictions.
• Recall (or Sensitivity): The ratio of true positive predictions to the total number of the actual positive instances, indicating the model’s ability to identify all positive instances.
• F1-Score: The harmonic mean of precision and recall, providing a balance between the two metrics.
• Support: The number of actual occurrences of the class in the test dataset.

Additionally, the classification report usually includes an average or weighted average of these metrics across all classes, providing an overall summary of the model’s performance.

Confusion Matrix – the model predicted “the other way round”

A confusion matrix is comparing the predicted labels to the true labels and then seeing which ones the model gets confused. Shows “False Negatives” and “False Positives” – when the model predicted, “the other way round”.

True positives and true negatives, along with false positives and false negatives plotted against each other. Useful where a model is getting “confused”, e.g. how often it predicts the right and wrong classes.

In the above matrix:

• TP (True Positive): The number of instances correctly predicted as positive.
• TN (True Negative): The number of instances correctly predicted as negative.
• FP (False Positive): The number of instances incorrectly predicted as positive (false alarm).
• FN (False Negative): The number of instances incorrectly predicted as negative (miss).

From the confusion matrix, various performance metrics can be calculated, such as accuracy, precision, recall (sensitivity), specificity, F1 score, etc., which provide insights into how well the model is performing across different classes.

# 'confusion_matrix()' - Compute confusion matrix to evaluate the accuracy of a classification.

from sklearn.metrics import confusion_matrix

conf_matrix = confusion_matrix(y_test, y_preds)

Confusion Matrix results in the context of predicting heart disease:

• True positive -> model predicts 1 when the truth is 1
• False positive -> model predicts 1 when the truth is 0
• True negative -> model predicts 0 when the truth is 0
• False negative -> model predicts 0 when the truth is 1

# Example output for the 'confusion_matrix()` function
array([
[20, 6],
[ 2, 33]
])

It’s better to visualize the above example result. We can do it using ConfusionMatrixDisplay.

# Confusion Matrix visualization.
from sklearn.metrics import ConfusionMatrixDisplay

ConfusionMatrixDisplay.from_predictions(y_true=y_test,
                                         y_pred=y_preds);

NOTE: In this article, I’m just barely scratching the surface. This topic needs more reading and research on your own. I’m still at the beginning of my learning process of AI & ML!