Machine Learning Maverick – My Path of Learning

Preface

In this post, I will focus on EDA – Exploratory Data Analysis. EDA is done in the first step “Data – load & analyze” of the workflow. The next required steps and the technology stack are described in detail in scikit-learn – Machine Learning Models in Python – Workflow & Introduction.

NOTE: In the scikit-learn – Machine Learning Models in Python – Workflow & Introduction I assumed that we have ideal data: no missing values, all values are numeric. I will load a different dataset with missing and non-numeric values in this blog.

Used datasets

1. scikit-learn […] Workflow & Introduction (previous post)
   • Dataset Heart Attack Analysis Prediction – Kaggle rashikrahmanpritom
2. scikit-learn […] Exploratory Data Analysis & Workflow (current post)
   • https://www.kaggle.com/datasets/mirichoi0218/insurance

EDA – Exploratory Data Analysis

Exploratory Data Analysis (EDA) is a vital step in machine learning to understand data, uncover patterns, and prepare it for modeling. Key tasks include summarizing data, visualizing distributions, detecting outliers, analyzing correlations, and handling missing values. EDA ensures data quality, guides preprocessing, and informs feature engineering to improve model performance.

Loading data

# Import necessary library
import pandas as pd

# Get the data with medical records
# Load the data from CVS into pandas DataFrame
insurence_df = pd.read_csv("dataset_medical-insurance-cost_missing-values_kaggle-mirichoi0218.csv")

# Display loaded data - first 5 rows
insurence_df.head()

Data overview

# Check data types of columns to know how to handle them
insurence_df.dtypes

age           int64
sex          object
bmi         float64
children      int64
smoker       object
region       object
charges     float64
dtype: object

Filling missing values

# Verify missing values relative to data types - before filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)

# Fill missing values
insurence_df["bmi"] = insurence_df["bmi"].fillna(insurence_df["bmi"].mean())
insurence_df["charges"] = insurence_df["charges"].fillna(insurence_df["charges"].mean())

# Verify missing values relative to data types - after filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)

         Column data type  Number of missing values
age                 int64                         0
sex                object                         0
bmi               float64                         0
children            int64                         0
smoker             object                         0
region             object                         0
charges           float64                         0

Converting non-numeric data to numeric

# Columns with text values 'yes'/'no' converted to numbers 1/0
insurence_df['smoker'] = insurence_df['smoker'].map({'yes': 1, "no": 0})

# Display the changed data - first 5 rows
insurence_df.head()

# Columns with text values 'male'/'female' converted to numbers 1/0
insurence_df['sex'] = insurence_df['sex'].map({'male': 1, "female": 0})

# Display the changed data - first 5 rows
insurence_df.head()

Converting categorical data to numeric type

NOTE: Below is presented “the easiest” way, but not necessarily the ideal one. An alternative to the solution below with ‘region_frequency’ is to use: from sklearn.preprocessing import OneHotEncoder and from sklearn.compose import ColumnTransformer.

# Calculate the frequency of occurrences for each category
region_frequency = insurence_df['region'].value_counts()

print(region_frequency)

region
southeast    364
southwest    325
northwest    325
northeast    324
Name: count, dtype: int64

Yes, I see the problem with two different regions (southwest and northwest) becoming one after using value_counts() to return the same value.

# Assigning the frequency of occurrences to each category
insurence_df['region_encoded'] = insurence_df['region'].map(region_frequency)

# Display the changed data - first 5 rows
insurence_df.head()

# Remove column 'region' leave the new column 'region_encoded'
insurence_df = insurence_df.drop("region", axis=1)

# Display the changed data - first 5 rows
insurence_df.head()

Below we can see the difference between data types in DataFrame before (on the left) and after (on the right) EDA – Exploratory Data Analysis.

age           int64
sex          object
bmi         float64
children      int64
smoker       object
region       object
charges     float64
dtype: object

age                 int64
sex                 int64
bmi               float64
children            int64
smoker              int64
charges           float64
region_encoded      int64
dtype: object

The whole code in one place

Only EDA

# WORKFLOW - 1. Data - load & analyze
# With EDA - Exploratory Data Analysis

# Import necessary library
import pandas as pd

# Get the data with medical records
# Load the data from CVS into pandas DataFrame
insurence_df = pd.read_csv("dataset_medical-insurance-cost_missing-values_kaggle-mirichoi0218.csv")

# Display loaded data - first 5 rows
insurence_df.head()

# Check data types of columns to know how to handle them
insurence_df.dtypes

# Verify missing values relative to data types - before filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)

# Fill missing values
insurence_df["bmi"] = insurence_df["bmi"].fillna(insurence_df["bmi"].mean())
insurence_df["charges"] = insurence_df["charges"].fillna(insurence_df["charges"].mean())

# Verify missing values relative to data types - after filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)


# Converting non-numeric data to numeric

# Columns with text values 'yes'/'no' converted to numbers 1/0
insurence_df['smoker'] = insurence_df['smoker'].map({'yes': 1, "no": 0})

# Display the changed data - first 5 rows
insurence_df.head()

# Columns with text values 'male'/'female' converted to numbers 1/0
insurence_df['sex'] = insurence_df['sex'].map({'male': 1, "female": 0})

# Display the changed data - first 5 rows
insurence_df.head()


# Converting categorical data to numeric type

# Calculate the frequency of occurrences for each category
region_frequency = insurence_df['region'].value_counts()

print(region_frequency)

# Assigning the frequency of occurrences to each category
insurence_df['region_encoded'] = insurence_df['region'].map(region_frequency)

# Display the changed data - first 5 rows
insurence_df.head()

# Remove column 'region' leave the new column 'region_encoded'
insurence_df = insurence_df.drop("region", axis=1)

# Display the changed data - first 5 rows
insurence_df.head()

The whole Workflow with EDA

# WORKFLOW - 1. Data - load & analyze
# With EDA - Exploratory Data Analysis

# Get the data with medical insurance cost
# Load the data from CVS into pandas DataFrame
insurence_df = pd.read_csv("dataset_medical-insurance-cost_missing-values_kaggle-mirichoi0218.csv")

# Display loaded data - first 5 rows
insurence_df.head()

# EDA - Exploratory Data Analysis
# Import necessary library
import pandas as pd

# Display loaded data - first 5 rows
insurence_df.head()

# Check data types of columns to know how to handle them
insurence_df.dtypes

# Verify missing values relative to data types - before filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)

# Fill missing values
insurence_df["bmi"] = insurence_df["bmi"].fillna(insurence_df["bmi"].mean())
insurence_df["charges"] = insurence_df["charges"].fillna(insurence_df["charges"].mean())

# Verify missing values relative to data types - after filling them
info_df = pd.DataFrame({
    'Column data type': insurence_df.dtypes,
    'Number of missing values': insurence_df.isna().sum()
})

print(info_df)

# Columns with text values 'yes'/'no' converted to numbers 1/0
insurence_df['smoker'] = insurence_df['smoker'].map({'yes': 1, "no": 0})

# Display the changed data - first 5 rows
insurence_df.head()

# Columns with text values 'male'/'female' converted to numbers 1/0
insurence_df['sex'] = insurence_df['sex'].map({'male': 1, "female": 0})

# Display the changed data - first 5 rows
insurence_df.head()

# Calculate the frequency of occurrences for each category
region_frequency = insurence_df['region'].value_counts()

print(region_frequency)

# Assigning the frequency of occurrences to each category
insurence_df['region_encoded'] = insurence_df['region'].map(region_frequency)

# Display the changed data - first 5 rows
insurence_df.head()

# Remove column 'region' leave the new column 'region_encoded'
insurence_df = insurence_df.drop("region", axis=1)

# Display the changed data - first 5 rows
insurence_df.head()

# WORKFLOW - 2. Model - choose ML model

# Instantiate LinearRegression to create a Machine Learning Model
model = LinearRegression()


# WORKFLOW - 3. Split the data - features, labels; training, testing

# X - training input samples, features
# X - contains all features/columns except 'charges'
X = insurence_df.drop("charges", axis=1)
X.head()

# y - training input labels, the desired result, the target value
# y - contains only the label/column 'charges'
y = insurence_df["charges"]
y.head()


# Split the data into training and test sets
# train_test_split() function splits arrays or matrices into random train and test subsets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)


# WORKFLOW - 4. Train & Predict - ML model

# fit() function builds a forest of trees from the training set (X, y)
model.fit(X_train, y_train)


# WORKFLOW - 5. Evaluate - ML model

# Predicting labels
# predict() function predicts class for X
y_pred = model.predict(X_test)

# score() function returns the mean accuracy on the given test data and labels
model.score(X_test, y_test)

NOTE: In this article, I’m just barely scratching the surface. This topic needs more reading and research on your own. I’m still at the beginning of my learning process of AI & ML!