Machine Learning Maverick – My Path of Learning

In this article, I will go deeper into the step “The data” from the Machine Learning Workflow. In the previous article, How do I work with data using a Machine Learning Model? I described three of the six steps.

All the data for the Machine Learning Model needs to be numerical. Preparing the data involves filling in missing values, and changing all non-numerical values into numbers, e.g.: text into categories or integers, string dates split into days, months, and years as integers, and boolean yes/no as 0 and 1.

In the previous articles, I was working with the perfect data where no preparation was needed. In the real world perfect data doesn’t exist, you will always have to work with the data. This step has its name, the Exploratory Data Analysis (EDA) is the most important task to conduct at the beginning of every data science project.

Loading the data

We need to load the data for Exploratory Data Analysis (EDA). The data set used for this article is the “Apartment Prices in Poland” – https://www.kaggle.com/datasets/krzysztofjamroz/apartment-prices-in-poland/data.

# Importing the tools
import pandas as pd

# Load the data into pandas DataFrame
data_frame = pd.read_csv("apartments_rent_pl_2024_01.csv")

# Let's see what data we have
data_frame.head()

id	city	type	squareMeters	rooms	floor	floorCount	buildYear	latitude	longitude	centreDistance	poiCount	schoolDistance	clinicDistance	postOfficeDistance	kindergartenDistance	restaurantDistance	collegeDistance	pharmacyDistance	ownership	buildingMaterial	condition	hasParkingSpace	hasBalcony	hasElevator	hasSecurity	hasStorageRoom	price
2a1a6db97ff122d6bc148abb6f0e498a	szczecin	blockOfFlats	52	2	3	3	2008	53.4605350947556	14.5454160213269	4.26	2	0.753	1.049	0.595	0.674	0.229	2.2	0.307	condominium	brick		yes	yes	no	no	no	3500
368e16142922433c709e6921a3b8f2a8	szczecin	blockOfFlats	70	3	7	11	1985	53.3784846	14.6570885	8.1	21	0.3	0.111	0.317	0.071	0.068		0.086	condominium	concreteSlab	premium	no	yes	yes	no	no	2900
655acd54eb518a718fd1f59fd7161c61	szczecin	blockOfFlats	43	2	3	3	1960	53.4259458	14.5594895	0.3	68	0.072	0.73	0.589	0.114	0.016	0.405	0.338	condominium	concreteSlab	premium	no	yes	no	no	yes	2900
99f85bbc55d110aa5a79ad4c7d5a0562	szczecin	blockOfFlats	46.7	2	1	4	1980	53.4469902	14.5568514	2.64	14	0.196	0.715	0.744	0.248	0.258	1.562	0.096	condominium	concreteSlab	premium	no	no	no	no	yes	2400
601b83a79fb6b89fe8e488bfcdd5e872	szczecin		56.2	3	1	4		53.4411	14.5491	2.12	24	0.126	0.223	0.148	0.215	0.367	0.732	0.05	condominium			no	no		no	no	3000

As we can see even in the first five rows we have missing values, in the form of empty cells.

Dealing with missing values

First, we need to identify data types for each column in the loaded data set. We need numeric values, any data type other than object is good.

# Checking columns data types to know how to handle missing values
data_frame.dtypes

Below we have a list of column names and their data types, e.g. column id is of type object.

id                       object
city                     object
type                     object
squareMeters            float64
rooms                   float64
floor                   float64
floorCount              float64
buildYear               float64
latitude                float64
longitude               float64
centreDistance          float64
poiCount                float64
schoolDistance          float64
clinicDistance          float64
postOfficeDistance      float64
kindergartenDistance    float64
restaurantDistance      float64
collegeDistance         float64
pharmacyDistance        float64
ownership                object
buildingMaterial         object
condition                object
hasParkingSpace           int64
hasBalcony                int64
hasElevator               int64
hasSecurity               int64
hasStorageRoom            int64
price                     int64
dtype: object

Identify missing values

Before we even start filling missing values we need to know in which columns and how many missing values we have.

For this task, I’ve used dtypes property and isna() method, I packed the result of those two into pandas DataFrame to see it as a columns.

# Checking data types vs NaN values - before and after filling missing data
info_df = pd.DataFrame({
    'Data Type': data_frame.dtypes,
    'Missing Values': data_frame.isna().sum()
})

print(info_df)

Below we have a list of columns with its Data Type and number of Missing Values for each column.

                     Data Type  Missing Values
id                      object               0
city                    object               0
type                    object            2203
squareMeters           float64               0
rooms                  float64               0
floor                  float64            1030
floorCount             float64             171
buildYear              float64            2492
latitude               float64               0
longitude              float64               0
centreDistance         float64               0
poiCount               float64               0
schoolDistance         float64               2
clinicDistance         float64               5
postOfficeDistance     float64               5
kindergartenDistance   float64               7
restaurantDistance     float64              24
collegeDistance        float64             104
pharmacyDistance       float64              13
ownership               object               0
buildingMaterial        object            3459
condition               object            6223
hasParkingSpace         object               0
hasBalcony              object               0
hasElevator             object             454
hasSecurity             object               0
hasStorageRoom          object               0
price                    int64               0

As we can see we have many missing values, e.g. column buildYear has 2492 missing values.

When we try to create a Machine Learning Model based on the DataFrame …

X = data_frame.drop("price", axis=1)
y = data_frame["price"]
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
np.random.seed(42)
from sklearn.ensemble import RandomForestClassifier
model = RandomForestClassifier()
model.fit(X_train, y_train)

… we will get an exception.

ValueError                                Traceback (most recent call last)

<ipython-input-15-345ee3a9038d> in <cell line: 17>()
     15 model = RandomForestClassifier()
     16 # 'fit()' - Build a forest of trees from the training set (X, y).
---> 17 model.fit(X_train, y_train)
     18 # 'predict()' - Predict class for X.
     19 y_preds = model.predict(X_test)

/usr/local/lib/python3.10/dist-packages/pandas/core/generic.py in __array__(self, dtype)
   1996     def __array__(self, dtype: npt.DTypeLike | None = None) -> np.ndarray:
   1997         values = self._values
-> 1998         arr = np.asarray(values, dtype=dtype)
   1999         if (
   2000             astype_is_view(values.dtype, arr.dtype)

ValueError: could not convert string to float: '1e1ec12d582075085f740f5c7bdf4091'

Filling missing values

Before we create a Machine Learning Model we need to fill in missing values even if they are numerics.

Numeric values

Filling missing numeric columns with mean() values isn’t the best idea, but as a starting point, it is good enough.

For this task, I’ve used two methods fillna() and mean() for specific columns, e.g.: column floor, data_frame["floor"], and used the parameter inplace=True to avoid reassigning value to the column.

# Dealing with missing values
# Filling NaN values
data_frame["floor"].fillna(data_frame["floor"].mean(), inplace=True)
data_frame["floorCount"].fillna(data_frame["floorCount"].mean(), inplace=True)
data_frame["buildYear"].fillna(data_frame["buildYear"].mean(), inplace=True)

# Without parameter inplate=True
# data_frame["buildYear"] = data_frame["buildYear"].fillna(data_frame["buildYear"].mean())

Non-numeric values

When we deal with non-numeric values the worst thing we can do is to fill in missing values with the same value. What do I mean by that?

First, I check the unique values for specific column.

# Checking non-numeric columns unique data to fill NaN
print(f"Condition: {data_frame['condition'].unique()}")

Condition: ['premium' 'low']

We don’t want all our apartments to be only premium or only low. Filling missing values with a single value is a very bad idea.

That’s why I use the below code to find unique values for specific columns and then randomly apply this value to the column.

unique_conditions = data_frame["condition"].dropna().unique()
data_frame["condition"] = data_frame["condition"].apply(
    lambda x: np.random.choice(unique_conditions) if pd.isna(x) else x)

The same can be applied to other columns, e.g.: city, for its values.

Cities: ['szczecin' 'gdynia' 'krakow' 'poznan' 'bialystok' 'gdansk' 'wroclaw' 'radom' 'rzeszow' 'lodz' 'katowice' 'lublin' 'czestochowa' 'warszawa' 'bydgoszcz']

Convert non-numeric data into numeric

Since we have filled in all missing values we can start converting them into numbers because ALL the data for the Machine Learning Model needs to be numerical.

Text into numbers

Sometimes it’s easy to change text into numbers. In the data set, we use, the id column contains text 2a1a6db97ff122d6bc148abb6f0e498a, in this case, we can change it into the number hexadecimal form. The same goes with boolean values like yes/no, we can convert them into 0 and 1.

# Convert non-numeric data into numeric

# id column type 'str' into 'int'
data_frame["id"] = data_frame["id"].apply(
    lambda x: int(x, 16) if isinstance(x, str) else x)

# columns with 'str' yes/no into bool
data_frame['hasParkingSpace'] = 
    data_frame['hasParkingSpace'].map({'yes': 1, "no": 0})

Dates into numbers

Even dates are stored in text form, e.g.: 2024-06-10, we need to split each part, the year, the month, and the day into separate variables/columns. The column is in a different data set city_rentals_wro_2007_2023.csv from the same “Apartment Prices in Poland” – https://www.kaggle.com/datasets/krzysztofjamroz/apartment-prices-in-poland/data.

# Convert non-numeric data into numeric
# changing column 'date_listed' of type 'str' into separate numbers
data_frame['date_listed'] = pd.to_datetime(data_frame['date_listed'])

# create new columns for year, month, and day
data_frame['year'] = data_frame['date_listed'].dt.year
data_frame['month'] = data_frame['date_listed'].dt.month
data_frame['day'] = data_frame['date_listed'].dt.day

# drop the original 'date' column if you wish
data_frame = data_frame.drop('date_listed', axis=1)

year	month	day
2019	9	27
2023	6	18
2023	2	16
2016	2	18
2015	12	9

The above table shows added columns after converting the column date_listed.

Categories into numbers

Text data can be changed into categories and then into numbers, the below code does it very well. I’m not going into many details, I use existing libraries and their classes the OneHotEncoder and the ColumnTransformer, all available in scikit-learn.

How did I figure out which column may be treated as a category? It’s related to the process described earlier Filling missing values – Non-numeric values, and it’s a part of the Exploratory Data Analysis (EDA).

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

# Turn the categories into numbers
categorical_features = ["city", "type", "ownership", "buildingMaterial", "condition"]
one_hot = OneHotEncoder()
transformer = ColumnTransformer([("one_hot", one_hot,
                                  categorical_features)],
                                remainder="passthrough")

transformed_X = transformer.fit_transform(X)
transformed_df = pd.DataFrame(transformed_X)

	0	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16	17	18	19	20	21	22	23	24	25	26	27	28	29	30	31	32	33	34	35	36	37	38	39	40	41	42	43	44
0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	1	1	0	1	0	5.59648010207413E+037	52	2	3	3	2008	53.4605350947556	14.5454160213269	4.26	2	0.753	1.049	0.595	0.674	0.229	2.2	0.307	1	1	0	0	0
1	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	1	0	1	0	1	7.25160657366022E+037	70	3	7	11	1985	53.3784846	14.6570885	8.1	21	0.3	0.111	0.317	0.071	0.068	1.29419493153976	0.086	0	1	1	0	0
2	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	1	0	1	0	1	1.34723498913507E+038	43	2	3	3	1960	53.4259458	14.5594895	0.3	68	0.072	0.73	0.589	0.114	0.016	0.405	0.338	0	1	0	0	1
3	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	1	0	1	0	1	0	1	2.04661433596736E+038	46.7	2	1	4	1980	53.4469902	14.5568514	2.64	14	0.196	0.715	0.744	0.248	0.258	1.562	0.096	0	0	0	0	1
4	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	1	0	0	1	1	0	0	1	1.27748749886723E+038	56.2	3	1	4	1998.34355001684	53.4411	14.5491	2.12	24	0.126	0.223	0.148	0.215	0.367	0.732	0.05	0	0	0	0	0

Before transforming the DataFrame we had 28 columns now we have 44 columns without human-readable column names, instead we have only numbers as column names.

ALL the data is numerical, we accomplished EDA and ended up with the DataFrame ready to be used in the Machine Learning Model.

The source code

Below we can find all the source code necessary for preparing the data for using it with a Machine Learning Model.

Steps covered:

1. Loading the data
2. Dealing with missing values
3. Identify missing values
4. Filling missing values
   • Numeric values
   • Non-numeric values
5. Convert non-numeric data into numeric
6. Text into numbers
7. Dates into numbers
8. Categories into numbers

# Importing the tools
import pandas as pd
import numpy as np

data_frame = pd.read_csv(csv_file_name)

# Dealing with missing values
# Filling NaN values
data_frame["floor"].fillna(data_frame["floor"].mean(), inplace=True)
data_frame["floorCount"].fillna(data_frame["floorCount"].mean(), inplace=True)
data_frame["buildYear"].fillna(data_frame["buildYear"].mean(), inplace=True)
data_frame["schoolDistance"].fillna(data_frame["schoolDistance"].mean(), inplace=True)
data_frame["clinicDistance"].fillna(data_frame["clinicDistance"].mean(), inplace=True)
data_frame["postOfficeDistance"].fillna(data_frame["postOfficeDistance"].mean(), inplace=True)
data_frame["kindergartenDistance"].fillna(data_frame["kindergartenDistance"].mean(), inplace=True)
data_frame["restaurantDistance"].fillna(data_frame["restaurantDistance"].mean(), inplace=True)
data_frame["collegeDistance"].fillna(data_frame["collegeDistance"].mean(), inplace=True)
data_frame["pharmacyDistance"].fillna(data_frame["pharmacyDistance"].mean(), inplace=True)

unique_types = data_frame["type"].dropna().unique()
data_frame["type"] = data_frame["type"].apply(lambda x: np.random.choice(unique_types) if pd.isna(x) else x)

data_frame["ownership"].fillna("condominium", inplace=True)

unique_bms = data_frame["buildingMaterial"].dropna().unique()
data_frame["buildingMaterial"] = data_frame["buildingMaterial"].apply(
    lambda x: np.random.choice(unique_bms) if pd.isna(x) else x)

unique_conditions = data_frame["condition"].dropna().unique()
data_frame["condition"] = data_frame["condition"].apply(
    lambda x: np.random.choice(unique_conditions) if pd.isna(x) else x)

unique_hes = data_frame["hasElevator"].dropna().unique()
data_frame["hasElevator"] = data_frame["hasElevator"].apply(
    lambda x: np.random.choice(unique_hes) if pd.isna(x) else x)

# Convert non-numeric data into numeric
# id column type 'str' into 'int'
data_frame["id"] = data_frame["id"].apply(lambda x: int(x, 16) if isinstance(x, str) else x)
# columns with 'str' yes/no into bool
data_frame['hasParkingSpace'] = data_frame['hasParkingSpace'].map({'yes': 1, "no": 0})
data_frame['hasBalcony'] = data_frame['hasBalcony'].map({'yes': 1, "no": 0})
data_frame['hasElevator'] = data_frame['hasElevator'].map({'yes': 1, "no": 0})
data_frame['hasSecurity'] = data_frame['hasSecurity'].map({'yes': 1, "no": 0})
data_frame['hasStorageRoom'] = data_frame['hasStorageRoom'].map({'yes': 1, "no": 0})

# X - training input samples, features
X = data_frame.drop("price", axis=1)

from sklearn.preprocessing import OneHotEncoder
from sklearn.compose import ColumnTransformer

# Turn the categories into numbers
categorical_features = ["city", "type", "ownership", "buildingMaterial", "condition"]
one_hot = OneHotEncoder()
transformer = ColumnTransformer([("one_hot", one_hot,
                                  categorical_features)],
                                remainder="passthrough")

transformed_X = transformer.fit_transform(X)
transformed_df = pd.DataFrame(transformed_X)
transformed_df.to_csv("saved_transformed_df.csv")

# y - training input labels, the desired result, the target value
y = data_frame["price"]

Below we can find all the source code necessary for creating a Machine Learning Model based on the prepared data.

# Import 'train_test_split()' function
# "Split arrays or matrices into random train and test subsets."
from sklearn.model_selection import train_test_split

# Split the data into training and test sets
X_train, X_test, y_train, y_test = train_test_split(transformed_X, y, test_size=0.2)

# Setup random seed - to have the same results, me and you
np.random.seed(42)

# Import the LinearRegression estimator class
from sklearn.linear_model import LinearRegression

# Instantiate LinearRegression to create a Machine Learning Model
model = LinearRegression()

# 'fit()' - Build a forest of trees from the training set (X, y).
model.fit(X_train, y_train)

# 'predict()' - Predict class for X.
y_preds = model.predict(X_test)

NOTE: In this article, I’m just barely scratching the surface. This topic needs more reading and research on your own. I’m still at the beginning of my learning process of AI & ML!