pandas, what pandas 🐼
pandas is an open-source library for Python that enables data manipulation and analysis.
Why pandas
The pandas library is a powerful tool for data manipulation, analysis, and exploration in Python. Its intuitive data structures, rich set of functions, and integration with other libraries make it a popular choice for a wide range of data-related tasks.
Pandas is a tool that simplifies working with structured data, which is a crucial aspect of machine learning and data science.
Import the library
# importing the pandas library & checking its version
import pandas as pd
print(f"pandas version: {pd.__version__}")
Know available Data Types
pandas.DataFrame– (in most cases) a two-dimensional data table (array) with rows and columns.pandas.Series– a one-dimensional column of data.
The Series type can be created using pd.Series() and passing a Python list to it.
# Creating a series of fruit names fruits = pd.Series(["Apple", "Banana", "Orange"]) # Creating a series of fruit prices prices = pd.Series([2.5, 3.5, 3.0])
The DataFrame type can be created using pd.DataFrame() and passing a Python dict to it.
# Creating a DataFrame using dict and previously created series
market_df = pd.DataFrame(
{
"Fruit name": fruits,
"Fruit price": prices
}
)
For the above example, dictionary keys became the column headers and the values of both Series became the values in the DataFrame. The two series have to be the same size before being combined into a data frame.
| Fruit name | Fruit price | |
| 0 | Apple | 2.5 |
| 1 | Banana | 3.5 |
| 2 | Orange | 3 |
Import Data
To import data using Pandas, you can use functions like pd.read_csv() for CSV files, pd.read_excel() for Excel files, or pd.read_sql() for data from a SQL database. All the data read from different data sources are available as a DataFrame, which is pandas primary data structure.
Tip: If a Google Sheet is public, pd.read_csv() can read it via a URL.
# Importing Data from a File
car_sales_csv_file = pd.read_csv("car-sales.csv")
# "Displaying" data
car_sales_csv_file
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | Toyota | White | 150043 | 4 | $4,000.00 |
| 1 | Honda | Red | 87899 | 4 | $5,000.00 |
| 2 | Toyota | Blue | 32549 | 3 | $7,000.00 |
| 3 | BMW | Black | 11179 | 5 | $22,000.00 |
# Importing Data from a Google Sheet, URL
car_sales_url = pd.read_csv("https://docs.google.com/spreadsheets/d/UNIQUE_ID/export?format=csv")
car_sales_url
Note: Add query parameter format=csv – URL/export?format=csv
Export Data
After making changes to the data, you can export and save it so others can access the modifications.
pandas allows exporting DataFrame to a .csv format using .to_csv() or a spreadsheet format using .to_excel().
# Exporting: Saving the DataFrame 'car_sales_csv_file' to a new file
car_sales_url.to_csv("export_car_sales_csv_file.csv")
Running the above code will save a file named export-car-sales_url.csv in the current directory.
Exploring Data in DataFrame
It’s very important to explore the data we imported into a DataFrame. pandas has many useful built-in functions.
# Show the data in the DataFrame - simply write the name of DataFrame variable car_sales_csv_file
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | Toyota | White | 150043 | 4 | $4,000.00 |
| 1 | Honda | Red | 87899 | 4 | $5,000.00 |
| 2 | Toyota | Blue | 32549 | 3 | $7,000.00 |
| 3 | BMW | Black | 11179 | 5 | $22,000.00 |
# Show data types for each column in the DataFrame car_sales_csv_file.dtypes
Make object Colour object Odometer (KM) int64 Doors int64 Price object dtype: object
# Show data type for single column in the DataFrame car_sales_csv_file["Make"].dtypes
dtype('O')
# Show ststistical data for each column in the DataFrame car_sales_csv_file.describe()
| Odometer (KM) | Doors | |
| count | 10.000000 | 10.000000 |
| mean | 78601.400000 | 4.000000 |
| std | 61983.471735 | 0.471405 |
| min | 11179.000000 | 3.000000 |
| 25% | 35836.250000 | 4.000000 |
| 50% | 57369.000000 | 4.000000 |
| 75% | 96384.500000 | 4.000000 |
| max | 213095.000000 | 5.000000 |
# Show summary info about the DataFrame car_sales_csv_file.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 10 entries, 0 to 9 Data columns (total 5 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Make 10 non-null object 1 Colour 10 non-null object 2 Odometer (KM) 10 non-null int64 3 Doors 10 non-null int64 4 Price 10 non-null object dtypes: int64(2), object(3) memory usage: 528.0+ bytes
# Use math functions for the DataFrame car_sales_csv_file.mean(numeric_only=True)
Odometer (KM) 78601.4 Doors 4.0 dtype: float64
# Use math functions for the DataFrame car_sales_csv_file.sum(numeric_only=True)
Odometer (KM) 786014 Doors 40 dtype: int64
# Show a list of columns in the DataFrame car_sales_csv_file.columns
Index(['Make', 'Colour', 'Odometer (KM)', 'Doors', 'Price'], dtype='object')
# Show total numbers of elements in the DataFrame len(car_sales_csv_file)
10
Selecting Data from DataFrame
There are many ways to select and view data from the DataFrame, let’s explore some of them.
# Show the first 5 rows of data car_sales_csv_file.head()
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | Toyota | White | 150043 | 4 | $4,000.00 |
| 1 | Honda | Red | 87899 | 4 | $5,000.00 |
| 2 | Toyota | Blue | 32549 | 3 | $7,000.00 |
| 3 | BMW | Black | 11179 | 5 | $22,000.00 |
| 4 | Nissan | White | 213095 | 4 | $3,500.00 |
# Show a specified number of rows car_sales_csv_file.head(3)
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | Toyota | White | 150043 | 4 | $4,000.00 |
| 1 | Honda | Red | 87899 | 4 | $5,000.00 |
| 2 | Toyota | Blue | 32549 | 3 | $7,000.00 |
# Show the last 5 rows of data car_sales_csv_file.head()
| Make | Colour | Odometer (KM) | Doors | Price | |
| 5 | Toyota | Green | 99213 | 4 | $4,500.00 |
| 6 | Honda | Blue | 45698 | 4 | $7,500.00 |
| 7 | Honda | Blue | 54738 | 4 | $7,000.00 |
| 8 | Toyota | White | 60000 | 4 | $6,250.00 |
| 9 | Nissan | White | 31600 | 4 | $9,700.00 |
# Show a row with a specified index car_sales_csv_file.loc[4]
Make Nissan Colour White Odometer (KM) 213095 Doors 4 Price $3,500.00 Name: 4, dtype: object
# Show a row at a specified position car_sales_csv_file.iloc[4]
Make Nissan Colour White Odometer (KM) 213095 Doors 4 Price $3,500.00 Name: 4, dtype: object
# Let's see how this looks for diverse data
colors = pd.Series(
["white", "black", "red", "green", "blue"],
index=[0, 2, 3, 5, 3]
)
print(f"Index:\n {colors.loc[3]}, \nPosition:\n {colors.iloc[3]}")
Index: 3 red 3 blue dtype: object, Position: green
# Show data using a column name car_sales_csv_file["Odometer (KM)"]
0 150043 1 87899 2 32549 3 11179 4 213095 5 99213 6 45698 7 54738 8 60000 9 31600 Name: Odometer (KM), dtype: int64
# Show & filter data - column "Odometer(KM)" car_sales_csv_file[car_sales_csv_file["Odometer (KM)"] > 100000]
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | Toyota | White | 150043 | 4 | $4,000.00 |
| 4 | Nissan | White | 213095 | 4 | $3,500.00 |
# Show & filter data - column "Make" car_sales_csv_file[car_sales_csv_file["Make"] == "BMW"]
| Make | Colour | Odometer (KM) | Doors | Price | |
| 3 | BMW | Black | 11179 | 5 | $22,000.00 |
Simple Data Visualization
pandas use the matplotlib library and wrap it to visualize data using pandas DataFrame.
car_sales_csv_file["Odometer (KM)"].plot();
Data Manipulation
Using column data type properties/functions
# Altering the case of letters in a column car_sales_csv_file["Make"] = car_sales_csv_file["Make"].str.lower() # Show changed data car_sales_csv_file.head()
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | toyota | White | 150043 | 4 | $4,000.00 |
| 1 | honda | Red | 87899 | 4 | $5,000.00 |
| 2 | toyota | Blue | 32549 | 3 | $7,000.00 |
| 3 | bmw | Black | 11179 | 5 | $22,000.00 |
| 4 | nissan | White | 213095 | 4 | $3,500.00 |
Using column with function apply()
# Altering values for a column using 'apply()'
# Convert from kilometers to miles for the 'Odometer (KM)' column
car_sales_csv_file["Odometer (KM)"] =
car_sales_csv_file["Odometer (KM)"].apply(lambda x: x / 1.6)
# Show changed data
car_sales_csv_file.head()
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | toyota | White | 93776.875000 | 4 | $4,000.00 |
| 1 | honda | Red | 54936.875000 | 4 | $5,000.00 |
| 2 | toyota | Blue | 57689.166667 | 3 | $7,000.00 |
| 3 | bmw | Black | 6986.875000 | 5 | $22,000.00 |
| 4 | nissan | White | 133184.375000 | 4 | $3,500.00 |
# Altering values for a column using 'apply()'
# Convert from miles to kilometers for the 'Odometer (KM)' column
car_sales_csv_file["Odometer (KM)"] =
car_sales_csv_file["Odometer (KM)"].apply(lambda x: x * 1.6)
# Show changed data
car_sales_csv_file.head()
| Make | Colour | Odometer (KM) | Doors | Price | |
| 0 | toyota | White | 150043 | 4 | $4,000.00 |
| 1 | honda | Red | 87899 | 4 | $5,000.00 |
| 2 | toyota | Blue | 32549 | 3 | $7,000.00 |
| 3 | bmw | Black | 11179 | 5 | $22,000.00 |
| 4 | nissan | White | 213095 | 4 | $3,500.00 |
Column Manipulation
Using pandas we can manipulate columns in DataFrame:
- Add a new column using pandas.Series
- Add a new column using a Python list
- Use the existing column to create a new column.
- Delete the existing column.
Add column – pandas.Series
# Add a new column using 'pandas.Series' wheels_column = pd.Series([3, 3, 4, 4, 4, 3]) car_sales_csv_file["Wheels"] = wheels_column # Show a data with added column car_sales_csv_file
| Make | Colour | Odometer (KM) | Doors | Price | Wheels | |
| 0 | Toyota | White | 150043 | 4 | $4,000 | 3 |
| 1 | Honda | Red | 87899 | 4 | $5,000 | 3 |
| 2 | Toyota | Blue | 92302.666667 | 3 | $7,000 | 4 |
| 3 | BMW | Black | 11179 | 5 | $22,000 | 4 |
| 4 | Nissan | White | 213095 | 4 | $3,500 | 4 |
| 5 | Toyota | Green | 92302.666667 | 4 | $4,500 | 3 |
Add column – Python list
# Add a new column using Python 'list' type weights_column = [1.4, 1.3, 1.9, 2.0, 1.1, 1.5] car_sales_csv_file["Weight"] = weights_column # Show a data with added column car_sales_csv_file
| Make | Colour | Odometer (KM) | Doors | Price | Wheels | Weight | |
| 0 | Toyota | White | 150043 | 4 | $4,000 | 3 | 1.4 |
| 1 | Honda | Red | 87899 | 4 | $5,000 | 3 | 1.3 |
| 2 | Toyota | Blue | 92302.666667 | 3 | $7,000 | 4 | 1.9 |
| 3 | BMW | Black | 11179 | 5 | $22,000 | 4 | 2.0 |
| 4 | Nissan | White | 213095 | 4 | $3,500 | 4 | 1.1 |
| 5 | Toyota | Green | 92302.666667 | 4 | $4,500 | 3 | 1.5 |
Add column – using the existing one
# Add a new column using existing one
# New column name "Price per KM"
# Existing columns "Price" & "Odometer (KM)"
car_sales_csv_file["Price per KM"] =
car_sales_csv_file["Price"] / car_sales_csv_file["Odometer (KM)"]
# Show a data with added column
car_sales_csv_file
| Make | Colour | Odometer (KM) | Doors | Price | Wheels | Weight | Price per KM | |
| 0 | Toyota | White | 150043 | 4 | $4,000 | 3 | 1.4 | 0.026659 |
| 1 | Honda | Red | 87899 | 4 | $5,000 | 3 | 1.3 | 0.056883 |
| 2 | Toyota | Blue | 92302.666667 | 3 | $7,000 | 4 | 1.9 | 0.075837 |
| 3 | BMW | Black | 11179 | 5 | $22,000 | 4 | 2.0 | 1.967976 |
| 4 | Nissan | White | 213095 | 4 | $3,500 | 4 | 1.1 | 0.016425 |
| 5 | Toyota | Green | 92302.666667 | 4 | $4,500 | 3 | 1.5 | 0.048753 |
Delete the existing column
# Delete existing column with 'drop()' and 'axis=1'
car_sales_csv_file = car_sales_csv_file.drop("Price per KM", axis=1)
| Make | Colour | Odometer (KM) | Doors | Price | Wheels | Weight | |
| 0 | Toyota | White | 150043 | 4 | $4,000 | 3 | 1.4 |
| 1 | Honda | Red | 87899 | 4 | $5,000 | 3 | 1.3 |
| 2 | Toyota | Blue | 92302.666667 | 3 | $7,000 | 4 | 1.9 |
| 3 | BMW | Black | 11179 | 5 | $22,000 | 4 | 2.0 |
| 4 | Nissan | White | 213095 | 4 | $3,500 | 4 | 1.1 |
| 5 | Toyota | Green | 92302.666667 | 4 | $4,500 | 3 | 1.5 |
NOTE: In this article, I’m just barely scratching the surface. This topic needs more reading and research on your own. I’m still at the beginning of my learning process of AI & ML!
Leave a Reply