Welcome back to our Machine Learning Basics series! In our previous post, we built a simple linear regression model that achieved an R-squared score of only 0.123. While this gave us a good foundation, the model's predictive power was quite limited.

In this tutorial, we'll explore feature engineering - one of the most powerful techniques in machine learning. By creating new features from existing data, we'll dramatically improve our model's performance from an R-squared of 0.123 to 0.862!

What You'll Learn

By the end of this tutorial, you'll understand:

• What feature engineering is and why it matters

• How to create new features from domain knowledge

• The concept of interaction features

• How feature engineering can dramatically improve model performance

• The importance of data visualization in feature discovery

What is Feature Engineering?

Feature Engineering is the process of using domain knowledge to create new features (variables) from existing data that make machine learning algorithms work better. It's often considered more of an art than a science, requiring creativity and understanding of the problem domain.

Good features can:

• Capture important patterns in the data

• Make relationships more apparent to the model

• Significantly improve model accuracy

Getting Started

Let's begin by loading our dataset and necessary libraries:

# Install required modules and load the insurance dataset
!pip install pandas seaborn matplotlib numpy
import pandas as pd
import numpy as np
!curl -O https://raw.githubusercontent.com/stedy/Machine-Learning-with-R-datasets/refs/heads/master/insurance.csv
df = pd.read_csv('insurance.csv')
df.head()

We're using the same insurance dataset from the previous tutorial. Let's quickly remind ourselves what it contains.

Output:

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

Verifying Data Structure

Before we start engineering features, let's verify our dataset structure:

# Check the dataset structure and data count
df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   age       1338 non-null   int64
 1   sex       1338 non-null   object
 2   bmi       1338 non-null   float64
 3   children  1338 non-null   int64
 4   smoker    1338 non-null   object
 5   region    1338 non-null   object
 6   charges   1338 non-null   float64
dtypes: float64(2), int64(2), object(3)
memory usage: 73.3+ KB

Perfect! We have 1,338 records with no missing values.

Data Quality Check

Let's verify that our dataset only contains adult records, since this is health insurance data:

# Check if we have any children records
print(df[df['age'] < 18])

Output:

Empty DataFrame
Columns: [age, sex, bmi, children, smoker, region, charges, obese]
Index: []

Good! All records are for adults (age 18 and above), which makes sense for individual health insurance policies.

Feature Engineering: Creating the Obesity Flag

Now comes the exciting part - creating new features! Our first engineered feature will be an obesity flag based on medical guidelines.

According to the World Health Organization (WHO):

• Overweight: BMI ≥ 25

• Obese: BMI ≥ 30

Let's create this feature along with converting our categorical variables to numerical format:

# Convert 'male' to 1 and 'female' to 0 using the .replace() method
df['sex'] = df['sex'].replace({'male': 1, 'female': 0}).astype('int8')
df['smoker'] = df['smoker'].replace({'yes': 1, 'no': 0}).astype('int8')

# Lets add a flag for obesity
# As per WHO [https://www.who.int/news-room/fact-sheets/detail/obesity-and-overweight]
# For adults, WHO defines overweight and obesity as follows:
# overweight is a BMI greater than or equal to 25; and
# obesity is a BMI greater than or equal to 30.

# Use np.where to apply the conditional logic:
# Condition: df['bmi'] >= 30
# Value if True: 1
# Value if False: 0

df['obese'] = np.where(df['bmi'] >= 30, 1, 0).astype('int8')

# Print the modified DataFrame to show the result
print("\nDataFrame after converting 'male' to 1 and 'female' to 0:")
print(df)

Output:

DataFrame after converting 'male' to 1 and 'female' to 0:
      age  sex     bmi  children  smoker     region      charges  obese
0      19    0  27.900         0       1  southwest  16884.92400      0
1      18    1  33.770         1       0  southeast   1725.55230      1
2      28    1  33.000         3       0  southeast   4449.46200      1
3      33    1  22.705         0       0  northwest  21984.47061      0
4      32    1  28.880         0       0  northwest   3866.85520      0
...   ...  ...     ...       ...     ...        ...          ...    ...
1333   50    1  30.970         3       0  northwest  10600.54830      1
1334   18    0  31.920         0       0  northeast   2205.98080      1
1335   18    0  36.850         0       0  southeast   1629.83350      1
1336   21    0  25.800         0       0  southwest   2007.94500      0
1337   61    0  29.070         0       1  northwest  29141.36030      0

[1338 rows x 8 columns]

Notice our new obese column! We've now converted the continuous BMI variable into a binary flag. This can sometimes help models capture non-linear relationships more effectively.

Why Create an Obesity Flag?

While we already have BMI as a continuous variable, creating a binary obesity flag can help because:

• Medical research shows obesity (BMI ≥ 30) is a distinct risk category

• It captures a threshold effect that might be harder for linear models to detect

• It's based on domain knowledge from healthcare

Visualizing Relationships

Let's explore how age and charges are related with some visualizations. This helps us understand our data and discover potential new features.

# Explore charges vs age data
import seaborn as sns
from matplotlib import pyplot as plt
%matplotlib inline

df.plot(kind='scatter', x='age', y='charges', figsize=(10, 5)).set_title("Charges vs Age")

This scatter plot shows how insurance charges vary with age. Notice the distinct clusters - this suggests there might be important categorical factors affecting charges.

Discovering the Smoking Impact

Let's visualize how smoking status affects the relationship between age and charges:

# Explore the impact of age and smoking
g = sns.pairplot(data = df[['age', 'sex', 'bmi', 'children', 'smoker', 'charges']],
                 x_vars=['age'], y_vars=['charges'], aspect=1.5, hue='smoker')
g.fig.set_size_inches(10, 5)
plt.title("Impact of age and smoking on charges")

This visualization is revealing! We can see two distinct clusters:

• Non-smokers (blue): Lower charges that increase gradually with age

• Smokers (orange): Significantly higher charges with steeper age-related increases

This suggests that smoking has a major impact on insurance charges, and this impact might vary with age.

Exploring the Obesity Effect

Now let's examine how obesity affects the relationship:

# Explore the impact of age and obesity on charges
g = sns.pairplot(data = df[['age', 'sex', 'bmi','obese', 'children', 'smoker', 'charges']],
                 x_vars=['age'], y_vars=['charges'], aspect=1.5, hue='obese')
g.fig.set_size_inches(10, 5)
plt.title("Impact of age and obesity on charges")

Obesity also shows a clear effect on insurance charges, though perhaps not as pronounced as smoking.

Creating an Interaction Feature

Here's where feature engineering gets really powerful. We noticed that both smoking and obesity affect charges. But what about people who are both smokers and obese? This combination might have an amplified effect.

This is called an interaction feature - a new feature created by combining two or more existing features to capture their combined effect.

# Lets create a new feature which represents a product of smoker and obesity feature
df['smoker_obese'] = df['smoker'] * df['obese']
print("Number of customers who are both obese and smoke: ", df[df.smoker_obese == 1].shape[0])
print("Total number of customers: ", df.shape[0])

Output:

Number of customers who are both obese and smoke:  145
Total number of customers:  1338

About 10% of customers are both smokers and obese. This is a high-risk group that likely has significantly higher insurance charges.

What is an Interaction Feature?

An interaction feature captures the combined effect of two or more features. The mathematical operation here is multiplication:

• If someone is obese (1) AND a smoker (1): smoker_obese = 1 × 1 = 1

• If someone is only obese: smoker_obese = 1 × 0 = 0

• If someone is only a smoker: smoker_obese = 0 × 1 = 0

• If neither: smoker_obese = 0 × 0 = 0

This allows the model to assign a separate coefficient to this high-risk combination.

Preparing Features for Training

Now let's select our features for model training. Notice we're including our newly engineered features:

# Lets create new dataframes with the features and one with target
x = df[['age', 'bmi', 'sex', 'children', 'smoker', 'obese', 'smoker_obese']]
y = df['charges']

Our feature set now includes:

• Original features: age, bmi, sex, children, smoker

• Engineered features: obese, smoker_obese

Training the Improved Model

Let's train a linear regression model with our enhanced feature set:

# Lets train the model
from sklearn import linear_model

# Create a new Linear Regression model
lr = linear_model.LinearRegression()

# Train the model
lr.fit(x, y)

# Print the coefficients
ceoffs = pd.DataFrame(lr.coef_, x.columns, columns=['Coefficient'])
print(ceoffs)

Output:

               Coefficient
age             263.807602
bmi              98.637188
sex            -488.091970
children        515.971652
smoker        13431.633343
obese          -805.123043
smoker_obese  19734.622381

Understanding the New Coefficients

Let's interpret what these coefficients tell us:

• age (263.81): Each additional year adds ~$264 to charges

• bmi (98.64): Each BMI unit adds ~$99 to charges (note: much lower than before)

• sex (-488.09): Males have ~$488 lower charges than females (interesting!)

• children (515.97): Each child adds ~$516 to charges

• smoker (13,431.63): Smoking adds a whopping ~$13,432 to charges!

• obese (-805.12): Obesity flag alone actually shows negative effect (because the interaction term captures the real impact)

• smoker_obese (19,734.62): Being both a smoker AND obese adds an additional ~$19,735!

The smoker_obese coefficient is the highest, confirming our hypothesis that this combination is especially costly.

Making Predictions

Let's use our improved model to make predictions:

# Lets try to predict
predictions = lr.predict(x)
print(predictions)

scores = pd.DataFrame({'Actual': y, 'Predicted': predictions})
scores.head()

Output:

[16316.56695109  2422.88293888  6016.95162578 ...  2698.805796
  3205.41071655 27511.89173746]

        Actual     Predicted
0  16884.92400  16316.566951
1   1725.55230   2422.882939
2   4449.46200   6016.951626
3  21984.47061   5577.727872
4   3866.85520   5923.004906

Notice how much closer the predictions are to the actual values compared to our first model!

Evaluating the Improved Model

Now for the moment of truth - let's see how much our feature engineering improved the model:

from sklearn import metrics

print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y, predictions)))
print('Mean Absolute Error:', metrics.mean_absolute_error(y, predictions))
print('Mean Squared Error:', metrics.mean_squared_error(y, predictions))

print("Average Cost:", y.mean())
print("R-squared:", metrics.r2_score(y, predictions))

Output:

Root Mean Squared Error: 4490.387801338095
Mean Absolute Error: 2460.035500296957
Mean Squared Error: 20163582.606405977
Average Cost: 13270.422265141257
R-squared: 0.8624047908410836

Performance Comparison

Let's compare our improved model with the original:

What This Means

Our improved model explains 86.2% of the variance in insurance charges, compared to just 12.3% before. This is a dramatic improvement!

• RMSE dropped by 60%: Our predictions are now much more accurate

• MAE dropped by 73%: The average prediction error is just $2,460 instead of $8,982

• R-squared increased to 0.862: We now explain 86.2% of the variation in charges

This demonstrates the enormous power of feature engineering!

Key Takeaways

1. Feature engineering is powerful: Simple feature engineering improved R² from 0.123 to 0.862

2. Domain knowledge matters: Understanding obesity thresholds helped create meaningful features

3. Interaction features capture combined effects: The smoker_obese feature was crucial

4. Visualization guides feature creation: Plotting helped us discover the smoking and obesity patterns

5. Small datasets benefit greatly from good features: With only 1,338 records, feature engineering was essential

Why Did This Work So Well?

Our feature engineering succeeded because:

1. Domain-driven: We used medical knowledge (BMI ≥ 30 for obesity) to create meaningful categories

2. Captured non-linearity: The obesity flag helped the linear model capture threshold effects

3. Interaction effects: The smoker_obese feature captured the amplified risk of combined factors

4. Data-driven discovery: Visualization helped us identify which features to engineer

Next Steps

To further improve this model, you could:

1. Create more interaction features: Try age * smoker, bmi * age, etc.

2. Polynomial features: Create squared or cubed terms (age², bmi², etc.)

3. Encode region: We excluded region - adding it might help

4. Try other algorithms: Random Forest or Gradient Boosting might capture even more patterns

5. Cross-validation: Use proper train/test splits to validate performance

Conclusion

Congratulations! You've seen firsthand how powerful feature engineering can be. By adding just two simple features (obesity flag and smoker-obesity interaction), we improved our model's R² from 0.123 to 0.862 - a massive improvement!

This tutorial demonstrates a key principle in machine learning: Better features often matter more than better algorithms. Before reaching for complex deep learning models, invest time in understanding your data and engineering meaningful features.

Remember the workflow:

1. Explore your data through visualization

2. Apply domain knowledge to create meaningful features

3. Test interaction effects between important variables

4. Evaluate and iterate on your features

In our next post, we'll explore train-test splits, cross-validation, and how to properly evaluate model performance to avoid overfitting.

What's Next? Stay tuned for our next post where we'll explore proper model validation techniques and introduce regularization!

Have questions about feature engineering? Feel free to reach out or leave a comment below.

Welcome to the first post in our Machine Learning Basics series! In this tutorial, we'll dive into one of the most fundamental algorithms in machine learning: Linear Regression. We'll build a simple linear regression model to predict insurance charges based on various demographic and health factors.

Linear regression is an excellent starting point for anyone learning machine learning because it's intuitive, interpretable, and forms the foundation for many more complex algorithms.

What You'll Learn

By the end of this tutorial, you'll understand:

• How to prepare data for machine learning

• The basics of linear regression

• How to build and train a linear regression model

• How to evaluate model performance

• How to interpret model coefficients

The Dataset

We'll be working with a health insurance dataset that contains information about:

• Age: Age of the individual

• Sex: Gender (male/female)

• BMI: Body Mass Index

• Children: Number of children/dependents

• Smoker: Whether the person smokes (yes/no)

• Region: Geographic region

• Charges: Medical insurance charges (our target variable)

Getting Started

First, let's import the necessary libraries and load our dataset:

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn import linear_model, metrics
import numpy as np

# Load the insurance dataset
df = pd.read_csv('insurance.csv')
print(df.head())

Output:

   age     sex     bmi  children smoker     region      charges
0   19  female  27.900         0    yes  southwest  16884.92400
1   18    male  33.770         1     no  southeast   1725.55230
2   28    male  33.000         3     no  southeast   4449.46200
3   33    male  22.705         0     no  northwest  21984.47061
4   32    male  28.880         0     no  northwest   3866.85520

Let's examine the structure of our data:

df.info()

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1338 entries, 0 to 1337
Data columns (total 7 columns):
 #   Column    Non-Null Count  Dtype
---  ------    --------------  -----
 0   age       1338 non-null   int64
 1   sex       1338 non-null   object
 2   bmi       1338 non-null   float64
 3   children  1338 non-null   int64
 4   smoker    1338 non-null   object
 5   region    1338 non-null   object
 6   charges   1338 non-null   float64
dtypes: float64(2), int64(2), object(3)
memory usage: 73.3+ KB

This gives us important information about our dataset:

• 1,338 entries (rows)

• 7 columns with no missing values

• Mix of numerical (age, bmi, charges) and categorical (sex, smoker, region) data

Data Preprocessing

Machine learning algorithms work with numerical data, so we need to convert categorical variables to numerical format. This process is called encoding.

# Convert categorical variables to numerical
df['sex'] = df['sex'].replace({'male': 1, 'female': 0})
df['smoker'] = df['smoker'].replace({'yes': 1, 'no': 0})

# Print the modified DataFrame to show the result
print("\nDataFrame after converting 'male' to 1 and 'female' to 0:")
print(df)

Output:

DataFrame after converting 'male' to 1 and 'female' to 0:
      age  sex     bmi  children  smoker     region      charges
0      19    0  27.900         0       0  southwest  16884.92400
1      18    1  33.770         1       1  southeast   1725.55230
2      28    1  33.000         3       1  southeast   4449.46200
3      33    1  22.705         0       1  northwest  21984.47061
4      32    1  28.880         0       1  northwest   3866.85520
...   ...  ...     ...       ...     ...        ...          ...
1333   50    1  30.970         3       1  northwest  10600.54830
1334   18    0  31.920         0       0  northeast   2205.98080
1335   18    0  36.850         0       0  southeast   1629.83350
1336   21    0  25.800         0       0  southwest   2007.94500
1337   61    0  29.070         0       0  northwest  29141.36030

[1338 rows x 7 columns]

Perfect! Now we can see that:

• Sex: female = 0, male = 1

• Smoker: no = 0, yes = 1

Exploratory Data Analysis

Before building our model, it's crucial to understand the relationships in our data. Visualization helps us identify patterns and potential issues.

# Create pairplot to visualize relationships
sns.pairplot(df)
plt.show()

# Focus on relationships with our target variable (charges)
sns.pairplot(data=df[['age', 'bmi', 'children', 'smoker', 'sex', 'charges']],
             x_vars=['age', 'smoker', 'bmi', 'sex'],
             y_vars='charges',
             aspect=1)
plt.show()

These visualizations help us understand:

• Which variables might be good predictors of insurance charges

• Whether there are any obvious outliers

• The distribution of our data

Preparing the Data for Machine Learning

In machine learning, we separate our data into:

• Features (X): The input variables we use to make predictions

• Target (y): The variable we want to predict

# Select features (first 5 columns excluding region)
x = df.iloc[:, :5]  # age, sex, bmi, children, smoker
y = df.iloc[:, 6]   # charges

print(x.head())
print(y.head())

Output:

   age  sex     bmi  children  smoker
0   19    0  27.900         0       0
1   18    1  33.770         1       1
2   28    1  33.000         3       1
3   33    1  22.705         0       1
4   32    1  28.880         0       1

0    16884.92400
1     1725.55230
2     4449.46200
3    21984.47061
4     3866.85520
Name: charges, dtype: float64

Building the Linear Regression Model

Now for the exciting part - building our machine learning model!

# Create and train the linear regression model
lr = linear_model.LinearRegression()
lr.fit(x, y)

# Display the coefficients
coeffs = pd.DataFrame(lr.coef_, x.columns, columns=['Coefficient'])
coeffs

Output:

          Coefficient
age        241.263511
sex        660.859891
bmi        326.761491
children   533.168130
smoker     660.859891

Understanding the Coefficients

The coefficients tell us how much each feature influences the insurance charges:

• Age (241.26): For each additional year of age, insurance charges increase by ~$241

• Sex (660.86): Being male (vs female) increases charges by ~$661

• BMI (326.76): Each unit increase in BMI adds ~$327 to charges

• Children (533.17): Each additional child increases charges by ~$533

• Smoker (660.86): Being a smoker increases charges by ~$661

Making Predictions

Let's use our trained model to make predictions:

# Make predictions on our training data
predictions = lr.predict(x)
print(predictions)

# Compare actual vs predicted values
scores = pd.DataFrame({'Actual': y, 'Predicted': predictions})
scores.head()

Output:

[ 6240.68269989  9772.39705015 12999.76207347 ...  8923.93452889
  6037.01059213 16756.06111267]

        Actual     Predicted
0  16884.92400   6240.682700
1   1725.55230   9772.397050
2   4449.46200  12999.762073
3  21984.47061   9242.565695
4   3866.85520  11019.054388

Evaluating Model Performance

It's crucial to evaluate how well our model performs. We'll use several metrics:

# Calculate performance metrics
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y, predictions)))
print('Mean Absolute Error:', metrics.mean_absolute_error(y, predictions))
print('Mean Squared Error:', metrics.mean_squared_error(y, predictions))

print("Average Cost:", y.mean())
print("R-quared:", metrics.r2_score(y, predictions))

Output:

Root Mean Squared Error: 11336.133773688362
Mean Absolute Error: 8982.350383484953
Mean Squared Error: 128507928.93495792
Average Cost: 13270.422265141257
R-quared: 0.12306876681889345

Understanding the Metrics

• RMSE (11,336): On average, our predictions are off by about $11,336

• MAE (8,982): The average absolute error is about $8,982

• R-squared (0.123): Our model explains about 12.3% of the variance in insurance charges

What Does This Mean?

An R-squared of 0.123 means our simple model only explains about 12% of the variation in insurance charges. This suggests that:

1. The model is quite basic - there's room for improvement

2. Important features might be missing - perhaps we need more variables

3. The relationship might not be purely linear - we might need more sophisticated models

Key Takeaways

1. Linear regression is interpretable: We can easily understand how each feature affects the outcome

2. Data preprocessing is crucial: Converting categorical variables to numerical format is essential

3. Visualization helps: Exploring data relationships guides model building

4. Model evaluation is important: Metrics help us understand model performance

5. Simple models are a good starting point: Even basic models provide valuable insights

Next Steps

To improve this model, you could:

1. Feature engineering: Create new features or transform existing ones

2. Include more variables: Add the 'region' variable after proper encoding

3. Try different algorithms: Random Forest, Support Vector Machines, etc.

4. Handle outliers: Identify and address unusual data points

5. Cross-validation: Use better evaluation techniques

Conclusion

Congratulations! You've built your first machine learning model using linear regression. While this simple model has limitations (R² of 0.123), it demonstrates the fundamental machine learning workflow:

1. Data collection and exploration

2. Data preprocessing

3. Model training

4. Prediction and evaluation

This foundation will serve you well as you explore more advanced machine learning techniques. In our next post, we'll explore how to improve this model and introduce more sophisticated algorithms.

What's Next? Stay tuned for our next post where we'll explore multiple linear regression with feature engineering and better evaluation techniques!

Have questions about this tutorial? Feel free to reach out or leave a comment below.