Feature Engineering and Data Preparation for Machine Learning

Experienced ML engineers will tell you: most of the improvement in a real project comes from the data and the features, not from switching algorithms. A simple model with well-chosen features regularly beats a complex model fed raw columns.

Models only understand numbers

Take a dataset of used cars on an online marketplace: brand, fuel type, year, kilometres driven, city, number of owners, price. A model cannot multiply "Diesel" by a weight. Each kind of column needs a different treatment.

Categorical features: encoding

• One-hot encoding — one 0/1 column per category: fuel_Petrol, fuel_Diesel, fuel_CNG. The safe default for categories with no order.
• Ordinal encoding — numbers that respect a real order: owners "First" → 1, "Second" → 2, "Third+" → 3. Never use it for unordered categories — encoding cities as 1, 2, 3 tells the model Pune is "between" Delhi and Mumbai.
• High-cardinality categories (hundreds of car models, thousands of PIN codes): group rare values into "Other", or use target or frequency encoding carefully.

Numeric features: scaling

Kilometres driven ranges up to 2,00,000; number of owners up to 4. Models based on distances or gradients (KNN, SVM, logistic regression, neural networks) are dominated by the bigger number unless you scale.

• Standardisation (StandardScaler) — mean 0, standard deviation 1. The usual default.
• Min-max scaling — squeezes values into 0–1.
• Log transform — for skewed values like price or income, where a few huge values stretch the scale.

Tree-based models (decision trees, random forests, gradient boosting) do not need scaling — a small but frequently asked interview point.

Missing values

Impute numbers with the median, categories with the most frequent value or an explicit "Missing" category — and consider adding a 0/1 "was missing" flag, because missingness itself can carry information (a customer who skipped the income field may be different).

Creating new features

This is where domain knowledge shines:

• Car age = current year − manufacturing year (more meaningful than the year itself).
• Kilometres per year = km driven ÷ age (heavy use matters more than raw km).
• From dates: day of week, month, festive season, days since last purchase.
• Ratios: loan amount ÷ income, marks obtained ÷ maximum marks.

Data leakage: the silent killer

Leakage happens when the model gets information during training that it will not have when making real predictions. The result: a fantastic test score and a useless model.

• Target leakage: predicting whether a loan will default using a "collections_calls" column — calls only happen after a default. Ask of every feature: will I know this at prediction time?
• Train/test contamination: fitting the scaler or imputer on the whole dataset before splitting. The test set's statistics leak into training. Always fit preparation steps on the training data only.
• Time leakage: randomly splitting time-based data so the model trains on the future and is tested on the past. For forecasts, train on earlier dates and test on later ones.

Doing it properly: a Pipeline

scikit-learn's Pipeline and ColumnTransformer bundle preparation and model together, so preparation is always fitted on training data only — leakage-safe by design.

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import Ridge
from sklearn.model_selection import train_test_split

cars = pd.read_csv("used_cars.csv")
cars["age"] = 2026 - cars["year"]
cars["km_per_year"] = cars["km_driven"] / cars["age"].clip(lower=1)

numeric     = ["age", "km_driven", "km_per_year", "owners"]
categorical = ["brand", "fuel", "transmission", "city"]

prepare = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")), ("scale", StandardScaler())]), numeric),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]), categorical),
])
model = Pipeline([("prepare", prepare), ("regressor", Ridge(alpha=1.0))])

X, y = cars[numeric + categorical], cars["price"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
model.fit(X_train, y_train)                 # imputers, scaler and encoder learn from training data only
print("R² on test data:", round(model.score(X_test, y_test), 3))