Classification Models: Logistic Regression, Decision Trees and Random Forests

When a bank decides in seconds whether your card payment at 2 am in another city looks suspicious, a classifier is answering one question: which category does this example belong to? Three models handle the majority of real classification work on tabular data. Learn them well and you can solve — and explain — most problems you will meet as a fresher.

1. Logistic regression: a probability from a weighted sum

Despite the name, it is a classifier. Like linear regression, it computes a weighted sum of the features — then squeezes that number into a probability between 0 and 1 with the S-shaped sigmoid function. "Probability this transaction is fraud: 0.83." You then pick a threshold (often 0.5) to turn it into a yes/no decision.

• Strengths: fast, stable, gives probabilities, coefficients are explainable ("a new device raises the odds of fraud the most").
• Weaknesses: assumes a roughly linear relationship; needs scaling and feature work for complex patterns.

2. Decision tree: a flowchart learned from data

A tree asks yes/no questions, choosing at each step the question that best separates the classes:

Is the amount > ₹50,000?
├── yes → Is the device new?
│        ├── yes → FRAUD (risk 0.91)
│        └── no  → Is it a usual city? ...
└── no  → GENUINE (risk 0.03)

• Strengths: easy to explain and draw, handles non-linear patterns and mixed data, no scaling needed.
• Weaknesses: a deep tree memorises noise (overfitting), and small changes in data can produce a very different tree.

3. Random forest: many trees vote

Train hundreds of trees, each on a random sample of rows and considering a random subset of features at each split, then let them vote. Individual trees make different mistakes; averaging cancels much of the noise. This idea — combining many models — is called an ensemble.

• Strengths: strong accuracy with little tuning, robust to outliers, gives feature importance.
• Weaknesses: slower and less interpretable than one tree; larger to store.

Its close cousin, gradient boosting (XGBoost, LightGBM, CatBoost), builds trees one after another, each fixing the previous ones' mistakes. It often wins on tabular data — learn it after you are comfortable with random forests.

Train and compare all three

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, f1_score

X, y = load_breast_cancer(return_X_y=True, as_frame=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

models = {
    "Logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
    "Decision tree (depth 4)": DecisionTreeClassifier(max_depth=4, random_state=42),
    "Random forest (300 trees)": RandomForestClassifier(n_estimators=300, random_state=42),
}
for name, model in models.items():
    model.fit(X_train, y_train)
    pred = model.predict(X_test)
    print(f"{name:26s} accuracy={accuracy_score(y_test, pred):.3f}  F1={f1_score(y_test, pred):.3f}")

forest = models["Random forest (300 trees)"]
top = sorted(zip(forest.feature_importances_, X.columns), reverse=True)[:5]
for importance, feature in top:
    print(f"{feature:25s} {importance:.3f}")

Logistic regression is wrapped in a pipeline with scaling because it needs it; the trees do not. Comparing several models on the same split, with the same metrics, is standard practice.

Choosing between them

Imbalanced classes: the fraud problem

If 1 in 1,000 transactions is fraud, a model that always says "genuine" is 99.9% accurate — and useless. With imbalanced data:

• Judge models by precision, recall and F1, not accuracy (next lesson).
• Use stratify=y when splitting so both sets contain the rare class.
• Try class_weight="balanced" so mistakes on the rare class cost more.
• Tune the decision threshold instead of accepting 0.5.