ML Interview Part 3: Gradient Descent, Bias–Variance, Validation, Clustering, PCA, XGBoost, Neural Networks & Metrics

1. Q11. What is Gradient Descent and what are its variants?

Gradient descent is an optimisation algorithm that finds the model parameters (weights) that minimise a loss function. The gradient is the vector of partial derivatives of the loss with respect to each parameter; it points in the direction where the loss increases fastest. Moving a small step in the opposite direction reduces the loss.

Analogy: you stand on a foggy hill and want to reach the valley. You cannot see the valley, but you can feel the slope under your feet. You take a step downhill, feel the slope again, and repeat. The size of each step is the learning rate.

Update rule

θ ← θ − η · ∇J(θ)

θ = parameters (weights and bias)    η = learning rate    ∇J(θ) = gradient of the loss J

Linear regression with MSE:  J(w, b) = (1/n) Σ (ŷᵢ − yᵢ)²,  ŷᵢ = w·xᵢ + b
∂J/∂w = (2/n) Σ (ŷᵢ − yᵢ) · xᵢ        ∂J/∂b = (2/n) Σ (ŷᵢ − yᵢ)

One training loop

1. Initialise weights (zeros or small random values).
2. Predict ŷ for the training examples with the current weights.
3. Compute the loss (e.g. mean squared error or log loss).
4. Compute the gradient of the loss with respect to each weight.
5. Update every weight: w = w − learning_rate × gradient.
6. Repeat until the loss stops decreasing (convergence) or the epoch limit is reached.

Gradient descent for linear regression from scratch

import numpy as np

X = np.array([1, 2, 3, 4, 5], dtype=float)      # years of experience
y = np.array([3, 5, 7, 9, 11], dtype=float)     # salary in lakhs (y = 2x + 1)

w, b, lr = 0.0, 0.0, 0.01
for epoch in range(2000):
    y_hat = w * X + b
    error = y_hat - y
    dw = (2 / len(X)) * np.dot(error, X)
    db = (2 / len(X)) * error.sum()
    w -= lr * dw
    b -= lr * db

print(round(w, 3), round(b, 3))   # ≈ 2.0, 1.0

Momentum and Adam

Momentum:  v ← β·v + ∇J(θ);   θ ← θ − η·v                  (β ≈ 0.9)

Adam:      m ← β₁·m + (1 − β₁)·g          (mean of gradients)
           s ← β₂·s + (1 − β₂)·g²         (mean of squared gradients)
           m̂ = m / (1 − β₁ᵗ),  ŝ = s / (1 − β₂ᵗ)   (bias correction)
           θ ← θ − η · m̂ / (√ŝ + ε)       (β₁ = 0.9, β₂ = 0.999, ε = 1e-8)

Choosing the learning rate: too small and training crawls; too large and the loss oscillates or explodes (diverges). Plot the training loss per epoch. Common practice is to start with 0.001 for Adam or 0.01–0.1 for SGD and use a schedule (step decay, cosine decay or warm-up) that lowers it over time.

Feature scaling matters: if one feature ranges 0–1 and another 0–1,000,000, the loss surface becomes a long narrow valley and gradient descent zig-zags. Standardise features (mean 0, standard deviation 1) before training.

2. Q12. What is the Bias–Variance trade-off?

The expected prediction error of a model on new data can be split into three parts: bias, variance and irreducible noise.

Bias is error from wrong or overly simple assumptions. A straight line fitted to curved data has high bias: it misses the pattern on training and test data alike.

Variance is error from sensitivity to the particular training sample. A very deep decision tree memorises the training data; train it on a slightly different sample and its predictions change a lot.

Noise is randomness in the data itself that no model can remove.

Decomposition (squared error)

E[(y − f̂(x))²] = Bias[f̂(x)]² + Var[f̂(x)] + σ²

Bias[f̂(x)] = E[f̂(x)] − f(x)     how far the average model is from the truth
Var[f̂(x)]  = E[(f̂(x) − E[f̂(x)])²]  how much the model changes between training sets
σ²          = irreducible noise

Analogy: archery. High bias, low variance: every arrow lands close together but far from the bullseye. Low bias, high variance: arrows scatter all around the bullseye. The goal is arrows clustered on the bullseye. As model complexity increases, bias falls and variance rises; total error is U-shaped and the best model sits at the bottom of the U.

3. Q13. What are Overfitting and Underfitting, and how do you prevent them?

Overfitting: the model learns the training data too well, including noise and coincidences, and performs poorly on new data. Example: a student memorises last year's question paper word for word and fails when the questions are rephrased.

Underfitting: the model is too simple to capture the real pattern and performs poorly even on training data. Example: a student who only read the chapter titles.

Preventing overfitting

1. Get more training data or use data augmentation (images: flips, crops; text: back-translation).
2. Use cross-validation to measure performance honestly.
3. Regularise: L2 (Ridge) shrinks weights, L1 (Lasso) sets some to zero, Elastic Net combines both.
4. Simplify the model: fewer features, shallower trees (max_depth, min_samples_leaf), fewer layers.
5. Early stopping: stop training when validation loss stops improving.
6. Dropout in neural networks: randomly switch off neurons during training.
7. Ensembles: bagging (Random Forest) averages away variance.
8. Remove leakage: make sure no feature contains information from the future or from the target.

Regularised loss

Ridge (L2):  J = MSE + λ · Σ wⱼ²
Lasso (L1):  J = MSE + λ · Σ |wⱼ|
Elastic Net: J = MSE + λ₁ · Σ |wⱼ| + λ₂ · Σ wⱼ²

Seeing overfitting with polynomial degree

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)
X = np.sort(rng.uniform(0, 1, 40)).reshape(-1, 1)
y = np.sin(2 * np.pi * X).ravel() + rng.normal(0, 0.2, 40)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, random_state=0)

for degree in (1, 4, 15):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression()).fit(X_tr, y_tr)
    print(degree, mean_squared_error(y_tr, model.predict(X_tr)), mean_squared_error(y_te, model.predict(X_te)))
# degree 1: both errors high (underfit); degree 4: both low; degree 15: train ≈ 0, test large (overfit)

ridge = make_pipeline(PolynomialFeatures(15), Ridge(alpha=1e-3)).fit(X_tr, y_tr)   # regularisation tames degree 15

4. Q14. What is Cross-Validation and how do you tune hyperparameters?

A single train/test split gives a noisy estimate: a lucky split looks great, an unlucky one looks bad. K-fold cross-validation splits the training data into k equal folds, trains k times, each time holding out a different fold for validation, and averages the k scores.

Parameters are learned from data (weights, tree split points). Hyperparameters are set before training (learning rate, max_depth, number of trees, k in KNN, C in SVM). Cross-validation is how we choose hyperparameters without touching the final test set.

Grid search and randomized search

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV, StratifiedKFold, train_test_split
from scipy.stats import randint

X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, test_size=0.2, random_state=42)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

grid = GridSearchCV(RandomForestClassifier(random_state=42),
                    {"n_estimators": [100, 300], "max_depth": [5, 10, None]},
                    cv=cv, scoring="f1", n_jobs=-1).fit(X_train, y_train)

rand = RandomizedSearchCV(RandomForestClassifier(random_state=42),
                          {"n_estimators": randint(100, 600), "max_depth": randint(3, 20), "min_samples_leaf": randint(1, 10)},
                          n_iter=30, cv=cv, scoring="f1", random_state=42, n_jobs=-1).fit(X_train, y_train)

print(grid.best_params_, grid.best_score_)
print(rand.best_estimator_.score(X_test, y_test))   # test set used once, at the very end

5. Q15. How does K-Means clustering work?

K-means is an unsupervised algorithm that groups n data points into k clusters so that each point belongs to the cluster with the nearest centroid (mean). It minimises the within-cluster sum of squared distances, also called inertia.

Example: a retail chain clusters customers by annual spend and visits per month. K-means might find three groups: occasional low spenders, regular mid spenders and frequent high spenders, each receiving a different marketing offer.

Algorithm (Lloyd's)

1. Choose k and place k initial centroids (k-means++ spreads them out for better results).
2. Assignment step: assign every point to its nearest centroid (Euclidean distance).
3. Update step: move every centroid to the mean of the points assigned to it.
4. Repeat assignment and update until assignments stop changing or the movement is below a tolerance.

Objective

minimise  J = Σ (j = 1 … k) Σ (xᵢ ∈ Cⱼ) ‖xᵢ − μⱼ‖²

Cⱼ = points in cluster j      μⱼ = centroid (mean) of cluster j
Time per iteration: O(n · k · d)   n = points, d = features

K-means with the elbow and silhouette methods

from sklearn.cluster import KMeans
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import silhouette_score

X_scaled = StandardScaler().fit_transform(customers[["annual_spend", "visits_per_month"]])

for k in range(2, 9):
    km = KMeans(n_clusters=k, n_init=10, random_state=42).fit(X_scaled)
    print(k, round(km.inertia_, 1), round(silhouette_score(X_scaled, km.labels_), 3))

km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X_scaled)
customers["segment"] = km.labels_

6. Q16. How do DBSCAN and Hierarchical Clustering work?

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) groups points that are packed closely together and marks points in sparse regions as noise. It does not need k, finds clusters of any shape and handles outliers naturally.

Two parameters: eps (ε), the radius of a neighbourhood, and min_samples, the minimum number of points inside that radius for a point to be dense.

DBSCAN algorithm

1. Pick an unvisited point and find all points within eps.
2. If it is a core point, start a new cluster and add all its neighbours.
3. For every neighbour that is also a core point, add its neighbours too (expand the cluster).
4. If it is not a core point, mark it as noise for now (it may later become a border point).
5. Repeat until every point is visited.

Example: delivery GPS points in a city. DBSCAN finds dense hotspots (markets, office parks) of irregular shape and labels isolated pings as noise, something K-means cannot do because it forces every point into a round cluster.

Hierarchical clustering builds a tree of clusters (a dendrogram). Agglomerative (bottom-up) is the common form: start with every point as its own cluster and repeatedly merge the two closest clusters until one remains. Cutting the dendrogram at a chosen height gives the clusters.

DBSCAN and agglomerative clustering

from sklearn.cluster import DBSCAN, AgglomerativeClustering
from sklearn.neighbors import NearestNeighbors
from scipy.cluster.hierarchy import linkage, dendrogram
import numpy as np

# Pick eps from the k-distance plot: sort each point's distance to its 5th neighbour and look for the knee
distances, _ = NearestNeighbors(n_neighbors=5).fit(X).kneighbors(X)
k_dist = np.sort(distances[:, -1])

db = DBSCAN(eps=0.5, min_samples=5).fit(X)
print(set(db.labels_))            # -1 = noise

agg = AgglomerativeClustering(n_clusters=4, linkage="ward").fit(X)
Z = linkage(X, method="ward")      # dendrogram(Z) draws the merge tree

7. Q17. What is PCA and why do we reduce dimensions?

Principal Component Analysis (PCA) transforms correlated features into a smaller set of new uncorrelated features called principal components. The first component points in the direction of maximum variance in the data, the second in the direction of maximum remaining variance perpendicular to the first, and so on.

Why reduce dimensions: the curse of dimensionality (distances become less meaningful, data becomes sparse), faster training, less overfitting, removing multicollinearity, and visualising high-dimensional data in 2-D or 3-D.

Example: a house dataset has carpet area, built-up area, number of rooms and number of bathrooms, all strongly correlated. PCA may combine them into one "size" component that keeps most of the information.

PCA step by step

1. Standardise every feature (mean 0, standard deviation 1); PCA is driven by variance, so scale matters.
2. Compute the covariance matrix of the features (d × d).
3. Compute its eigenvectors (directions) and eigenvalues (variance along each direction).
4. Sort components by eigenvalue, largest first.
5. Keep the top k components that explain enough variance (e.g. 95%).
6. Project the data onto those k components: Z = X · Wₖ.

Key formulas

Covariance matrix:      C = (1 / (n − 1)) · Xᵀ X          (X standardised)
Eigen decomposition:    C · v = λ · v
Explained variance of component i = λᵢ / Σ λⱼ
Projection:             Z = X · W_k     (W_k = first k eigenvectors as columns)

PCA in scikit-learn

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
import numpy as np

X_scaled = StandardScaler().fit_transform(X)
pca = PCA().fit(X_scaled)
cumulative = np.cumsum(pca.explained_variance_ratio_)
k = int(np.argmax(cumulative >= 0.95)) + 1
print(f"{k} components keep 95% of the variance")

pipe = make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression(max_iter=1000))

8. Q18. How do Gradient Boosting and XGBoost work?

Gradient Boosting builds an ensemble of shallow decision trees one after another. Each new tree is trained to predict the residual errors (more generally, the negative gradient of the loss) of the ensemble built so far, and its prediction is added with a small weight, the learning rate.

Example: predicting house prices. Tree 1 predicts the average price, ₹60 lakh, for a house actually worth ₹75 lakh, leaving a residual of +15. Tree 2 learns that houses in this locality are under-predicted and adds +10 × 0.1. Each tree corrects a little more of the remaining error.

Gradient boosting for regression

1. Start with a constant prediction F₀(x) = mean(y).
2. Compute residuals rᵢ = yᵢ − F(xᵢ) (the negative gradient of squared error).
3. Fit a small tree hₘ(x) to the residuals.
4. Update: Fₘ(x) = Fₘ₋₁(x) + η · hₘ(x).
5. Repeat for M trees, or stop early when the validation loss stops improving.

XGBoost (Extreme Gradient Boosting) is an optimised implementation with: second-order gradients (Newton steps); L1 and L2 regularisation on leaf weights in the objective; built-in handling of missing values (learns the best direction for missing data at each split); column and row subsampling; parallel split finding with histogram methods; and tree pruning based on gain.

LightGBM grows trees leaf-wise with histogram binning (very fast on large data); CatBoost handles categorical features natively with ordered target statistics.

XGBoost objective and split gain

Obj = Σ l(yᵢ, ŷᵢ) + Σ Ω(fₖ),   Ω(f) = γ·T + ½·λ·Σ wⱼ²     T = leaves, w = leaf weights

Optimal leaf weight:  wⱼ* = − Gⱼ / (Hⱼ + λ)         G = sum of gradients, H = sum of hessians
Gain = ½ [ G_L²/(H_L+λ) + G_R²/(H_R+λ) − (G_L+G_R)²/(H_L+H_R+λ) ] − γ

XGBoost with early stopping

from xgboost import XGBClassifier

model = XGBClassifier(
    n_estimators=2000, learning_rate=0.05, max_depth=5,
    subsample=0.8, colsample_bytree=0.8, reg_lambda=1.0,
    eval_metric="auc", early_stopping_rounds=50,
)
model.fit(X_train, y_train, eval_set=[(X_valid, y_valid)], verbose=False)
print(model.best_iteration)
importance = model.get_booster().get_score(importance_type="gain")

9. Q19. How do Neural Networks and Backpropagation work?

A neural network is layers of neurons. Each neuron computes a weighted sum of its inputs plus a bias and passes it through a non-linear activation function. Stacking layers lets the network learn complex non-linear relationships: early layers learn simple features, later layers combine them.

Without non-linear activations, any number of layers collapses into a single linear transformation, so activations are essential.

One layer

z = W · x + b          W = weight matrix, b = bias vector
a = g(z)               g = activation function

ReLU(z)    = max(0, z)                 hidden layers (default)
Sigmoid(z) = 1 / (1 + e^(−z))          binary output probability
Softmax(zᵢ)= e^(zᵢ) / Σⱼ e^(zⱼ)          multi-class probabilities
Tanh(z)    = (e^z − e^(−z)) / (e^z + e^(−z))

Training: forward pass, loss, backpropagation, update

1. Forward pass: compute each layer's output from inputs to prediction.
2. Loss: compare prediction with the true label (MSE for regression, cross-entropy for classification).
3. Backpropagation: apply the chain rule from the output layer back to the input layer to get ∂Loss/∂W for every weight.
4. Update: an optimiser (SGD, Adam) moves every weight against its gradient.
5. Repeat over mini-batches and epochs.

Chain rule for one weight

∂L/∂w = ∂L/∂a · ∂a/∂z · ∂z/∂w

Output layer with sigmoid + binary cross-entropy:  ∂L/∂z = a − y
Hidden layer:  δˡ = (Wˡ⁺¹)ᵀ · δˡ⁺¹ ⊙ g'(zˡ),    ∂L/∂Wˡ = δˡ · (aˡ⁻¹)ᵀ

A small network in PyTorch

import torch
from torch import nn

model = nn.Sequential(
    nn.Linear(20, 64), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(64, 32), nn.ReLU(),
    nn.Linear(32, 1),                 # logits; sigmoid is inside the loss
)
loss_fn = nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(20):
    for xb, yb in train_loader:
        optimizer.zero_grad()
        loss = loss_fn(model(xb).squeeze(1), yb.float())
        loss.backward()               # backpropagation computes all gradients
        optimizer.step()              # gradient descent update

10. Q20. Which evaluation metrics do you use: Precision, Recall, F1, ROC-AUC, PR-AUC?

Accuracy alone misleads on imbalanced data. If 1% of transactions are fraud, a model that always says "not fraud" is 99% accurate and completely useless. The confusion matrix shows exactly what kind of mistakes a classifier makes.

Metrics from this matrix

Accuracy  = (TP + TN) / total        = (15 + 950) / 1000 = 0.965
Precision = TP / (TP + FP)           = 15 / 45  = 0.333   of flagged transactions, how many are fraud
Recall    = TP / (TP + FN)           = 15 / 20  = 0.750   of all fraud, how much we caught (sensitivity, TPR)
Specificity = TN / (TN + FP)         = 950 / 980 = 0.969
F1        = 2 · P · R / (P + R)      = 2·0.333·0.75 / 1.083 = 0.462
FPR       = FP / (FP + TN)           = 30 / 980 = 0.031

ROC curve: plot True Positive Rate (recall) against False Positive Rate as the decision threshold moves from 1 to 0. AUC = 1 is perfect, 0.5 is random guessing. PR curve: plot precision against recall; its baseline equals the positive rate (0.02 here), so it shows weaknesses that ROC-AUC hides on rare-event problems.

Threshold tuning: models output probabilities; the default 0.5 threshold is rarely optimal. Choose the threshold from the PR curve using business costs, e.g. "flag when probability > 0.2 because a missed fraud costs ₹50,000 and a manual review costs ₹200".

Computing metrics and choosing a threshold

from sklearn.metrics import (confusion_matrix, classification_report, roc_auc_score,
                             average_precision_score, precision_recall_curve)
import numpy as np

proba = model.predict_proba(X_test)[:, 1]
pred = (proba >= 0.5).astype(int)

print(confusion_matrix(y_test, pred))
print(classification_report(y_test, pred, digits=3))
print("ROC-AUC", roc_auc_score(y_test, proba), "PR-AUC", average_precision_score(y_test, proba))

precision, recall, thresholds = precision_recall_curve(y_test, proba)
f1 = 2 * precision * recall / (precision + recall + 1e-12)
best = thresholds[np.argmax(f1[:-1])]
print("threshold with best F1:", round(best, 3))