Data ScienceIntermediate16 min readUpdated March 2025

Feature Engineering

Better features often beat fancier algorithms. Learn to encode categories, scale numbers, create new signals from dates and text, and build leak-free preprocessing pipelines with scikit-learn.

Why Features Matter

Models only learn from the inputs you give them. Feature engineering transforms raw columns into representations that expose useful patterns — and is frequently the biggest lever on model quality.

Common Transformations

Match the transformation to the data type:

Scaling — Standardize or min-max scale numeric features for distance- and gradient-based models.
One-hot encoding — Convert nominal categories into binary columns.
Ordinal encoding — Map ordered categories (low/medium/high) to integers.
Date parts — Extract day of week, month, is_weekend, days since an event.
Binning and log transforms — Tame skew or capture non-linear effects.
Interactions — Ratios and products, e.g. price per square foot.

A Leak-Free Preprocessing Pipeline

Fit transformations on training data only, then apply them to test data. A scikit-learn `ColumnTransformer` inside a `Pipeline` makes this automatic:

python

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.ensemble import RandomForestRegressor

numeric = ["sqft", "bedrooms", "age"]
categorical = ["neighborhood", "style"]

preprocess = ColumnTransformer([
    ("num", Pipeline([
        ("impute", SimpleImputer(strategy="median")),
        ("scale", StandardScaler()),
    ]), numeric),
    ("cat", Pipeline([
        ("impute", SimpleImputer(strategy="most_frequent")),
        ("onehot", OneHotEncoder(handle_unknown="ignore")),
    ]), categorical),
])

model = Pipeline([
    ("prep", preprocess),
    ("rf", RandomForestRegressor(n_estimators=200, random_state=42)),
])
model.fit(X_train, y_train)
print("R^2:", model.score(X_test, y_test))

Creating New Features

Domain knowledge produces the most valuable features:

python

df["order_date"] = pd.to_datetime(df["order_date"])

df["dow"] = df["order_date"].dt.dayofweek
df["is_weekend"] = df["dow"].isin([5, 6]).astype(int)
df["price_per_sqft"] = df["price"] / df["sqft"]
df["log_income"] = np.log1p(df["income"])

# Days since customer's previous order
df = df.sort_values(["customer_id", "order_date"])
df["days_since_prev"] = df.groupby("customer_id")["order_date"].diff().dt.days

Key Takeaways

Good features frequently outperform complex algorithms.
Fit preprocessing on training data only to avoid data leakage.
Pipelines make preprocessing reproducible and safe.
Domain knowledge drives the most useful engineered features.

Contact Us

Have a question or feedback? Fill out the form below or reach us directly at support@nvaitraining.com