Print ( 'Test MSE: %.3f' % mean_squared_error ( y_test, search. So what do most people do? They score the model against their holdout set: 1įrom trics import mean_squared_error # Evaluate: Search = RandomizedSearchCV ( GradientBoostingRegressor ( random_state = 42 ), scoring = 'neg_mean_squared_error', n_iter = 25, param_distributions = hyper, cv = cv, random_state = 12, n_jobs = 4 ) # Fit the grid searchĪt this point, we’ve fit a valid model, and we want to know how it performs.

Hyper = # Define our CV class (remember to always shuffle!)Ĭv = KFold ( shuffle = True, n_splits = 3, random_state = 1 ) # Define our estimator X, y = load_boston ( return_X_y = True ) X_train, X_test, y_train, y_test = train_test_split ( X, y, random_state = 42 ) # Define (roughly) our hyper parameters Here’s a quick problem setup: 1įrom sklearn.datasets import load_boston from sklearn.model_selection import train_test_split from sklearn.model_selection import RandomizedSearchCV from sklearn.model_selection import KFold from sklearn.ensemble import GradientBoostingRegressor from scipy.stats import randint import numpy as np # Load the data Most of the time I see data dredging, it’s in the context of evaluating a grid search. In a practical sense, it’s when you repeatedly expose your holdout set to your model while continuing to make adjustments. What’s data dredging?Īlso commonly called “p-hacking,” data dredging is essentially the practice of allowing your test or validation set to inform decisions around your model-building or hyper-parameter tuning.

This time, I want to cover an equally egregious practice: data dredging. Last time, we saw how covariate shift can be accidentally introduced by (seemingly harmlessly) applying a fit_transform to your test data. In the spirit of my last post, I want to continue talking about some common mistakes I see among machine learning practitioners.

Data dredging p-hacking