It is a well known fact that data scientists spend the majority of their time cleaning and wrangling data before they begin to start to model a problem. For a well defined problem with relatively clean data and knowledge of the problem domain, the time taken is far less. But when tasked with a problem that we aren’t as knowledgeable about, how can a data scientist make the best use of his time to determine whether a problem is solvable with the data provided?

Forbes findings confirm the above statement.

Enter Rapid Prototyping

This is my personal take on determining whether a project can be completed given the data at hand.

The goal of Rapid Prototyping is simple:

What is the simplest and fastest model implementation that will give us a baseline working prototype?

The concern any data scientist will have now is that: We want to build a model but we have yet to focus on that data. And without knowledge of the problem domain, how can we determine what features we should generate and use to implement this solution? I’ll answer the question of feature selection shortly, but first let’s look at how we’ll automatically generate meaningful features.

Deep Feature Synthesis

Invented by MIT and first showcased in 2015, Deep Feature Synthesis was originally designed to speed up the process of building predictive models on multi-table datasets.

Deep feature Synthesis has three key concepts:

     1. Deriving features from relationships in the data
     2. Features are generated by using simple mathematics across datasets
     3. New features are created from previously derived features

for more reading on this, check out the FeatureLabs blog post.

Our main reason for using Deep Feature Synthesis for Rapid Prototyping is to take away the need for the knowledge of the problem domain by fully automating the feature generation. DFS can create very complex features by stacking primitives (basic operations on the data) within a singular dataset and across relational tables.

As for how we’ll select features, we’ll use the primitives generated by DFS and remove any highly correlated features and use this as our feature set.

Rapid Prototyping with Deep Feature Synthesis by Example - The Titanic

I thought it apt as my maiden (pun intended) post to use the Titanic dataset to demonstrate the idea of rapid prototyping. The goal here is to create the simplest baseline model to prove that this problem can be solved. For those unfamiliar with the Titanic data (albeit, if you’re reading this you probably know exactly what I’m talking about), the goal is to predict whether a passenger survived given a set of features. A binary classification problem.

We start by reading in the data and taking a quick look at what’s there:

train = pd.read_csv("train.csv")
test = pd.read_csv("test.csv")

train.head()

Now we look at rapid prototyping, lets impute data in the most simplest way or drop it if we need to.

# Fill the age with the median value

median_age_train = train.Age.median()

# Fill missing Age, forward fill embarked, drop what we may not need for rapid prototyping
train.Age.fillna(median_age_train, inplace=True)
train.Embarked.fillna(method='ffill', inplace=True)
train.drop(['Name', 'Cabin', 'Ticket'], axis=1, inplace=True)

median_age_test = test.Age.median() # set median value

# fill NAN data
test.Age.fillna(median_age_test, inplace=True)
test.drop(['Name', 'Cabin', 'Ticket'], axis=1, inplace=True)

The second step after imputing is to automate feature engineering and to do this we use the featuretools package for deep feature synthesis. The featuretools package has two types of primitives, namely aggregation and transformation:

Aggregation

primitives = ft.list_primitives()
pd.options.display.max_colwidth = 100
primitives[primitives['type'] == 'aggregation'].head(primitives[primitives['type'] == 'aggregation'].shape[0])

Transformation

primitives[primitives['type'] == 'transform'].head(primitives[primitives['type'] == 'transform'].shape[0])

The aggregation features are very simple, while the transformation features tend to be a bit more complex. Stacking these automated features on top of each other creates more complex features that may be better predictors. The idea here is that we want to abstract ourselves away from needing domain knowledge in the short term as, if the problem can be solved relatively simply, we can spend more time developing deeper, domain specific features after we’ve proved the problem can be solved.

Deep Feature Synthesis - Coded

# Create the full dataset with both training and test data
full = train.append(test)
passenger_id=test['PassengerId']

We need to do a bit of cleanup on categorical variables to apply deep feature synthesis, our initial features need to be numeric.

# replace missing Fare
full.Fare.fillna(full.Fare.mean(), inplace=True)

# Encode Gender
full['Sex'] = full.Sex.apply(lambda x: 0 if x == "female" else 1)

# Encode Embarked
full['Embarked'] = full['Embarked'].map( {'S': 0, 'C': 1, 'Q': 2} ).astype(int)

# replace all other missing with 0
full.fillna(0, inplace=True)

Next we create the entity set, this defines the DataFrame and what each variable data type is (the default is a continuous numeric)

# We create an entity set
es = ft.EntitySet(id = 'titanic')

es = es.entity_from_dataframe(entity_id = 'full', dataframe = full.drop(['Survived'], axis=1), 
                              variable_types = 
                              {
                                  'Embarked': ft.variable_types.Categorical,
                                  'Sex': ft.variable_types.Boolean
                              },
                              index = 'PassengerId')

es

Entityset: titanic
  Entities:
    full [Rows: 1309, Columns: 8]
  Relationships:
    No relationships

We then normalize the entries, this isn’t normalization in the traditional data science sense. This is a creation of relationships between the large dataset and lookups into the mapped features:

es = es.normalize_entity(base_entity_id='full', new_entity_id='Embarked', index='Embarked')
es = es.normalize_entity(base_entity_id='full', new_entity_id='Sex', index='Sex')
es = es.normalize_entity(base_entity_id='full', new_entity_id='Pclass', index='Pclass')
es = es.normalize_entity(base_entity_id='full', new_entity_id='Parch', index='Parch')
es = es.normalize_entity(base_entity_id='full', new_entity_id='SibSp', index='SibSp')
es

Entityset: titanic
  Entities:
    full [Rows: 1309, Columns: 8]
    Embarked [Rows: 3, Columns: 1]
    Sex [Rows: 2, Columns: 1]
    Pclass [Rows: 3, Columns: 1]
    Parch [Rows: 8, Columns: 1]
    SibSp [Rows: 7, Columns: 1]
  Relationships:
    full.Embarked -> Embarked.Embarked
    full.Sex -> Sex.Sex
    full.Pclass -> Pclass.Pclass
    full.Parch -> Parch.Parch
    full.SibSp -> SibSp.SibSp

What we’ve done here is defined entities for each of the features and related them to the DataFrame we use. These new entities contain unique values of the features within the original DataFrame. We can now run the deep feature synthesis:

features, feature_names = ft.dfs(entityset = es, 
                                 target_entity = 'full', 
                                 max_depth = 2)
len(feature_names)

112

Within a few seconds we’ve generated 112 features from the 5 we originally had! Some of these may not be useful and we’d want to remove any variables that are highly correlated (or collinear).

# Threshold for removing correlated variables
threshold = 0.95

# Absolute value correlation matrix
corr_matrix = features.corr().abs()
upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))
upper.head(50)

A brief look at the features created shows that DFS has created some very simple features, this is because we set max_depth = 2 for more complex features we can increase this. We then remove one of the two features that are highly correlated. this brings the number of features that we plan to use down to 64.

Rapid XGBoost

Our next step is to build a simple classification model. I chose XGBoost purely because of it’s speed and accuracy, however, you could use a Logistic Regression, LightGBM or any other binary classification algorithm.

From here we get our full dataset,

features_positive = features_filtered.loc[:, features_filtered.ge(0).all()]

features_positive

split it into training and testing set and append our survived column onto our training set (this test set is the one we’ll use for the Kaggle submission)

train_X = features_positive[:train.shape[0]]
train_y = train['Survived']

test_X = features_positive[train.shape[0]:]

Split the training set into our training and testing split for model validation,

X_train, X_test, y_train, y_test = train_test_split(train_X, train_y, test_size=0.2, random_state=42)

Run our XGBoost with some very standard parameters.

gbm = xgb.XGBClassifier(max_depth=4, n_estimators=300, learning_rate=0.05, random_state=42)
gbm.fit(train_X, train_y)
cross_val_score(gbm,train_X, train_y, scoring='accuracy', cv=10).mean()

0.8294841675178753

An 83% accuracy on 10 fold cross-validation is pretty good! Checking the precision and recall:

print(classification_report(y_test, gbm_pred))

              precision    recall  f1-score   support

           0       0.90      0.92      0.91       105
           1       0.89      0.85      0.87        74

   micro avg       0.89      0.89      0.89       179
   macro avg       0.89      0.89      0.89       179
weighted avg       0.89      0.89      0.89       179

Really not bad at all! Submitting this onto Kaggle puts us in the top 77% of users (which isn’t all that great) but given that none of the features we used were defined by a human with domain knowledge, I’d say this is very good and definitely proves that this is a solvable problem with a lot of room for improvement.

Conclusion

In my workflow, when I need to decide whether a problem is worth working on, rapid prototyping is a large part of my process and a lot of this code is simple boilerplate code that I’ve found online and adapted to the problems I need to solve. One of the richest features of the featuretools package hasn’t been showcased here. Namely that ability to provide relations between multiple datasets that could be predictors of your main problem. Imagine with the above problem we were able to link a passenger’s name to their medical history, this could have been a valuable and these features would be automatically generated for us.

This is only the beginning though. If you apply your domain knowledge on top of a problem and then use DFS, you may be able to eek out that extra bit of accuracy to your model and thus not only is this valuable for rapid prototyping but also to be added as a tool in your data science toolbox.

You can view the full notebook over on my GitHub Page