Organise preprocessing results in layers

I had to work with a dataset with over 50GB of raw data when doing analysis on a recent study. For downstream analyses, the dataset needed to be prepared with various preprocessing steps. Some features were derived through multiple preprocessing steps and could also be configured differently (think of associating data from one sensor with another one).

In the first iteration, I derived the preprocessed features necessary for my analysis directly from the raw master dataset, which is a flat unnormalized table (timestamp, sensor_name, data). Naturally, this proved to be highly inefficient, especially when iterating fast over approaches. It obviously makes more sense to cache some of the preprocessed features, especially ones which are read more often and do not change much.

Instead, I chose to layer the preprocessed features, so that layers only need to be recomputed when needed. This allowed me to change a downstream feature without having to recompute the entire dataset.

Additionally, I applied versioning to the feature layers with hive partitioning: feature_a/version=1 or predicted_labels/model=baseline. Using DuckDB or Arrow, I can then read only the desired version for downstream processing, or compare performance characteristics, etc.

While this approach sounds trivial, it requires thinking about how the data is used downstream in analysis and by other researchers. It is helpful to manually note down a dependency graph, so it immediately becomes clear which features need to be rebuilt after changing parameters in an upstream feature. If implemented with care and reproducibility in mind (which is hopefully the case when building the preprocessing), only the base layer needs to be shipped or sent to colleagues. All other layers can then be computed on their machines. It also allows for the exact opposite, when work is only required on one of the downstream layers.

A Real-World Example

In our study, we have — among other sensor data — collected the logical representation of user interfaces: UI trees. We want to predict a label for each UI tree, for example, to identify infinite scrolling behaviour or to check whether a chat is displayed to the participant. Let's say that we want to compare different classifiers (random forests, gradient boosting or deep learning) and features derived from the UI trees.

Organising the data in layers, we can save our data like this:

1. Master_Data (the big table with all sensor data)
2. UI_Trees (extracted UI trees)
3. UI_Tree_Features (features derived from UI trees)
4. UI_Tree_Labels (labels predicted on UI trees)

And if because we are using hive partitioning, we can save the results for comparison like this:

• Comparing feature engineering approaches: UI_Tree_Features/approach=a
• Predicting labels using different classifiers UI_Tree_Labels/features=a/model=rf

For our UI_Tree_Featues layer, we could also pull up the versioning in the root folder. It might be useful when we want to indicate users (like other researchers or yourself in 6 months) that the feature engineering approaches differ widely. It could also be useful when many downstream consumers depend on it, and when different features are used in different downstream layers.

When I want to test out a new classifier using the same predictors (UI_Tree_Features) I used for another classifier, I can save the results in a new folder: UI_Tree_Labels/features=a/model=xgboost.