Calculating Feature Importance with Permutation to Explain the Model — Income Prediction Example

Feature importance measures the contribution of the features for prediction. It can be used for feature selection and XAI(Explainable AI) tasks. The permutation technique might be one of the most intuitive methods used in feature importance calculation. In this post, I'd like to share how permutation is used for calculating feature importance.

1. Permutation Explained
2. Permutation Codes
3. Feature Importance in a simple income prediction project

1. Permutation Explained

Before talking about permutation, let's think in this way. If I want to know the importance of a feature, I can just remove that feature from the dataset and check the model's accuracy. The discrepancy in the accuracy between this model(without the feature) and the original one reflects its importance. The permutation is similar to this technique, but instead of removing the feature, we use a random value of that feature to measure the discrepancy in the model's accuracy. In other words, shuffle the value of the feature you want to measure, checking how much the accuracy deviates from the original one.

2. Permutation Codes

Not exactly a code section, but the logic is:

(1) Train our model and record the accuracy on the validation set.
(2) Shuffle values for a feature, use the model to…