Can we inspect a model while it learns, detect an unwanted decision rule, and use that evidence to change training? I explored this with Random Forests on Colored MNIST. The aim was to encourage recognition of digit shape while keeping the original RGB images unchanged.

First, look at the shortcut

Most training digits have a color associated with their label. That makes color a convenient predictor, even though a digit's identity should survive a change of color.

Original test observations: color-aligned digits on the top row and handwritten 9s in different colors on the bottom row.

These are original dataset images. The bottom row contains different handwritten 9s, not recolored versions of the same image. No training or test observation was edited to produce this comparison.

The original experiment exposed the failure: the forest was excellent on examples that followed the training color association and much weaker when that association broke. Green 9s were particularly revealing because green was associated with 4. A model could repeatedly predict 4 for a recognizable 9.

Weighting helped, but the shortcut remained

The first intervention increased the influence of rare digit-color combinations. Full joint weighting gives equal total training weight to each combination; it changes what the forest is rewarded for fitting without rewriting any image.

The initial weighting experiment compares aligned and conflicting test accuracy for the exploratory subset and the full-data weighted model.

Bars are means across three forest seeds; whiskers show their standard deviation. The first two configurations use the same training subset. The final configuration also has more distinct training examples, so that comparison includes additional data.

The improvement was substantial, but green 9s still failed. Large weights gave the few available examples more influence without adding handwriting diversity. This motivated a more direct question: could a diagnostic of unwanted behavior guide the tree while it was being built?

Turn an explanation into a training signal

This is a white-box approach: we can inspect candidate branches and influence which one becomes part of the tree. The analogy to interpretability-guided training in neural models is the feedback loop, rather than a shared architecture.

Training feedback loop: read unchanged images, inspect candidate branches, guide tree growth, and evaluate the resulting forest.

At each node, the custom learner measures how much a candidate branch reveals about color within the same true digit. This conditional-information diagnostic avoids directly punishing digit discrimination just because color and digit are correlated in the training set.

The learner rewards digit Gini gain and subtracts a penalty for the measured color dependence. It then accepts the best feasible branch, or keeps a leaf when no candidate has positive utility. Color annotations supervise this process during training; prediction receives RGB pixels alone.

This particular diagnostic is our experimental adaptation, related to research on unwanted-attribute criteria in tree learning. It is a proxy for shortcut use, not a direct detector of a shape mechanism.

See a branch decision change

The audit records the ordinary Gini candidate and the feedback-guided choice on the same observations at the current node. Here is an actual root-node intervention:

Actual root-node audit: two candidate pixel-channel tests on the same displayed green 9, with their measured digit gains and color-information penalties.

The boxes locate the candidate pixel tests on an original image. The statistics were calculated over the training observations reaching the root, not from this image alone. The diagram compares two candidates at one node; it does not reconstruct an entire alternative tree.

Feedback accepted a slightly smaller immediate digit gain in exchange for a lower measured color association. This shows that the signal changes training decisions. Whether that change improves generalization must be checked separately.

What changed in held-out performance?

I compared ordinary Gini, static weighting, and weighting with feedback using the same custom tree search. Tree settings and bootstrap seeds were matched; only the training intervention changed. Validation selected the penalty before the three-seed test evaluation.

Matched custom-forest comparison showing conflicting accuracy, aligned accuracy, color-following predictions, and green-9 accuracy.

Adding feedback improved conflicting accuracy by 2.3 percentage points over the matched static-weighting control. It also reduced predictions that followed the misleading color, with a small loss in aligned accuracy. The separate weighted sklearn reference still performed better overall; the custom learner uses a different approximate search and capacity.

RF evidence maps: an activation-like view

The next question is visual: which pixel tests contributed to a prediction?

A Random Forest has no convolutional activation maps, so these are not Grad-CAM images. Instead, I followed the actual tree paths and attributed each child-minus-parent change in the probability of digit 9 to the pixel-channel feature tested at that branch. Averaging across trees gives an exact accounting identity:

P(9) = forest root prior + sum of path contributions

Original digit images alongside signed pixel maps showing path contributions to the probability of digit 9 for ordinary Gini, static weighting, and training feedback.

Red increases P(9); blue decreases it. All maps explain the same target class and share a symmetric logarithmic color scale, which makes small contributions visible. The faint outline marks the original digit footprint. Titles show each model's prediction and P(9). The examples deliberately include a recovered green 9, a persistent failure, and an orange comparison case; they are illustrative seed-42 cases rather than a random sample.

This produces the kind of spatial explanation often sought from activation visualizations, while using the forest's own internal computations. The method reads original images and tree paths: it does not recolor, mask, or perturb the input. The contributions plus the root prior were checked against the forest's predicted probability.

These maps are path-dependent probability contributions, not causal effects or SHAP values. Correlated pixels, repeated tests, and the order of branches affect where credit lands. A heatmap over a digit's strokes is therefore insufficient proof of shape learning.

Keep the RGB channels visible

Combining channels into one spatial map can hide an important distinction. A tree tests a particular channel at a particular pixel, so two forests can use the same location for different reasons.

Separate red-, green-, and blue-channel probability-contribution maps for the same recovered green 9 under the three custom forests.

These panels explain the same image and target class on one symmetric logarithmic scale, with a faint digit outline for orientation. White means zero net contribution. A channel with zero input intensity can still contribute because a branch can test for the absence of that channel. The colors of the heatmap encode positive or negative contributions, not the image's RGB color.

The useful observation is how the allocation of prediction credit changes. The maps do not establish that one channel corresponds to color and another to shape; both kinds of information are mixed in RGB pixels.

The failures that averages conceal

The group heatmaps provide the wider context for those selected explanations:

Mean test accuracy for every digit-color combination, with the green-9 group outlined in each model.

Each cell averages the same test group across three seeds. The outlined cell is digit 9 with green, color annotation 4. The static and guided controls use identical custom-tree settings.

Feedback improved green-9 accuracy, but most examples in that group were still misclassified. Stronger penalties also degraded validation performance. Lower measured color dependence is useful evidence about the intervention, rather than a guarantee of a better model.

What this experiment establishes

We built an inspectable feedback loop inside tree growth and obtained a modest improvement over its matched weighted control. The images stayed unchanged, and the explanations expose both the branch decisions affected by training and the pixel tests contributing to selected predictions.

The remaining limits matter. Rare groups provide little handwriting diversity, the validation set omits some digit-color combinations, and these test data had already been inspected in earlier experiments. The results are exploratory. The original high-accuracy shape reference also binarizes grayscale images, so matching it with unchanged RGB input remains a separate target.

The next step is to improve the diagnostic and test a frozen procedure on independent data. An internal explanation can become an actionable training signal, but both the explanation and the resulting generalization need validation.

Reproduce the walkthrough

The project repository accompanies the original experiment. The follow-up notebooks are RF_weight_mitigation.ipynb and RF_interpretability_training.ipynb; interpretability_forest.py contains the learner.

The new visualizations are generated by generate_rf_visuals.py, using the recorded full-data configuration and seed 42. rf_explanations.py implements the read-only path contributions. The generator verifies that refitted models reproduce the recorded prediction accuracy, checks the contribution identity, and records the selected filenames and probabilities in rf_training_feedback_results/visuals/explanation_metadata.json.