Significance Testing

AddMaple can highlight where a segment is performing higher or lower than expected in any pivot table that compares two categorical groupings (including multi-select tags). We run z-score based significance tests for every cell so you can see which intersections are worth a second look without exporting to a separate stats package.

Turn on significance shading

1. Build the table you want to analyse. Significance is available for stacked tables with at least two segment columns and more than one category per axis.
2. Open the chart actions menu (the three-dot More menu) and toggle Significance Testing.
3. We automatically switch the table into percentage view (% of column) so you can compare segments on the same base size. Count view is disabled while significance is on.
4. A legend appears underneath the table explaining the colours that are now applied to the cells.

Read the colours

• Warmer shading means the observed share for that segment is higher than expected; cooler shading means it is lower.
• The deeper the colour, the more confident we are in the effect. Neutral (white/grey) cells are not statistically convincing.
• The legend shows five swatches: directional up/down, reliable up/down, and the neutral baseline. These match the colours shown in the table.
• Hover any coloured cell to see a tooltip with the z-score, unadjusted p, Holm-adjusted p, the percentage-point difference from expectation, and Cohen's h effect size.
• We use the same color settings as set for Opion Scales. Learn how to customize these colors here.

Why colours don’t change when you switch “% of column / % of row / % of all”

Significance is computed from the raw contingency counts, which don’t change when you change how the table displays percentages. Switching the displayed denominator only changes the formatting of the numbers, not the underlying counts or the chi-square residuals — so the z-scores (and therefore colours) stay the same.

What the tiers mean

AddMaple places each cell into one of three tiers before colouring it:

• Directional (Tier 1): Unadjusted p ≤ 0.10. Treat these as hints; they may disappear after correction.
• Reliable (Tier 2): Holm-adjusted p ≤ 0.05, but the effect size is below our practical threshold. These are statistically solid but small.
• Reliable & meaningful (Tier 3): Holm-adjusted p ≤ 0.05 and a meaningful lift of at least 5 percentage points or Cohen's h ≥ 0.20. These get the strongest shading.

For multi-select columns we relax the practical thresholds slightly (4pp / h ≥ 0.12) so commonly overlapping tags are still surfaced.

How we calculate it

• We build a contingency table of the two pivoted columns and run a chi-square test.
• Each cell's colour is based on the standardized residual (z-score) from that test.
• We correct p-values within each column using Holm-Bonferroni so repeated comparisons stay conservative.
• We suppress strong tiers when sample sizes are too small (expected counts < 5 or totals < 20) to avoid unstable highlights.
• Numeric vs categorical pivots (e.g. scalar scores against segments) still generate the same contingency table behind the scenes so the z-scores remain comparable, even while we also run the appropriate t-test or ANOVA.

Tips & limitations

• Significance only appears when every column has more than one category and the table is not filtered down to a single row or column.
• Charts keep significance on when you swap between pivoted table and chart views, but we only colour the table itself.
• Turning the toggle off restores your previous count/percentage settings.
• Combine with filters to focus on segments with enough responses before trusting the directional hints.
• If you apply respondent weights in the pivot builder, those weights flow into the contingency counts and z-scores, so the shading and tiers reflect the weighted contribution of each response.

With significance testing enabled you can scan for reliable lifts or drop-offs in seconds, then dive into the supporting rows or excerpts to understand why those differences exist.