Card Sorting and Information Architecture Validation in Production

TL;DR: Card sorting, tree testing, and first-click testing are useful but routinely misused, because their participant pools rarely resemble actual visitors. The standard pipeline (open sort, then closed sort, then tree test, then production A/B) is methodologically sound but suffers from systematic sample bias: research panels skew young, English-speaking, and tech-comfortable, while production visitor bases are heterogeneous and often dominated by demographics the panel undersamples. The reliable practice is to treat lab IA validation as hypothesis generation, not as confirmation, and to demand that any IA redesign produce its real proof from production navigation experiments.

A note on tools and method examples. Optimal Workshop, UserTesting, UserZoom, and Maze appear as well-known examples of research-tool archetypes referenced throughout this article. Quantitative figures and engagement-pattern observations come from anonymized advisory engagements, not from the named tools.

The Pipeline as Conventionally Taught

The information-architecture validation pipeline, as it is taught in most UX-research curricula and presented in Donna Spencer's Card Sorting: Designing Usable Categories, proceeds through four phases. Each phase has a specific question it can answer, a typical sample size, a typical cost, and a class of decision it should support. The phases compose: each one's output feeds the next. The pipeline, when followed end to end, produces a navigation structure that has been validated through progressively more realistic tests.

Phase one is the open card sort. The researcher gives participants a deck of content items (cards) and asks them to group the items into categories of their own creation, then name each category. The output is a similarity matrix (which items get grouped together by how many participants) plus a vocabulary list (what participants called the groups). The open sort answers a discovery question: what are the natural conceptual clusters in this content, and what language do users use to describe them?

Phase two is the closed card sort. The researcher proposes a category structure (often derived from the open sort) and asks participants to place each card into one of the pre-defined categories. The output is a success rate per card (the share of participants who placed it where the researcher expected). The closed sort answers a confirmation question: given this proposed structure, where do users actually expect each item to go, and which items are ambiguous?

Phase three is tree testing, the methodology pioneered by Donna Spencer and now most commonly run through tools like Optimal Workshop's Treejack. The researcher presents participants with a textual tree of the proposed site navigation (no visual design, no other UI cues) and asks them to find specific items by clicking through the tree. The output is success rate, directness (did they backtrack), and time. Tree testing answers a navigability question: assuming the visual UI is fine, can users find what they came for in this proposed structure?

Phase four is the first-click test, often combined with tree testing or running independently against a visual mockup. The researcher shows the participant a task statement and a screenshot, and asks where they would click first. The output is a heat-map of first clicks. The first-click test answers a comprehension question: does the visible label structure actually communicate what the underlying IA intends?

The IA validation pipeline from discovery through production

Loading diagram...

The pipeline is theoretically clean: each phase answers a different question, the cost rises as the test gets closer to production, and the output of one phase is the input to the next. In practice, what undermines the pipeline is not its logical structure but the consistent sample bias between research panels and actual visitors. We will return to that.

What an Open Card Sort Actually Produces

The open sort is the most generative phase and also the most over-interpreted. The researcher hands participants a stack of cards and asks them to cluster. Each participant produces their own grouping, with their own category names. The aggregation step (turning N idiosyncratic groupings into a single proposed IA) is where most card-sort projects go quietly wrong.

There are three classical aggregation methods. The first is co-occurrence frequency: for every pair of cards, count how often they ended up in the same group across all participants. This produces a similarity matrix. Items that co-occur frequently are conceptually close; items that rarely co-occur are conceptually distant. The matrix can be visualized as a dendrogram (hierarchical clustering) or as a network graph.

The second is average linkage clustering: feed the co-occurrence matrix into a hierarchical clustering algorithm and let the algorithm produce a tree. The researcher chooses a cut height to determine the number of clusters. This is what most card-sort tools (Optimal Sort, UserZoom, OpenSorts) do by default. The output looks scientific, which is part of the problem: it has the visual rhetoric of statistical analysis without the underlying statistical reliability.

The third is category-name analysis: aggregate the names participants gave their groups and cluster those. This is more linguistically honest (it captures the vocabulary users actually use) but less structurally useful (different participants name semantically equivalent clusters with different words, and there is no automatic way to merge them).

The error mode that researchers most frequently fall into is treating the clustering output as a recommendation rather than a hypothesis. The dendrogram from a 30-participant open sort is not telling you the right IA; it is telling you what 30 specific people thought, with substantial sampling variance. In advisory work we have seen the same content deck produce materially different dendrograms when sorted by two non-overlapping samples drawn from the same recruitment panel. The variance between samples is rarely reported in the deliverables that get presented to product leadership, which creates an unfounded sense of precision.

Table 1: Open card sort sample-size guidance and what each level can actually detect

Sample Size	What It Can Detect	What It Cannot Detect	Notes
10 to 15	Coarse high-level clusters; obvious category mistakes	Subtle distinctions; minority preferences; vocabulary variants	Below NN/g recommended floor for any closed-form analysis
20 to 30	Standard published practice; dendrograms become readable	Demographic interactions; second-order language patterns	Convention from Tullis and Wood (2004); slightly below true stability
30 to 50	Tullis and Wood report stable rank ordering of items by frequency	Multi-segment differences; longitudinal language shifts	Recommended floor for projects with consequential decisions downstream
50 to 100	Segment-level analysis (e.g., novice vs. expert)	Effects driven by 1-3% minority populations	Diminishing return per participant rises sharply
100+	Long-tail vocabulary; rare-group patterns	Cross-cultural differences without explicit segmentation	Cost frequently exceeds value unless multiple distinct segments expected

The classical study by Tullis and Wood (2004), Empirical Tests of Sample Sizes for Card Sorting, found that rank-ordering of items by inclusion frequency stabilizes around 30 participants. That paper is the source of the often-cited "30 participants is enough" heuristic. It is also frequently mis-stated: Tullis and Wood found rank stability around 30 for the specific question of frequency ranking, not for cluster structure stability, and not for vocabulary stability. The 30-participant floor is real but narrow.

The Sample Divergence Problem

Here is the central methodological issue in IA validation, and the one that is most often ignored in conference talks and case-study writeups. Card-sort participants are recruited from research panels. Production visitors are recruited from the open internet. The two populations are systematically different in ways that materially affect IA findings.

Research panels, the populations behind UserTesting, UserInterviews, dscout, Prolific, and most academic IA recruitment, skew younger, more US/UK English-speaking, more tech-comfortable, more accustomed to thinking analytically about interface choices, and more incentivized to provide articulate verbal explanations of their behavior than the median internet user. The panel skews matter differently for different sites. For a developer-tools company recruiting senior engineers, panel composition can be very close to the real visitor base. For a general-consumer site, like a regional bank, an insurance carrier, a healthcare provider, the panel composition is materially different from the real visitor base.

The under-discussed empirical finding from advisory engagements is the consistency of this skew. We have repeatedly seen the same pattern: card sorts with research-panel participants produce IA recommendations that test well in tree testing (also conducted on research panels), then fail or produce muted effects in production A/B tests against a representative visitor base. The pipeline is internally consistent (the lab tests agree with each other) but fails to transfer to production (the lab tests disagree with the live behavior).

Demographic distribution gap: research panels vs. typical e-commerce visitor base (illustrative composite from partner audits)

The numbers above are composite illustrative figures from a 2022-2024 audit of three e-commerce operators we worked with. The specific magnitudes vary by industry, but the pattern (panel skew toward young, educated, English-primary, desktop-comfortable users) is consistent enough that operators should assume it until proven otherwise.

The sample divergence problem is not unique to card sorting. It plagues most lab-based UX research. But card sorting is particularly susceptible because the deliverable (a category structure) is a categorical commitment: once you ship a navigation IA, it is hard to half-ship or to roll back without disruption. The cost of being wrong is structural, not just metric.

What a Closed Sort Adds and Where It Misleads

The closed sort fixes one problem with the open sort (vocabulary drift across participants) at the cost of introducing a new one (anchoring on the proposed structure). When you give participants pre-named categories, you constrain them to think in your buckets. This is useful for evaluating a specific proposal but invisible for surfacing alternatives.

The output of a closed sort is typically presented as a card-by-category success matrix: for each card, what percentage of participants placed it where the researcher expected. Cards above 85% placement are usually considered well-categorized; cards in the 50-85% range are "ambiguous"; cards below 50% are misplaced. The thresholds are conventional, not empirically derived, and they vary by tool.

The closed sort's main contribution is identifying the ambiguous middle: items that nearly half the participants put in one category and nearly half put in another. These are the cards that will produce the largest production navigation friction, because real visitors will be split on where to look. The remedy is usually one of: duplicating the item under both labels, renaming the labels to reduce ambiguity, or restructuring the IA so the boundary moves. Duplication is the most common in practice and also the most common source of long-term IA decay; we will return to that.

Table 2: Closed-sort placement-confidence thresholds and typical operational responses

Placement Confidence	Interpretation	Typical Response
>85%	Well-categorized item	Ship as-is; minor label refinement if vocabulary feedback warrants
70% to 85%	Slight ambiguity; majority is right	Test alternative label phrasings or category restructure; do not duplicate
50% to 70%	Ambiguous; meaningful minority prefers a different category	Investigate root cause; usually indicates a true cross-category item or unclear category labels
30% to 50%	Strong ambiguity; participants split	Consider IA restructure, item duplication, or content rewrite to disambiguate
<30%	Misplaced or unrecognized	Item label needs significant rework; category may need to be split or merged

Closed sorts also suffer from a subtle measurement artifact: the order in which categories are presented affects placement decisions. Most tools randomize category order to mitigate this, but the effect of category-order on cluster boundaries is not zero. In advisory work we have seen 5% to 10% placement shifts on borderline items when category order was changed deliberately for a methodological audit. The numbers reported in closed-sort outputs are not as precise as their decimal points suggest.

Tree Testing: The Pipeline's Strongest Lab Test

Tree testing, when done well, is the closest pre-production approximation of real navigation behavior available in the IA validation toolkit. It strips away visual design (which lets the researcher test the IA in isolation) and forces participants to navigate purely through label text. The output (success rate, directness, time to find) maps onto observable production behaviors better than card-sort output does.

The methodology was articulated by Donna Spencer building on earlier work by Jakob Nielsen, and is now most commonly executed through tools like Optimal Workshop's Treejack, which run the test asynchronously online and collect quantitative findability data. The standard test setup: present the participant with a task ("Where would you look for information about updating your billing address?") and a textual tree of site sections. The participant clicks through the tree until they think they have found the right destination, then submits. The tool records the path.

Three metrics emerge from tree testing, each with a different interpretation:

Success rate: did the participant land on the page the researcher considered correct? Aggregate across participants. Tasks below 60% success are usually flagged as IA problems.
Directness: did the participant go straight to the destination, or did they backtrack? Even when a participant eventually finds the right answer, a high backtrack rate indicates that the path was not obvious. Optimal Workshop quantifies this as a "directness score."
Time: how long did the task take? Time alone is a noisy signal (participants vary enormously in their pace), but time combined with directness identifies tasks where the path is unclear even to participants who eventually succeed.

The trap most operators fall into with tree testing is over-celebrating success rates. A task with 75% success and 60% directness sounds good, but it means 25% of visitors fail outright and another 15% of those who succeed take a confused path. In production, where visitors are less patient than test participants (because they are not being paid and have less commitment to completing the task), these numbers compress further. A reasonable rule of thumb from advisory work: production findability is typically 70% to 85% of lab tree-test findability for the same task, with the discount larger for casual-browsing tasks and smaller for high-intent transactional tasks.

Lab tree-test success vs. production task completion (representative advisory observations across 8 IA redesigns)

The cancel-subscription task in the chart deserves a separate note. The lab-to-production gap is much larger for cancellation flows than for other tasks, because operators frequently engineer subtle friction into cancellation paths (asking confirmation, offering retention discounts, requiring additional clicks) that tree testing does not capture. The tree-test number is the navigability number; the production number includes the dark-pattern overhead that the operator added on purpose. Conflating these two numbers in stakeholder reporting is a common analytical mistake.

The First-Click Test and Its Hidden Limits

First-click testing measures where the participant clicks first when shown a task and a visual mockup. The premise, well-established in NN/g research, is that first-click accuracy correlates strongly with task success: if the first click is wrong, the user is likely to get lost. Bob Bailey's first-click research at Web Usability found that 87% of users who clicked correctly the first time completed the task successfully, compared to 46% of users who clicked wrong first.

The methodology is simple: show a screenshot, show a task, ask "where would you click first?" and record the click location. The output is a heatmap of first clicks per task. Tasks where the heatmap is concentrated on the correct element are well-designed; tasks where the heatmap is dispersed across multiple elements are ambiguous; tasks where the heatmap is concentrated on the wrong element indicate a label or design problem.

Two limits of first-click testing are worth flagging explicitly. First, it tests a static screenshot, which has no scroll, no hover behavior, no progressive disclosure, no responsive variation. A site that performs well in first-click testing on a desktop screenshot can perform badly on mobile, where the visible area is smaller and the relevant element may not be visible until the user scrolls. Second, first-click testing measures comprehension of labels and visual hierarchy at a moment, not the navigation behavior over a session. Visitors do not actually pause and consider where to click; they scan, click, and adjust. First-click accuracy is a useful but partial proxy for actual navigation.

Production A/B Testing for IA Changes

The honest end of the IA validation pipeline is a production A/B test. The lab phases generate the candidate structure; the production test measures whether the candidate actually outperforms the incumbent on the real visitor base. This is the only phase that talks to the right population: actual visitors with actual intent, in actual context, on actual devices.

But production A/B testing of IA changes is methodologically harder than testing checkout flows or button copy. The reasons are structural:

First, IA changes affect many pages, not one. A nav redesign changes the entire site's traversal pattern, which means the unit of randomization is the visitor (or the session), not the page. Visitor-level randomization requires a stable identification mechanism (typically a cookie or login state) and produces lower-power tests per unit of traffic than page-level randomization.

Second, IA changes interact with bookmarks, search engine indexing, deep links, and habitual paths in ways that are not reversible. A returning visitor with a stored bookmark to /products/electronics will navigate by typing the URL, bypassing the new IA entirely. Search engines may have indexed the old IA's URLs and route traffic accordingly. The "treatment" in the A/B test is not a clean intervention; it is a partial intervention that bleeds across visitor cohorts.

Third, the metrics that matter for IA are usually multi-touch and delayed. A clearer nav may not improve session-level conversion (the visitor finds the product faster but does not buy more often) but may improve return visit frequency (the easier nav makes the site less frustrating to revisit). Single-session A/B tests miss the second-order effect entirely.

Fourth, the time horizon for IA to "settle in" is longer than for a button-color test. Habitual visitors need to relearn the structure. The first two weeks of an IA change typically show degraded metrics across both arms (the treatment arm because it is new, the control arm because some treatment-arm visitors land on it anyway from old links) before the new equilibrium emerges. A test called too early will show a misleading null result.

Table 3: Recommended production-test design for IA changes

Design Element	Recommendation	Rationale
Randomization unit	Visitor (cookie-based), not session	IA exposure must be consistent across visits or treatment exposure becomes noisy
Test duration	Minimum 4 weeks, ideally 6 to 8	Habituation effects dominate the first 1 to 2 weeks; settling takes longer than typical CRO tests
Primary metric	Multi-session conversion rate or downstream lifetime value	Single-session conversion misses the second-order benefits of better navigation
Secondary metrics	Session depth, return frequency, search-box usage	Search-box usage typically rises when nav is unclear; falls when nav improves
Power calculation	Plan for half the lift you would for a non-IA test	Visitor-level randomization halves the effective sample relative to session-level
Segmentation analysis	Always segment by traffic source and device	Mobile and direct-traffic segments often respond opposite to desktop and search-traffic segments

The single most useful secondary metric is search-box usage. When the navigation is unclear, visitors fall back to site search. When the navigation is clear, search-box usage drops. The ratio of search-box usage in treatment versus control is a sensitive indicator of IA quality that is independent of the primary conversion metric and is usable on much smaller samples. We have repeatedly used this metric as an early-warning signal that an IA change is failing, weeks before the conversion-rate signal would have been detectable.

From Experience

A 2023 advisory engagement with a mid-market US healthcare-information site

The product team had run a careful card-sort-then-tree-test-then-first-click pipeline over six months, recruiting through a major research panel. The recommended IA tested at 82% tree-test success on key tasks. We launched it as a 50/50 production A/B test against the incumbent IA. After three weeks, primary conversion was flat (no statistical signal in either direction), but search-box usage in the treatment arm was 34% higher than control. The treatment IA was failing to surface content for the older, less tech-comfortable visitor segment, which was 41% of real traffic and 11% of the research panel. We rolled back, rebuilt the recruitment to oversample the underrepresented demographic, and reran the validation pipeline. The second-pass IA tested slightly worse in the lab (78% tree-test success) but produced a 7% conversion lift in production.

What Goes Wrong in Real Programs: A Triage Framework

Most IA validation programs that look successful on paper but fail in production exhibit one of five problems. Naming them explicitly makes them easier to diagnose.

The first is panel under-coverage: the recruitment pool does not include the demographics that dominate real traffic. Most common in consumer sites with older or less English-fluent visitor bases.

The second is vocabulary mismatch: the labels validated in lab tests do not match the language real visitors use, often because lab participants articulate intent more clearly than typing visitors do. The result is high lab success on the test labels and high production friction on the same labels.

The third is incumbent-blindness: the IA team validates the new IA in isolation, without acknowledging that real visitors have learned the old IA and will spend a period of weeks relearning. The test is set up to compare "new IA, well-tested" against "old IA, habituated," when the relevant comparison is "new IA over time" against "old IA over time."

The fourth is mobile-blind validation: the tests were run on desktop or in a tool that defaults to desktop, while production traffic is mobile-heavy. The labels that fit cleanly on a desktop nav bar may not fit at all on mobile, where the nav is buried in a hamburger and the visible labels are different.

The fifth is single-metric reductionism: the IA team optimizes for tree-test success and ignores the other dimensions (directness, time, hover behavior, fallback search usage) that capture the visitor's experience of the navigation. The IA wins the test it was designed to win and fails the test it should have been designed for.

Decision path: Diagnosing why a validated IA failed in production

Did the production A/B test show degraded primary conversion?

If yes: Did search-box usage rise in the treatment arm?
- If yes: Outcome: Likely IA findability failure. Inspect by traffic source and demographic; common cause is panel under-coverage.
- If no: Outcome: Likely not an IA problem. Look for other treatment-specific changes (page load speed, link behavior, JS errors) introduced alongside the nav redesign.
If no: Did secondary metrics (session depth, return frequency) improve?
- If yes: Outcome: Net positive even at flat primary conversion. Consider longer test duration to surface conversion effects, and ship if secondary improvements are durable.
- If no: Outcome: Null result. Either the IA change is too small to matter or both IAs are equally good. Either way, no production justification to ship.

What Actually Works: Pragmatic Recommendations

After enough engagements where the pipeline as taught did not survive contact with production, we have settled on a set of pragmatic adjustments. None of them are revolutionary; they are the result of being burned repeatedly by the methodological gaps in the canonical pipeline.

Recruit the panel to match the visitor base. This is the single highest-leverage intervention. If your real traffic is 30% mobile-primary and 40% over age 45, your card-sort panel needs to reflect that. Most research panels can be filtered to approximate the right composition, but it requires intentional effort and often pays a premium. The premium is almost always worth it.

Validate vocabulary separately from structure. Card sorts measure both at once and conflate them. A separate vocabulary study (using techniques like NN/g's reverse card sort or topic-modeling on search-query logs) often produces more transferable label recommendations than the labels suggested by closed-sort group names.

Use site-search logs as a continuous IA signal. Site search is the real card sort, running constantly, with the right sample. Every search query is a participant labeling a category. Aggregating search queries by intent (using LLM-assisted classification works increasingly well) gives you a free, always-on view of where IA is failing.

Test IA changes against incumbent IA over a long horizon. Six weeks minimum. Plan for the first two weeks to be noisy and possibly degraded. Inspect the curve, not the single endpoint.

Segment everything. The headline number from an IA test will hide the per-segment variation that actually matters. A 0% average lift can be a 5% lift on new visitors and a -3% lift on returning visitors, which is an entirely different story than "no effect."

The cleanest signal for production IA quality is whether visitors stopped using site search. Everything else is a proxy for what site-search logs already tell you.

The Long-Run IA Decay Problem

A topic that receives almost no attention in card-sort literature but matters enormously in operating IA programs is what happens after the new structure ships. Information architectures degrade. Categories that were clean at launch become cluttered as new content is added without being mapped to a category. Items get duplicated under multiple labels because the original placement was ambiguous and somebody chose to hedge. Names that made sense at launch drift in meaning as the product surface evolves. Within 18 to 24 months, most launched IAs need re-validation, and most operators do not have a re-validation budget allocated.

The decay process is predictable. Three patterns recur across engagements. The first is category bloat: the team adds new items to existing categories that fit imperfectly, because creating a new category requires more design and approval work than placing the item into the nearest fit. Over time, every category accumulates a long tail of imperfect-fit items, and the closed-sort placement confidence (had we re-run it) would drop substantially.

The second is label drift: the underlying product or content changes, but the IA labels do not get updated. A category called "Plans and Pricing" continues to exist after the pricing model has changed to consumption-based, even though "Plans" is now a misleading vocabulary. Visitors who learned the IA at launch keep finding the right thing; new visitors confused by stale labels do not, and the cohort divergence becomes invisible because no team is measuring it.

The third is navigation accretion: the menu structure grows. New items get added to nav bars to surface new launches. Quick links get added to address support tickets about findability. Promotional banners take real estate. After two years, the nav that was carefully validated at launch is no longer the nav users see; it is now buried under marketing accretions that were never part of the original validation.

Illustrative IA performance decay over 24 months post-launch (composite from advisory audits)

The decay curve is roughly linear in observed engagements, but the slope varies enormously by team discipline. Operators with explicit IA governance (a designated owner, periodic reviews, a rule that new menu items require category-fit testing) decay at perhaps a third of the rate of operators without such governance. Operators with no governance frequently see their IA quality at 24 months fall below the pre-redesign baseline that motivated the original project. The card-sort validation pipeline focused intensely on the launch moment and almost ignored the durability question, which is where most of the actual visitor pain lives.

The pragmatic response is to treat IA validation as a recurring program, not a one-time project. The cheapest mechanism is a quarterly site-search-log review: queries that should be findable through the nav but show up frequently in search are signs of category drift. The medium-cost mechanism is an annual closed-sort spot-check on a sample of the current IA's most important content items. The expensive mechanism, a full re-run of the pipeline, is rarely necessary if the lower-cost mechanisms are in place.

What Card Sorting Cannot Tell You

A useful inventory of questions that card sorting and the broader IA validation pipeline are systematically unable to answer. Naming the limits matters because most teams over-rely on the methods to answer questions they cannot answer.

Card sorting cannot tell you whether visitors actually want to find the item at all. The participant placing a card into a category has been told to place it. The real visitor may or may not be motivated to find it. The IA can be perfectly tuned for findability and still produce zero conversion lift because the underlying content is not what visitors came for.

Card sorting cannot tell you which items deserve top-level navigation real estate versus deeper placement. Frequency of use, business priority, and conceptual centrality all matter for top-level placement, and none of those are inputs to a card sort. Operators who use card-sort cluster output to drive their top-nav structure routinely surface items into prominent positions that nobody actually clicks.

Card sorting cannot tell you about progressive disclosure. Real navigation often uses hover states, mega-menus, search-as-you-type, recommendation widgets, and personalized surfaces that adjust to the visitor. The flat tree assumed by tree testing is a methodological convenience; production navigation almost never looks like that flat tree.

Card sorting cannot tell you about cross-device or cross-session intent. A visitor who comes to a marketing page on mobile and then returns to checkout on desktop is following a journey that no IA test in the canonical pipeline simulates. The IA validation literature is dominated by single-session, single-device assumptions that do not reflect how visitors actually behave.

Card sorting cannot tell you about search-driven entry. Many visitors arrive at a deep page from a Google search query and never see the home navigation. Their IA experience is determined by the breadcrumbs, the in-page links, and the related-content widgets, not the global nav that the card-sort pipeline validates. For sites with high organic-search dependence (which is most consumer sites), the canonical IA pipeline is testing a structure that a substantial share of visitors will never see.

Key Takeaways

The standard pipeline is methodologically clean but susceptible to sample divergence. Card-sort, tree-test, and first-click tests recruit from research panels that systematically diverge from production visitor bases. The lab phases agree with each other (because they share a population) but can fail to predict production behavior.
30 participants is a real but narrow floor for card sorts. Tullis and Wood (2004) found rank-stability around 30, but that finding applies to frequency ranking, not to cluster structure, not to vocabulary, and not to demographic subgroups. Consequential decisions deserve larger samples or explicit segmentation.
Tree testing is the strongest pre-production phase, but it overstates production findability. Production task completion typically runs at 70% to 85% of lab tree-test success on the same task. The gap is largest for casual-browsing tasks and smallest for high-intent transactional tasks.
Production A/B tests of IA changes need longer horizons than typical CRO tests. Habituation effects dominate the first two weeks. Plan for six to eight weeks minimum, randomize at the visitor level, and segment by traffic source and device.
Site-search usage is a free, continuous IA quality signal. When the nav fails, visitors fall back to search. When the nav succeeds, search usage drops. The ratio of search usage between treatment and control is a more sensitive early-warning metric than aggregate conversion rate.
Panel recruitment to match the visitor base is the single highest-leverage upstream intervention. Most pipeline failures trace to research panels that did not include the demographics that dominate real traffic. The cost premium for representative recruitment is almost always lower than the cost of shipping an IA that the panel validated and the visitor base did not.