5 statistical mistakes analysts make: check the conclusion

A correct query can lead to an incorrect conclusion. The figures reconcile and the chart looks good, yet the recommendation goes beyond what we actually know. It is easy to confuse more events with greater effectiveness, an average with most customers’ experience or an association with a cause. Alongside checking code, I therefore need to check the reasoning.

I will explain five mistakes using examples you can recalculate manually. All numbers and business situations are synthetic. They do not describe my clients or real campaign results. The useful habits are identifying the observation, denominator and population, then deciding how far the conclusion can extend. These questions apply regardless of which analytical tool you use.

Mistake one: treating association as causation

Imagine a shop where weeks with larger advertising budgets also have higher sales. Advertising may have helped. The company may also have increased spending before holidays, when purchase demand was already rising. Two numbers moving together do not establish the mechanism. Claiming that the budget caused a particular sales increase needs more support than two lines moving in the same direction.

I would begin with timing and the decision process. When was the budget set? Was the team responding to an earlier rise in demand? Did prices, stock availability or order measurement change simultaneously? These questions belong at the start of the analysis. Without them, it is difficult to establish what we are comparing and which explanations remain possible.

NIST describes scatter plots as tools for examining relationships, with an explicit distinction from causal proof. I would also inspect points by period or group. One unusual week can dominate the picture, while a shared trend can create an apparently strong association. A correlation coefficient alone does not reveal the relationship’s full shape or measurement problems.

Adding control variables to a regression may help describe the data, but does not automatically establish causality. Variable selection, event order and model assumptions need justification. An important factor may be unmeasured. In that case, describing the limitation and proposing a better study design is more useful than giving the slide a more confident title.

When the decision permits it, consider an experiment with random assignment and a metric defined beforehand. Assignment quality, missing observations and interactions between groups still matter. Observational methods are available when experiments are impossible, but each has conditions. For a straightforward report, it may be enough to say that an association is present and its possible explanations need investigation.

Mistake two: letting one average describe everything

Suppose we examine one hundred completed deliveries. Eighty took exactly one day and twenty took exactly seven days. The total is eighty times one plus twenty times seven, or 220 days. Dividing by one hundred gives a mean of 2.2 days. The median of this dataset is one day.

No delivery took exactly 2.2 days. That does not make the mean wrong; it shows why the mean cannot describe the complete customer experience. Most people waited briefly, while a substantial group waited much longer. If the promise was at most two days under the same time definition, twenty deliveries missed it. A median of one day also hides that problem when reported alone.

The question determines which measure is useful. A mean can help with aggregate workload; a median can help describe a typical observation. Assessing a service promise requires the share meeting the deadline and information about delays. NIST’s discussion of location measures explains why the distribution matters when interpreting a typical value.

Before adding more indicators, inspect a simple distribution. Is there one group, a long tail or more than one process? Local and international deliveries may operate under different conditions. Choose breakdowns because they address the question, rather than because one group looks attractive. My SQL histogram guide explains how to inspect counts across a distribution.

Also check which orders were eligible for the average. If it includes completed deliveries only, the longest-waiting open orders have no final duration yet. Excluding them can make a report improve while customers’ situation worsens. Show open orders and their current age separately. Do not invent an end time, but do not let those orders disappear from the process description either.

Mistake three: treating sample size as a guarantee

Nine positive responses out of ten surveys give ninety percent. Ninety out of one hundred also give ninety percent. These are identical descriptive proportions with different amounts of information. One additional answer has much greater influence in the small dataset. I would show the numerator and denominator before inviting the reader to compare percentages alone.

I would not use a rule saying nothing can be learned below thirty observations and everything becomes reliable above that threshold. The question, variability, selection process, independence and required precision all matter. NIST distinguishes these aspects of sample adequacy. A small dataset may expose a specific process defect without supporting a precise population estimate.

A large dataset can still be poorly selected. If only people who completed a purchase see a survey, it does not describe visitors who abandoned the process. If the happiest or most frustrated people are especially likely to respond, increasing the count does not remove that mechanism. Establish the target population, who could respond and who actually appears in the data.

A thousand events from ten users are different from a thousand independently selected users. Repeated observations of the same person may be dependent, affecting how uncertainty should be estimated. Counting every row as the sample size can create false confidence. The sampling unit and the table’s row unit are not always identical.

When planning a study, define the smallest difference relevant to the decision and the risks you will accept. NIST’s sample-size guidance relates the choice to required precision and variability, among other factors. A confidence interval must suit the data and assumptions. Adding the word “confidence” cannot repair a faulty sampling process.

There is more than one source of uncertainty

A confidence interval addresses a particular kind of uncertainty under a method and its assumptions. It does not automatically include measurement errors, an incorrect customer definition or every consequence of nonresponse. A narrow interval for the wrong quantity still answers the wrong question. Establish whether you need a mean, proportion, difference or prediction for an individual future observation.

In the classical interpretation, a ninety-five percent confidence level concerns the procedure: across repeated sampling under the assumptions, about that proportion of calculated intervals would contain the true parameter. NIST explains this for the mean. It does not mean that ninety-five percent of deliveries lie inside an interval for their mean duration. Individual durations require a distribution or a different interval.

A business report can state that the material describes observed cases without yet providing a stable estimate for the entire process. You might recommend more data, a limited trial or a monitored decision. The consequences of error affect that choice. I would not add a formal interval merely to make a descriptive report appear more scientific.

Mistake four: percentages without a comparison basis

In one period, a shop has two purchasing sessions out of one hundred sessions. In the next, it has three out of three hundred. Each session counts at most once as converting, regardless of order count. Converting sessions increased from two to three, a fifty percent rise. The conversion rate nevertheless fell from two percent to one percent. Both statements are true.

The rate fell by one percentage point, or fifty percent of its initial value. These expressions answer different questions. I would report the before and after values, percentage-point difference and underlying counts. A dramatic percentage without its denominator can suggest an effect the data do not establish.

Do not calculate relative growth from a zero baseline. If there were previously no conversions and now there are three, show the count change and corresponding session counts. Replacing a division-by-zero error with “100% growth” does not fix the calculation. A very small baseline also calls for a plain explanation of scale rather than a dramatic headline.

Check consistent periods and definitions. Comparing a week with a month as though only effectiveness changed is misleading. Sessions need the same counting rule and conversions must belong to the appropriate group. If event tracking changed, a jump may concern measurement. Reconcile the inputs before commenting on customer behaviour.

When the overall rate rises while both groups decline

Consider two synthetic periods and two mutually exclusive session segments: new and returning visitors, classified by an unchanged rule. Each period contains one thousand sessions. The table gives purchasing sessions and all sessions in each segment. This is neither a randomised trial nor a measurement of a specific website change’s effect.

For returning visitors, the rate falls from twenty to eighteen percent. For new visitors, it falls from five to four percent. Overall, period A has 65 converting sessions out of one thousand, or 6.5 percent. Period B has 110, or eleven percent. In this configuration, the overall rise comes from the greater share of the higher-converting segment despite declines in both segment rates.

This is a reason to inspect composition before announcing improvement. It does not mean the breakdown explains why everything changed. We still do not know why segment shares changed or why their rates declined. The aggregate answers a question about all traffic, while the segments describe its parts. Both levels are useful when their interpretations remain distinct.

Mistake five: selecting data for a predetermined answer

Imagine a team expecting confirmation of campaign success. An analyst inspects many periods and segments, then shows only the best result. The arithmetic may be correct. The problem is that the audience cannot see the other comparisons or selection process. The resulting impression of certainty is stronger than the underlying evidence.

Before analysing, record the main question, primary metric, period, exclusions and stopping rule. This does not prevent discovering new patterns. It helps distinguish a planned check from a hypothesis developed after looking at the data. An interesting newly discovered segment can be reported as exploration and investigated using subsequent observations.

Be particularly careful with repeated testing and reporting only small p-values. The ASA statement on p-values stresses transparent reporting and context; statistical significance does not measure effect size or practical importance. A conclusion should not depend solely on whether one value crosses a threshold. Nor does every descriptive table require a statistical test.

I would also write down an argument against my own recommendation. What could show the interpretation is wrong? Perhaps the result disappears after returns, concerns one unusual day or depends on a user-identification error. Examine the most plausible alternatives before deciding, and keep the checks proportionate to the consequences of an error.

Ask another person to move from the table to a conclusion without knowing the desired answer. Focus the review on the reasoning that connects the data to the recommendation. Ask them to identify where an assumption must be added. If readers understand the metric differently, clarify its definition. If they agree about the figures but disagree about the explanation, name that uncertainty in the recommendation.

Checks before sending a report

Start with one sentence defining the population, period and observational unit. Then check each figure’s definition, denominator and missing-data rule. Delivery time needs a start and end; conversion needs a session and success definition. Otherwise, a correct formula can answer a different question from the one the recipient intended.

Compare the aggregate with the distribution and important segments. Check whether extreme observations or a changing composition alter the picture. Show missing values separately rather than silently turning them into zeros. My first personal data project demonstrates that distinction. Small tables need these checks too, even when they initially appear too simple to contain a trap.

Finally, read the verbs in your conclusions. “Increased,” “is associated with” and “caused” need different evidence. A recommendation can propose action despite uncertainty, provided it identifies its basis and what should be monitored. Complete knowledge is not necessary for a useful report, but findings and hypotheses need clear boundaries.

An exercise with three headlines

Take these sentences: “Deliveries take about two days,” “Conversion increased by fifty percent” and “The new campaign improved sales.” Rewrite each using what the examples actually establish. For the first, include the 2.2-day mean and seven-day group. For the second, separate converting-session counts from the rate. For the third, remove the causal claim unsupported by the study design.

Then identify one missing item that would improve the answer: session counts, open-order status or how campaign exposure was assigned. If you cannot identify one, revisit the question. A useful statistical review ends with a specific improvement to the data, method or language of the conclusion, rather than a collection of warnings alone.