All articles

KajoData / Career & business

How I check AI analysis: from a business question to a repeatable report

Using a fictional store, I show how to define the question, catch revenue multiplied by a join, and preserve checked calculations for recurring reports.

Kajo RudzińskiPublished 9 min read

Data passes an inspection lens and a control gate before reaching a finished chart.

You give AI a sales file and receive charts, a summary and a recommendation to increase campaign spending. It looks like finished work. Then someone in the meeting asks whether revenue includes returns. You do not know. The chart does have a lovely shade of blue, though.

I am interested in the stage between receiving a report and using it for a decision. I use AI while developing KajoData and appreciate how quickly it can help turn an idea into something that works. With analysis, I also want to explain where a number came from and what it cannot establish.

I start with the decision and separate three questions

Consider the fictional store from the source material. April revenue was PLN 100,000 and May revenue was PLN 120,000. A new campaign launched in May. The 20% increase follows from those figures. Claiming the campaign caused it requires more evidence.

Prices might have changed, a large customer might have arrived, or demand might be seasonal. Those are possible explanations. An association alone does not establish causation, a distinction also discussed in the NIST statistics handbook. Two monthly totals do not let me select one story.

Add another detail from the example. After product costs, the store had PLN 30,000 left in April and PLN 24,000 in May. Both figures are before advertising and other operating costs. Revenue rose while that remaining amount fell. I still do not know the cause or the store’s full financial result.

I therefore separate three questions: what changed, why it changed, and whether the activity was worthwhile. Each can need different data. To assess increasing a particular campaign’s budget, I need more than two sales totals: spending, returns, product costs and the rules for attributing purchases to advertising.

A list of missing data can be an honest first result. I prefer that to a confident recommendation which goes beyond the evidence. My article on data analysis methods develops the work of turning a business problem into an analytical question.

I write down the context before delegating

Knowledge of the business is concrete here. You may know when returns are recognised and what order statuses mean. You may also know that finance and marketing use customer differently. Accounts, purchases and paid purchases without returns describe different populations.

A column name does not contain all those agreements. I would write down the metric definition, period, date field and included statuses. Are we using order date, payment date or settlement date? I also ask when the data can be considered complete for this report.

My instruction to AI would be ordinary language: I want to assess whether the data supports increasing the campaign budget. First describe the available data and the meaning of one row. List missing information and assumptions requiring confirmation. Show the calculation, separating observations from possible explanations.

I do not need a collection of impressive titles for the model to adopt. I need a description of the work and the conditions for accepting it. If advertising cost is absent, I want that gap made visible rather than a default quietly inserted into a calculation.

I can revise this note after speaking to the report user. Next time, I do not need to explain every rule again. I can also see where a definition changed and decide whether the figures from two periods remain comparable.

For the example store, I would specify the first stage’s output too: a monthly comparison under the agreed definition, a description of missing data and hypotheses to investigate. I would reserve the budget recommendation for a stage with the necessary evidence. The recipient can then understand why the report ends with a question rather than a firm instruction.

After joining, I check whether I am still counting orders

In the video’s example, one order is worth PLN 100. A separate table contains its three products. Joining on order ID produces three rows, each showing the full PLN 100 order amount. Summing them gives PLN 300. The store has tripled revenue exclusively inside the report.

The matching rule explains this behaviour. PostgreSQL 18 documentation states that an INNER JOIN produces a row for each pair satisfying the join condition. One record can consequently appear beside several matching records from the other table. Valid syntax cannot detect a mistaken interpretation of an amount.

Before fixing the report, I establish its purpose. For total order revenue, I want the correct amount counted once per order. For a product breakdown, I need line amounts or an agreed allocation rule. I cannot silently assign the whole basket value to every product.

Removing repeated amounts can also be a bad repair. Two separate orders may each be worth PLN 100, and both belong in the total. Meanwhile, the three lines of one order can be genuine records. I do not want to delete them simply because their order identifier repeats.

I need the meaning of a row before and after the join. That determines how I calculate the result. If grain and cardinality are unfamiliar terms, I explain them alongside other analytical concepts.

A balance compares one order tile with three copies, beside separate product blocks.

I choose checks that can expose the likely error

I do not assume I must redo the whole analysis. I would choose checks likely to reveal a significant problem. For the store, I would start with revenue and order counts compared against the appropriate source export. Both sides need the same period and definitions.

A matching total helps, but does not prove the whole analysis is right. Errors can partly cancel each other out, and an accurate amount says little about campaign attribution. I want to check the calculation and the limits of its interpretation separately.

Next I would trace cases chosen for the risks: an ordinary order, an order with several products, a return and a transaction at the month boundary. For each, I inspect where it entered the report and which rules applied. An easy order without a return tells me little about return handling.

As an additional example, suppose someone orders on the last day of a month and returns the product in the next month. First I ask which treatment we agreed for that situation. Then I follow the single order through the source and report. If the monthly view differs from the daily one, I try to explain the difference using those rules. An unexplained discrepancy calls for further investigation.

I also want access to the calculation, whether as a formula, query or table of intermediate steps. That gives me something concrete to compare with the source. Asking whether everything is definitely correct provides a weak reference point. NIST’s 2024 generative AI profile describes the risk of confidently presented false content and reasoning.

Finally, I read the conclusions sentence by sentence. I can report the observed revenue increase. If its cause is unresolved, I label an explanation as a hypothesis. For a decision with greater consequences, I would expand the checks and involve another person. The scale of review should reflect what an error could affect.

For recurring reports, I preserve the calculations and expose gaps

Suppose the store wants the report every week. I want to preserve the checked definitions, filters and return treatment. If I restart with a general request to analyse the file each time, I add a new question: did the business change, or did the calculation change?

AI can help prepare a solution that carries out the agreed steps. I want to check those steps across several periods and keep the working calculation. A model can help describe changes, while every significant number in the summary should remain traceable to the process output.

I also decide what happens when data is missing. If the returns file has not arrived, the report should not quietly suggest that no returns occurred. I want an explicit incomplete-data notice or the relevant calculation to stop. The recipient can then see which result is not yet ready to use.

I would distinguish receiving a valid empty file for a period from receiving no file at all. The first can indicate that no events occurred; the second leaves the information missing. Showing the same zero on a chart would hide that distinction. Deciding how to handle it belongs in the report design, before scheduling regular delivery.

I would begin with one report. For an example procedure, I would record sources, definitions, required files, checks and the person accepting the result. When a rule changes, I can deliberately assess comparability across periods. Automation preserves errors as readily as correct steps, so I want to understand what will repeat.

I practise with data I can trace myself

Create a dozen or so fictional orders. Include an order with several products, a return and an event near a month boundary. Define the meaning of a row and the sales measure. The small scope lets you trace the result without using company data.

Ask AI to analyse the dataset under those rules. Then check the main amount and the selected cases. Mark summary statements you cannot substantiate. If the model attributes growth to a campaign without the relevant data, identify the missing justification.

Next improve the instruction or the calculation. Record whether the problem came from missing context, execution or interpretation. Those require different repairs. A request for more cautious wording will not fix multiplied revenue, and a correct join will not supply evidence about why sales grew.

This is why I still teach analytical fundamentals, Excel and SQL. They help me recognise a problem and assess a proposed solution. At the end of the exercise, I want to explain where the number came from, which rules were used, and which decision that number alone still cannot settle.

Key takeaways

Is asking AI to double-check the report sufficient?

That request may help, but I also need checks against source data and the calculation itself. A model can repeat the same mistaken assumption during another review.

Why can revenue increase after joining tables?

One order can match several order lines. If its full amount appears beside each line, summing that amount repeats the revenue. The calculation needs the appropriate grain.

When would I automate an analysis prepared with AI?

After agreeing definitions and filters, tracing the result and checking cases relevant to the report. I would preserve the calculations and decide how missing data should be reported before scheduling regular runs.

AI was used in an automated way while working on this article. If you want to learn how to do it too, check out this project.

Kajo Rudziński

Newsletter KajoData

Let’s keep in touch.

I write about data analysis, learning and changing careers. My newsletter and the sign-up form are in Polish. If that works for you, I’d be glad to see you there.