The following is a guest post from Sarthak Gupta, a data scientist at Amazon. Opinions are the author’s own.
Most artificial intelligence projects in finance are not stalling because the technology is weak. They are stalling because we hand a model more than it can handle and then trust what comes back.
An MIT study of 300 enterprise AI deployments found that about 95% delivered no measurable impact on the P&L. Every CFO who sees that number wants to know which 5% worked, and most of the answers point to strategy or talent. One of the quieter reasons is unglamorous, and it is sitting inside your own data.
Leadership has been sold a clean story: point AI at all your data and get answers. Finance data does not cooperate with that story. It is enormous, it is interlinked, and it runs back years — the general ledger, the sub-ledgers beneath it, the transaction detail beneath those, the filings, the contracts, the board decks that reference all of it. A single reconciliation can reach across half of that at once.
Hand a model more than it can process, and it does not stop to ask for room. It works with the slice that fits and answers in the same confident tone it would use if it had read every line. Nothing on the screen tells you it read only a fraction.
Why size breaks the model
Picture the model's working memory as a desk. You can spread only so many documents across it. Push past that and pages slide off the edge, and the model never tells you which ones are left.
There is a documented version of this that researchers call "lost in the middle": give a model a long document and it leans on what sits near the beginning and the end, and gets unreliable on what sits in the middle. The figure you need is often on page 44 of a 200-page 10-K, in exactly the stretch the model handles worst.
This is measurable in finance, not just in the lab. One finance-specific benchmark, built from real 10-Ks, 10-Qs and earnings reports, found that even leading models, handed the relevant pages, answered wrong or refused on roughly four out of five questions.
These were questions a first-year analyst could answer with the filing open. Models have improved since. The structural limit has not: independent testing of today's frontier models found that each one grew less accurate as the input grew longer. More data past a point does not buy more accuracy. It buys more confident error.
The standard vendor answer to all of this is a bigger context window: load the whole data room, the pitch goes, and the problem disappears. It does not. The same testing that shows models losing accuracy with length is run on those larger windows. A model that can technically accept a million tokens still reads the middle of them poorly, and it now costs more and runs slower to hand you the same shaky answer. A bigger desk does not help if the habit is still to pile everything onto it. The teams that get this right put less in front of the model, not more.
There is a second problem that lands harder on a finance team than on an engineer. Ask the same question of the same filing twice, and you can get two different answers. In a spreadsheet, a formula returns the same result every time, and that is the whole point of controls. A model does not promise that. A number you cannot reproduce is a number you cannot sign, and it is a number you cannot hand an auditor and call evidence.
What a wrong number costs
None of this would matter if the errors announced themselves. They do not. A bad figure in a spreadsheet gets caught because someone owns the tab and someone checks it. A bad figure from an AI tool arrives clean, formatted and sure of itself, and it tends to skip the scrutiny a hand-built number would draw.
Say an FP&A team loads three years of transactions and asks for a driver-based forecast. The model cannot hold all of it while quietly working from a fraction, and the forecast that reaches the board rests on a slice no one knew was a slice. Or a deal team asks a model to pull the termination and liability terms out of a 90-page contract, and the one clause that matters sits on a page it skimmed.
The cost is not embarrassment. It is a covenant tripped, a quarter of guidance missed, a bid won at a price that loses money. In finance, the wrong number does not stay a mistake. Someone acts on it.
What to do about it
The size of the data is a design constraint you plan around, not one you wish away. These four principles hold up.
1. Break the work down the way you close the books. You do not reconcile the whole company in one enormous spreadsheet. You close each sub-ledger, tie it out, then roll it up. AI needs the same handling: split a large question and a large dataset into parts small enough to check, verify each part, then combine them. It is good close discipline under a new name.
2. Pull, do not dump. You would never hand an analyst every file in the building and ask for the DSO trend. You pull the receivables aging and the revenue detail. Scope the question, give the model only the records that bear on it, and it has a real chance. Feed it the whole archive, and it drowns. The pilots that returned something usually started here, pointed at one report and one question, not at the entire data lake.
3. Fix the data before you point AI at it. A model cannot reason its way around figures trapped in forty inconsistent spreadsheets. Cleaning and centralizing the numbers is the precondition, not the afterthought. It is the least exciting line item in the project and the one that decides whether the project works.
4. Put a named person on the numbers that carry weight, and draw the line in advance. Anything bound for the board, a lender, a customer or any figure above a materiality threshold you set clears a validation check and a signature before it moves. Everything below that line can run unattended. It is the habit engineering already uses, where a change that touches production trips a flag before it ships.
3 questions to ask before you buy
Capability is what the vendor sells. A number you can defend is what you inherit. Before the next deployment, put three questions to the vendor, and do not accept a slide in reply.
1. How do you break my data into pieces? If the answer is "we have a large context window, just load everything," you have found the problem, not the solution.
2. How do you pull only what the question needs? A tool that retrieves the right records for the question beats one that swallows the archive, every time.
3. How do you show me the number is right and repeatable? Same question, same data, same answer, with a trail I can follow. If they cannot show you that, you are not buying a control. You are buying a demo.
The teams getting real money out of AI are not the ones running the biggest model. They are the ones who accepted that their data is bigger than any model, and built for it.
