Part 1 of my post highlighted that the time to analysis has compressed. However, the work around picking the right problem and defining the problem well etc. has not. If anything, the other steps have become more complicated. Part 2 is about two things:
Protecting credibility through Audit in the AI era.
Leveraging scaling and automation to compress the other steps.
Auditing
Auditing is not a new discipline and existed long before LLMs. If you have worked in FP&A, in banking, or in corp dev, you probably had a manager who trained or grilled you until you learned to audit your model comprehensively. So, what is different with an LLM? What has changed is that we now operate at a much higher level of instruction and abstraction.
A financial model can be inspected cell by cell. The logic is visible and the source data is clear. However, when you ask an LLM to do a cohort analysis or a variance analysis, traceability becomes substantially harder. In fact, mostly, you have to explicitly demand traceability and verification from an LLM.
This can be a hard lesson to learn. I have seen an analysis walked back after the wrong conclusion had already reached senior leadership. It was a very important topic and that made the analysis failure even more painful for that team.
Where the whole process broke was not the analysis per se, but insufficient audit. The team relied on the LLM’s ability to run the numbers, and the output supported a hypothesis a leader already held. So they ran with it, because the conclusion also confirmed what the room already believed. Nobody interrogates a number that tells them they were right :). That step is reserved for numbers that directionally look wrong.
So how do I approach it? Here is my audit template:
Foundational checks, for every analysis:
Make sure you did the Control step during the definition phase well. (check Part 1 for details)
Check the metric / calculation to the aligned definition. If there is no agreed definition, that is the first thing to fix.
Have the LLM show how the calculation implements that definition, and verify it
Inspect the query or formula behind any critical numbers
Calculate one important figure yourself to verify
Reconcile to the right authoritative source: closed accounting books for financials OR data warehouse for operational metrics like pipeline
State period, currency, FX basis
Separate fact from inference. Label hypotheses as hypotheses
How deep you want to go with these checks requires judgment. You can always pursue additional checks for very high impact work. Everything above, plus:
Start a fresh chat with only the raw data and the question. Do not show it the first answer. Otherwise you risk anchoring the second review on the original reasoning.
Human peer review by someone who can audit the SQL query, not just the findings. Usually someone in Analytics, but the qualification is the skill, not the team
Falsification: what would make this wrong? Go check that
Red flags, stop and verify:
Round numbers that are unlikely
Percentages that do not reconcile to the underlying counts
A metric that changed definition mid-analysis
Most importantly, ownership applies at every level. Whoever runs the analysis signs off on it. They decide how far to take verification and they carry the confidence. Going deeper is an incremental investment and a judgment call, not a rule.
Scale and automate
Scaling and automating shrinks every step in the sequence. A definition agreed once and kept in a central .md file makes the next problem faster to define. A packaged analysis makes the next one faster to run, and lets a peer find it before they rebuild it. We need to stop treating each analysis as a one-off, and this is one of the biggest ways to create leverage through AI.
A new agent does not have memory. It can only rely on repository rules and skills that are centrally available, so anything you do not write down dies with the chat. This is where the decision spec from Part 1 comes back. Not all four of its layers may be worth keeping every single time.
The steps I use are: run the analysis, audit it, then ask what part of this someone else can reuse. Package that and create a skill out of it. Route definition changes to the person who owns and maintains the central file. Then distribution kicks in and the next person does not start from nothing. However, if everyone starts doing this, how do you make sure that there is a level of fidelity and credibility to what is being pushed centrally? This is where the approval process comes in.
On approval, the rule is clear. You audit your own work. However, if you are going to build something that is going to get used centrally or by your peers, it should get a second pair of eyes before it gets shared.
However, it is easier said than done. This is not guidance you share in one meeting. It needs consistent training, office hours by experts, and to be brought to the team in a format that they can absorb without overwhelming them. And it is worth being honest about the size of it. This is overwhelming for every single person, and that makes it tricky.
The flywheel
Early on the audit and analysis steps are heavy, and they are heavy because the scale investment has not been made yet. Every output is bespoke, so every output gets defined and audited from scratch. However, if the leaders start early to invest in scale and automation then every other step shrinks. That frees time for problem definition, and for the higher-impact problems you were not getting to. Which is why this is not really a sequence. What you scale and automate at the end feeds back into the problems you pick, and into how you define them next time.
Finance and strategy and similar analytics roles are going through a critical transition in the age of AI, where the role itself will change substantially. The generation before us had to adjust to the move into Excel. My generation never had to do that. However, this AI transition is our version of it.
People who are on the frontier of taking advantage of this transition will not only retain their jobs better, they will command a higher compensation premium in the market. They will be the most sought-after finance professionals for the next few years. This is going to be the topic of my next post.