We regularly receive requests from investment teams for datasets that aren’t available off the shelf from data providers and therefore have to be built and sourced directly.

Mining production data is a useful case study because the raw data is public but scattered across websites, PDFs, and spreadsheets. Companies use different units, fiscal calendars, product names, and reporting bases.

In this post, we show the process of going from unstructured company filings to a structured dataset that could be used in systematic research.

Github repo

Live app

All the production information is public, but it is scattered across inconsistent websites and reports.

When talking to central data teams at hedge funds or to data providers directly, building a new dataset that is provably correct and has reliable updates always sounded like a very big challenge.

For each company, I want to extract production figures that are comparable:

• What was produced
• Which operation produced it
• Which period it covers

The hard part is normalization since every region and company reports differently (if not SEC):

• Different units across reports like copper in kt, million pounds, or wet metric tonnes
• Fiscal years don't align (calendar year vs June FY vs September FY)
• Some report on a payable basis, others contained metal, others equity-adjusted
• Product naming is inconsistent ("copper concentrate" vs "cu conc" vs "SX-EW cathode")
• Important details are sometimes hidden in headings, footnotes, and surrounding text (e.g. ownership percentages, and reporting methods)

The "old" way of doing this would be to write a bespoke ETL pipeline for each company.

The "new" way that I tried is using LLMs to generate, monitor, and maintain deterministic ETL code. An agent then runs the pipelines and jumps in whenever the script fails and needs to adapt. The idea was to have self-healing data pipelines: when a website or PDF layout changes, an agent investigates, fixes, and tests the extraction or transformation code. If it can’t figure it out, it escalates to a me for review.

Scraping code monitors company websites and captures new reports.

We can then extract the raw production figures from the reports. A mix of traditional PDF parsing and gemini-3.7-flash worked very well here. The extraction also returns the location in the source (e.g. page 123, table X, row Z, cell Y) which is very helpful for QA and source grounding.

To normalize the data, I choose between different transformation strategies:

• can the data be parsed as is?
• can I generate deterministic transformation code, e.g. a regex mapper
• last resort if no deterministic approach is possible: use an LLM to map the data

Then we validate the data against various QA rules. We treat every extracted value as wrong until it passes validation: guilty until proven innocent.

If you have a similar dataset in mind, get in touch with us. We can show you how to set up the full process in Kadoa for your sources, schema, validation rules, and update schedule.

Adrian Krebs

Co-Founder & CEO of Kadoa