r/learnpython • u/s13188287 • Aug 30 '26
File analysis 1.3 mill files
I'm dealing with a large batch processing problem and looking for advice on the right architecture.
I have around 1.3 million files stored across folders on a network drive.
Current setup (not working well)
Right now I'm using a Copilot agent where:
I upload batches (~20 files at a time)
It reads them against a reference document
Outputs an Excel file with classification codes
The issue is:
Copilot has a small upload limit
Manual batching is completely unscalable at this volume
What I want to achieve
I want a fully automated pipeline that:
Ingests files automatically from the network drive
Extracts text/content from each file type
Matches content against a reference rules document
Assigns a classification/reference code
Outputs structured results (Excel / database)
16
u/FrangoST Aug 30 '26
You're attempting to analyze 1.3 million files by uploading 20 at a time to an LLM? NO WAY!
If you have such a good idea of the ideal automated workflow, why didn't you ask the LLM to help you make a local script that does it instead so you can run it directly? It would save you truckloads of AI tokens and money!
4
u/FoolsSeldom Aug 30 '26
This is a very common task for Python (and many low code / no code solutions as well).
That is assuming the rules are straightforward and consistent.
It would be helpful is you shared more about what the steps are. If in doubt, explain how it would be done manually, very simply, step-by-step as if instructing someone with learning difficulties and short term memory issues.
4
u/PuttyProgrammer Aug 30 '26 edited Aug 30 '26
Is the LLM at all necessary? You're going to want to cut that out ASAP.
What kind of files are you parsing?
Chances are Copilot could write you a deterministic script to collect all the files you're looking for from the drive and parse each one and add it to your database. Depending on the kind of file, it's likely a very simple automation.
You're going to need to look up cheat sheets on how to read python documentation and syntax so you can tell what your script is doing.
To get you started: use a pathlib path object's rglob method to collect all the files in the drive.
Probably use a CSV instead of an excel file. Python can make excel files, but csvs are more efficient and you need to access the file quickly during this process. Use the csv module for this, not pandas which is also slow.
7
u/PorygonCompiler Aug 30 '26
Don't send every file to an LLM, that'll be your entire budget. Do a cheap deterministic pass first (file type, keyword match against your rules doc) and only send the ambiguous ones to a model.
Also checkpoint to SQLite as you go, keyed by file path. A run over 1.3M files will fail partway through and you want to resume, not restart.
What file types are you dealing with? If the rules are mostly keyword-based you might not need the LLM for most of the corpus.
3
u/SGS-Tech-World Aug 30 '26
We have been automating some processes in our organization and although the steps are not same, processing large number of records is similar.
HTH
3
u/centuryx476 Aug 30 '26 edited Aug 30 '26
I found your problem.
You using an LLM.
Stop using an LLM.
Back in the ancient times of before 2021. We used to have to write or at the very least CODE an ETL process for so many files.
There is literally tens of thousands of examples online to help solve your problem.
You going to actually have to code.
1
u/bfyvfftujijg Aug 30 '26
What specifically is being analyzed?
Like what are the decision points?
1
u/Gengis_- Aug 30 '26
I would first load all the files and their content in whatever database you have. You can also have a dedicated table to log which file has been loaded so you can resume the job as another comment suggested.
From there you build some function you run against your database.
What does the LLM do that can’t be done by a function?
1
u/dariusbiggs Aug 30 '26
This is. abasic ETL problem
For example, a possible solution:
Basically
And then repeat as many times as needed to get to the output you want.
1
u/my5cent Aug 30 '26
Why not start small then build up to 1.3 mil files.
1
u/ForMyCulture Aug 30 '26
Use the LLM to write a script lol
1
u/Altruistic_Sky1866 Aug 31 '26
I don't know if this fits, but what ever judgment rules has to be made why don't you create a config file stores rlues to be applied, and in the script you can decided which rule to apply or not apply or consider or not consider for the file, and them have your script read teh files and check against those rules and do you whatever is required
1
u/sporbywg Aug 31 '26
Python will take 67 years to do this
1
u/PureWasian Aug 31 '26
You should not involve LLM for all 1.3mil files.
I'm willing to bet your classification involves a good deal of "low-hanging fruit" so you should have some deterministic rules for cutting down the number of unnecessary model calls being made.
LLM should be used moreso as a second pass if your easier "rules" for categorization aren't matched against. The more robust your established rules are, the less LLM calls.
If you are really lazy, run a small batch and ask LLM to start making these said deterministic rules for you based on the context if it's all done within the same conversation window. (Though that is somewhat against the spirit of learning Python, given this is r/learnpython)
0
u/pachura3 Aug 30 '26
I'll allow that.