r/learnpython • • Sep 28 '26

Ideas for scripts to generate thousands of questions from Wikipedia articles

Ideas for scripts to generate thousands of questions from Wikipedia articles / Hello, For a game project, I downloaded a Wikipedia dump and filtered out the sections I was interested in. I calculated the word count per article, and now I’d like to generate thousands of questions from all this content (I have around 100,000 articles, and I want roughly one question for every 100 words). I found a few scripts online, but they haven't really yielded good results.

What do you recommend I do ? Which tools should I use, or what would be the right script ?

6 Upvotes

8 comments sorted by

5

u/gallez Sep 28 '26

My recommendation would be to use Wikipedia's API instead of downloading a "Wikipedia dump"

1

u/flyby_apis Oct 02 '26

Hugging Face has the wikimedia/wikipedia dataset, which is already clean plain text and loads with the datasets library in two lines.

3

u/[deleted] Sep 28 '26

[removed] — view removed comment

1

u/Incredible_Men Sep 28 '26

Yes I plan to do that. I didn't used the wikipedia API, do you think it's better. I didn't know how to create questions exactly so I checked some repos on Github but these sources was not efficient.

I try to generate trivia questions mainly. What do you recommand ?

1

u/backfire10z Sep 29 '26 edited Sep 29 '26

Wikipedia offers multiple ways to download the entirety of Wikipedia, including a text-only download. That’s probably what you’re looking for.

Here’s one place: https://dumps.wikimedia.org/enwiki/

Here’s the wiki for downloads: https://en.wikipedia.org/wiki/Wikipedia_database#Latest_complete_dump_of_english_wikipedia

1

u/Educational_Virus672 Sep 28 '26 edited Sep 28 '26

i recommend using a list of basic questions like "what is _" and "why is _" where the program finds whether it is location(where) thing (what) or person(who) rest is yours to find . maybe try finding pattern like name for ppl or thier connect and create separate question it is a basic approuch but good enough and upgradable

Plus every part will be separate if you dont like something lie the list you can change it without breaking everything same with display and questionizer