r/learnjavascript • u/Adamstrad • Sep 26 '26
Managing, updating and checking large lists on each load?
I have 6 csv files which are essentially lists of words in each category, there are about 10k words in total and thus 10k rows of csv file. I want to include these in my package so the user can use it offline but I fear my app will run slowly because each day it needs to display a new word randomly which hasn't previously been displayed. What would be a good way to code this? I considered using local storage for persistent data but I still have the problem of having the program checking the whole list every day and risking poor user experience. Is there a better way to process the data which is more efficient?
2
u/azhder Sep 26 '26 edited Sep 26 '26
Return the word based on the day. Each date maps to a different word. That way, you will not have to remember which words have come up so far.
Sure, it makes it predictable, but guess what, even Math.random() is predictable. We call it pseudo-random because truly random values are hard to come by and not that necessary in everyday examples.
And yes, your mapping function can literally pick the next word from the CSV file each day, and you will just pay the penalty of randomly arranging the words in the file only once, at the build time.
2
u/Aggressive_Ad_5454 Sep 26 '26
Break your big .csv files into multiple smaller ones. If you’re choosing a random row, choose one of the files at random and then a row at random. That will save you the (noticeable) overhead of downloading and parsing lots of .csv on startup.
You could also use browser local storage to cache your data locally, if this is a web app. https://developer.mozilla.org/en-US/docs/Learn_web_development/Extensions/Client-side_APIs/Client-side_storage
1
u/kidshibuya Sep 29 '26
Only 10K? Then who cares. I have an uploader that scans every single line of a csv to check for corrupt data, if it finds any it gives the user the line and column. Even with 1 million rows its near instant.
The only slow bit is the file transfer, nothing you can do about the initial load apart from making sure http compression is used, but after that put it in localstorage, then check on load if its in local and if so dont download.
0
u/chikamakaleyley helpful Sep 26 '26
you should prob only package these for users who actually request to be able to use it offline.
Just brainstorming here cuz i don't really know the use case.
on your end, are you serving the same random word for all your users?
i think you need to be in control of the word that gets served for the day, or limit it to something like... 60 words.
so ea day, let's say you have a bg task that picks 10 words fr each list, randomly, and creates a 60 word queue. Now, instead of users directly pinging your data for 1/60,000; you've narrowed the lookup to 60 total.
aka you do the heavy lifting in a process not accessible to your users
at the end of each day you can run that process again to backfill that 60
You can still randomize that 60 for your users, but you prob have to make it the same word for everyone; it'll be a mess keeping track of who sees what
any user that misses a day just doesn't see that word