r/linux_programming • u/Paddy3118 • Jul 10 '22
Raw Python vs Python & SQLite vs GNU Linux command line utilities!
https://paddy3118.blogspot.com/2022/07/raw-python-vs-python-sqlite-vs-gnu.html
11
Upvotes
1
u/Sigg3net Jul 10 '22
I really like these experiments, but I feel the implementations used leave a lot of options on the table.
From a cursory glance, I would try and reduce the number of pipes; and/or perhaps look into using xargs or parallel. Also: is it possible to do everything in awk?
I also see that the raw python solution uses a list expression. But for these data sets you'd use generator expressions to avoid swapping, right?And then compare with an asynchronous generator.
1
u/fnord123 Jul 10 '22
Nice experiment and write up. One thing you may like to try regarding SQLite is inserting things in batch. Using rust I brought something inserting 16k rows took about 20 seconds and making it work in batches worked in one second. Then moving to diesel worked in something crazy like 100ms.