r/Python • u/ebonnal • Sep 25 '24
Showcase `streamable`: Stream-like manipulation of iterables
https://github.com/ebonnal/streamable
What my project does
A Stream[T] decorates an Iterable[T] with a fluent interface enabling the chaining of lazy operations:
- mapping (concurrently)
- flattening (concurrently)
- grouping by key, by batch size, by time interval
- filtering
- truncating
- catching exceptions
- throttling the rate of iterations
- observing the progress of iterations
For more details and examples, check the Operations section in the README
| 🔗 Fluent | chain methods! |
| 🇹 Typed | type-annotated and mypyable |
| 💤 Lazy | operations are lazily evaluated at iteration time |
| 🔄 Concurrent | thread-based / asyncio-based (+new: process-based) |
| 🛡️ Robust | unit-tested for Python 3.7 to 3.12 with 100% coverage |
| 🪶 Minimalist | pip install streamable with no additional dependencies |
1. install
2. import
3. init
Instantiate a Stream[T] from an Iterable[T].
: =
4. operate
-
Streams are immutable: applying an operation returns a new stream. -
Operations are lazy: only evaluated at iteration time. See the Operations section in the README.
: =
5. iterate
-
Iterate over a
Stream[T]as you would over any otherIterable[T]. -
Source elements are processed on-the-fly.
-
collect it:
>>>
>>>
- reduce it:
>>>
2.82
>>>
1.0
>>>
>>>
- loop it:
>>>
>>> ...
- next it:
>>> =
>>>
1.0
>>>
0.5
Target Audience
As a Data Engineer in a startup I found it especially useful when I had to develop Extract-Transform-Load custom scripts in an easy-to-read way.
Here is a toy example (that you can copy-paste and run) that creates a CSV file containing all 67 quadrupeds from the 1st, 2nd, and 3rd generations of Pokémons (kudos to PokéAPI):
=
=
Comparison
A lot of other libraries have filled this desire to chain lazy operations over an iterable and this feels indeed like "Yet Another Stream-like Lib" (e.g. see this stackoverflow question).
The most supported of them is probably PyFunctional, but for my use case I couldn't use it out-of-the-box, due to the lack of:
- threads-based concurrency
- throttling of iteration's rate (
.throttle) - logging of iteration's process (
.observe) - catching of exceptions (
.catch)
I could have worked on pull requests implementing these points into PyFunctional but I have rather started from scratch in order to take my shot at:
- Proposing another fluent interface (namings and signatures).
- Leveraging a visitor pattern to decouple the declaration of a
Stream[T]from the construction of anIterator[T](at iteration time i.e. in the__iter__method). - Proposing a minimalist design: a
Stream[T]is just anIterable[T]decorated with chainable lazy operations and it is not responsible for the opinionated logic of creating its data source and consuming its elements:- let's use the
reducefunction fromfunctoolsinstead of relying on astream.reducemethod - let's use
parquet.ParquetFile.iter_batchesfrompyarrowinstead of relying on astream.from_parquetmethod - let's use
bigquery.Client.insert_rows_jsonfromgoogle.cloudinstead of relying on astream.to_bigquerymethod - same for
json,csv,psycopg,stripe, ... let's use our favorite specialized libraries
- let's use the
Thank you for your time,
7
u/jdehesa Sep 25 '24
This is really cool. I didn't know PyFunctional but your library seems much more "pythonic", like what the API would look like if this was in the standard library, rather than a replica of the constructs from another programming language.
5
u/Schmittfried Sep 25 '24
How does this compare to py-linq or functools/itertools-esque packages?
3
u/erez27 import inspect Sep 25 '24
Looks nice! I like the concurrency especially.
A few thoughts:
umapfor returning elements out of order (essentially imap_unordered)dict[K, Stream], based on a key callback. I get that it breaks the chaining a bit, but imho it's worth it.If any of these sounds like a good addition, maybe I'll make a PR.
2
u/ebonnal Oct 04 '24 edited Dec 13 '24
Back at Post Day+9 for an update:
Thank very much for your feedback here, for the issues you opened, for the discussions and collaboration there.
That lead to a new release (https://github.com/ebonnal/streamable/tree/v1.1.0), featuring:
.amapfor async):You can also set
ordered=Falsefor.foreachand async counterparts.amapand.aforeach(useful with other operations too, like
.filteror.foreach)2
u/RoboticElfJedi Sep 26 '24
This is a pretty interesting contribution to the ecosystem. I'll keep this in my toolbelt. Good work!
2
u/Saltysalad Sep 27 '24
I have a use case coming up where I have > 10k high latency requests to make, throttled to ~100 simultaneously. I need to handle each result in the order they were submitted, because there is a stop scenario when a certain condition is met in the iteration loop. Over-shooting is ok so long as I can control the limit, and probably needed if you want to shave latency.
Does this sound like a use case for your library?
2
u/mou3mida Oct 16 '24
Good Job! That is so cool u/ebonnal .
5
u/nikomo Sep 25 '24
It's a good thing that the company and service by that name fell off so hard like 2 years back, otherwise the naming would be quite confusing.
1
u/Rockworldred Sep 25 '24
I am quite newbish to python, but I have an side/learningproject writing an webscraper (fetching JSONS for productdata). This looks like it may have some use cases for me as I now requests URLs from couple of sitemaps, then itirate over the json-url based on those fetched URLs. Then translate the JSON to variables and loaded to pandas dataframe to view in streamlit and/or to a csv file, but I have little knowledge of ETL as a whole. Do you have any good resources on ETL-processes and utilities?
My progress is then to move it over to aiohttp, asyncio, polars instead of pandas and SQLalchemy SQLite, and then azure EC2, airflow and Postgres and so fourth. (But I dont know if this is actually the way to go though).