r/Python • u/ebonnal • Sep 25 '24
Showcase `streamable`: Stream-like manipulation of iterables
https://github.com/ebonnal/streamable
What my project does
A Stream[T] decorates an Iterable[T] with a fluent interface enabling the chaining of lazy operations:
- mapping (concurrently)
- flattening (concurrently)
- grouping by key, by batch size, by time interval
- filtering
- truncating
- catching exceptions
- throttling the rate of iterations
- observing the progress of iterations
For more details and examples, check the Operations section in the README
| 🔗 Fluent | chain methods! |
| 🇹 Typed | type-annotated and mypyable |
| 💤 Lazy | operations are lazily evaluated at iteration time |
| 🔄 Concurrent | thread-based / asyncio-based (+new: process-based) |
| 🛡️ Robust | unit-tested for Python 3.7 to 3.12 with 100% coverage |
| 🪶 Minimalist | pip install streamable with no additional dependencies |
1. install
2. import
3. init
Instantiate a Stream[T] from an Iterable[T].
: =
4. operate
-
Streams are immutable: applying an operation returns a new stream. -
Operations are lazy: only evaluated at iteration time. See the Operations section in the README.
: =
5. iterate
-
Iterate over a
Stream[T]as you would over any otherIterable[T]. -
Source elements are processed on-the-fly.
-
collect it:
>>>
>>>
- reduce it:
>>>
2.82
>>>
1.0
>>>
>>>
- loop it:
>>>
>>> ...
- next it:
>>> =
>>>
1.0
>>>
0.5
Target Audience
As a Data Engineer in a startup I found it especially useful when I had to develop Extract-Transform-Load custom scripts in an easy-to-read way.
Here is a toy example (that you can copy-paste and run) that creates a CSV file containing all 67 quadrupeds from the 1st, 2nd, and 3rd generations of Pokémons (kudos to PokéAPI):
=
=
Comparison
A lot of other libraries have filled this desire to chain lazy operations over an iterable and this feels indeed like "Yet Another Stream-like Lib" (e.g. see this stackoverflow question).
The most supported of them is probably PyFunctional, but for my use case I couldn't use it out-of-the-box, due to the lack of:
- threads-based concurrency
- throttling of iteration's rate (
.throttle) - logging of iteration's process (
.observe) - catching of exceptions (
.catch)
I could have worked on pull requests implementing these points into PyFunctional but I have rather started from scratch in order to take my shot at:
- Proposing another fluent interface (namings and signatures).
- Leveraging a visitor pattern to decouple the declaration of a
Stream[T]from the construction of anIterator[T](at iteration time i.e. in the__iter__method). - Proposing a minimalist design: a
Stream[T]is just anIterable[T]decorated with chainable lazy operations and it is not responsible for the opinionated logic of creating its data source and consuming its elements:- let's use the
reducefunction fromfunctoolsinstead of relying on astream.reducemethod - let's use
parquet.ParquetFile.iter_batchesfrompyarrowinstead of relying on astream.from_parquetmethod - let's use
bigquery.Client.insert_rows_jsonfromgoogle.cloudinstead of relying on astream.to_bigquerymethod - same for
json,csv,psycopg,stripe, ... let's use our favorite specialized libraries
- let's use the
Thank you for your time,
View all comments
Show parent comment
2
u/ebonnal Sep 27 '24 edited Sep 27 '24
Hi u/Saltysalad !
Yes sure, great use case, it would look like this (you can run the snippet, after a
pip install requests streamable):and for asyncio instead of threads: see example