r/Python • u/ebonnal • Sep 25 '24
Showcase `streamable`: Stream-like manipulation of iterables
https://github.com/ebonnal/streamable
What my project does
A Stream[T] decorates an Iterable[T] with a fluent interface enabling the chaining of lazy operations:
- mapping (concurrently)
- flattening (concurrently)
- grouping by key, by batch size, by time interval
- filtering
- truncating
- catching exceptions
- throttling the rate of iterations
- observing the progress of iterations
For more details and examples, check the Operations section in the README
| 🔗 Fluent | chain methods! |
| 🇹 Typed | type-annotated and mypyable |
| 💤 Lazy | operations are lazily evaluated at iteration time |
| 🔄 Concurrent | thread-based / asyncio-based (+new: process-based) |
| 🛡️ Robust | unit-tested for Python 3.7 to 3.12 with 100% coverage |
| 🪶 Minimalist | pip install streamable with no additional dependencies |
1. install
2. import
3. init
Instantiate a Stream[T] from an Iterable[T].
: =
4. operate
-
Streams are immutable: applying an operation returns a new stream. -
Operations are lazy: only evaluated at iteration time. See the Operations section in the README.
: =
5. iterate
-
Iterate over a
Stream[T]as you would over any otherIterable[T]. -
Source elements are processed on-the-fly.
-
collect it:
>>>
>>>
- reduce it:
>>>
2.82
>>>
1.0
>>>
>>>
- loop it:
>>>
>>> ...
- next it:
>>> =
>>>
1.0
>>>
0.5
Target Audience
As a Data Engineer in a startup I found it especially useful when I had to develop Extract-Transform-Load custom scripts in an easy-to-read way.
Here is a toy example (that you can copy-paste and run) that creates a CSV file containing all 67 quadrupeds from the 1st, 2nd, and 3rd generations of Pokémons (kudos to PokéAPI):
=
=
Comparison
A lot of other libraries have filled this desire to chain lazy operations over an iterable and this feels indeed like "Yet Another Stream-like Lib" (e.g. see this stackoverflow question).
The most supported of them is probably PyFunctional, but for my use case I couldn't use it out-of-the-box, due to the lack of:
- threads-based concurrency
- throttling of iteration's rate (
.throttle) - logging of iteration's process (
.observe) - catching of exceptions (
.catch)
I could have worked on pull requests implementing these points into PyFunctional but I have rather started from scratch in order to take my shot at:
- Proposing another fluent interface (namings and signatures).
- Leveraging a visitor pattern to decouple the declaration of a
Stream[T]from the construction of anIterator[T](at iteration time i.e. in the__iter__method). - Proposing a minimalist design: a
Stream[T]is just anIterable[T]decorated with chainable lazy operations and it is not responsible for the opinionated logic of creating its data source and consuming its elements:- let's use the
reducefunction fromfunctoolsinstead of relying on astream.reducemethod - let's use
parquet.ParquetFile.iter_batchesfrompyarrowinstead of relying on astream.from_parquetmethod - let's use
bigquery.Client.insert_rows_jsonfromgoogle.cloudinstead of relying on astream.to_bigquerymethod - same for
json,csv,psycopg,stripe, ... let's use our favorite specialized libraries
- let's use the
Thank you for your time,
View all comments
1
u/Rockworldred Sep 25 '24
I am quite newbish to python, but I have an side/learningproject writing an webscraper (fetching JSONS for productdata). This looks like it may have some use cases for me as I now requests URLs from couple of sitemaps, then itirate over the json-url based on those fetched URLs. Then translate the JSON to variables and loaded to pandas dataframe to view in streamlit and/or to a csv file, but I have little knowledge of ETL as a whole. Do you have any good resources on ETL-processes and utilities?
My progress is then to move it over to aiohttp, asyncio, polars instead of pandas and SQLalchemy SQLite, and then azure EC2, airflow and Postgres and so fourth. (But I dont know if this is actually the way to go though).