Release v0.1.1 (What’s new?).

Documentation Status!--None.svg?style=social!--None.svg?style=social

Welcome to unistream Documentation#

đź“” See Full Documentation HERE.

The library at hand provides a powerful abstraction for data producers and consumers—clients responsible for interfacing with stream processing systems. These clients handle the task of sending data to stream processing systems and pulling data from various streams, including but not limited to Apache Kafka, Apache Pulsar, AWS Kinesis, AWS SQS, and AWS CloudWatch logs.

This library offers a comprehensive set of business-critical features out of the box:

  1. Efficient Record Buffering: It intelligently groups records into micro-batches, optimizing network bandwidth utilization for enhanced performance.

  2. Data Integrity: To ensure data integrity, the library leverages a local write-ahead log mechanism, mitigating the risk of unexpected errors and data loss during the data transfer process.

  3. Automatic Retrying: With an integrated automatic retry mechanism using exponential backoff strategies, transient errors are managed seamlessly, contributing to a robust and reliable data transfer process.

  4. Checkpoint Management: The library automatically handles checkpoints, allowing for the storage of consumption progress and processing status for each record. This feature enhances traceability and fault tolerance, especially in critical business use cases.


unistream is released on PyPI, so all you need is to:

$ pip install unistream

To upgrade to latest version:

$ pip install --upgrade unistream

Table of Content#

About the Author#

(\ (\
( -.-)o

Sanhe Hu is a seasoned software engineer with a deep passion for Python development since 2010. As an author and maintainer of 20+ open-source projects, I bring a wealth of experience to the table. As a Senior Solution Architect and Subject Matter Expert in Amazon Web Services, Cloud Engineering, DevOps, Big Data, and Machine Learning, I thrive on helping clients with platform design, enterprise architecture, and strategic roadmaps.

Talk is cheap, show me the code:

API Document#