We are excited to announce StreamNative Platform 1.0, a cloud-native, unified messaging and streaming platform powered by Apache Pulsar. StreamNative Platform provides a complete, declarative API-driven experience for deploying and self-managing Apache Pulsar in your private environments. With StreamNative Platform, we’ve packaged our enterprise expertise with StreamNative Cloud to help you build your own private cloud Pulsar service.
Whether you are an agile development team that needs to get up and running with Apache Pulsar quickly, or a central infrastructure team that is responsible for enabling your engineering team to build messaging and streaming applications, StreamNative Platform may be the right fit for you. StreamNative platform enables you to:
With StreamNative Platform, you can achieve the simplicity, flexibility, and efficiency of the cloud without the burden of complex infrastructure operations. We provide all the components of a complete platform—ready out of the box with enterprise-grade configurations.
The StreamNative Platform is built on Apache Pulsar to allow developers to transition from traditional silos and monolithic applications, to modern microservices and messaging and streaming applications to increase agility and accelerate time to market.
The overall architecture of StreamNative Platform is illustrated in the following figure.
In this blog, we highlight the enterprise features for StreamNative Platform, including:
Below we provide a deep dive on each.
Built on Apache Pulsar 2.8, StreamNative Platform brings strong transactional guarantees to Pulsar. Transactional guarantees make it easier than ever to write real-time, mission-critical messaging and streaming applications. From tracking ad views to processing financial transactions, you can do it all in real-time and reliably with Pulsar Transaction. You no longer have to develop with lost or duplicated data in mind.
Pulsar PMC member and StreamNative Engineering Lead, Penghui Li, reviews this functionality in detail in the recent blog, Exactly-once Semantics with Transactions in Pulsar. Read this blog to learn more about the exactly-once semantics support in Pulsar.
Developed by OVHCloud and StreamNative, Kafka-on-Pulsar (KoP) has become one of the most popular protocol handlers in the Apache Pulsar community. Companies including Tencent, Bigo, and Dada Nexus have deployed KoP at internet-scale to migrate their existing Kafka applications to Pulsar.
StreamNative Platform includes the GA release of Kafka-on-Pulsar to enable Kafka-API users to build event streaming applications on a streaming platform architected for Kubernetes. The GA release of Kafka-on-Pulsar includes the following features:
Pulsar Functions and Pulsar IO have been proven to be two powerful building blocks for developing messaging and event streaming applications. However, running and orchestrating multiple functions and connectors at a large scale is not an easy task. The complexity is increased when the number of functions and connectors increases.
StreamNative Platform leverages Function Mesh to simplify building serverless event streaming applications. The key benefits include:
Once Pulsar is up and running within a large team, it’s critical to keep an eye on who is touching data and what they’re doing with it. Structured Audit Logs, which is GA on StreamNative Platform, provides an easy way to track user/application access so you can identify potential anomalies and bad actors.
Structured Audit Logs enable you to capture audit logs in a set of dedicated Pulsar topics, either on a local or a remote cluster, including:
With audit events safely stored in Pulsar topics, you can use Pulsar integrated tools, like Pulsar Functions, Pulsar SQL, and Flink SQL, to process and analyze them. Additionally, you can offload audit events to external data lakes or data warehouses (like Snowflake or Databricks) for analysis using Pulsar IO connectors.
StreamNative Platform provides high-level declarative APIs by extending the Kubernetes API through Custom Resource Definitions to support the management of Pulsar services. As a user, you can interact with the Custom Resource Definition by defining a Custom Resource that specifies the desired state. Then the StreamNative Platform will take care of the rest.
StreamNative Platform provides a set of Custom Resource Definitions to deploy and manage Pulsar components: ZooKeeper, BookKeeper, Pulsar Broker, Pulsar Proxy, and StreamNative Console.
The declarative API enables you to leave the infrastructure handling to software automation, freeing you to focus on your core business applications.
With StreamNative, you can utilize Kubernetes-native interfaces, integrations, and scheduling controls to operate consistently and cost-effectively alongside other applications and data systems.
Initially, we used Helm to provide a simple configuration abstraction on top of Kubernetes to allow you to define a declarative spec as a Helm values yaml file (around 60% to 70% of Pulsar users are using Pulsar or StreamNative Helm charts to deploy Pulsar on Kubernetes).
However, after working to provide a fully managed Pulsar service (StreamNative Cloud) and thinking about how to provide automation and packaged best practices, we found that Helm templates were not the right architecture choice. Helm did not provide important features for running a stateful storage service such as the ability to control deployment sequences and add additional operations between deployment steps.
As a result, we moved to an industry standard and aligned on providing a Kubernetes-native interface with Custom Resource Definitions and Controllers. A Kubernetes-native experience provides a reliable API-driven approach with custom resources and leverages the ecosystem tooling and features inherent to Kubernetes. With this approach, you do not need specialized knowledge of how the applications are deployed, such as how to configure storage and network for stateful services.
Each Pulsar resource configuration spec is defined as a Kubernetes-native Custom Resource Definition, and each Pulsar resource provides an extensible configuration interface:
The StreamNative team has experience running some of the largest Pulsar deployments in the world and operating StreamNative Cloud. StreamNative Platform brings a cloud-native experience for running Apache Pulsar workloads in on-premises environments. It provides an enterprise-ready deployment of Apache Pulsar that enhances Pulsar’s elasticity, ease of operations, and resiliency.
StreamNative Platform is a strong fit for the following use cases: