We are excited to announce that StreamNative is open-sourcing "MQTT on Pulsar" (MoP). MoP brings the native MQTT protocol support to Apache Pulsar by introducing an MQTT protocol handler on Pulsar brokers. Similar to KoP, MoP is also an implementation of the pluggable protocol handler. By adding the MoP protocol handler in your existing Pulsar cluster, you can migrate your existing MQTT applications and services to Pulsar without modifying the code. This enables MQTT applications to leverage Pulsar’s multi-layer system architecture (separation of compute and storage) and powerful features, such as infinite event stream retention with Apache BookKeeper and tiered storage.
Apache Pulsar is a cloud-native, distributed messaging and streaming platform that manages hundreds of billions of events per day. Pulsar was originally developed and deployed inside Yahoo as the consolidated messaging platform connecting critical Yahoo applications such as Yahoo Finance, Yahoo Mail, and Flickr, to data. Pulsar was contributed to open source by Yahoo in 2016 and became a top-level Apache Software Foundation project in 2018.
Apache Pulsar is a multi-tenant, high-performance solution for server-to-server messaging, including features such as native support for multiple clusters in a Pulsar instance, seamless geo-replication of messages across clusters, low publish and end-to-end latency, seamless scalability to over a million topics, and guaranteed message delivery with persistent message storage provided by Apache BookKeeper, among others.
Currently, Apache Pulsar is used in a wide variety of industries. Enterprises, such as Tencent, Verizon Media, Splunk, ChinaMobile, and BIGO, have deployed Apache Pulsar to achieve their business goals. For more use cases, click here.
Message Queuing Telemetry Transport (MQTT) is a lightweight publish-subscribe messaging transport protocol. MQTT is built on the TCP/IP protocol and was created by IBM in 1999. It is ideal for connecting remote devices with a small code footprint and minimal network bandwidth to provide real-time and reliable messaging services. Today, as a low-overhead and low-bandwidth real-time communication protocol, MQTT today is adopted widely across multiple industries, such as the Internet of Things (IoT), small microcontrollers, automotive, etc.
Apache Pulsar provides a unified messaging model for both queueing and streaming workloads. Pulsar implemented its own Protobuf-based binary protocol to provide high performance and low latency. This choice of Protobuf makes it convenient to implement Pulsar clients and the project already supports Java, Go, Python, and C++ languages alongside third-party clients provided by the Pulsar community.
Because Apache Pulsar’s multi-tenancy and the overall architecture with Apache Bookkeeper is able to simplify operations, an increasing number of companies are exploring a way to shift and build the foundation of their services on Pulsar. However, to adopt Pulsar’s new unified messaging protocol, existing applications written using other messaging protocols have to be rewritten.
To address this, StreamNative has been working on a number of new projects to make this transition easier for organizations adopting Pulsar. Earlier this year, StreamNative announced KoP (Kafka-on-Pulsar) and AoP (AMQP-on-Pulsar) protocol handlers to facilitate the migration to Pulsar from Kafka and AMQP.
Over the past several months, StreamNative received a lot of inbound requests for help migrating services from MQTT to Pulsar and recognized the need to also support MQTT protocol natively on Pulsar. StreamNative invested engineering time and effort to introduce a general protocol handler framework in Pulsar that would allow developers who use the MQTT protocol to use Pulsar.
MoP is implemented as a pluggable protocol handler that can support native MQTT protocol on Pulsar by leveraging Pulsar features such as Pulsar topics, cursors etc. The diagram below illustrates a Pulsar cluster with the MoP protocol handler. Both the MQTT Proxy and MQTT protocol handler can run along with Pulsar brokers.
MQTT has defined three Quality of Service (QoS) levels:
Currently, the MoP protocol handler only supports QoS level 0 and QoS level 1. In future releases, QoS level 2 will be supported.
The MoP Proxy is an optional component for MoP. It extends MoP to multiple nodes to realize horizontal expansion of services. The MoP Proxy is mainly used to forward messages delivered between the MQTT Client and the Pulsar broker. Therefore, the MQTT Client only needs to connect to the MoP Proxy to send and receive data, regardless of the Pulsar broker to which topics are dispatched.
The MoP Proxy can sense the status of Pulsar brokers. Once a Pulsar broker is disconnected or unreachable, the MoP Proxy will send messages from the MQTT Client to a new Pulsar broker.
The following figure illustrates the MoP Proxy service workflow.
At present, the MoP Proxy works with the Pulsar broker. Users could choose whether to start the MoP Proxy service through the related configuration. For details, see here.
For any problems in the use of the MoP protocol handler, you can create an issue in the MoP repository. We will reply to you as soon as possible. Meanwhile, we look forward to your contribution to MoP.
Have something to say about this article? Share it with us on Twitter or contact us.