Powering Federated Learning at Tencent with Apache Pulsar
Executive Summary
Tencent Angel PowerFL is a distributed federated learning platform which can support trillions of concurrent training. Angel PowerFL has been widely used in Tencent Financial Cloud, Advertising Joint Modeling, and other businesses. The platform requires a stable and reliable messaging system with guaranteed high performance and data privacy. After investigating different solutions and comparing several messaging queues, Angel PowerFL adopted Apache Pulsar as the data synchronization solution in Federated Learning (FL).
In this blog, the Tencent Angel PowerFL team shares how they built federated communication based on Pulsar, the challenges they encountered with Pulsar, and how they solved those problems and contributed to the Pulsar community. Tencent’s use of Pulsar in production has demonstrated it provides the stability, reliability, and scalability that the machine learning platform requires.
Download and review our white paper for an in-depth look at why Federated Learning at Tencent chooses Apache Pulsar.
Chao Zhang is a senior software engineer at the Tencent Data Platform Department. He specializes in Angel PowerFL Federated Communication / PowerFL on K8S. Together with the MQ team of the Tencent Data Platform Department, he introduces Apache Pulsar to the Angel PowerFL platform and starts new chapters for Pulsar adoption in machine learning.
StreamNative is a global team of knowledgeable experts in Apache Pulsar, Apache BookKeeper, and messaging and streaming.
If you have any questions or suggestions, click here to send us an email directly. We’ll reply to you as soon as possible.
