Unified Stream Manager: Manage and Monitor Apache Kafka® Across Environments

If you’re running Confluent Platform or our new offering, Confluent Private Cloud, on-premises, you have your reasons: data sovereignty, regulatory compliance, or maybe a phased cloud migration. Your on-prem Apache Kafka® isn’t going anywhere. It’s a critical part of your infrastructure. But let’s be honest: You’ve probably looked at the advanced, centralized capabilities in Confluent Cloud and wished you could bring some of those governance and observability capabilities into your own data centers. Managing a hybrid environment like this is a tough job for any type of software. The reality for most platform teams is a patchwork of tools. You have one management experience for your public cloud services, another for your private cloud, and yet another for your on-prem data centers.

This fragmentation isn't just an inconvenience; it creates real operational drag. For platform architects, enforcing consistent governance across this fractured landscape is a constant struggle. For operators and developers, troubleshooting a failing producer can feel like searching for a needle in a haystack of disconnected clusters, each with its own monitoring dashboard and ruleset.

Today at Current New Orleans, we’re excited to change that. We’re announcing that Unified Stream Manager (USM) is now generally available with the release of Confluent Platform 8.1!

USM is designed to unify how you manage and monitor all your Confluent clusters regardless of where they are. All your clusters, whether on-prem, in a private cloud, or across public clouds, are displayed in a single unified view.

To leverage USM, you’ll connect your Confluent Platform private environment to a Confluent Cloud environment through a secure private network connection. This connection is initiated from the private environment, sharing only metadata and telemetry with your cloud environment, and ensures that all your Kafka data and all client workloads remain securely in your private environment.

Let’s break down the core capabilities that make this possible.

A Single Source of Truth for All Your Schemas 

For developers, schemas are the lifeblood of reliable applications. For platform teams, they’re the foundation of good governance. In a hybrid world, keeping schemas in sync is a major headache. USM solves this by making Confluent Cloud Schema Registry the single source of truth for your entire organization. But this doesn’t mean a painful migration. Your existing on-premises Schema Registry isn’t going away; it simply becomes a smart, read-only cache that forwards all new schema writes to the cloud. No changes are needed for your existing clients or applications, so you get global schema consistency and local performance without disruption.

This unified approach lets you enforce consistent data contracts across your entire infrastructure. And with built-in client-side field level encryption (CSFLE), you can now encrypt sensitive fields right at the source, protecting personally identifiable information (PII) before it ever lands in Kafka.

Find and Fix Problems Faster With Unified Observability

For operators, the day-to-day challenge is visibility. When something breaks, how quickly can you pinpoint the cause? USM consolidates the view of your distributed systems into one central location, dramatically cutting down your mean time to resolution (MTTR).

Having this single operational view gives you the power to:

• Monitor your entire hybrid platform from a single view. Get a comprehensive look at the health and performance of all your on-prem Kafka brokers, topics, connectors, and clients directly within the Confluent Cloud user interface (UI)—with no more jumping between different monitoring tools.

• Discover all your data in a unified catalog. Data Portal allows developers to find, understand, and use topics and schemas from across your entire infrastructure. With metadata tagging and a global search, it’s never been easier to discover and trust the data you need.

• Trace data streams end to end to debug issues in minutes. This is the game-changer for troubleshooting. You can now visually follow the path of your data across clients, topics, and connectors—even in on-prem clusters—to instantly pinpoint the root cause of failures. What used to take hours of manual log-diving can now be done in minutes.

Unified Oversight With Built-In Privacy

The first question on everyone's mind is: How does this connection work, and is it secure? We built USM with a trust-first, security-first mindset. The architecture is simple but powerful, with two main components: the USM agent and the USM cloud console.

The USM agent is a lightweight component you deploy in your on-prem Confluent Platform environment (version 8.1 or later is required). The agent securely collects and streams operational data and metadata—like cluster health, topic configurations, and schemas—to Confluent Cloud over a secure, private network connection using AWS PrivateLink. It’s designed as a single, secure egress point for this information so that your message data never leaves your environment.

The USM Cloud Console is your central hub within the Confluent Cloud UI. It aggregates all the data from the agent and gives you that unified view of your entire hybrid landscape. You can see all your clusters in one dashboard, monitor their health, trace data lineage, and manage governance policies without having to switch between different tools.

USM’s architecture keeps all message data—including PII—securely within your environment, sending only essential operational metadata to Confluent Cloud over private networking. This minimizes your security footprint, avoids exposing on-prem data to public cloud terms, and ensures that your streaming workloads keep running even if the cloud connection is interrupted—so you get unified oversight without compromising data privacy or operational resilience.

This secure, hybrid connectivity is also the foundation for Confluent Private Cloud, our new offering for customers who want to bring cloud-like capabilities to their private environments.

Ready to Unify Your Streaming World?

Getting started with USM is straightforward. To begin, your entire Confluent Platform deployment needs to be on version 8.1. The setup involves deploying the USM agent alongside your cluster, which can be automated using the Confluent for Kubernetes (CFK) operator or Ansible playbooks.

For secure communication between your on-prem environment and Confluent Cloud, you must have AWS PrivateLink established. At launch, this is supported for Amazon Web Services (AWS) private networking, and we plan to expand to Azure and Google Cloud Platform in the future. Please note that USM requires a Stream Governance Advanced package.

You can start your journey today by downloading Confluent Platform 8.1. Once connected, you can configure your cluster to sync telemetry, metadata, and schemas for a fully governed, observable hybrid architecture. Welcome to the future of hybrid data streaming!

Additional Enhancements in Confluent Platform 8.1

Confluent Platform 8.1 is built on Apache Kafka version 4.1.0, reinforcing our core capabilities as a data streaming platform. Below are the release highlights, and you can find additional details about the features in the release notes. For more details about Apache Kafka 4.1.0, read the release blog post.

Key enhancements to Confluent Platform in the latest 8.1 release: 

• Control Center introduces powerful new visibility tools, including historical consumer lag graphs to spot trends and client-specific performance metrics for rapid troubleshooting. Additionally, new active-active support provides high availability for Control Center itself, ensuring that your management and monitoring tools are always resilient. For the full details on the latest in Control Center, check out the release notes.

• Queues for Kafka is now in Open Preview. If you have feedback or would like to test on non-production clusters that can be upgraded to production when this feature is in General Availability, reach out to your Confluent account team. Learn more.

• Microsoft Azure User Assigned Managed Identity (UAMI) for Kafka OAuth authentication is now supported, so users can securely connect to identity providers, such as Microsoft Entra, without managing static client IDs or secrets. Authentication is automated using Azure’s identity management and is available by updating to the latest Java or librdkafka-based client. Learn more.

Confluent Platform for Apache Flink® Updates

Confluent Platform for Apache Flink® now supports Apache Flink® versions 2.0.0 and 2.1.0, with 2.0.0 introducing compatibility with Java 21. This ensures that users can leverage the latest Flink features and performance improvements, while migration from older Confluent Manager for Apache Flink® (CMF) versions to 2.1.0 is fully supported. Notable features in Confluent Platform for Apache Flink® include:

• Manage savepoints via REST API, triggering them on SQL statements or Flink applications and importing from open source Flink for easy migration and streamlined disaster recovery.

• Python developers can now use PyFlink and TableAPI in Confluent Platform for Apache Flink® (Flink versions 2.0.0 and 2.1.0), bringing official support for building and running Flink jobs in Python.

• CMF can now persist its metadata in PostgreSQL or Microsoft SQL Server. This external storage option improves disaster recovery and reliability, enabling faster and more robust recovery of Flink job metadata.

• Flink SQL on Confluent Platform now lets you customize inferred database tables with ALTER TABLE and adds built-in machine learning functions for real-time anomaly detection and forecasting—all in your streaming SQL pipelines.

Confluent for Kubernetes (CFK) Updates

CFK provides a declarative API-driven control plane to deploy and manage Confluent Platform on Kubernetes. CFK 3.1.0 allows you to deploy and manage Confluent Platform versions 7.3.x–8.1.x on Kubernetes versions 1.26–1.34 (OpenShift 4.13–4.19). For details on installing CFK and Confluent Platform, see the documentation on how to deploy CFK and how to deploy Confluent Platform using CFK.

Confluent Control Center (Legacy) was deprecated and removed from Confluent Platform 8.0.0 and above. Use CFK to deploy the new Control Center. For full details on the latest in CFK, check out the release notes.

Confluent Ansible Updates

Confluent Ansible offers a simple way to configure and deploy Confluent Platform. Confluent Ansible 8.1.0 allows you to deploy Confluent Platform version 8.1.0. This release supports Ansible Core versions 9.x–11.x and Python versions 3.10 and above. For more information, see Prerequisites for Installing Confluent Platform With Ansible Playbooks. For full details on the latest in Confluent Ansible, check out the release notes.

Get Started Today

Managing a hybrid data streaming landscape doesn't have to be so complex. With USM, you can get the visibility and control you need to manage your entire environment with confidence. Download Confluent Platform 8.1 today to get started with the only cloud-native and comprehensive platform for data in motion, built by the original co-creators of Apache Kafka.

Before you upgrade to Confluent Platform 8.1, consult the official Apache Kafka 4.1.0 upgrade guide, which has detailed, step-by-step instructions for performing the upgrade, considerations for rolling upgrades, and crucial information about potential breaking changes or compatibility issues that may arise during the upgrade process.

The preceding outlines our general product direction and is not a commitment to deliver any material, code, or functionality. The development, release, timing, and pricing of any features or functionality described may change. Customers should make their purchase decisions based on services, features,   and functions that are currently available.

Confluent and associated marks are trademarks or registered trademarks of Confluent, Inc.

Apache®, Apache Flink®, Flink®, Apache Kafka®, Kafka®,  and the Kafka and Flink logos are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries. No endorsement by the Apache Software Foundation is implied by the use of these marks. All other trademarks are the property of their respective owners.