Secure Shared Services with Data Streaming: OAuth, Client Quotas, and more

As demand for real-time everything continues to increase, organizations are expanding their use of data streaming with Apache Kafka® across many internal/external-facing applications built in the cloud. But there’s a trade-off: businesses have to choose between protecting application performance and maintaining sustainable infrastructure costs. The ideal solution is to run apps side by side on shared infrastructure—a shared services or multi-tenant model. This model still brings challenges, though, such as a lack of native tools to support secure application identity and access management and a lack of resource utilization control across applications.

With that in mind, we bring you the latest features from Confluent Cloud. You’ll find new capabilities that enable organizations to build a secure shared services platform for data streaming and maximize efficiency. Organizations can now centrally manage which applications need to gain access, what level of access is needed for each across the resource hierarchy, and the amount of resources an application can consume within Confluent Cloud.

Here’s an overview of the latest features—read on for more details:

• Centralized identity management (OAuth)

• Enhanced role-based access control (RBAC)

• Cloud Client Quotas

• Other new features

Join us to see the new features in action in the Q1 Launch demo webinar

Centralized identity management: OAuth support on Confluent Cloud

OAuth is a secure, industry-standard, cloud-native authentication protocol that allows an application to access resources hosted by another service provider, such as Confluent Cloud. Confluent Cloud now supports OAuth, which allows customers to integrate with their own third-party identity provider to centralize account management across all of their cloud services. This can avoid months of work redoing identities within every cloud for every application. They can pick an OAuth-compliant identity provider of their choice, and maintain a single source of truth for all of their applications. This feature comes with automation built-in, allowing users to easily plug into any existing workflows within an organization. 

“Security is a first-class citizen in our platform and is included in every feature we release. The ability to apply least privileged access across our platform is a strong requirement when adding technologies to our tech stack,” said Jonathan Kropp, Director of Architecture at Extend. “The combination of Confluent’s OAuth and access control allows us to grant the least amount of access needed for each feature to do its job. No more, no less. This batteries-included approach to authentication/authorization has made securing Kafka simple and straightforward, allowing us to focus on decoupling data generation and retention from data consumption in our highly scalable distributed systems.” 

With OAuth support for Confluent Cloud, you can now:

• Reduce operational burden by managing application identities and credentials through your own identity provider and a cloud-native authentication standard

• Enhance security with industry standard-based short-lived credentials (tokens) and move away from authentication methods such as API keys that potentially do not fit into security policies of your organization

• Support autonomy through integration with RBAC and automation via REST APIs

Enhanced role-based access control: Granular access controls extended to all cloud features

We started our RBAC journey by enabling access controls to critical resources like production environments and sensitive clusters (Cluster Role-Based Access Control), then expanded the scope to include granular access controls at individual Kafka resources, including topics, consumer groups, and transactional IDs (RBAC at scale). Today, we have further expanded the scope of RBAC to now include all Confluent Cloud resources. Some of the newer capabilities include RBAC for Schema Registry, Connect, and ksqlDB. 

With RBAC now available at all levels in the resource hierarchy, you can leverage a consistent access control approach across users and applications, which enables better security and faster time to production. 

Some of the key highlights of RBAC include:

• Onboard architects, operators, and developers fast with ability to add users and set roles using the Confluent Cloud UI, the Confluent Cloud CLI, or Confluent APIs

• Enable your organization's controls to ensure data protection with enforced permissions to perform CRUD operations (read, write, update, and delete) across all resources

• Centrally manage and moderate all users and applications with ability to add, edit, or remove users and permissions as your business requires

Cloud Client Quotas: Cloud-native resource utilization controls for shared services, multi-tenant deployments

When deploying a shared services model for data streaming, just one poorly behaved application has the ability to drive significant over-utilization on the cluster and consequently drop performance for every other neighboring app. This is a problem for platform operators who require high confidence that any new architectural model can support mission-critical performance levels for every single application running on it. This presents a risk to the shared infrastructure approach that could leave net-new use cases waiting on the sidelines if otherwise dependent upon additional infrastructure spend or a complex workaround. 

Cloud Client Quotas on Confluent Cloud enable businesses to easily deliver enterprise-wide access to highly performant Apache Kafka while reducing operational complexity and cost. Because Confluent Cloud can provide a single Kafka cluster that supports up to 20GBps of overall throughput and infinite data storage, customers can build and manage a multi-tenant, shared services data streaming platform in the cloud with no trade-offs on individual application performance—each is configured with custom throughput controls. Cloud Client Quotas are available on Dedicated clusters and can be applied to either a Confluent Cloud service account or third-party OAuth identity via the Cloud Console UI, API, CLI, or Terraform.

Cloud Client Quotas provides a cloud-native solution for safe, cost-effective resource sharing and allows Confluent customers to:

• Ensure precise, side-by-side application performance with individually curated ingress/egress throughput controls and monitoring for every workload and application

• Expand Kafka while reducing operational complexity and cost with a shared services model for data streaming running on an individual GBps+ cluster

• Quickly accelerate data streaming innovation when teams reuse high-quality data available on internally shared clusters in an easy, self-service fashion

Other new features in the Confluent Cloud launch

Cloud Service Quotas: With enterprises expanding their usage of data streaming across applications, centralized platform teams need to be aware of limitations for resources and operations within Confluent Cloud. With Cloud Service Quotas, enterprises can access and budget quota limits for all Kafka resources (org., environment, networking, clusters, and more). Further, users can manage notifications for service quota events with the Confluent Cloud Console or with the REST API.

Audit logs for critical Kafka operations: This update includes audit log events for all critical Kafka operations, promoting transparency and enabling customers to proactively mitigate security risks. Audit logs allow users to capture, protect, and preserve Kafka authentication actions, authorization actions, and organization operations into topics in Standard and Dedicated clusters.

AsyncAPI: AsyncAPI is an open source, machine-readable specification language with growing popularity and support within the event-driven ecosystem. Confluent's AsyncAPI tool allows customers to easily capture their Confluent Cloud implementation and provide internal teams with a concise, shared document of their data streaming architecture.

Self-serve networking options: Self-serve provisioning for Transit Gateway attachment resources via UI/API/Terraform simplifies on-prem or multi-peering scenarios with enhanced troubleshooting and automation capabilities. There’s also the option to choose to migrate from peering to Transit Gateway to simplify complex multi-network peering patterns.

Terraform support for Cluster Linking: Terraform support for Cluster Linking is now generally available for Confluent Terraform Provider as we continue to expand the scope across all cloud resources.

Start building with new Confluent Cloud features

Ready to get started? Remember to register for the Q1 ʼ23 Launch demo webinar on February 22 where you’ll learn firsthand from our product managers how to put these new features to use. 

And if you haven’t done so already, sign up for a free trial of Confluent Cloud. New sign-ups receive $400 to spend within Confluent Cloud during their first 30 days. Use the code CL60BLOG for an additional $60 of free usage.*