Workflow and Infrastructure

• KCP automates the migration lifecycle by first scanning existing Kafka deployments—including topics, schemas, ACLs, and connectors—to generate cost and metrics reports.0:42
• The tool generates Terraform configurations for necessary target infrastructure, such as AWS Private Link, VPC Peering, and Transit Gateway, to prepare the Confluent Cloud environment.
• Data replication relies on cluster linking, which maintains byte-for-byte topic replication and globally consistent offsets between the source MSK or Apache Kafka cluster and the destination.3:23

Migration and Cutover

• Migration groups serve as the primary unit of coordination, allowing operators to move related topics and client applications as a logical cluster rather than performing a site-wide cutover.4:37
• The CC Gateway acts as an intelligent proxy that handles traffic routing and authentication translation, enabling clients to keep their original credentials while connecting to the destination cluster using API keys fetched from secrets stores like AWS Secrets Manager or HashiCorp Vault.6:33
• In testing, this architecture achieved a cutover window of less than 30 seconds, during which producers buffer writes until the new gateway route is active, allowing consumers to resume from their exact previous offset.6:09

Roadmap and Scope

• Current KCP capabilities are limited to migrating all traffic for a given migration group, though the team is working toward future updates that would support producer-only, consumer-only, and principal-level migrations.7:31
• While the tool promises to reduce coordination burden and manual tech debt, its reliability in diverse real-world environments is not yet established, and finer-grained migration features remain under development.