Terraform Your Whole Kafka Platform

Which Terraform provider manages Kafka? Usually you pick your Kafka vendor's, or a generic one like Mongey/kafka if you're self-managed or multi-vendor. Under that question there are three layers: creating the cluster, creating the topics, governing the platform (ownership, policies, access).

Conduktor also has a Terraform provider, and it plays a different game: not spinning up clusters, but one consistent way to terraform all your Kafka clusters and keep them governed:

• Who owns which topic?
• What rules must it respect?
• What can this team create without asking anyone?

Everything you declare in HCL is the same model the UI, the API and MCP read and act on. Terraform is one door into the platform, not a side channel.

Like AWS, whose new aws_msk_topic resource takes a cluster_arn instead of bootstrap servers, we rely on the control-plane model: don't dial brokers from CI, manage Kafka resources through a control-plane API. It offers more flexibility, more control, and richer resource types than the raw Kafka primitives.

Why Terraform needs to talk to a control plane

Letting Terraform talk to your Kafka infrastructure is about managing risks:

• Your Terraform runner needs a route to every broker of every cluster, with TLS, SASL or IAM configured in CI: Every new cluster is a new networking and credentials problem in your pipeline.
• The credentials Terraform holds are Kafka admin credentials: Your CI system becomes the most privileged Kafka client in the company.
• The model stops at the Kafka protocol. partitions = 12, "retention.ms" = "60000", an ACL for a principal: Nothing says who owns the topic, what it contains, or what a team is allowed to create.

And Terraform itself doesn't make Kafka changes safe. AWS's own MSK tutorial changes a topic from 50 partitions to 10; Kafka can't shrink partitions, so Terraform drops and recreates the topic, data included. The plan just says 1 to add, 1 to destroy.

The Kafka protocol has no concept of ownership, so no provider speaking that protocol can express it. What teams build instead is a central Terraform repo that the platform team owns, with custom resources, and every topic request becomes a PR someone else must review:

"They'll go in and have to make a PR to that repo. It's not super intuitive on how to do that. So they'll end up pinging our team anyway." — Platform engineer, consumer-lending company

Same story with the vendor providers. They're excellent inside their own platform (the Confluent provider covers environments, networking, RBAC, Schema Registry, Flink) and they stop at their platform's boundary. Run vanilla Kafka, Confluent Cloud and MSK side by side and you're maintaining several provider configurations and security models, each with its own idea of what a topic is.

Terraform to govern the platform

The Conduktor provider is built for the platform layer. Terraform talks HTTPS to Conduktor Console, and Console manages the clusters connected to it: MSK, Confluent, Aiven, Redpanda, self-managed, Gateway virtual clusters. One Terraform model, whatever the underlying Kafka: clusters, topics, policies, applications.

# Register an existing MSK cluster so Console can govern what runs on it.
resource "conduktor_console_kafka_cluster_v2" "msk" {
  name = "payments-msk"
  spec = {
    display_name = "Payments (MSK, eu-west-1)"
    bootstrap_servers = "b-1.xxxxx.kafka.eu-west-1.amazonaws.com:9198"
    properties = {
      "sasl.jaas.config" = "software.amazon.msk.auth.iam.IAMLoginModule required awsRoleArn='arn:aws:iam::123456789123:role/MSK-role';"
      "security.protocol" = "SASL_SSL"
      "sasl.mechanism"    = "AWS_MSK_IAM"
    }
  }
}

# A Kafka virtual cluster registers the same way and Terraform governs it like any other.
resource "conduktor_console_kafka_cluster_v2" "payments_vc" {
  name = "payments-vc"
  spec = {
    display_name = "Payments (virtual cluster)"
    bootstrap_servers = "gateway.internal:6969"
    properties = {
      "security.protocol" = "SASL_SSL"
      "sasl.mechanism"    = "PLAIN"
    }
    kafka_flavor = {
      gateway = {
        url = "https://gateway.internal:8888"
        user = "admin"
        password = var.gateway_admin_password
        virtual_cluster = "payments"
      }
    }
  }
}

A topic is still a topic, but it carries the platform model with it. An owner, labels, a description that shows up in the catalog:

resource "conduktor_console_topic_v2" "orders" {
  name = "click.orders.events"
  cluster = conduktor_console_kafka_cluster_v2.msk.name
  labels = {
    domain = "orders"
    criticality = "C1"
  }
  description = "Order lifecycle events, one message per state change."
  spec = {
    partitions = 6
    replication_factor = 3
    configs = {
      "cleanup.policy" = "delete"
      "retention.ms"   = "604800000"
    }
  }
  lifecycle {
    prevent_destroy = true
  }
}

And the rules a topic must respect are themselves Terraform resources. This is the part no protocol or vendor provider has an equivalent for:

resource "conduktor_console_topic_policy_v1" "orders_rules" {
  name = "orders-topic-rules"
  spec = {
    policies = {
      "metadata.name" = {
        match = { pattern = "^click\\.(?<event>[a-z0-9-]+)\\.(avro|json)$" }
      }
      "metadata.labels.criticality" = {
        one_of = { values = ["C0", "C1", "C2"] }
      }
      "spec.configs.retention.ms" = {
        range = { min = 3600000, max = 604800000 }
      }
    }
  }
}

Attach that policy to an application and give the team an API key scoped to their self-service application instance, not an admin key.

The dev doesn't ping the platform team. They don't PR into a central repo they don't understand. They apply Terraform in their own repo, with their own credentials, and the control plane validates it against the policies: naming convention enforced, retention bounded, criticality label mandatory. terraform apply fails with a clear message the developer can act on, no reviewer needed.

All-in on Confluent Cloud?

Even all-in on Confluent Cloud, the same gap is there. The Confluent provider is excellent at provisioning Confluent Cloud itself, so keep it for that. The platform layer sits above it, and once the infrastructure is solid it's where your teams spend their time: who owns what, which rules apply, what each team can create on its own.

With Conduktor on top of Confluent Cloud, that layer becomes Terraform too. The rule is a resource, enforced on every path into the cluster (UI, CLI, API, or the provider above). And spend rolls up to the applications you declare, so the bill reads by team, not by service account.

Beyond topics

The full resource list goes further than topics: users, groups and permissions, service accounts, subjects, connectors, applications and application instances, partner zones, and on the Gateway side virtual clusters, interceptors and service accounts. More than twenty typed resources as of v1.2.2.

What it doesn't do

Conduktor is a control layer, not an infra layer, it means that:

• It does not create the cluster. No brokers, no VPC, no MSK Serverless. You keep hashicorp/aws or confluentinc/confluent for the infrastructure layer. The pairing is: the cloud provider creates the cluster, the Conduktor provider governs what runs on it.
• It requires Conduktor Console. The provider is the IaC surface of the platform, not a standalone Kafka client.
• Some Console features land in the API before they get a typed Terraform resource. There's an experimental conduktor_generic resource for those, and we don't recommend it in production yet.

The cluster and the topics have had Terraform providers for years. The platform layer, who owns what, which rules hold, what each team can ship on its own, is the one still living in a wiki. It belongs in code too. Browse the Conduktor provider on the Terraform Registry, or talk to us about governing your Kafka as code.