AWS MSK Authentication: OIDC and mTLS Without Touching the Broker

"Every other service we run takes an Okta token. Kafka is the one thing that won't." — Platform engineer at a financial-services company

A team comes to you wanting to put a new application on the shared AWS MSK cluster. Their service already authenticates to everything else with an OIDC token from the company identity provider, whether that's Okta, Entra ID, or whatever the standard is. They assume Kafka works the same way, and it doesn't: MSK will not take that token, and now the onboarding conversation is about credentials instead of the feature they wanted to ship.

This is the second post in a series on the problems teams hit operating Kafka on AWS MSK, drawn from our AWS MSK Journey webinar. The first post was about networking. This one is about the problem teams hit right after it: authentication.

What AWS MSK accepts

MSK supports three ways for a Kafka client to authenticate, and that list is closed:

• IAM (AWS_MSK_IAM): the client signs the connection with AWS credentials. Clean if the client is an AWS principal with an IAM role, off the table if it isn't.
• SASL/SCRAM: username and password, with the secret stored in AWS Secrets Manager.
• Mutual TLS: client certificates, but the trust anchor must be an AWS Private CA. You can't point it at your own corporate PKI.

There is no fourth option, and in particular MSK does not accept OIDC or OAuth bearer tokens from a third-party identity provider. So if your organization has standardized on OIDC for everything that isn't Kafka, as many have, the broker can't participate in that scheme.

🚫 "We'll just federate our IdP into MSK like we do for the rest of the platform."

There's nowhere to plug the IdP into the broker itself; the three mechanisms above are the whole surface. You may notice MSK's own IAM client libraries use Kafka's OAUTHBEARER mechanism, but the bearer token there is an AWS SigV4 signature, not a JWT from your IdP, so the only issuer the broker trusts is still AWS.

Each also has a catch once you're past a single team:

• IAM only speaks AWS credentials. A client outside AWS can still get them by federating an outside identity into one: an OIDC workload through STS AssumeRoleWithWebIdentity, a certificate-based one through IAM Roles Anywhere. Both work, but the client still ends up presenting an AWS credential you have to vend and rotate, with a dedicated IAM role per identity, for workloads that don't live on AWS, including partners you'd rather not hand an AWS identity at all. Terminating at a proxy skips that exchange entirely: the client keeps its own OIDC token or certificate and never holds an AWS credential.
• SCRAM works anywhere, but now you own a growing pile of shared secrets in Secrets Manager. You can automate rotation with a Lambda, but you're still distributing passwords to every app, and each one can leak.
• mTLS is tied to AWS Private CA. You can chain ACM Private CA under your corporate root as a subordinate, but issuance still flows through ACM PCA, so AWS's CA service mints certificates for your clients rather than the PKI workflow you already run. And MSK doesn't honor certificate revocation lists, so a compromised cert stays valid until you fence it off with ACLs and security groups.

None of this is a bug. MSK is a managed broker, and the broker's job is to run Kafka, not to be your identity layer. The mismatch is that authentication is where your organization already has strong opinions and existing infrastructure, and the broker can't meet them.

Terminate the client's scheme at a proxy

The way out is to stop trying to teach MSK new authentication methods, and instead put a Kafka-aware proxy in front of it. The client authenticates to the proxy using whatever scheme your organization standardized on; the proxy authenticates to MSK using one of the three methods it does accept. The two halves are decoupled.

Conduktor Gateway sits between the client and the cluster. It presents a listener that speaks OIDC or mTLS to the client, validates that identity, maps it to a Kafka principal, and then opens its own connection to MSK using AWS_MSK_IAM. The client never needs an AWS identity, and MSK never needs to understand your IdP.

Where this runs. Terminating OIDC or mTLS and mapping it to a principal is a Conduktor Gateway capability, beyond the basic SASL passthrough in Gateway Community. If you're weighing which edition fits, talk to us.

Gateway runs in one of two modes, and the distinction matters here:

• KAFKA_MANAGED: authorization is delegated to the backing cluster, so existing Kafka principals and ACLs keep working. Good for a gradual migration where MSK stays the source of truth.
• GATEWAY_MANAGED: Gateway owns authentication and authorization, with its own service accounts and ACLs. This is what unlocks features like virtual clusters, and it's where identities that MSK could never represent (an OIDC subject, or a certificate from your own CA) become first-class.

For OIDC and mTLS clients against MSK, GATEWAY_MANAGED is the requirement. mTLS termination only exists in this mode. KAFKA_MANAGED can forward a SASL credential to the backing cluster, but forwarding only works when that cluster can validate the credential, and MSK can't validate a token from your IdP, so there's nothing to forward it to.

One thing to plan for: in this mode Gateway becomes the authorization source of truth for those clients, so their access is governed by Gateway ACLs rather than MSK-side ACLs. That's the point (it's how a non-AWS identity gets Kafka permissions at all), but it concentrates the audit trail too: MSK and CloudTrail see only Gateway's IAM role for all proxied traffic, so per-client attribution lives in Gateway's audit log, and that's where access reviews for these clients happen.

Using OIDC and mTLS with AWS MSK: the client and the proxy

Each walkthrough has two halves: what the application developer sets on the client, and what the platform team sets on the Gateway. The promise is that the client half is ordinary, so we'll start there.

Before you copy. These snippets are representative, not production config: the environment variables and client properties are real, and the hostnames, paths, and secrets are yours to fill in. Gateway config is shown in the per-listener (GATEWAY_LISTENER_*) style introduced in Gateway 3.20; on earlier versions the same settings live in the global network configuration, so check your version first.

OIDC

On the client (the developer's side), it's a standard Kafka OAUTHBEARER login pointed at your existing IdP. No AWS SDK, no IAM role, no MSK-specific code:

security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required;
sasl.oauthbearer.client.credentials.client.id=my-app
sasl.oauthbearer.client.credentials.client.secret=${OIDC_CLIENT_SECRET}
sasl.oauthbearer.scope=kafka
sasl.oauthbearer.token.endpoint.url=https://idp.example.com/oauth2/token
sasl.login.callback.handler.class=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginCallbackHandler

This is the same OAuth client-credentials flow the service already uses for its HTTP and database calls. The Kafka client fetches a token from the IdP and refreshes it on its own before it expires, so a short-lived token doesn't drop your producers mid-stream. If you're on Spring Kafka, these go under spring.kafka.properties.* and you don't write a callback handler yourself. The only infrastructure change is pointing bootstrap.servers at the Gateway endpoint.

Mind the Kafka version. The top-level sasl.oauthbearer.* credential properties need Kafka 4.1+ (KIP-1139); on 3.1–4.0, put clientId, clientSecret, and scope inside the JAAS line, a form 4.1 deprecates but accepts. The callback handler path is the 3.4+ location; 3.1–3.3 use the org.apache.kafka.common.security.oauthbearer.secured package, and below 3.1 Kafka has no built-in OIDC (KIP-768), so everything here assumes clients on 3.1 or newer. Two more traps: Entra ID won't issue a client-credentials token without a scope (its convention is api:///.default), and Kafka won't expand a bare ${OIDC_CLIENT_SECRET}, so wire the secret through EnvVarConfigProvider (Kafka 3.5+, declared under config.providers, referenced as ${env:OIDC_CLIENT_SECRET}) or your secrets tooling.

On the Gateway (the platform team's side), you declare that Gateway owns identity and authorization, tell it where to fetch the IdP's signing keys, pin the issuer and audience so a token minted for something else won't work, and choose which claim becomes the principal. With per-listener config, you set the mode and ACL switches yourself; Gateway doesn't infer them:

# Gateway owns authentication and authorization for these clients
GATEWAY_SECURITY_MODE=GATEWAY_MANAGED
GATEWAY_ACL_ENABLED=true

# Client-facing listener speaks SASL_SSL / OAUTHBEARER
GATEWAY_LISTENER_EXTERNAL_SECURITY_PROTOCOL=SASL_SSL

# Server certificate the listener presents to clients
GATEWAY_SSL_KEY_STORE_PATH=/etc/gateway/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD=<secret>
GATEWAY_SSL_KEY_PASSWORD=<secret>

# Validate the JWT against the IdP
# (values from our Keycloak reference architecture)
GATEWAY_OAUTH_JWKS_URL=https://keycloak.cdk-deps.svc.cluster.local/realms/conduktor-realm/protocol/openid-connect/certs
GATEWAY_OAUTH_EXPECTED_ISSUER=https://oidc.localhost/realms/conduktor-realm
GATEWAY_OAUTH_EXPECTED_AUDIENCES="[account]"

# Which claim becomes the Kafka principal
# (Keycloak puts the client ID in azp; Okta and Entra typically use sub)
GATEWAY_OAUTH_SUB_CLAIM_NAME=azp

Gateway validates the token at connect time and refreshes the IdP's signing keys on the GATEWAY_OAUTH_JWKS_REFRESH interval, so key rotation at the IdP doesn't take you down. The claim you nominate becomes a Gateway service account, the identity you write ACLs against: azp here, because Keycloak puts the OAuth client ID there on client-credentials tokens; with Okta or Entra you'd map sub. Declare it as a GatewayServiceAccount of type EXTERNAL: externalNames holds the claim's value (my-app here, an opaque 0oa4… for an Okta sub) and metadata.name is the friendly name ACLs and audit logs show. The two can coincide, as below, but don't have to:

apiVersion: gateway/v2
kind: GatewayServiceAccount
metadata:
  name: my-app
spec:
  type: EXTERNAL
  externalNames: ["my-app"]

mTLS with your own CA

On the client, the application presents a certificate from your PKI and trusts the Gateway listener's certificate. These are ordinary Kafka SSL properties:

security.protocol=SSL
ssl.keystore.location=/etc/app/client.keystore.p12   # your client cert + key
ssl.keystore.password=${KEYSTORE_PW}
ssl.keystore.type=PKCS12
ssl.truststore.location=/etc/app/truststore.p12      # trusts the Gateway listener cert
ssl.truststore.password=${TRUSTSTORE_PW}
ssl.truststore.type=PKCS12

Keep the type lines: Kafka defaults to JKS, and .p12 files without them fail at startup.

On the Gateway, you require client authentication on the listener, trust your CA in the truststore (not AWS Private CA), and define how the certificate subject becomes a principal:

# Gateway owns authentication and authorization for these clients
GATEWAY_SECURITY_MODE=GATEWAY_MANAGED
GATEWAY_ACL_ENABLED=true

# Client-facing listener requires a client certificate
GATEWAY_LISTENER_EXTERNAL_SECURITY_PROTOCOL=SSL
GATEWAY_LISTENER_EXTERNAL_SSL_CLIENT_AUTH=REQUIRE

# Server certificate the listener presents to clients
GATEWAY_SSL_KEY_STORE_PATH=/etc/gateway/keystore.jks
GATEWAY_SSL_KEY_STORE_PASSWORD=<secret>
GATEWAY_SSL_KEY_PASSWORD=<secret>

# Trust your own CA, not AWS Private CA
GATEWAY_SSL_TRUST_STORE_PATH=/etc/gateway/truststore.jks
GATEWAY_SSL_TRUST_STORE_PASSWORD=<secret>
GATEWAY_SSL_TRUST_STORE_TYPE=jks

# Extract the principal from the certificate subject (DN)
GATEWAY_SSL_PRINCIPAL_MAPPING_RULES=RULE:^CN=(.*?),.*$/$1/

The certificate now comes from the PKI you already operate, and the trust anchor lives in a truststore you control. What that doesn't buy you is one-certificate revocation: the truststore holds your CA, not individual client certificates, and Gateway's model here is truststore plus principal mapping, with no CRL or OCSP checking. Revoking a single compromised certificate therefore means fencing its principal with Gateway ACLs, or reissuing under a rotated intermediate if the compromise is wider. The difference from MSK is that each of those levers is yours.

The second hop is the same for both

Whichever scheme the client used, Gateway connects to MSK as a Kafka client using IAM, inheriting a role from the ECS task, EKS pod (via IRSA), or EC2 instance it runs on:

KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=AWS_MSK_IAM
KAFKA_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Gateway passes anything prefixed KAFKA_ to its upstream connection as an ordinary Kafka client property. The callback handler line signs the handshake with AWS credentials; without it the connection to MSK fails.

The client's scheme and MSK's scheme never have to agree, because they never meet.

Latency and availability

A proxy in the data path draws two fair objections. Both have concrete answers once you look at where the work happens.

Latency. Gateway validates authentication when a client opens a connection, not on every produce or fetch, so token and certificate checks stay out of your throughput hot path. The remaining cost is one extra network hop, which we measured in the Conduktor Gateway performance benchmark. Co-locate Gateway with the cluster and it stays small.

A security reviewer will ask what happens when a token expires mid-connection. By default, nothing: Gateway supports KIP-368 re-authentication, but GATEWAY_AUTHENTICATION_CONNECTION_MAX_REAUTH_MS defaults to 0, so Gateway only re-checks the credential when the client reconnects. Set it if you want token lifetimes to bound connection lifetimes.

Availability. On a diagram the proxy looks like a single point of failure, but Gateway instances run as a horizontally scaled pool behind a load balancer. An instance holds no per-client session state, so a client whose instance dies reconnects through another one. Gateway externalizes shared state for advanced features to internal Kafka topics, with instances coordinating through Kafka's group-membership protocol, the same primitives your consumers already rely on. Putting authentication at the proxy doesn't put your uptime on a single box.

Why this is worth doing

The immediate win is that onboarding stops stalling on credentials: a team points its existing OIDC or mTLS setup at a new bootstrap endpoint and connects. The longer-term win is that every caller becomes a real, named principal at the proxy, whatever front-door scheme they used.

That matters when your security team asks: who can read what? Once identities are first-class principals, you can answer it across both AWS IAM identities and Kafka principals, and enforce it with ACLs in one place. The AWS MSK partnership page covers how Conduktor fits alongside MSK. And if OIDC or mTLS in front of your own cluster is the control you're after, we'd love to talk.