I've been working on creating an MSK Serverless (kafka) cluster. The serverless product is relatively new, but the documentation for it is confusing; the operating model of "serverless" is fundamentally different from MSK proper. Both are part of the same product family, so when searching it's difficult to separate the two. As you would expect with a "serverless" product, the whole software stack needed to run kafka is abstracted away and managed by AWS. It all gets deployed into a managed VPC that you can't access, and a bootstrap broker is exposed to a number of your VPCs using VPC endpoints.

I could not get my head around how the routing worked. It turned out to be much simpler than I had blown it up to be. The bootstrap host and all the broker hostnames will resolve within the VPCs that the cluster is attached to. The hostnames are subdomains of c3.kafka-serverless.{region}.amazonaws.com. If you have multiple linked networks, then the broker hostnames won't resolve because their DNS records are private. In this configuration, you have to expose your VPC DNS to the other networks, and configure those other networks or devices to forward requests to your resolver. The product that enables this is Route 53 inbound resolver. There's also an outbound resolver which allows forwarding DNS requests from your VPC to other DNS resolvers within other networks.

You can link VPCs and networks together using lots of different strategies. How you link your networks and VPCs together is out of the scope of this article; the assumption is that you have completed the work to make your VPC accessible to your device somehow. You can find the private IP address of your bootstrap broker and connect to it from your machine (check your VPC Endpoints). Finally, this also assumes you have set up the Route 53 inbound resolvers for your VPC.

It took a bit of digging and luck to find a solution that didn't require installing dnsmasq or configuring bind. I really didn't want to screw around with the DNS on my machine more than I had to. The system configuration utility scutil allows you to change your network configuration, including to add additional resolvers. The utility documentation isn't particularly easy to find or digest, but I found an example that was close enough to get it to do what I needed it to do.

Assuming you have deployed resolvers in three subnets in us-east-1, and the IP addresses assigned to them are 192.1.0.1, 192.2.0.1 and 192.3.0.1:

sudo esutil
> open
> d.init
> d.add ServerAddresses * 192.1.0.1 192.2.0.1 192.3.0.1
> d.add SupplementalMatchDomains * kafka-serverless.us-east-1.amazonaws.com
> set State:/Network/Service/kafka-serverless.us-east-1.amazonaws.com/DNS
> quit

You should then be able to see the resolver listed when you run esutil --dns. You should also be able to resolve the bootstrap host, which is the only hostname that AWS gives you, along with all the kafka brokers that form the cluster. I've not yet attempted to add multiple resolvers (i.e. multiple VPC + resolver + cluster combinations), but the name of the key needs to be unique. You could change the above command to VPC+region, whatever makes it clear.

set State:/Network/Service/vpc-0123456789.us-west-2/DNS

You may also wish to add multiple space-separated SupplementalMatchDomains to map more service endpoints if you have other use-cases to cover.