A Dev Kafka Broker for the Price of a Bad Coffee
Running a real Kafka broker for Sentinel's ingestion layer — without the managed service price tag
Deploying Kafka on Fly.io
Before I could wire Kafka into Sentinel’s ingestion layer, I needed a Kafka broker that wasn’t going to cost me $50/month to leave running while I figured things out.
AWS MSK? Overkill for dev work. I need something in the middle — a real broker, running on real infrastructure, that I could spin up and shut down for pennies.
So I put Kafka on Fly.io.
Why Fly.io for a Dev Kafka
This isn’t a production setup yet. This is housekeeping.
Sentinel’s ingestion service publishes to Kafka. To develop and test that integration, I need a broker I can hit from my local machine without running a JVM locally or paying for managed infrastructure I’m not using yet.
Fly.io gives me three things that matter here:
- Cheap machines that stop billing when stopped. A shared-cpu-1x with 1GB RAM runs about $5/month. Stop it, and you pay $0 for compute. The 10GB volume costs $1.50/month regardless — pennies.
- fly proxy for local access. Just fly proxy 9092 and your local tools connect as if the broker were on localhost.
- Persistent volumes. Kafka needs disk. Fly volumes survive machine restarts.
The whole point: a real Kafka broker I can reach from kcat and my Go service, without the overhead of a managed service or the networking headaches of Docker.
KRaft Mode: No Zookeeper, One Process
I’m running Kafka 3.9.2 in KRaft mode. This is worth explaining because it’s an architectural decision, not just a configuration toggle.
Traditional Kafka needs Zookeeper — a separate distributed coordination service — to manage broker metadata, leader election, and cluster membership. KRaft moves all of that into Kafka itself. The broker and the controller run in a single process.
For a single-node dev broker, this is the obvious choice. No second process, no second failure domain. But it’s also the forward-looking one. KRaft has been production-ready since Kafka 3.3, and Kafka 4.0+ drops Zookeeper support entirely. If you’re still deploying with Zookeeper in 2026, you’re building on deprecated infrastructure.
Combined mode — KAFKA_PROCESS_ROLES=controller,broker — means one process handles both roles. The controller quorum has one voter: itself. Self-elected leader.
Image Selection: Three Attempts, Two Failures
This is the part nobody writes about. The twenty minutes you spend debugging before you write a single line of actual configuration.
Attempt 1: bitnami/kafka:3.9 — Failed.
Attempt 2: confluentinc/cp-kafka:7.7.1 — Failed.
Attempt 3: apache/kafka:3.9.2 via [build] image — The official Apache image. Referenced it directly in fly.toml.
What actually worked: A one-line Dockerfile.
FROM apache/kafka:3.9.2
With [build] dockerfile = “Dockerfile” in fly.toml.
The Configuration
Here’s the full fly.toml:
app = ‘kafka-fly-dev’
primary_region = ‘dfw’
[build]
dockerfile = ‘Dockerfile’
[env]
KAFKA_NODE_ID = ‘0’
KAFKA_PROCESS_ROLES = ‘controller,broker’
KAFKA_CONTROLLER_QUORUM_VOTERS = ‘0@localhost:9093’
KAFKA_LISTENERS = ‘PLAINTEXT://:9092,CONTROLLER://:9093’
KAFKA_ADVERTISED_LISTENERS = ‘PLAINTEXT://127.0.0.1:9092’
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP = ‘CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT’
KAFKA_CONTROLLER_LISTENER_NAMES = ‘CONTROLLER’
KAFKA_AUTO_CREATE_TOPICS_ENABLE = ‘true’
KAFKA_LOG_RETENTION_HOURS = ‘24’
KAFKA_HEAP_OPTS = ‘-Xmx512m -Xms256m’
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR = ‘1’
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR = ‘1’
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR = ‘1’
CLUSTER_ID = ‘MkU3OEVBNTcwNTJENDM2Qk’
[[mounts]]
source = ‘kafka_data’
destination = ‘/opt/kafka/kafka-logs’
[[vm]]
memory = ‘1gb’
cpu_kind = ‘shared’
cpus = 1
Every setting is there for a reason. A few worth calling out:
KAFKA_HEAP_OPTS = ‘-Xmx512m -Xms256m’ — Kafka runs on the JVM. The default heap allocation would eat the entire 1GB VM and get OOM-killed. Cap it.
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR = ‘1’ — The default is 3. With one broker, Kafka can’t replicate to three nodes. Single node means replication factor of 1 everywhere.
CLUSTER_ID — A fixed cluster ID so the broker identity survives restarts. Without this, Kafka generates a new one on every boot, and the data on your persistent volume becomes orphaned.
KAFKA_LOG_RETENTION_HOURS = ‘24’ — Messages deleted after 24 hours. This is a dev broker. I don’t need a week of retention eating disk space.
Deploy: Three Commands
# Create the app
fly launch --no-deploy --copy-config --yes
# Deploy with a 10GB volume
fly deploy --volume-initial-size 10
# Connect locally
fly proxy 9092 -a kafka-fly-dev
That’s it. Broker is running. Local access is live.
Testing with kcat
kcat is the right tool here. Natively installed via Homebrew.
brew install kcat
# Check cluster metadata
kcat -b 127.0.0.1:9092 -L
# Produce
echo ‘{”event”:”signup”,”name”:”silverwoi”}’ | kcat -b 127.0.0.1:9092 -t test-events -P
# Consume all messages
kcat -b 127.0.0.1:9092 -t test-events -C -e
I tried running Kafka UI in Docker (provectuslabs/kafka-ui). It failed but we can skip it since terminal kcat usage is enough.
The Terraform Detour
I tried automating this with Terraform it also failed. I moved on.
Cost
Machine (shared-cpu-1x, 1GB) ~$5/month or $0 if stopped
Volume (10GB) $1.50/month
Stop the machine when you’re not using it:
fly machine stop <machine-id> --app kafka-fly-dev
fly machine start <machine-id> --app kafka-fly-dev
Fly only bills for running machines. The volume costs $1.50/month regardless. A dev Kafka broker for the price of a bad coffee.
Why This Matters for Sentinel
This isn’t a pure Sentinel episode. It’s the infrastructure underneath one.
My ingestion service — the gRPC streaming pipeline I built in the last episode — publishes to Kafka. To develop and test that integration locally, I needed a broker I could reach without mocking the entire Kafka client. Now I have one. fly proxy 9092, point the Go service at 127.0.0.1:9092, and I’m producing real messages to real topics.
The next episode connects the ingestion service to this broker and possible insert to the db (we’ll see how much I can work on this this coming week). That’s where the architecture stops being a diagram and starts moving data.
Senior Tip
- Why this matters in production: Dev/test infrastructure that closely mirrors production catches integration bugs that mocks never will. A real Kafka broker — even a single-node one — exercises serialization, networking, and consumer group behavior that a fake client can’t simulate.
- What juniors underestimate: How much time goes into infrastructure that isn’t “the project.” Setting up a dev Kafka broker isn’t glamorous work. But without it, you’re either mocking everything or paying for managed services you don’t need yet, learning how to provision your own infrastructure is a great skill to posses.
- What hiring managers notice: Whether your dev environment shows the same intentionality as your architecture. If your system design says “Kafka” but your testing says “I used a mock,” that’s a gap.
I’m building this project in public as if my goal was trying to get hired in 2026.
If you’re building your own “hire-me” project, subscribe and build alongside.
Build Like You’re Already Senior.