Kafka vs NATS for Go Event Pipelines: How to Choose

Hamza AnisAuthor

September 26, 2026·11 min read

Choose Kafka when the event log is a record: you need long retention, replay into warehouses and analytics, stream processing, and an ecosystem of connectors, and you can afford to run it properly. Choose NATS with JetStream when events are traffic between services: low-latency messaging, request/reply, fan-out, and persistence where you need it, all from a single small binary a small team can operate. For most Go teams building real-time product features, NATS JetStream is the simpler start. For data platforms where the log is the system of record, Kafka earns its weight.

We build real-time data pipelines on Kafka, NATS, and NSQ with Go workers. Our OroTracker build, for example, uses Go streaming ingestion over NATS and NSQ for live gold and silver prices. Here's how we decide between them.

Records vs traffic

The question that settles most of these decisions is: what happens to an event after it's been processed?

• If it must stay available for weeks or months, so a new consumer can replay history, a warehouse can reload, or you can rebuild state from the log, you're treating events as

  records

  . That's Kafka's home ground.

• If an event matters mostly until it's handled, and persistence is there for reliability rather than history, you're moving

  traffic

  . NATS was built for that, with JetStream adding durability where required.

Architecture in one paragraph each

Kafka is a distributed, partitioned commit log. Topics are split into partitions; each partition is an ordered, append-only log replicated across brokers. Consumers in a consumer group divide partitions between them and track their position with offsets, so any consumer can rewind and replay. Since Kafka 4.0, clusters run in KRaft mode without ZooKeeper, which removes one moving part, but brokers, partitions, replication, and rebalancing still need real operational care. See the Kafka documentation.

NATS is a lightweight messaging system built around subjects with wildcards (prices.gold, prices.>). Core NATS is fire-and-forget pub/sub and request/reply with very low latency. JetStream adds persistence: streams capture subjects to disk with retention limits, and durable consumers track delivery and acknowledgements. It also includes a key-value store and object store. All of it ships in one server binary. See the JetStream docs.

Delivery semantics and ordering

Whichever you choose, write idempotent handlers. At-least-once means duplicates will happen: during rebalances, redeliveries, and deploys.

Operating each

This is where most teams actually feel the difference:

• Kafka:

  multiple brokers, partition planning (partitions are hard to reduce later), replication and ISR monitoring, consumer-group rebalances, disk and retention sizing, and usually a schema registry. Managed Kafka removes much of the toil at a significant price.

• NATS:

  one binary, a three- or five-node cluster for JetStream, stream and consumer configuration as code. Far less to operate. Its ecosystem of connectors and stream processors is smaller than Kafka's.

Choosing an event backbone for a real-time feature? We design and load-test pipelines that process millions of events per minute. Book a technical discovery call and we'll pressure-test your choice against your traffic.

Go consumers for both

Kafka with franz-go

franz-go is a pure-Go Kafka client, with no cgo. Here's a consumer-group worker that commits only after processing and dead-letters failures:

package main

import (
	"context"
	"log"
	"os"
	"strings"

	"github.com/twmb/franz-go/pkg/kgo"
)

func runKafka(ctx context.Context, handle func(context.Context, []byte, []byte) error) error {
	cl, err := kgo.NewClient(
		kgo.SeedBrokers(strings.Split(os.Getenv("KAFKA_BROKERS"), ",")...),
		kgo.ConsumerGroup("price-alerts"),
		kgo.ConsumeTopics("prices"),
		kgo.DisableAutoCommit(), // commit only what we've processed
	)
	if err != nil {
		return err
	}
	defer cl.Close()

	for {
		fetches := cl.PollFetches(ctx)
		if fetches.IsClientClosed() || ctx.Err() != nil {
			return ctx.Err()
		}
		fetches.EachError(func(topic string, partition int32, err error) {
			log.Printf("fetch error %s/%d: %v", topic, partition, err)
		})

		fetches.EachRecord(func(r *kgo.Record) {
			if err := handle(ctx, r.Key, r.Value); err != nil {
				// Park the poison message instead of blocking the partition.
				dlq := &kgo.Record{Topic: "prices.dlq", Key: r.Key, Value: r.Value}
				if err := cl.ProduceSync(ctx, dlq).FirstErr(); err != nil {
					log.Printf("dead-letter failed: %v", err)
				}
			}
		})

		if err := cl.CommitUncommittedOffsets(ctx); err != nil {
			log.Printf("commit failed: %v", err)
		}
	}
}

Records within a partition arrive in order. To process partitions in parallel while keeping per-key order, run one goroutine per partition (franz-go exposes per-partition iteration via fetches.EachPartition).

NATS JetStream with nats.go

With the jetstream package, the stream and durable consumer are declared in code, and each message is acknowledged explicitly:

package main

import (
	"context"
	"errors"
	"time"

	"github.com/nats-io/nats.go"
	"github.com/nats-io/nats.go/jetstream"
)

var errPoison = errors.New("unprocessable message")

func runJetStream(ctx context.Context, handle func(context.Context, string, []byte) error) error {
	nc, err := nats.Connect(nats.DefaultURL)
	if err != nil {
		return err
	}
	defer nc.Drain()

	js, err := jetstream.New(nc)
	if err != nil {
		return err
	}

	stream, err := js.CreateOrUpdateStream(ctx, jetstream.StreamConfig{
		Name:     "PRICES",
		Subjects: []string{"prices.>"},
		Storage:  jetstream.FileStorage,
		Replicas: 3,
		MaxAge:   7 * 24 * time.Hour, // keep a week for replay
	})
	if err != nil {
		return err
	}

	cons, err := stream.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
		Durable:    "price-alerts",
		AckPolicy:  jetstream.AckExplicitPolicy,
		AckWait:    30 * time.Second,
		MaxDeliver: 5, // then stop redelivering; alert on the advisory
	})
	if err != nil {
		return err
	}

	cc, err := cons.Consume(func(msg jetstream.Msg) {
		if err := handle(ctx, msg.Subject(), msg.Data()); err != nil {
			if errors.Is(err, errPoison) {
				msg.Term() // never redeliver
				return
			}
			msg.NakWithDelay(5 * time.Second) // retry later
			return
		}
		msg.Ack()
	})
	if err != nil {
		return err
	}
	defer cc.Stop()

	<-ctx.Done()
	return ctx.Err()
}

Both workers follow the same rules: acknowledge or commit only after the side effect succeeded, park poison messages instead of blocking, and make handle idempotent.

The decision table

Whichever you pick, size it for peak, not average, and add backpressure so producers slow down before consumers fall over. Both are on our free Backend Scaling Checklist.

Sizing, lag, and backpressure

Most pipeline incidents we're called into aren't broker failures. They're consumers falling behind. Plan for that from the start:

• Kafka partitions cap consumer parallelism.

  A consumer group can't have more active consumers than the topic has partitions. Pick a partition count with headroom for peak, because adding partitions later changes key-to-partition mapping and breaks per-key ordering during the transition.

• JetStream in-flight limits are your throttle.

  MaxAckPending

  bounds how many unacknowledged messages a consumer holds. Set it to what your workers can process within

  AckWait

  , or you'll see redelivery storms under load.

• Alert on lag, not just errors.

  Kafka consumer lag (log-end offset minus committed offset) and JetStream's pending count are the leading indicators. Page on sustained growth, not on a single spike.

• Scale workers on lag.

  Autoscaling consumers on CPU misses I/O-bound workers that are waiting on a slow database. Scale on queue depth or lag instead.

• Make producers feel the pressure.

  Bound the producer-side buffers and return errors (or slow down) when the broker or downstream is saturated, rather than growing memory until the process dies.

Load test the whole path, producer to broker to consumer to database, at peak rates before launch. The broker is rarely the weakest link.

Running both

At scale, many teams run both: NATS as the fast connective tissue between services, and Kafka as the durable log that feeds analytics. A bridge service (in Go, naturally) moves the events that matter into Kafka. It's a sensible end state, but not a sensible starting point. Start with one, and add the second when a concrete need appears. The same "extract when justified" rule applies to services, as we argue in modular monolith vs microservices. For the worker runtime itself, see Go vs Rust vs Node.js for high-throughput APIs.

FAQ

Is NATS a replacement for Kafka?

Sometimes. NATS with JetStream covers persistent streams, replay, and at-least-once delivery for many event-driven systems with far less operational weight. Kafka remains the better fit when the event log is a long-lived system of record feeding analytics, warehouses, and stream processing.

Which is faster, Kafka or NATS?

Both handle very high message rates when tuned, so raw speed rarely decides it. NATS typically offers lower latency for request/reply and fan-out; Kafka excels at sustained high-volume, durable log throughput with long retention. Benchmark with your message sizes, durability settings, and replication factor.

Does NATS guarantee message ordering?

JetStream stores messages in order within a stream, but processing order depends on your consumer: redeliveries and multiple in-flight messages can reorder work. For strict per-key ordering, partition subjects by key and limit in-flight messages per consumer, or design handlers to be order-tolerant.

Can Go work well with Kafka?

Yes. Pure-Go clients such as franz-go support consumer groups, transactions, and manual offset commits without cgo, and Go's concurrency model suits per-partition worker pools.

Should we run both Kafka and NATS?

It's a reasonable architecture at scale: NATS for low-latency service-to-service messaging and fan-out, Kafka for the durable event log that feeds analytics. For a startup, start with one. Usually NATS JetStream if the team is small, or Kafka if the log itself is the product.

Build the pipeline once, correctly

We build real-time data processing systems in Go on Kafka, NATS, and NSQ, designed to handle millions of events per minute and load-tested before launch. Book a technical discovery call and we'll help you choose, and size, the right backbone.

Written by Hamza Anis

CEO & Senior Backend Engineer at CapregSoft. Building high-performance backends and AI automation systems for funded startups and mid-market teams.