Event‑Driven Notification Service with Go & WebSockets

TL;DR
– Real‑time push beats polling by up to 80 % latency (Cloudflare, 2023).
– Go’s goroutine model lets a single server juggle tens of thousands of WebSocket streams with megabyte‑scale RAM.
– Choose Kafka for ordered, replayable events; pick Redis Pub/Sub for fire‑and‑forget alerts.
– Keep connection state in Redis, run a heartbeat timer, and use a circuit‑breaker when back‑pressure spikes.
– Deploy each component in its own container, expose Prometheus metrics, and let Kubernetes auto‑scale the hub pods.

Before you start, you need:

• Go 1.22 or newer installed (go version).
• Docker 26+ and a running Kubernetes cluster (minikube works for demos).
• A running Kafka 3.5 (or Redis 7) instance reachable from your dev box.
• Basic knowledge of HTTP, WebSockets, and JSON/ProtoBuf serialization.
• git, make, and kubectl on your PATH.

Introduction: The Need for Real‑Time, Scalable Notification Services

A sudden surge of chat messages slammed the backend of a popular e‑learning platform. Users stared at a blank screen while the server polled every five seconds, each request adding milliseconds of delay. The incident turned a friendly learning experience into a source of frustration, and the ops team spent hours chasing phantom latency spikes.

Switching to a push‑based model could have cut the round‑trip time dramatically. Cloudflare measured an 80 % latency reduction when developers replaced HTTP polling with WebSockets. The lesson? When users expect instant feedback—think collaborative docs, stock tickers, or live game scores—polling simply cannot keep up.

Event‑driven architecture supplies the glue that binds producers (user actions, sensor updates) to consumers (notification engines, analytics). By decoupling sources from sinks, you gain horizontal scalability, fault isolation, and the ability to replay or enrich streams later.

💡 Pro Tip: Treat every UI‑visible change as an event. Even a “read receipt” qualifies, and turning it into a message lets you track delivery guarantees later.

Core Architecture: Event‑Driven Principles and Components

The backbone of any robust real‑time notification service resembles a highway network: producers drop cars onto the road, a central dispatcher (the message broker) routes them, and multiple exits deliver the payload to waiting passengers.

Event Producers and Consumers

• Producers – API servers, IoT gateways, or background jobs that emit JSON or Protobuf events.
• Consumers – The WebSocket hub, analytics pipelines, or email/SMS fallbacks.

Both sides should be idempotent; a consumer must tolerate duplicate deliveries without side effects, because at‑least‑once semantics are the default for Kafka and often inevitable for Redis Pub/Sub under failover.

The Message Broker / Event Bus (Kafka vs. Redis Pub/Sub)

Pick Kafka when you need strict ordering (e.g., financial alerts) or the ability to reprocess past events. Opt for Redis when you favor ultra‑low latency and can tolerate occasional gaps.

The Connection Manager & WebSocket Hub

The hub acts as a connection pooling layer. It stores live sockets in a thread‑safe map, groups them by user ID, and pushes messages downstream. Gorilla WebSocket v1.5.0 provides a battle‑tested implementation.

The Notification Service API Layer

REST endpoints accept subscription requests, token validation, and optional filtering parameters. The API writes a “subscription” record to Redis, enabling stateless scaling of hub pods.

// main.go — Go 1.22, Gorilla WebSocket v1.5.0
package main

import (
    "context"
    "encoding/json"
    "log"
    "net/http"
    "time"

    "github.com/gorilla/websocket"
    "github.com/redis/go-redis/v9"
)

var (
    upgrader = websocket.Upgrader{
        ReadBufferSize:  1024,
        WriteBufferSize: 1024,
        // Allow all origins for demo – lock down in prod.
        CheckOrigin: func(r *http.Request) bool { return true },
    }
    rdb = redis.NewClient(&redis.Options{
        Addr: "redis:6379",
        // Use DB 2 for connection metadata.
        DB: 2,
    })
)

// subscriptionMsg defines the JSON payload a client sends to join a room.
type subscriptionMsg struct {
    UserID    string   `json:"user_id"`
    Channels  []string `json:"channels"`
    AuthToken string   `json:"auth_token"`
}

// wsConn wraps a Gorilla connection with bookkeeping fields.
type wsConn struct {
    *websocket.Conn
    send   chan []byte
    userID string
}

// readPump continuously reads messages from the client.
func (c *wsConn) readPump(ctx context.Context) {
    defer func() {
        c.Close()
        close(c.send)
    }()
    c.SetReadLimit(512)
    c.SetReadDeadline(time.Now().Add(60 * time.Second))
    c.SetPongHandler(func(string) error {
        c.SetReadDeadline(time.Now().Add(60 * time.Second))
        return nil
    })
    for {
        _, msg, err := c.ReadMessage()
        if err != nil {
            if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway) {
                log.Printf("unexpected close: %v", err)
            }
            return
        }
        // Process subscription request.
        var sub subscriptionMsg
        if err := json.Unmarshal(msg, &sub); err != nil {
            log.Printf("invalid subscription payload: %v", err)
            continue
        }
        // Simple token validation placeholder.
        if sub.AuthToken == "" {
            log.Println("missing auth token")
            continue
        }
        // Store mapping in Redis for stateless hub scaling.
        key := "user:" + sub.UserID + ":channels"
        if err := rdb.SAdd(ctx, key, sub.Channels...).Err(); err != nil {
            log.Printf("redis error: %v", err)
        }
        c.userID = sub.UserID
        // Acknowledge.
        ack := []byte(`{"type":"sub_ack"}`)
        c.send <- ack
    }
}

// writePump sends queued messages to the client, honoring backpressure.
func (c *wsConn) writePump(ctx context.Context) {
    ticker := time.NewTicker(30 * time.Second)
    defer func() {
        ticker.Stop()
        c.Close()
    }()
    for {
        select {
        case msg, ok := <-c.send:
            c.SetWriteDeadline(time.Now().Add(10 * time.Second))
            if !ok {
                // Channel closed, send close frame.
                _ = c.WriteMessage(websocket.CloseMessage, []byte{})
                return
            }
            w, err := c.NextWriter(websocket.TextMessage)
            if err != nil {
                return
            }
            _, _ = w.Write(msg)
            if err := w.Close(); err != nil {
                return
            }
        case <-ticker.C:
            // Send ping to keep connection alive.
            c.SetWriteDeadline(time.Now().Add(10 * time.Second))
            if err := c.WriteMessage(websocket.PingMessage, nil); err != nil {
                return
            }
        case <-ctx.Done():
            return
        }
    }
}

// serveWS upgrades HTTP to WebSocket and registers the connection.
func serveWS(w http.ResponseWriter, r *http.Request) {
    conn, err := upgrader.Upgrade(w, r, nil)
    if err != nil {
        log.Printf("upgrade failed: %v", err)
        return
    }
    ctx, cancel := context.WithCancel(r.Context())
    defer cancel()
    client := &wsConn{
        Conn: conn,
        send: make(chan []byte, 256), // Buffered to absorb bursts.
    }
    go client.readPump(ctx)
    go client.writePump(ctx)
}

The snippet above shows graceful shutdown (context cancellation), heartbeat handling, and a buffered send channel that provides simple back‑pressure. In production you would attach a circuit‑breaker around the Redis calls and enrich error logs with request IDs.

Deep Dive: Implementing the WebSocket Connection Hub in Go

A hub that supports thousands of concurrent users must juggle three responsibilities: register/deregister sockets, broadcast events efficiently, and protect against runaway memory usage. The following design mirrors the classic “hub” pattern but adds connection pooling tricks that keep CPU overhead low.

Managing Connections with Goroutines and Channels

The hub maintains three channels: register, unregister, and broadcast. Each event runs through a selector loop, guaranteeing single‑threaded map access without mutexes.

// hub.go — Go 1.22, no external dependencies.
type Hub struct {
    // Registered connections indexed by user ID.
    conns map[string]map[*wsConn]struct{}

    // Channels for internal coordination.
    register   chan *wsConn
    unregister chan *wsConn
    broadcast  chan broadcastMsg
}

// broadcastMsg carries payload and target user IDs.
type broadcastMsg struct {
    users []string
    data  []byte
}

// newHub initializes the struct.
func newHub() *Hub {
    return &Hub{
        conns:      make(map[string]map[*wsConn]struct{}),
        register:   make(chan *wsConn),
        unregister: make(chan *wsConn),
        broadcast:  make(chan broadcastMsg, 1024),
    }
}

// run starts the event loop; only one goroutine touches the maps.
func (h *Hub) run() {
    for {
        select {
        case c := <-h.register:
            if h.conns[c.userID] == nil {
                h.conns[c.userID] = make(map[*wsConn]struct{})
            }
            h.conns[c.userID][c] = struct{}{}
        case c := <-h.unregister:
            if users, ok := h.conns[c.userID]; ok {
                delete(users, c)
                if len(users) == 0 {
                    delete(h.conns, c.userID)
                }
            }
        case msg := <-h.broadcast:
            for _, uid := range msg.users {
                for conn := range h.conns[uid] {
                    // Non‑blocking send; drop if buffer full.
                    select {
                    case conn.send <- msg.data:
                    default:
                        // Back‑pressure: close and deregister.
                        close(conn.send)
                        delete(h.conns[uid], conn)
                    }
                }
            }
        }
    }
}

The broadcast case illustrates targeted notifications (iterate only over interested users). If a connection’s send buffer is full, the hub discards the message and tears down the socket—this prevents a single slow client from throttling the whole system.

Connection State, Heartbeats, and Graceful Shutdown

Each wsConn starts a read and write pump (see earlier). The hub registers the connection once the initial subscription succeeds. When the client disconnects, both pumps close their channels, the hub receives an unregister event, and resources are reclaimed.

⚠️ Warning: Never block on a per‑connection write; a dead client can stall the entire broadcast loop. Use the non‑blocking pattern shown above.

Broadcasting vs. Targeted Notifications

• Broadcast – Use when the event is truly global (e.g., system maintenance notice). The hub loops over every user map entry, which scales linearly but remains cheap because the operation stays in‑memory.
• Targeted – Build a slice of user IDs that match the event (e.g., chat room members). This approach keeps network traffic minimal and respects privacy boundaries.

Handling Events: From Source to User Screen

Once the hub is ready, the rest of the pipeline focuses on event ingestion, transformation, and delivery.

Subscribing to Event Streams

A dedicated consumer service subscribes to Kafka topics (or Redis channels) and pushes raw payloads into the hub.

// consumer.go — Kafka client using segmentio/kafka-go v0.4.28
package main

import (
    "context"
    "log"
    "time"

    "github.com/segmentio/kafka-go"
)

func startKafkaConsumer(h *Hub, topic string) {
    r := kafka.NewReader(kafka.ReaderConfig{
        Brokers:        []string{"kafka:9092"},
        GroupID:        "notif-service",
        Topic:          topic,
        MinBytes:       1e3,
        MaxBytes:       1e6,
        CommitInterval: time.Second,
    })
    go func() {
        defer r.Close()
        for {
            m, err := r.ReadMessage(context.Background())
            if err != nil {
                log.Printf("kafka read error: %v", err)
                continue
            }
            // Pass raw value to hub; conversion in next step.
            h.broadcast <- broadcastMsg{
                users: extractUserIDs(m.Value), // custom function.
                data:  m.Value,
            }
        }
    }()
}

If you prefer Redis, replace the reader with a Subscribe loop using go-redis/v9. The consumer must be idempotent; store the last processed offset in Redis to survive restarts.

Transforming and Enriching Events

Often the raw event lacks context (e.g., user name, locale). A lightweight enrichment step—perhaps a gRPC call to a user‑profile service—adds those fields before forwarding to the hub. Keep this step stateless; store caches in an in‑memory LRU (e.g., github.com/hashicorp/golang-lru).

Pushing to the WebSocket Hub and Client‑Side Handling

On the front end (nileshblog.tech), a simple JavaScript client opens a socket and registers interest:

// client.js – runs on nileshblog.tech
const ws = new WebSocket('wss://api.nileshblog.tech/ws');
ws.addEventListener('open', () => {
    ws.send(JSON.stringify({
        user_id: localStorage.getItem('uid'),
        channels: ['news', 'comments'],
        auth_token: getJwt()
    }));
});
ws.addEventListener('message', (ev) => {
    const data = JSON.parse(ev.data);
    if (data.type === 'notification') {
        renderToast(data.payload);
    }
});
ws.addEventListener('close', () => {
    // Reconnect with exponential backoff.
    setTimeout(() => location.reload(), 2000);
});

The client handles graceful degradation: if the socket closes, the UI falls back to a short‑polling fallback for a few seconds before trying again. This ensures users never see a blank screen.

Critical Engineering Considerations & Trade‑offs

Designing for production forces you to confront subtle trade‑offs that rarely appear in tutorial‑level code.

Scalability and Horizontal Scaling Strategies

• Stateless hubs – Store connection metadata (user → connection IDs) in Redis. Any pod can pick up a user’s session after a restart, eliminating sticky sessions.
• Sharding – Split the user space by hash‑modulo and route each shard to a dedicated hub replica. Use a front‑end load balancer (NGINX 1.25) with the hash $remote_addr consistent; directive.
• Autoscaling – Expose the hub’s process_cpu_seconds_total metric to Prometheus, then configure a HorizontalPodAutoscaler (HPA) that scales when CPU > 70 %.

Message Delivery Guarantees: At‑Least‑Once vs. At‑Most‑Once

Kafka naturally offers at‑least‑once delivery; duplicates may appear if a consumer restarts after a commit failure. Mitigate with idempotent consumers—store a hash of the message ID per user and drop repeats. Redis Pub/Sub, by contrast, behaves as at‑most‑once: if a subscriber disconnects, the message vanishes. Choose based on tolerance for missed alerts.

Backpressure and Connection Throttling

When a sudden traffic spike floods the hub, the buffered send channels fill quickly. The non‑blocking select pattern discards excess data and closes the offending connection. Additionally, implement a circuit‑breaker (e.g., github.com/sony/gobreaker) around external calls (Redis, gRPC) to prevent cascading failures.

// breaker.go – circuit breaker example
var cb = gobreaker.NewCircuitBreaker(gobreaker.Settings{
    Name:        "RedisWrite",
    MaxRequests: 5,
    Interval:    30 * time.Second,
    Timeout:     10 * time.Second,
})

func safeRedisAdd(ctx context.Context, key string, members ...string) error {
    _, err := cb.Execute(func() (interface{}, error) {
        return nil, rdb.SAdd(ctx, key, members...).Err()
    })
    return err
}

When the breaker opens, the hub temporarily drops new subscriptions, logs the state, and returns a friendly “service busy” response to the API caller.

Monitoring, Metrics, and Observability

Expose a /metrics endpoint with Prometheus client (github.com/prometheus/client_golang v1.16). Track:

• ws_active_connections – Gauge of current sockets.
• ws_messages_sent_total – Counter per user/channel.
• ws_backpressure_events_total – Counter of dropped messages due to full buffers.

Grafana dashboards (Grafana 10) can visualise spikes and correlate them with Kafka lag (kafka_consumer_lag). Alert on lag > 5000ms to trigger scaling.

⚠️ Warning: Do not expose raw error messages over the WebSocket; they can leak internal details. Instead, send a generic "type":"error" envelope with a code that the client can map to a user‑friendly string.

Deployment and Performance Optimizations

A clean Docker image and a Kubernetes manifest keep the service portable across clouds.

Containerizing with Docker and Orchestrating with Kubernetes

# Dockerfile – Go 1.22, Alpine 3.20
FROM golang:1.22-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o notifsvc ./cmd/main

FROM alpine:3.20
RUN adduser -D appuser
USER appuser
COPY --from=builder /app/notifsvc /usr/local/bin/notifsvc
EXPOSE 8080
ENTRYPOINT ["notifsvc"]

Kubernetes Deployment (excerpt):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ws-hub
spec:
  replicas: 3
  selector:
    matchLabels:
      app: ws-hub
  template:
    metadata:
      labels:
        app: ws-hub
    spec:
      containers:
      - name: hub
        image: ghcr.io/nilesh/notification-hub:v1.2.0
        ports:
        - containerPort: 8080
        env:
        - name: REDIS_ADDR
          value: redis:6379
        - name: KAFKA_BROKERS
          value: kafka:9092
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
        resources:
          limits:
            cpu: "500m"
            memory: "256Mi"
          requests:
            cpu: "200m"
            memory: "128Mi"

The readinessProbe prevents the Service load‑balancer from sending traffic to a pod before it finishes connecting to Redis and Kafka.

Load Testing and Benchmarking Scenarios

Use k6 (v0.54) to simulate 20k concurrent WebSocket connections:

import ws from 'k6/ws';
import { check, sleep } from 'k6';

export const options = {
  stages: [{ duration: '2m', target: 20000 }],
};

export default function () {
  const url = 'wss://api.nileshblog.tech/ws';
  const response = ws.connect(url, {}, function (socket) {
    socket.on('open', function () {
      socket.send(JSON.stringify({
        user_id: `user_${__VU}`,
        channels: ['loadtest'],
        auth_token: 'test-token'
      }));
    });
    socket.on('message', function (msg) {
      check(msg, { 'got notification': (m) => m.includes('loadtest') });
    });
    socket.setTimeout(function () {
      socket.close();
    }, 30000);
  });
  sleep(1);
}

Observe CPU, memory, and latency in Grafana. If the hub’s ws_backpressure_events_total climbs, consider increasing send buffer size or adding more replicas.

Cost‑Benefit Analysis: Self‑Hosted vs. Managed Services

For a modest startup, managed Pub/Sub speeds time‑to‑market. Once traffic peaks past a few hundred thousand messages per second, self‑hosting Kafka can shave a few milliseconds and lower per‑message cost.

Common Errors & Fixes

Conclusion: Key Takeaways and Further Evolution

• Push‑based communication with WebSockets slashes latency compared to polling, especially for high‑frequency UI updates.
• Pairing Go’s lightweight concurrency with a solid event bus gives you a foundation that scales horizontally without massive memory footprints.
• Selecting the right broker hinges on ordering and durability needs; Kafka wins for strict guarantees, Redis shines for pure speed.
• Guard against back‑pressure early, make every consumer idempotent, and surface metrics to a monitoring stack.
• Containerize, observe, and let Kubernetes do the heavy lifting—your code stays lean, your ops stay sane.

My take: While the buzz around server‑sent events (SSE) and GraphQL subscriptions is loud, nothing matches the raw efficiency of a well‑engineered Gorilla WebSocket hub when you need deterministic ordering and sub‑millisecond latency. Invest time in the connection manager; it pays dividends as traffic grows.

Frequently Asked Questions

Why use Go (Golang) for a WebSocket notification service?

Go’s lightweight goroutines and channel primitives let a single binary juggle tens of thousands of long‑lived connections. The runtime’s scheduler keeps memory usage predictable, and the compiled binary runs fast enough to stay under typical cloud CPU limits.

When should I choose Redis Pub/Sub over Apache Kafka for the event bus?

Pick Redis Pub/Sub when you need ultra‑low latency and can tolerate occasional message loss or out‑of‑order delivery—think transient alerts or chat “typing” indicators. Choose Kafka for any scenario that requires replayability, strict ordering, or guaranteed persistence, such as financial transaction notifications.

How do you handle a user being connected from multiple devices simultaneously?

The hub maintains a map of userID → []*wsConn. When an event arrives, it iterates over the slice and pushes the payload to each socket. Store the map in Redis so any hub instance can reconstruct the list after a restart, avoiding sticky‑session constraints.

Call to Action

Enjoyed the deep dive? Have questions about scaling your own notification service on nileshblog.tech? Drop a comment below, share the article on social media, or subscribe to the newsletter for more production‑grade Go patterns and system‑design walk‑throughs. Let’s build real‑time experiences together!

Author Bio:
I’m Nilesh Raut, a Software Development Engineer with 2+ years of experience, specializing in Go, JavaScript, Python, Docker, Kubernetes, Git, Jenkins, microservices, and system design (LLD/HLD), backed by a strong foundation in data structures and algorithms. Alongside my engineering journey, I bring 4+ years of hands‑on experience in SEO, where I’ve worked extensively on content strategy, keyword research, technical SEO, and organic growth, helping products and businesses scale efficiently by aligning solid technology with search‑driven performance.