Building a Stateful FIX Engine on Kubernetes

Alexander Hagemann
Date : 08 Jul, 2025
Categories : Devops
Tags : Fix Kubernetes Dotnet Eventhub Logging

Introduction

Financial Information eXchange (FIX) remains the de facto standard for electronic trading. Each message in a FIX session carries stateful sequencing to guarantee delivery and order. When we host a FIX engine in a Kubernetes cluster, we must reconcile the inherently stateful nature of FIX with Kubernetes’ stateless design philosophy. Trades must never be lost. Pods can restart or shift between nodes at any time. A naive setup, where messages are kept solely in memory or ephemeral storage, risks data loss during failures. The goal of this article is to demonstrate a resilient approach that stores every message on disk via a Persistent Volume Claim (PVC), tails the log with Fluent Bit, and forwards it to Azure Event Hub for downstream consumers. We’ll build our sample FIX engine using .NET and QuickFIX/n, but the concepts apply to any language or FIX library.

This post is intentionally thorough. We will walk through the reasoning behind each architectural choice, show full YAML manifests for Kubernetes, and discuss how multiple consumers can use the same Event Hub stream for persistence and processing. By the end, you’ll have a blueprint for running a stateful FIX service in a cloud native environment that embraces failure and scales horizontally.

The Importance of State in FIX Trading

Every FIX session uses message sequence numbers to detect gaps, duplicates, and out-of-order deliveries. Brokers and exchanges rely on these numbers to confirm receipt and to replay messages on recovery. If we lose a message, even momentarily, we risk inconsistent trade states and compliance issues. A stateful FIX engine maintains its last known sequence numbers and persistent logs of all messages sent and received. When the engine restarts, it must load this state from disk so it can continue the session seamlessly.

Traditional FIX engines often run on bare metal or virtual machines with local disks. In Kubernetes, containers default to ephemeral filesystems. When the pod restarts, all local data vanishes. This is fine for stateless microservices but disastrous for trading applications. The first step is to attach a Persistent Volume to the FIX container so that log files survive restarts. We also ensure that each engine instance uses its own volume to avoid cross-session contamination.

While QuickFIX/n, our example .NET engine, supports various storage types, the simplest and most reliable is the file store. It appends every message to disk before acknowledging it to the counterparty. Because I/O is sequential, performance remains high. The engine uses these log files both for state recovery and for auditing purposes. Our architecture builds on this foundation: the log file is the source of truth, and everything else streams from it.

Logging to a Persistent Volume

A Persistent Volume Claim in Kubernetes is an abstraction over network-attached storage. It provides the resilience of disk while allowing pods to be rescheduled across nodes. Below is a basic YAML definition for a PVC suitable for FIX logs. This example uses a storage class with ReadWriteOnce access, ensuring that only one pod mounts the volume at a time:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fix-log-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: fast-ssd

You can adjust the storage class to your cloud provider. Some trading environments prefer network file systems with high throughput. Others use managed block storage. The key point is that the PVC retains data even if the pod restarts or migrates.

With the volume created, we mount it into the FIX engine container. QuickFIX/n expects a directory path for its file store. In our example we use /data as the mount point. The C# configuration looks like this:

var settings = new SessionSettings("fix.cfg");
settings.SetString("FileStorePath", "/data");

When the engine writes to /data, those files reside on the persistent volume. Kubernetes ensures they remain intact across container lifecycles. If the pod is rescheduled, Kubernetes reattaches the PVC to the new node before the container starts. Upon restart, QuickFIX/n reads the log to restore sequence numbers and session state.

Kubernetes Deployment with a Fluent Bit Sidecar

We run Fluent Bit as a sidecar container inside the same pod as the FIX engine. A sidecar shares the pod’s lifecycle and volume mounts, making it straightforward to access the log file. The pod template below defines two containers: fix-engine and fluent-bit. Both share the fix-log volume, which mounts the PVC created earlier.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: fix-engine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fix-engine
  template:
    metadata:
      labels:
        app: fix-engine
    spec:
      containers:
        - name: fix-engine
          image: myregistry/fix-engine:latest
          env:
            - name: FIX_LOG_PATH
              value: /data
          volumeMounts:
            - name: fix-log
              mountPath: /data
        - name: fluent-bit
          image: cr.fluentbit.io/fluent/fluent-bit:2.2
          env:
            - name: EVENTHUB_CONNECTION
              valueFrom:
                secretKeyRef:
                  name: eventhub-secret
                  key: connection-string
          volumeMounts:
            - name: fix-log
              mountPath: /data
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc
      volumes:
        - name: fix-log
          persistentVolumeClaim:
            claimName: fix-log-pvc
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config

This manifest keeps the engine and the log forwarder tightly coupled while still adhering to Kubernetes best practices. If the engine restarts, so does Fluent Bit. Because both containers share the same volume, Fluent Bit immediately continues tailing the file from where it left off.

The fluent-bit-config ConfigMap referenced above contains Fluent Bit’s configuration file. We’ll detail that in the next section.

Fluent Bit Configuration

Fluent Bit is a lightweight log processor ideal for sidecar deployments. It can read local files, parse or transform them, and ship the data to a variety of destinations. The Azure Event Hubs output plugin is well suited for streaming structured logs. Our ConfigMap below defines an input that tails fix.log and an output that forwards each line to Event Hub:

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  labels:
    app: fix-engine
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
        Parsers_File parsers.conf

    [INPUT]
        Name              tail
        Path              /data/fix.log
        Tag               fix
        Refresh_Interval  1

    [OUTPUT]
        Name       azure_event_hubs
        Match      fix
        Event_Hub  trading-stream
        Connection_String  ${EVENTHUB_CONNECTION}
  parsers.conf: |
    [PARSER]
        Name   fix
        Format none

Fluent Bit reads the log line by line. Because FIX messages are single-line records, we disable parsing and forward the raw lines. Event Hub clients can parse the FIX messages as needed. The [SERVICE] section flushes records every second to minimize latency without overloading the Event Hub API.

Publishing to Azure Event Hub

Azure Event Hub is a scalable streaming platform similar to Kafka. It acts as a durable log of all FIX traffic. Each record is appended to a partition, and consumers track their own offsets. The output plugin in Fluent Bit requires a connection string that grants send rights to the hub. We stored it in a Kubernetes Secret called eventhub-secret and referenced it in the deployment manifest.

Once messages arrive in Event Hub, we gain powerful decoupling. The FIX engine no longer needs to communicate directly with every downstream system. Instead, each consumer reads from the hub at its own pace. If a consumer lags or fails, it can catch up by replaying messages from its last checkpoint. The FIX engine simply keeps writing to its log file, and Fluent Bit ensures that log is replicated to Event Hub.

Multiple Consumers: Persistence and Processing

In many trading systems, we maintain two broad categories of consumers. The first is a persistence service that stores messages in long-term archival storage, such as a relational database or blob storage. This provides a tamper-evident audit trail for compliance and post-trade analysis. The second category is real-time processing. These consumers validate orders, route them to downstream venues, or trigger business logic such as risk checks and analytics.

Because Event Hub supports multiple consumer groups, we can run these services independently. The persistence worker might buffer batches to maximize throughput, while the processing worker handles records immediately to reduce latency. Each consumer group maintains offsets separately, so they never interfere with each other. If one consumer fails, it does not affect the others. Scaling up is as simple as adding more consumer instances and partitions.

To illustrate, here is a basic .NET worker that consumes from Event Hub using the Azure.Messaging.EventHubs SDK and writes messages to a database. The code uses a separate connection string for the storage checkpoint manager:

var consumerGroup = "$Default";
var connection = Environment.GetEnvironmentVariable("EVENTHUB_CONNECTION");
var blobStorage = Environment.GetEnvironmentVariable("CHECKPOINT_STORE");
var client = new EventProcessorClient(new BlobContainerClient(blobStorage), consumerGroup, connection, "trading-stream");

client.ProcessEventAsync += async args =>
{
    var message = Encoding.UTF8.GetString(args.Data.Body.ToArray());
    await SaveToDatabase(message);
    await args.UpdateCheckpointAsync(args.CancellationToken);
};

await client.StartProcessingAsync();

With a similar approach, another consumer might parse FIX messages into application objects and execute domain-specific logic. Because these consumers run independently of the FIX engine, we can deploy updates or scale them without touching the trading pod.

Designing for Failure

The combination of a file-backed message store and a streaming platform enables robust failure recovery. Consider a scenario where the FIX engine container crashes. Kubernetes restarts the pod and reattaches the PVC. QuickFIX/n reads the existing log files, loads its last sequence numbers, and resumes the session without renegotiation. Fluent Bit also restarts and resumes tailing the log from the last line it processed. No messages are lost because they were written to disk before the crash.

If Fluent Bit cannot reach Event Hub—for example, during a network outage—it buffers records locally. Once connectivity returns, it sends the queued messages in order. Azure Event Hub itself is highly available, but we can further protect against outages by using Geo-DR or replicating to multiple regions.

Even if an entire node fails, Kubernetes reschedules the pod to another node and reattaches the PVC. Because the log file remains intact on network storage, the FIX engine recovers instantly. This design embraces Kubernetes’ ephemeral nature while maintaining the stateful guarantees required by FIX.

Operational Considerations

Running a FIX engine in production involves more than just shipping logs. We must monitor resource usage, set proper requests and limits, and secure access to secrets. Here are some operational tips:

Resource Management: FIX engines are sensitive to latency. Allocate CPU and memory with generous headroom, and pin the pod to a node pool with low interference. Using requests and limits in the deployment ensures the container gets the resources it needs.
Liveness and Readiness Probes: Implement health endpoints in the FIX engine to indicate session state. Kubernetes can restart the pod if it becomes unhealthy and wait for the session to log on before routing traffic.
Secret Management: Store sensitive data such as Event Hub connection strings in Kubernetes Secrets. Use role-based access control to restrict who can read or modify them.
Rolling Updates: Because each FIX session uses its own volume, we can roll out new versions by creating a new deployment and gradually shifting sessions. Blue/green deployments ensure we can roll back quickly if something goes wrong.
Metrics and Logging: Fluent Bit can also ship metrics to Prometheus or other monitoring systems. Track message throughput, error rates, and resource consumption to catch issues early.

These practices help maintain stability and observability as the system scales.

Complete Example

Below is a more detailed deployment manifest combining the PVC, the deployment, and the ConfigMap. While real-world setups often include additional Kubernetes objects—such as Secrets, Services, and Ingress—the following example illustrates the core pieces in a single file for clarity:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fix-log-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: fast-ssd
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  labels:
    app: fix-engine
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Log_Level    info
    [INPUT]
        Name              tail
        Path              /data/fix.log
        Tag               fix
        Refresh_Interval  1
    [OUTPUT]
        Name       azure_event_hubs
        Match      fix
        Event_Hub  trading-stream
        Connection_String  ${EVENTHUB_CONNECTION}
  parsers.conf: |
    [PARSER]
        Name   fix
        Format none
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fix-engine
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fix-engine
  template:
    metadata:
      labels:
        app: fix-engine
    spec:
      containers:
        - name: fix-engine
          image: myregistry/fix-engine:latest
          env:
            - name: FIX_LOG_PATH
              value: /data
          volumeMounts:
            - name: fix-log
              mountPath: /data
        - name: fluent-bit
          image: cr.fluentbit.io/fluent/fluent-bit:2.2
          env:
            - name: EVENTHUB_CONNECTION
              valueFrom:
                secretKeyRef:
                  name: eventhub-secret
                  key: connection-string
          volumeMounts:
            - name: fix-log
              mountPath: /data
            - name: fluent-bit-config
              mountPath: /fluent-bit/etc
      volumes:
        - name: fix-log
          persistentVolumeClaim:
            claimName: fix-log-pvc
        - name: fluent-bit-config
          configMap:
            name: fluent-bit-config

This manifest demonstrates how to package the FIX engine and the Fluent Bit sidecar together. In a real setup you would also add a Service to expose the FIX session, as well as Secrets for sensitive credentials. The same pattern scales to multiple sessions by deploying additional pods, each with its own PVC.

Conclusion

Running a stateful FIX engine in Kubernetes might seem counterintuitive at first, but with the right design it offers tremendous flexibility and resilience. By persisting messages on a PVC, we guarantee that no trade is lost. Fluent Bit tails the log and streams it to Azure Event Hub, where multiple consumers can process the data independently. Kubernetes handles restarts and scheduling, while Event Hub ensures durable delivery and fan-out to any number of downstream services. Together, these tools provide a scalable platform for modern electronic trading that still honors the rigorous reliability requirements of the financial industry.

Scaling the Architecture

Once the basics are in place, scaling becomes the next concern. A single FIX session often suffices for small trading operations, but most institutions connect to multiple brokers and venues simultaneously. Kubernetes makes it easy to spin up additional pods, each running its own FIX engine instance with its own PVC. By labeling pods with the session name, you can target upgrades or restarts selectively. Horizontal Pod Autoscaling is typically unnecessary because session counts are static, but you can automate provisioning through Helm charts or custom controllers.

Event Hub also scales horizontally through partitions. If you expect very high message volume, configure the hub with multiple partitions and assign each FIX session to a partition key. Consumers then distribute work across partitions while retaining ordering within each session. When scaling to dozens of pods, keep an eye on storage IOPS to ensure PVC performance does not become a bottleneck. Many cloud providers offer premium storage tiers to meet low-latency requirements.

Monitoring and Alerting

Visibility is crucial in a trading environment. Beyond basic container metrics, you should track FIX-specific indicators such as message throughput, session logon status, and sequence number gaps. Expose metrics from the FIX engine using Prometheus-compatible endpoints. Fluent Bit can forward its own metrics to Prometheus as well, allowing you to monitor log shipping latency and buffer sizes.

Grafana dashboards can visualize these metrics for operations teams. Alertmanager rules should trigger when the engine disconnects unexpectedly, when log shipping lags behind by more than a few seconds, or when Event Hub reports throttling. Integrating logs with a SIEM solution helps meet compliance requirements and provides a unified view of trading activity across the cluster. All of these observability components fit naturally into Kubernetes via Helm charts or operators.

Security Considerations

Trading infrastructure handles sensitive financial data, so security must be top of mind. Network policies in Kubernetes restrict which pods can talk to one another. You might create a namespace dedicated to trading and lock down ingress and egress. Use secrets and encryption for all connections, including TLS termination between the FIX engine and its counterparties.

Role-based access control ensures that only authorized personnel can deploy new versions or access the persistent logs. Consider encrypting the PVC at rest if your storage backend supports it. Fluent Bit can also mask or filter sensitive fields before sending messages to Event Hub, minimizing exposure should downstream systems be compromised. Finally, review audit logs regularly to ensure that configuration changes are tracked and reversible.

Advanced Recovery Scenarios

Despite best efforts, unexpected situations still occur. Exchanges might disconnect, network partitions can isolate pods, or entire clusters might need to fail over. Because FIX sessions are sensitive to sequence numbers, the engine must reconcile state with its counterparties after any disruption. QuickFIX/n automatically handles resend requests, but only if the log files are intact. Storing them on a PVC means we can spin up the engine in a new cluster if necessary and resume exactly where we left off.

Disaster recovery plans often involve replicating PVC snapshots to a secondary region. Cloud providers typically offer snapshot features that capture block-level storage at regular intervals. In the worst case, you can restore from a snapshot and redeploy the engine with minimal downtime. Similarly, Event Hub supports Geo-DR namespaces that replicate data across regions. Consumers connect to the secondary namespace if the primary becomes unavailable, ensuring continuity of service.

Another scenario involves manual replay of messages. Suppose a downstream consumer experiences a bug and loses data. Because Event Hub stores every message for the retention period, you can provision a new consumer, seek back to the desired offset, and reprocess historical data without affecting the main trading flow. This flexibility is invaluable for debugging and audit tasks.

Summary and Next Steps

We covered a lot of ground in this guide, but the ideas extend even further. You could automate the entire deployment with Helm charts, parameterizing the broker endpoints and PVC sizes. Another improvement is to integrate a message queue between the FIX engine and downstream services, providing additional buffering and decoupling. Companies operating at global scale might run multiple Kubernetes clusters across regions with a global traffic manager, routing sessions to the nearest active cluster. With a consistent logging strategy and Event Hub replication, all of these clusters can feed the same analytics and persistence pipelines.

For teams new to Kubernetes, start small. Deploy a single FIX session with its own volume and sidecar, then monitor behavior under failover scenarios. Gradually add more sessions and consumers as confidence grows. Over time, you will develop runbooks for common issues, from network glitches to engine upgrades. By embracing cloud native principles without sacrificing state, you gain a trading platform that is both resilient and easy to evolve.