Kafka versus Azure EventHub: Evidence-Based Platform Strategy

Executive summary

I analyzed approximately a year of Kafka utilization and modeled comparable EventHub volume using published pricing. The evidence showed that optimizing the existing Kafka platform was a stronger economic path than replacing it.

My role and scope

I performed the analysis, created the cost model, and personally authored the executive summary describing cost, scalability, and operating tradeoffs. The recommendation went to the vice president responsible for the final decision.

The business problem

Leadership was considering Azure EventHub as a potential replacement for Kafka. The decision needed to determine whether EventHub’s scaling model justified replacement after accounting for Walmart’s real demand pattern and the disk, compute, and network cost of highly utilized Kafka clusters.

Strategy and product decisions

I scanned every Kafka cluster with Prometheus and calculated minimum, average, and maximum utilization. I modeled comparable volume using EventHub’s published pricing, then presented a decision grounded in observed utilization and economics.

Current Microsoft documentation confirms pricing based on capacity and usage dimensions but does not establish the exact 2019 rates used internally. The public case therefore preserves only the method and relative result. See Azure Event Hubs pricing and Event Hubs quotas and limits.

Organizational and cross-functional leadership

The work demonstrates managing up through evidence: challenging a preferred direction, making actual usage visible, and giving an executive a clear set of tradeoffs rather than arguing from technology preference.

Measured result

The vice president used the executive summary to retain Kafka and invest in optimizing the existing environment, beginning with disk. The internal summary is no longer available, so this page does not claim a savings total or reconstruct historical rates from current pricing.

This 2019 decision is separate from the later Kafka Connect transformation, Lenses engagement, and close to $3 million annual connector-licensing reduction.

Leadership lessons

Strategic platform decisions should be tested against observed demand, the full cost model, and the organization’s actual optimization options. A concise executive narrative is most useful when leaders can see the assumptions, relative economics, and limits of the evidence.

Related projects