Kafka Connect: Zero-to-Scale Product Growth and Open-Source Economics

Executive summary

Before a formal product owner joined in 2021, I supplied the product and engineering leadership needed to establish the platform. Afterward, I remained the accountable engineering leader and product partner. When licensed connector costs threatened continued growth, I personally authored an executive alternatives analysis and turned it into a prioritized open-source migration program.

My role and scope

• Defined how an inherited client capability would become a supported platform product.
• Managed the team that built the platform and tenant base from zero.
• Defined priorities with tenants, product, program, and technical leaders.
• Quantified adoption, connector use, vendor exposure, and migration scope.
• Owned the vendor engagement and negotiations, with Legal retaining final contract approval.
• Directed inventory, feature-parity, and rollout strategy with the development team.

The business problem

Walmart needed a standardized way to deploy and operate Kafka Connect integrations at enterprise scale, removing operational work from tenants while controlling infrastructure and licensing costs. Leaving Confluent required a verified inventory, feature parity, a credible build or vendor option, migration support, capacity, executive sponsorship, and a phased risk plan.

Strategy and product decisions

POC-to-platform productization

The team added observability, inventory tracking, centralized logging, self-hosted deployments, a client-as-a-service experience, access controls, audit logging, latency tracking, and guidance for avoiding portfolio-overlap anti-patterns. The first tenant arrived in January of FYE2021.

Build-versus-buy as a portfolio decision

I helped scope an approximately $500,000 one-time engagement with Lenses to produce 10 open-source connectors. For a Kafka-to-Kafka use case, I worked with a major tenant, engineers, and product stakeholders on an internally built connector that progressed into production.

Organizational and cross-functional leadership

I used inventory, Prometheus data, and custom JavaScript analysis to establish actual connector use, parity scope, affected tenants, required investment, displaced roadmap work, and expected annual benefit. This reframed a technical preference as a product and investment decision executives, users, and engineers could evaluate together.

I aligned leadership, product, users, and engineers on return, priority, feature parity, inventory, rollout, and migration risk, and maintained that alignment through full migration.

Execution and operating model

• Lenses produced all 10 connectors.
• Migration was approximately 90% complete at the FYE25 evaluation point.
• Migration reached 100% in December 2024, approximately six months before I left Walmart.
• Optional operational support would have required a much smaller recurring annual fee.

Measured results

• Production deployments increased from 823 to 1,932, or 135%, in one measured year.
• The established production tenant base increased from 47 to 66, or 40%, in that year.
• The platform supported 12,106 production pods at the FYE25 measurement point.
• A major internal tenant accounted for approximately 55% of the platform at that time while the remaining portfolio also grew.
• The completed open-source program produced close to $3 million in net annual licensing reduction after external investment and operating costs.

Leadership lessons

Platform growth depends on product economics as much as technical capability. The durable decision was not simply to replace licensed connectors, but to build the evidence, product boundaries, executive support, vendor leverage, and migration path needed to change the economics without disrupting tenants.

Related projects