Enterprise Streaming Platform: Zero-to-Scale Growth, Reliability, and Economics

Executive summary

Before Walmart hired a formal product owner in 2021, I led the product and engineering decisions required to establish the platform. Afterward, I partnered with product and program leaders while retaining accountability for engineering strategy, execution, reliability, and operating outcomes.

The platform offered an alternative to higher-cost Databricks and DataProc processing. Internal analysis estimated Databricks at approximately twice the cost for relevant workloads and DataProc at about 10% more. The program created an estimated $12 million in annual cost avoidance.

My role and scope

• Defined platform and product direction with engineering, product, and program partners.
• Connected tenant requirements to milestones, operational requirements, and adoption plans.
• Made roadmap tradeoffs using adoption, reliability, cost, support demand, risk, and capacity.
• Influenced infrastructure, security, observability, finance, and tenant stakeholders.
• Developed engineers and technical leaders while remaining accountable for platform outcomes.
• Facilitated discovery, scoped the MVP, launched proofs of concept, and iterated from tenant feedback toward general availability.

The business problem

Walmart needed an enterprise path for ML pipelines, Medallion and Lambda architectures, CDC, and other streaming use cases without requiring every tenant to build and operate a separate platform or depend exclusively on higher-cost managed processing services. The product had to be economically compelling, reliable, supportable by a constrained organization, and straightforward to adopt.

Strategy and product decisions

I translated the business need into product boundaries, engineering milestones, tenant migrations, operating standards, shared-infrastructure dependencies, and measurable outcomes. Golden-path deployment templates and phased delivery balanced self-service flexibility with security, reliability, and operational simplicity.

Verified adoption and operating outcomes

• Documented counts reached 37 production Spark tenants and 66 production Kafka Connect tenants; Kafka Connect explicitly excludes its largest tenant. These counts are not added because some organizations used more than one product.
• Tenants could provision an environment in approximately five minutes.
• Governance included golden signals, dashboards, change gates, access controls, SLA monitoring, RACI documents, key management, and chargeback.
• Operations included 24×7 on-call coverage, incident review, vulnerability remediation, and recovery planning.

Organizational and cross-functional leadership

During severe headcount reduction, the organization contracted from approximately 14 contributors to about four engineers. I had not acted early enough on capacity, alert volume, and burnout signals. I took accountability by pausing new feature work for a quarter and resetting the roadmap around uptime, alert quality, incident reduction, and recovery.

I communicated capacity, delayed commitments, and operating risk directly to product, SRE, security, and leadership partners. I also helped hire three engineers within three months. The platform completed FYE2023 with zero financially impacting incidents; later controls required RCA for every P0/P1/P2 incident within two weeks and maintained zero vulnerabilities older than 60 days. In a later year-over-year comparison, average P1 paging declined approximately 10%, from 8.77 to 7.85 per week, and support tickets declined approximately 4%, from 642 to 616.

Execution and operating model

ArgoCD and GitOps

I owned planning, scope, and cross-functional delivery for the critical path from MVP to general availability. My technical lead and I co-owned the initial ArgoCD strategy; after he left, I carried the program through completion, including governance reviews across Security, Kubernetes, and Dev Tools.

The team launched Spark Streaming by the FYE2020 deadline. Managed ArgoCD later reached 100% adoption. The pattern expanded to Postgres, AMQ, and Observability, and Dev Tools reported hundreds of tenants beyond Data Platform.

Trustworthy chargeback

I identified a defect that billed frequently restarted executors as though they ran for a full hour. I worked with product to define a prorated model and influenced the third-party chargeback team to prioritize it. After delivery in FYE2026, one major pipeline team resumed profiling and reduced its node-pool cost by approximately 30%, roughly $5 million in pipeline-specific savings. This was separate from both bin-packing savings and migration-related cost avoidance.

Measured results

I personally analyzed Prometheus and capacity trends, identified inefficient Kubernetes placement, and worked with my Technical Lead to prioritize and deliver bin-packing without impacting tenant workload SLAs.

Leadership lessons

Roadmap delivery cannot come at the expense of team health and platform trust. I now treat staffing coverage, alert load, operational toil, and burnout signals as product and capacity-planning inputs. Success is measured through adoption, tenant outcomes, operating health, and economics—not feature output alone.

Related projects