Executive summary
Before Walmart hired a formal product owner in 2021, I led the product and engineering decisions required to establish the platform. Afterward, I partnered with product and program leaders while retaining accountability for engineering strategy, execution, reliability, and operating outcomes.
The platform offered an alternative to higher-cost Databricks and DataProc processing. Internal analysis estimated Databricks at approximately twice the cost for relevant workloads and DataProc at about 10% more. The program created an estimated $12 million in annual cost avoidance.
My role and scope
- Defined platform and product direction with engineering, product, and program partners.
- Connected tenant requirements to milestones, operational requirements, and adoption plans.
- Made roadmap tradeoffs using adoption, reliability, cost, support demand, risk, and capacity.
- Influenced infrastructure, security, observability, finance, and tenant stakeholders.
- Developed engineers and technical leaders while remaining accountable for platform outcomes.
- Facilitated discovery, scoped the MVP, launched proofs of concept, and iterated from tenant feedback toward general availability.
The business problem
Walmart needed an enterprise path for ML pipelines, Medallion and Lambda architectures, CDC, and other streaming use cases without requiring every tenant to build and operate a separate platform or depend exclusively on higher-cost managed processing services. The product had to be economically compelling, reliable, supportable by a constrained organization, and straightforward to adopt.
Strategy and product decisions
I translated the business need into product boundaries, engineering milestones, tenant migrations, operating standards, shared-infrastructure dependencies, and measurable outcomes. Golden-path deployment templates and phased delivery balanced self-service flexibility with security, reliability, and operational simplicity.
Support tickets, onboarding drop-off, and tenant interviews showed teams stalling at provisioning, deployment, and compliance. I sequenced self-service provisioning, golden-path templates, managed ArgoCD, tenant-level metrics, and HIPAA/PII support around those blockers. In 2024, annual customer growth reached at least 76% and workload growth reached 52%, with more than 500 active users. Earlier onboarding improvements eliminated approximately 80 hours of weekly toil. Customer organizations, individual active users, and running workloads were distinct measures of adoption.
Verified adoption and operating outcomes
- Documented counts reached 37 production Spark tenants and 66 production Kafka Connect tenants; Kafka Connect explicitly excludes its largest tenant. These counts are not added because some organizations used more than one product.
- Tenants could provision an environment in approximately five minutes.
- Governance included golden signals, dashboards, change gates, access controls, SLA monitoring, RACI documents, key management, and chargeback.
- Operations included 24×7 on-call coverage, incident review, vulnerability remediation, and recovery planning.
Organizational and cross-functional leadership
FYE2023 was a critical year of headcount reduction: the organization contracted from approximately 14 contributors to about four engineers. I had not acted early enough on capacity, alert volume, and burnout signals. I took accountability by pausing new feature work for a quarter and resetting the roadmap around uptime, alert quality, incident reduction, and recovery.
I communicated capacity, delayed commitments, and operating risk directly to product, SRE, security, and leadership partners. I also helped hire three engineers within three months. The platform completed FYE2023 with zero financially impacting incidents; later controls required RCA for every P0/P1/P2 incident within two weeks and maintained zero vulnerabilities older than 60 days. In a later year-over-year comparison, average P1 paging declined approximately 10%, from 8.77 to 7.85 per week, and support tickets declined approximately 4%, from 642 to 616. By my final year, overall paging had fallen to an average of four pages per week.
As our reliability practices matured, monitoring change failure rate helped us maintain the 99.95% availability SLA and meet a 99.99% SLO, with observed uptime averaging 99.999%—five nines. Playbooks and workflow automation brought mean time to recovery (MTTR) for toil-related incidents below five minutes. I accepted a few additional minutes of lead time for changes to incorporate code-based quality and security monitoring into delivery; this reduced deployment rework and improved security.
Execution and operating model
ArgoCD and GitOps
I owned planning, scope, and cross-functional delivery for the critical path from MVP to general availability. My technical lead and I co-owned the initial ArgoCD strategy; after he left, I carried the program through completion, including governance reviews across Security, Kubernetes, and Dev Tools.
Compliance policies for disaster recovery (DR), job health monitoring, and availability required a consistent way to define, monitor, and restore workloads across clusters. We used GitOps as the source of truth (SoT), reconciliation to align running workloads with declared state, and cluster abstraction to provide a consistent tenant deployment path.
The team launched Spark Streaming by the FYE2020 deadline. Managed ArgoCD later reached 100% adoption. First-time deployments took 15 minutes or less, the platform supported hundreds of deployments per hour, and it monitored thousands of updates a day. Recovery objectives were below 30 minutes RTO and five minutes RPO. The pattern expanded to Postgres, AMQ, and Observability, and Dev Tools reported hundreds of tenants beyond Data Platform.
The shared deployment path increased platform users’ trust and enabled more frequent deployments with low change-failure risk. Adoption grew from zero to more than 3,500 streams in just over five years. Users’ confidence in making frequent, safe changes helped support that growth.
Because this was an early platform decision, its individual impact on deployment frequency and adoption was difficult to isolate. Monitored updates describe platform activity, rather than distinct production releases; recovery objectives are targets. The adoption total describes streaming workloads, including Spark jobs and Kafka Connect deployments.
Trustworthy chargeback
I identified a defect that billed frequently restarted executors as though they ran for a full hour. I worked with product to define a prorated model and influenced the third-party chargeback team to prioritize it. After delivery in FYE2026, one major pipeline team resumed profiling and reduced its node-pool cost by approximately 30%, roughly $5 million in pipeline-specific savings. This was separate from both bin-packing savings and migration-related cost avoidance.
Measured results
I personally analyzed Prometheus and capacity trends, identified inefficient Kubernetes placement, and worked with my Technical Lead to prioritize and deliver bin-packing without impacting tenant workload SLAs.
From platform signals to decisions
Support tickets, onboarding drop-off, and tenant interviews helped identify where teams stalled. The observability examples trace those signals through delivery and reliability decisions and investments in self-service, connector economics, and trusted chargeback, with the resulting adoption, operational, and cost outcomes.
Leadership lessons
Roadmap delivery cannot come at the expense of team health and platform trust. I now treat staffing coverage, alert load, operational toil, and burnout signals as product and capacity-planning inputs. Success is measured through adoption, tenant outcomes, operating health, and economics—not feature output alone.
Continue the conversation
Review my LinkedIn profile or contact me at [email protected].