Organizational change is about shifting mindsets and enabling people to operate at a higher level than before.
It’s successful when teams don’t just adopt change, they own it and feel like they are part of driving the transformation, not just complying with it.
$130M+ in delivered cost-savings
3 Large Scale 0-1 Organizational Deployments
4 New Key Performance Metrics Developed to Production
30+ Analysts and Program Managers mentored to enhance technical acumen and statistical literacy
Worked cross functionally with over 13 teams including Operations, Engineering, Data Science, Business Intelligence, Software Development, Finance, and others.
Regularly report business reviews to VP and Executive Level Leadership
The Out of Buffer Entitlement Review (OOBER) is a centralized dashboard that unifies 13 cross functional teams across Amazon’s NACF (North America Customer Fulfillment) network, over 180 fulfillment centers, to collaborate on out-of-buffer performance and align on a single source of truth for operational metrics.
The OOBER dashboard aims to provide a comprehensive and reliable platform for tracking key out-of-buffer indicators, identifying root causes, and driving continuous improvement across the organization.
In current state, there is no large-scale mechanism for granular idle time investigation. Currently, iGraphs are currently the primary source of truth for WIP health and root causing lack of work in front of operations. These investigations take a manual user between 15 and 45 minutes based on the complexity of the scenario. ARC typically handles approximately 80 investigations per week with a total initial investigation time of 60 labor hours per week and many of these requests are not single-investigation resolutions. There is open conversation between SME’s and Operations in these investigation SIM’s and can require upwards of four investigations to occur throughout a week dependent on the information provided by the FC.
On an operations level, there is no tool that allows operators to deep dive the root cause of the idle time that faces their inductors and packers which prohibits innovation and degrades our ability to invent a scalable solution.
The OOBER dashboard will provide teams across the organization with a powerful tool to drive improvements in their respective areas of ownership. By centralizing key out-of-buffer metrics and offering detailed data insights, the dashboard empowers teams to take a data-driven approach to problem-solving and process optimization. Here's how teams can leverage the OOBER dashboard:
Identify Out-of-Buffer Root Causes: The dashboard offers a comprehensive view of metrics such as Buffer Limits, Low Backlog, Sort Imbalance, and Pick/Pack/Induct Rate Accuracy, allowing teams to pinpoint the primary contributors to out-of-buffer events.
Measure the Impact of Process Changes: With the 5-minute interval data available in the dashboard, teams can closely monitor the effects of any process improvements or software updates they implement, enabling them to quickly assess the impact of their initiatives on out-of-buffer performance.
Foster Collaboration and Transparency: The universal access and reliability of the OOBER dashboard encourage cross-functional collaboration among teams, as they can work together to develop holistic solutions and align on the same set of out-of-buffer performance metrics.
Empower Data-Driven Decision-Making: The OOBER dashboard provides teams with a centralized and trusted source of data, enabling them to make informed decisions based on facts rather than assumptions, and prioritize their improvement efforts effectively.
Identify and Replicate Best Practices: By analyzing the data in the OOBER dashboard, teams can identify high-performing sites, processes, or practices that can be replicated across the organization, accelerating the pace of improvement and fostering a culture of continuous optimization.
By leveraging the OOBER dashboard, teams can gain a deeper understanding of their out-of-buffer performance, make data-driven decisions, and collaborate more effectively to drive meaningful improvements in their owned areas, ultimately enhancing the overall efficiency and effectiveness of the NACF network.
OOBER is intended to be the primary reporting mechanism used to track and root cause periods of tote starvation in the Outbound ARS network. Opposed to Autoflow buffer targets, OOBER focuses on PickingPickedAtDestination work to measure idle time that occurs explicitly when there are low totes post-divert.
The OOBER dashboard consolidates data from multiple sources into a single, accessible platform, ensuring that all stakeholders are working from the same set of information. Moreover, the emphasis on 5-minute interval data granularity allows for more in-depth analysis and a deeper understanding of the dynamic nature of the Autoflow network's out-of-buffer performance.
The key benefits of the OOBER dashboard are:
Universality: The dashboard is available to all relevant teams, ensuring that everyone has access to the same data and metrics.
Reliability: The data is sourced and maintained by the Central Flow and ARC Business Intelligence teams, providing a trusted and authoritative source of information.
Transparency: The dashboard empowers operations managers to identify and address out-of-buffer issues, encouraging self-serve deep-dive solutions through the available data.
The OOBER dashboard is designed to serve a diverse audience across Amazon’s NACF network, including:
Operations Teams: From site-level managers to regional and national leaders, the dashboard equips operations teams with the data and insights they need to identify out-of-buffer root causes, optimize processes, and drive operational excellence.
Central Flow (OB) and ACES: These teams, responsible for overseeing the Autoflow network and process improvement initiatives, will utilize the OOBER dashboard as a single source of truth for out-of-buffer performance monitoring and benchmarking.
Functional Teams: Groups like AFT (Automation & Flow Technology), PE (Process Engineering), MSP (Mechatronics & Sustainability), and LT (Lean Transformation) will leverage the dashboard's data to inform their strategic decisions, R&D efforts, and process optimization initiatives.
Launch Teams: The AROD (Amazon Robotics Operations Deployment) team, responsible for new network deployments, will use the OOBER dashboard to establish baseline out-of-buffer performance metrics and track the progress of their launch efforts.
Support Functions: Teams such as PA (Pack Automation), ET (Emerging Technologies), and iDEA will utilize the dashboard to align on performance objectives, identify opportunities for innovation, and collaborate with operational counterparts.
a. INITIAL FINDINGS
OOBER has enabled the identification of disparity in several key idle time indicators. Below are the initial findings and contributors to increased idle (pack/induct) time in the network.
1. Process Path and Totes at Destination – past the MRS Divert – explain 88.84% (r-sq) instances of Pack and Induct Rate Degradation.
2. The ‘Low Backlog’ root cause is the primary driver of rate degradation while Idle with a mean rate of 416 UPH vs our maximum of 1048 UPH accounting for a 61% degradation in rates while processing under our 2-hour backlog limit.
3. The Standard Deviation of Pack/Induct Rates during Low Backlog periods is equal to the highest standard deviation root causes [Induct Overstaffing, and Induct Rate Accuracy] at 320.32 UPH, 4.8% from our highest observed STDEV, which emphasizes instability in processing during times of low backlog.
4. Idle Time occurs most aggressively in SingleSmall (SmartPac) and Multis (AFE) paths with an average of 12.5% and 3%, respectively.
5. Gen 11 is the largest contributor to idle time at 6.23% compared to the next highest at 5.04%
6. AFE Generation has the largest direct correlation to Idle Time at 3.41% for 2.5, and 3.07% at AFE 3.5’s.
b. STATISTICAL ANALYSIS
The statistical analysis conducted to find the deficiencies in Initial Findings includes ANOVA, Plot Screening, STDEV tests, and Multiple Regressions (Linear & Quadratic) to determine key subsets and integral inputs into Idle Time, and the observed result of processing while in ‘Low Backlog’ (Operations Driven Idle).
The number of Totes at Destination is the quantified work on a destination lane post-MRS-Divert.
Deviation by Process
PPAFE | 938 UPH | 184 SD | 19.6% of Rate
PPMix | 65 UPH | 13 SD | 20% of Rate
PPSingleSmall | 178 UPH | 38 SD | 21.3% of Rate
PPSmartPac | 445 UPH | 115 SD | 25.8% of Rate
Deviation by Root Cause
Low Backlog | 416 UPH | 320 SD | 76.9% of Rate
Overstaffing | 857 UPH | 320 SD | 37.3% of Rate
UnderPicking | 992 UPH | 192 SD | 19.3% of Rate
Rate Accuracy | 756 UPH | 252 SD | 33.3% of Rate
c. FINANCIAL ASSUMPTIONS & OVERALL BENEFITS
A: Financial
* *based on 20 operating hours per day, 7 days per week
MULTIS Self-Inflicted Idle Time (Low Backlog): 0.4%, 4.8 Minutes per Day per Inductor per FC
4.8 Minutes*10000 Avg AFE UPH*365 Days = 17,520,000 Units annualized capacity loss
1,752,000 units at an avg upgrade cost of $4.4/ea equates to $77.526 M Annualized
SINGLES Self-Inflicted Idle Time (Low Backlog): 3.41%, 40.92 Minutes per Day per Packer per FC
40.92 Minutes * 3000 Avg UPH (SP/Mix/SS)*365 Days = 44,807,400 Units annualized capacity loss
44,807,400 Units at an avg upgrade cost of $4.4/ea equates to $197 M Annualized
B: Non-Financial
Utilizing a FieldSense Survey feedback was gathered from 26 unique users to determine the efficacy and intuitiveness of the OOBER Dashboard. The summarized results are below.
Intuitiveness: 4 / 5 (80%)
Does This Fill a Business Gap: Yes (96%)
Do you have Confidence in the Data: 100%
L7 Intuitiveness Rating: 5 / 5
L4/5/6 Intuitiveness Rating: 4 / 5
Audience Net Promoter Score: 9.1 (91.2%)
· OOBER is expected to deploy on 10/1/2024 for a Q4 Launch
o This provides ample time for introduction, review, and familiarization prior to peak
Table 1: Deployment Timelines for Singles
Project Start Date:
08-01-2024
Initial Data Gathering
Testing
04-23 to 07-23
Proof of Concept (v1) – 8-12-2024
1st Revisions, CF Feedback (v2) – 8-22-2024
2nd Revisions, Closed Beta (v3) – 9-01-2024
3rd Revisions, Open Beta, Survey (v4) – 9-22-2024
Project Completion Date:
10-01-2024
OOBER slated to open for public consumption
Benefit Realization Date:
11-01-2024
Performance Improvement will be ready to test at 1 month of reporting and solution institution
There is currently no mechanism for large scale, granular deep-dives of idle time on the PickingPickedAtDestination due to a lack of physical work in front of associates. This dashboard drives the visibility and reporting of these high-cost operations-driven defect.
Risks: Time Under Totes and Time Out of Work
· Callout: Singles paths do not have a source of truth for no-tote idle time.
· Issue: There is a lack of granular and precise idle time data for Singles process paths. AFE’s have rectified this by using Fast Induct App signals for tote scans in front of an Inductor.
· What this causes: Less accurate idle time metric for Singles Paths
· How long has this problem existed: Forever
· How to manage it in current state: Both Time Out of Work and Time Under Totes are able to be selected allowing sites to deep dive each metric and look at their idle through multiple lenses.
Blockers: FC Adoption of the tool is paramount to success in understanding the cause of idle time and rectifying deficiencies observed.
Risk of Delayed Approval: There is currently no mechanism for measuring idle time on the destination lane. This is essential to begin reducing operations driven idle time prior to peak 2024.
Q1: How are future developments of Out of Buffer Reporting being communicated to stakeholders?
A1: ARC distributes a variety of Quicksight Reports. It will be essential for customers to create rules in their Inbox to filter these imperative reports out for viewing. Additionally, the Central Flow Risk team will be utilizing the Backlog Adherence Scorecard to accompany the Weekly Proactive Risk Report that is distributed to each region.
Q2: What are other influences to Idle instances?
A2: There are a variety of metrics that are direct inputs to Idle Time and are largely able to be root caused to planning discrepancies in Rate and/or Headcount.
Q3: What is the estimated annual cost that Amazon absorbs each year due to input driven idle?
A3: $274 Million
Report: Impact of PPW Breaches on DPMO in Sorted Operations
Based on these findings, we recommend implementing stricter PPW threshold monitoring and rapid staffing adjustments when approaching breach conditions to maintain quality standards.
Key Metrics Definitions
PPW (Packables Per Wall): Number of items that need to be sorted divided by the number of workers available
PPW Breach: Instances where PPW exceeds the established threshold for a facility (typically >80 for most facilities)
DPMO (Defects Per Million Opportunities): Number of defects per million sorting opportunities, calculated as (Defects/Opportunities) × 1,000,000
Executive Summary
This analysis examines the relationship between high PPW (Packables Per Worker) breaches and DPMO (Defects Per Million Opportunities) in Amazon's Sorted operations. After analyzing the data from two key datasets, we found:
Strong Positive Correlation: There is a statistically significant correlation between high PPW breaches and increased DPMO rates.
Quantifiable Impact: Facilities experiencing high PPW breaches (>80 packables per worker) show a 28% higher average DPMO compared to facilities operating within optimal PPW ranges.
Temporal Relationship: DPMO increases typically occur within 30-60 minutes following sustained high PPW breaches.
Regional Variations: The effect is most pronounced in the NA-East and EU regions, with NA-West showing more resilience to high PPW conditions.
Key Findings
1. Correlation Analysis
We analyzed the relationship between PPW values and DPMO rates across all facilities:
PPW Range
Avg DPMO
Sample Size
% Change from Baseline
<50 (Optimal)
2,345
1,823
Baseline
50-65 (Normal)
2,487
3,742
+6.1%
65-80 (High)
2,789
2,156
+18.9%
>80 (Breach)
3,002
1,478
+28.0%
>100 (Severe Breach)
3,598
612
+53.4%
Statistical Significance: Pearson correlation coefficient between PPW and DPMO: 0.73 (p-value < 0.001)
2. Time-Series Analysis
When analyzing how DPMO changes after PPW breaches:
Immediate Effect: 8% average increase in DPMO within 0-30 minutes of breach
Peak Effect: 28% average increase in DPMO within 30-60 minutes of breach
Recovery Period: DPMO typically returns to within 5% of baseline 2-3 hours after PPW returns to normal range
Conclusion
· The data clearly demonstrates a strong correlation between high PPW breaches and increased DPMO in Sorted operations. The relationship appears to be causal, with PPW breaches preceding DPMO increases. While regional variations exist, the pattern holds across all analyzed facilities.
· Managing PPW effectively through proper staffing and operational adjustments represents a significant opportunity to improve quality metrics and reduce defects in Sorted operations.
Report: CW1000 Capacity Optimization
Executive Summary
CW1000 - an AutoPack path for Medium (Mix) Units - launched with numerous mechanical issues that resulted in unreliable Machine Up Time. This resulted in inconsistent WIP in transit and at destination. As the machine would go down WIP would rise as we were still picking for two machines. As a machine came back up WIP would drain as we began picking for 1 machine.
This yo-yo effect created cause for creative solutions to the varying performance in the path and thus the Tote x Physical Constraint model was created. This utilized live UnitsPerTote to adjust the buffer range dynamically. Physical Constraint was based off the actual line length from MRS Divert to the Pack Station(s) which told Autoflow 'I always need between X and Y totes regardless of Headcount' which would protect MRS Recirculation by setting a top limit that would fit within the parameters of the conveyance.
Problem Statement
CW1000 Machines are under-processing to benchmark rate by 26% at AUS2 (368 v 500), 34% at AUS3 (330 v 500), and 18% in the ARS Network (411 v 500). ARC and MSP will be investigating the physical flow of the process path at AUS2 and AUS3 in order to drive network wide standardization improvements for the path. Additionally, other process paths will be reviewed for buffer health, and outlier cases such as high machine down time, or recurring jams. Several actions are planned to increase the pickable backlog in the CW1000 process path which shows a 66% correlation to outbound pack rates. By improving DME configurations we expect to increase pickable backlog by 0.3 hours, and thus allow pack rates to perform at higher levels without increasing DPMO/DEA risk. Buffer Configurations will test static levels for CW1000 which allows Autoflow to aim for static ranges rather than a moving target.
Success
A pilot was conducted utilizing Consuming Headcount x Unit configurations, our standard setup, and found numerous benefits. In a 6 day post change analysis CW1000 Pick to Slam Cycle Times improved by 14.74 minutes, Time in Buffer increased by 20%, Pack Rate improved by 31 UPH, MRS Recirculation improved by 2.6%, and Time OOW improved by 0.63%. These changes are tied to more accurate, and less volatile Autoflow Target Unit Rates produced by stable buffer ranges following the change of Tote to Unit configurations. Additionally, DME configurations, Max Capacity % and Machine Rate Configurations were increased to provide a more plan-able backlog for the process path in an effort to increase Hours of Work in backlog however no observable changes were recorded during the trial. These successes were launched network wide on 9/12/2024 and translated to $11.7 million in annualized savings in the NA ARS Network.
Report: Optimized Buffer Automation - NA AR Sortable
Setting buffer ranges directly impacts the stability of outbound (OB) operations. Due to performance variations over time, monitoring and adjusting buffers is crucial for consistent performance. Maintaining healthy buffers is a maintenance of performance, it is not a step increase of performance itself.
Historically, buffer ranges were established via site-specific analysis using nonstandard key performance indicators (KPIs) through a trouble ticket process. In 2022, ARC introduced Buffer Analysis Engine (BAE) to set buffers network-wide, using standardized KPIs and a 30-day update cycle based on recirculation, idle time, and minutes of work (MoW) metrics.
Currently, the buffer configuration approach (BAEv2) is fixed on using MoW to prevent idle time and MRS recirculation. BAEv2 has no upper limit, leading to buffer range increases if MRS recirculation is not maintained below 20%, negatively impacting OB cycle time. To solve this, cycle time was introduced as an indicator for a balanced buffer setup, optimizing speed, idle time, and rates.
BAEv2 tends to prioritize higher ranges due to its algorithm utilizing MRS recirculation, causing longer item dwell times and cycle times. Q1-2 2023 tests by a pizza team (ACES/ARC/AFT) at BOI2, DFW7 and LIT1 compared cycle-time optimized buffer ranges vs. MRS recirculation showing a 12.51% speed decrease in pick-to-MRS divert and 9.78% in MRS divert to induct, leading to a 43-minute (15.95%) reduction in FT SLA[1]. Using initial test findings, a series of pilots[2] were conducted to integrate cycle time into the buffer range setting calculations leading to a final pilot to confirm the baseline algorithm for the buffer automation[3] workstream and near-term scalability. The pilot was designed for an intervention analysis.
During the final 14-day pilot, from June 20, 2023, to July 3, 2023, sites attained a 9% improvement in cycle times and a 0.4% increase in rates. These improvements translate to an annualized benefit of $10.9MM for cycle time improvements through driving reduced FT SLA and ultimately increased glance views. Additionally, OB productivity improvement through rate enhancements can capture $12.0MM.
Network deployment of BAEv3 – Speed Optimized Buffers effective 8/16/2023
Reversion Criteria:
· Breaching Max Buffer increases cycle time (CT) and recirculation
· Breaching Min Buffer decreases rates and increases Out of Work Instances (OOW)
· A sustained 10% decrease in rates for 3 hours, or 5% over 24 hours
· A sustained 10% increase in CT for 3 hours or 5% over 23 hours
· A sustained 5% increase in time OOW for 8 hours
· A sustained 3% increase in recirc for 8 hours
The pilot graduation criteria included:
· Improved OB cycle time
· MRS Recirc maintained below (20%)
· Do no harm to OOW
· Do no harm OB throughput
Project Details
To align on a singular buffer model for NACF AR buffer automation and near-term scalability, two models were tested: ARC’s Buffer Analysis Engine v3 (BAEv3) and AFT Data Science Model (DSM):
BAEv3 - utilizes existing BAEv2 framework with the following logic changes;
· Compares OOW vs recirculation and cycle time to recommend increase or decrease
· Checks recommended ranges against guardrails[4] to prevent over-increasing or decreasing as seen in current production.
o avg((min_mow*minutecapacity)(1+p70_pct_ppad)) as min_guardrail
o avg((max_mow*minutecapacity)(1+p70_pct_ppad)) as max_guardrail
· These changes allow buffers to be hedged in accurate directions without egregious changes at a site level
DSM - Aligns buffer ranges towards minimum target units at destination by process path
· Minimum buffer is the sum of three buffers: In-picking, in-transit, and at-destination.
· Minimum buffer formula: (UPT-1) * rate/(pick_rate * 2 * pick_share) + ct25 * rate/60 + Minimum units at destination
· DSM model performs data retrieval, cleaning, and transformation to obtain in-picking and in-transit buffers.
· Minimum units at destination buffer is the at-destination buffer size above which one cannot observe a significant improvement of associate throughput.
· Process to determine minimum units at destination:
o Pull historical data (6 months horizon) on pack/induct throughput, at-destination buffer size, and detected headcount.
o Remove outlier observations from data and calculate at-destination buffer and throughput per associate.
o Bin at-destination units such that a statistical comparison between observed throughput per bin is possible (a bin must have at least 20 observations to prevent the kurtosis test from failing).
o Conduct a statistical comparison of throughput between adjacent at-destination bins.
§ Find the largest at-destination bin for which the comparison yields a significant, positive difference between throughputs.
§ Take this bin as an initial solution for the minimum buffer size.
§ Check n immediately larger bins if throughput can be improved more by a pre-selected threshold. (This is an additional heuristic to ensure robustness of the solution.)
§ If throughput under initial solution cannot be improved, return initial solution. Otherwise, return the solution of the added heuristic.
· Current maximum buffer limit formula: minimum * 130% (multiplication by 130% was recommended by ACES)
· Future maximum buffer limit formula: (UPT-1) * rate/(pick_rate * 2 * pick_share) + ct25 * rate/60 + upt * Max data science tote target
a. PILOT
Models were tested at the following sites: AGS1, OKC1, PSP1, YHM1, ABQ1, GRR1, ORF3, STL8. Sites selection was determined using a data science cluster analysis that compared current site performance, principal component analysis and generation type and additionally were not incorporated in another flow pilot. Each site spent separate weeks on BAEv3 vs DSM to gain variability, intentional intervention change, through the pilot. While decision framework incorporated primary and secondary effects by fractional volume of the process path by site. Initial findings displayed BAEv3 had a more positive impact on CT average (+900bps), while DSM had a more positive impact on rates (+8bps) vs BAEv3 (+4bps).
In reviewing additional KPIs, chuting rate performance on the DSM model displayed a 7bps improvement towards rate while induct remained flat[5]. By performing autoregressive integrated moving average (ARIMA[6]) analysis it was determined that despite DSM achieving better CT99 improvements it preformed 230bps worse in CT99.75 vs BAEv3. BAEv3 resulted in predominantly flat rate improvement across both chuting and induct[7]. Median Time in Buffer (TIB) improved for both DSM (+37bps) and BAEv3 (+34.5bps) vs control. Distribution of TIB under BAE remained flatter than under DSM, however TIB improvements between models is not statistically significant.
b. STATISTICAL ANALYSIS
Statistical Significance of BAEv3 average cycle time[8]:
During the post-intervention period, the response variable had an average value of approx. 61.47. By contrast, in the absence of an intervention, we would have expected an average response of 65.46. The 95% interval of this counterfactual prediction is [63.47, 67.32]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is -3.99 with a 95% interval of [-5.85, -2.00]. For a discussion of the significance of this effect, see below:
· Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 491.77. By contrast, had the intervention not taken place, we would have expected a sum of 523.67. The 95% interval of this prediction is [507.75, 538.57].
· The above results are given in terms of absolute numbers. In relative terms, the response variable showed a decrease of -6%. The 95% interval of this percentage is [-9%, -3%].
· This means that the negative (reduction) effect observed during the intervention period is statistically significant. If the experimenter had expected a positive effect, it is recommended to double-check whether anomalies in the control variables may have caused an overly optimistic expectation of what should have happened in the response variable in the absence of the intervention.
The probability of obtaining this effect by chance is very small (Bayesian one-sided tail-area probability p = 0). This means the causal effect can be considered statistically significant.
Statistical Significance of BAEv3 average outbound rate[9]:
During the post-intervention period, the response variable had an average value of approx. 530.49. In the absence of an intervention, we would have expected an average response of 531.88. The 95% interval of this counterfactual prediction is [524.94, 538.82]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is -1.39 with a 95% interval of [-8.32, 5.56]. For a discussion of the significance of this effect, see below:
· Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 4.24K. Had the intervention not taken place, we would have expected a sum of 4.26K. The 95% interval of this prediction is [4.20K, 4.31K].
· The above results are given in terms of absolute numbers. In relative terms, the response variable showed a decrease of -0%. The 95% interval of this percentage is [-2%, +1%].
· This means that, although it may look as though the intervention has exerted a negative effect on the response variable when considering the intervention period as a whole, this effect is not statistically significant, and so cannot be meaningfully interpreted. The apparent effect could be the result of random fluctuations that are unrelated to the intervention. This is often the case when the intervention period is very long and includes much of the time when the effect has already worn off. It can also be the case when the intervention period is too short to distinguish the signal from the noise. Finally, failing to find a significant effect can happen when there are not enough control variables or when these variables do not correlate well with the response variable during the learning period.
The probability of obtaining this effect by chance is p = 0.349. This means the effect may be spurious and would generally not be considered statistically significant.
c. ROLLOUT EVALUATION
As a result of the marginal difference between both models, deployment timeline must also be considered. The BAEv3 model can quickly deploy due to its inherent design. Whereas, integrating DSM through buffer automation requires an extended deployment period. Our proposal is for the AR network to transition to BAEv3 by 8/16, capitalizing on observed pilot benefits. Existing BAE upgrade cadence of 30-day performance period and accompanying updates will remain. Simultaneously, AFT is enhancing automated buffers within flow tools following the North Star vision:
· Short Term (<6mo)
ARC team will continue to use the existing dashboard (BAE) to monitor buffer performance as they are today. ARC team will transition to the DSM QuickSight dashboard to copy the automated buffer values and enter them manually in Outbound Flow Tracker. (ECD: Q3 2023)
· Medium Term (<18mo)
Step 1 will consist of adding an "Updated Buffers" tab with buffer information will be added to the Outbound Flow Tracker or AutoFlow (TBD), removing the need for a QuickSight dashboard. Then incorporate a button, "Save Buffer Size Limits," where the newest buffer settings will be applied, removing the need to manually enter in each buffer per process path per FC.
· Long Term (<36mo)
System automatically updates buffers for all FCs while also allowing users to manually override if necessary.
A: Financial
$10.9MM annualized savings towards OB cycle time
$12MM annualized savings through rate improvement
B: Non-Financial
TBD.
· NA deployment target is August 14th 2023
Table 1: Deployment Timelines
Project Start Date:
03-2023
Initial hypothesis testing (LIT1)
Testing
04 through 07-23
BOI2 Kaizen (Phase 1) – 4/1-4/15 2023
Phase 2 Buffer Pilot – 5/15-6/10 2023
Phase 3 Buffer Pilot – 6/20-7/3 2023
(Additional testing at BAPv2 sites: 05/01-07/16)
Project Completion Date:
8-16-2023
NA Deploys Wednesday, August 16th 2023
Benefit Realization Date:
9-16-2023
Financial impact / Savings beginning 1 month from launch
4. DISPOSAL/TRANSFER OF EXISTING ASSETS
N/A
1. Vendor Labor? ☐Yes ☒No
2. Internal Labor? ☒Yes ☐No
3. If Internal Amazon Labor is being considered/required to execute this change, please indicate from which team (s) (RME, Ops Engineering, ACES, Inbound Associates, etc.): ACES and AFT
4. Total Labor Hours: N/A
5. If Vendor Labor is being considered, which team is responsible for overseeing the vendor while in the Amazon building? N/A
6. Is downtime required for work? ☐Yes ☒No
7. If downtime is required, please provide the total hours: N/A
8. If global project, please indicate who will be responsible for executing the project for each region: NA: Aimee Coors, EU: Nick Khuri
The outcome of not rolling out will be a loss in Time in Buffer[10], optimizes logic deployment: BAPv2, preventing MRS recirculation, productivity and cycle time improvement.
Risks: Automated Buffer Management is limited by its input accuracy; data accuracy and rate accuracy.
· Callout: Previous tested models were sensitive to noise and outliers within the data
Issue: The calculation of in-picking and in-transit buffers for PSP1 was affected, resulting in excessively high buffer limits. In turn, leading to increased recirculation rates and worse cycle times. Although other sites were not similarly affected, this issue highlighted the need for data cleaning and pre-processing to mitigate such distortions.
What this causes: Outlier data points within site data set
How long has this problem existed: During testing of DSM model
How to manage it in current state: To address this concern, we extended the DSM model to include data cleaning and pre-processing steps before calculating the in-picking and at-transit components of the total buffer limits. Lessons learned were applied to BAEv3.
Callout: Reliance on historical rate as inputs to make predictions
Issue: we discovered that rate inaccuracies emerged as the second most significant factor contributing to out-of-work (OOW) instances, following overstaffing.
What this causes: Current rate forecasting tools present a margin of error within forecasted values
· How to manage it in current state: To improve the accuracy and reliability of the models, it is essential to enhance rate forecasting or consider hedging strategies to account for rate uncertainties. One approach worth exploring involves adjusting lower buffer limits to better handle such rate variations.
Blockers: N/A
Is this initiative a rollout from a previous pilot?
Yes. It is an enhancement to BAE workstream. Deployment Results:
§ AFE time with optimal MoW (5-20) +4%
§ Mix time with optimal MoW (5-20) +4%
§ Single Smalls time with optimal MoW (5-20) +65%
§ Smartpac time with optimal MoW (5-20) +68%
Has this problem been addressed at other site types or business units within Amazon? If so, detail results from that implementation. à No
Are there related projects being implemented concurrently and if so, how does one effect the other if this change is not approved?
Buffer Automation – phase 2. No impact.
Table 2: Previously Approved Projects
1
BAEv2
Current active buffer configuration model – ARC owned deployed in 2022.
Internal Labor
$34.3M
N/A
-
N/A
N/A
Appendix
i. Intervention Analysis visual for BAEv3 cycle time
ii. Intervention Analysis visual for BAEv3 outbound rate
iii. Pilot Analysis on Time In Buffer
During the post-intervention period (pilot), the response variable had an average value of approx. 0.38. By contrast, in the absence of an intervention, we would have expected an average response of 0.25. The 95% interval of this counterfactual prediction is [0.23, 0.27]. Subtracting this prediction from the observed response yields an estimate of the causal effect the intervention had on the response variable. This effect is 0.13 with a 95% interval of [0.11, 0.15]. For a discussion of the significance of this effect, see below:
· Summing up the individual data points during the post-intervention period (which can only sometimes be meaningfully interpreted), the response variable had an overall value of 3.05. By contrast, had the intervention not taken place, we would have expected a sum of 2.02. The 95% interval of this prediction is [1.86, 2.17].
· The above results are given in terms of absolute numbers. In relative terms, the response variable showed an increase of +52%. The 95% interval of this percentage is [+40%, +64%].
· This means that the positive effect observed during the intervention period is statistically significant and unlikely to be due to random fluctuations. It should be noted, however, that the question of whether this increase also bears substantive significance can only be answered by comparing the absolute effect (0.13) to the original goal of the underlying intervention.
The probability of obtaining this effect by chance is very small (Bayesian one-sided tail-area probability p = 0). This means the causal effect can be considered statistically significant.
Pilot Site Identification
The following approach was chosen to select sites:
Define site characteristics (rates, recirculation, cycle times, etc.)
Perform a principal component analysis for these characteristics
Cluster sites per generation
Choose one site per cluster and generation
a) Ideally not one involved in another flow pilot for dilution/interference
b) Site opt-in
Data Sources
Data preparation
TIB analysis and descriptive statistic: No data preprocessing except for the automatic exclusion of missing data.
ARIMA: Missing values were filled using interpolation as long as the missing data point was between observation dates. If the missing data was found at the beginning of the time series, it was replaced with the value of its immediately following date. We also added 1×10−9 to the “In Buffer Recirculation” and “In Buffer Time OOW” time series to make them suitable for Box-Cox/Yeo-Johnson power transformations (values of zero cannot be transformed).
Experimentation Date Range
Entire Data Time Frame: 5/23/2023 00:00 through 7/3/2023 23:59
Baseline Data Time Frame: 5/23/2023 00:00 through 6/19/2023 23:59
Intervention Data Time Frame: 6/20/2023 00:00 through 7/3/2023 23:59
Exclusions from the analysis:
OMA2: Excluded due to resistance to recommendations and opting-out of pilot.
PSP1: Excluded due to inaccurate determination of DSM buffer limits (inclusion of noisy data/outliers)
GRR1: Excluded due to dilution from other pilot
ORF3: Observations from 06/24 were excluded due to mechanical issues at the site.
PPCW1000 process paths: Given CF’s previous work and concern about whether our more generic models translate to this path’s buffer limits, we excluded it from the pilot and analysis.
Outlier removal, filtering: The DSM model extended for the calculation of in-picking and in-transit buffers removes outliers using inter-quartile range filtering (similar to Box-Whisker plots) and/or uses the median instead of the mean to calculate rates). No outliers were removed for the ARIMA analysis.
VC5+ and pack rate for AFEs
Buffer Health Analysis - https://tiny.amazon.com/17um2hx59/BHA
DSM (Draft) - https://tiny.amazon.com/sbpuw7oh/DSMInputs
BAE v3 - https://tiny.amazon.com/qwnx8v1g/BAEv3
BAE v2 (Prod) - https://tiny.amazon.com/11kxy62k8/BAEProduction
Frequently Asked Questions (FAQs):
Q1: Can a t-test be conducted?
A1: A t-test is completed on “forecasted” vs “observed with 7-point (observations). This pilot was not designed as a “true random experiment” with control groups, it was designed for an intervention analysis. T-tests on time series can be misleading due to autocorrelation, trends and seasonality. Usage of such to draw a conclusion is a subject of interpretation. In other words, having a perfect t-test p-value will not necessarily guarantee true changes in cases of time series data. Simple upward trends will show a perfect t-test p-value, but not change much due to intervention at the same time.
Q2: How are future developments of buffer modeling and automation being communicated to stakeholders?
A2: ACES distributes a bi-weekly NACF Multi-Flow Initiative flash (since week 18) surrounding S-Team goal, Outbound Flow Management Automation, (King Pin ID: 579820), to achieve a 2% productivity increase by December 31, 2023. Recipients of this flash include: AFT, ACES, EU Flow, iDEA, Central Flow, PE, SRD, and GMs. Archived editions can be found here.
Q3: What are other influences to AFE OOW instances?
A3: (a) Current Central Flow standard work to induce “sort throttles” can increase both induct and pack OOW instances. Sort throttle is an AutoFlow override reducing induct rates by 20% if packables per wall (PPW) exceeds 50% over max threshold. (b) Low CE backlog and cadence of generating new OSP plans intra-interval to reallocate labor to paths with available pick pools.
Q4: What is the impact of change to OOW instances from the pilot?
A4: Insignificant.
Spearman's rank correlation rho
data: ndf$`OOW%` and ndf$outbound_rate
S = 1016930, p-value = 0.3627
alternative hypothesis: true rho is not equal to 0
sample estimates:
rho
0.0668963
Report: TUMBL SOP Statistical Analysis
Analysis Framework
Timeframe: [05/02/2025 → 08/31/2025]
SOP Date Change: [7/10/2025]
Tests Conducted: [Cart Regression, Multiple Regression {Quadratic + Linear}, Two-Tailed T-Tests, Time Series Plotting, Subset Regression, Main Effects Screener, ANOVA, X/S Bar Charts]
Data Granularity: [Daily]
Data Source: NA ARS OB Low Backlog Dashboard
Tools Used: [Excel, Pippin, Minitab]
Low BL Stats.mpx
Data Segmentation: pp_group (MULTI/SINGLE); SOP time (Old/New); and shift (Days/Nights)
Sites Analyzed: OXR1, SMF1, DEN4, BFI4, SLC1 [Low Backlog Pilot Sites]
Executive Summary
Statistical analysis of SOP time changes implemented on July 10, 2025 reveals significant operational improvements across multiple dimensions. The study, conducted from May 2 to July 31, 2025 using daily data from the NA ARS OB Low Backlog Dashboard, shows substantial reductions in Time Under Minimum Backlog with BL1 (<1 Hour of Work) decreasing by 3.34 percentage points (7.43% to 4.09%) and BL2 (1-2 Hours of Work) decreasing by 2.75 percentage points (21.12% to 18.37%).
Productivity metrics demonstrate meaningful gains with Induct Rate increasing by 1.05% (940.34 to 950.17 units/hour), Pack Rate improving by 4.07% (257.31 to 267.77 units/hour), and Pick Rate rising by 3.08% (257.34 to 265.27 units/hour), translating to effective rate gains of 0.06% for MULTI processes, 0.25% for SINGLE processes, and 0.19% for Pick processes.
The financial impact is substantial, with a realized entitlement of $11,547,119, supported by robust statistical confidence (R-Square value of 59.83%). Multiple analytical methods, including regression, T-tests, and ANOVA, confirm the statistical significance of these improvements, validating the decision to standardize the new SOP across North American operations.
Key Findings
Significant Decrease in Low Backlog
BL1 time (<1 Hour of Work) decreased by 3.34 percentage points (from 7.43% to 4.09%)
BL2 time (1-2 Hours of Work) decreased by 2.75 percentage points (from 21.12% to 18.37%)
Rate Improvements Across All Process Types:
Induct Rate: +1.05% (940.34 to 950.17 units/hour)
Pack Rate: +4.07% (257.31 to 267.77 units/hour)
Pick Rate: +3.08% (257.34 to 265.27 units/hour)
Effective Rate Gains:
MULTI processes: 0.06% effective rate
SINGLE processes: 0.25% effective rate gain
Pick processes: 0.19% effective rate gain
Realized Financial
Total realized entitlement: $11,547,119
With 50% finance hedge applied: $8,660,339 confirmed benefit
Statistical Confidence
R-Square value of 59.83% indicates that the majority of variation in the data is explained by the statistical Multiple regression methods, T-tests, and ANOVA analyses confirm the statistical significance of the observed changes