Applying human-machine interaction design principles to retrofit existing automated freight planning systems

APPLYING HUMAN-MACHINE INTERACTION DESIGN PRINCIPLES TO RETROFIT EXISTING AUTOMATED FREIGHT PLANNING SYSTEMS

by

John Bishop Ravenel

M.S., Management, Stanford University Graduate School of Business, 2018 J.D., University of Richmond School of Law, 2004

B.A., Computer Science, Wofford College, 2001

SUBMITTED TO THE PROGRAM IN SUPPLY CHAIN MANAGEMENT IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

MASTER OF ENGINEERING IN SUPPLY CHAIN MANAGEMENT AT THE

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

JUNE 2019

C 2019 John Bishop Ravenel. All rights reserved.

The authors hereby grant to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created.

Signature of Author:

Certified By:

Executive Director,

Signature redacted

Departnent of Supply Chain Management May 10, 2019

Signature redacted

-- Dr. Eva Ponce Cueto

MITx MicroMasters Program in Supply Chain Management Thesis Advisor

Signature redacted

/{

JUN 18 2019

I I Professor Yossi Sheffi

Director, Center for Transportation and Logistics Elisha Gray II Professor of Engineering Systems Professor, Civil and Environmental Engineering

APPLYING HUMAN-MACHINE INTERACTION DESIGN PRINCIPLES TO RETROFIT EXISTING AUTOMATED FREIGHT PLANNING SYSTEMS

By

John Bishop Ravenel

Submitted to the Program in Supply Chain Management on May 10, 2019 in Partial Fulfillment of the

Requirements for the Degree of Master of Engineering in Supply Chain Management ABSTRACT

With the increased application of cognitive computing across the spectrum of industries,

companies strive to ready their people and machines for future system change. Based on resource constraints, business needs, and the speed of change, many companies may opt for system

augmentation rather than the adoption of entirely new systems. At the same time, changes in technology are increasing at paces never before realized. Against this backdrop, human actors and machines are working together interactively in new and increasing ways. Further, recent business model innovations, particularly in the retail space, have cast focus on logistics execution as a potential major competitive advantage.

In this context, we considered the conceptual question of how best to iteratively improve a logistics planning system, which is composed of both human and machine actors, to reduce transportation and labor costs and increase the ability of the organization to think and act

strategically. In order to front these current technological realities - the need to stage for agent based systems and cognitive computing, the likelihood of system retrofit over rebuild, the ever

increasing rate of change, and the rapid intertwining of human and machine roles - we proposed using human-machine interaction (HMI) design paradigms to retrofit an existing loosely coupled human-machine planning system. While HMI principles are normally applied to tightly coupled

systems such as jet airplanes, the HMI architectural design applied novelly in this case showed significant application to an existing loosely coupled planning system. In addition to meeting the realities of today's competitive landscape, the developed HMI framework is tailored to a retrofit

situation and also meets resiliency considerations. That novel conceptual proposal of HMI frameworks to an existing loosely coupled joint cognitive planning system shows tremendous promise to address these imminent realities. With regards to the particular freight planning

system considered, 71% of manual interventions were caused by the wrong sourcing facility being assigned to supply pallets to a customer. The remaining intervention causes were carrier changes 18%, customer restrictions 9%, and one change prompted by a data discrepancy.

Further, at a conceptual level, the application of HMI frameworks to an existing freight planning system was effective at isolating data and alignment incongruencies, displayed lower

communication costs than recurrent system rework processes, and tethered well with system resiliency factors.

Thesis Advisor: Eva Ponce Cueto

Title: Executive Director

ACKNOWLEDGMENTS

I would like to thank my thesis advisor, Dr. Eva Ponce Cueto, a fiercely passionate and focused scholar, researcher, teacher, and mentor, for her many hours of patient instruction over the last nine months. I would also like to thank Dr. Bruce Arntzen, the Executive Director of the MIT Supply Chain Management Program, for his steady and determined support of this project and my education and experience at MIT. Lastly, I offer my thanks and appreciation to the members of the thesis review committee for their attention and insights that richly developed the final

Table of Contents

List of Figures 1. Introduction

1.1 Company Business Model Overview

1.2 Company's Freight Planning System and Problem Description

1.3 Research Question and Hypothesis

1.4 Research Overview and Considerations

1.4.1 Transportation Planning Models

1.4.2 Systems Analysis: Human-Machine Interaction

2. Literature Review

2.1 Supply Chain Planning Methodologies

2.1.1 Centralized Planning Models

2.1.2 Distributed Agent-Based Models

2.2 Modeling Human-Machine Interaction in Planning Systems

2.3 Proposed Application of Human-Machine Design Paradigm

2.4 Literature Gaps

3. Methodology

3.1 Data Collection and Modeling

3.1.1 Automated System Evaluation

3.1.2 Human System Evaluation

3.1.3 Development ofAdditional System Data

3.1.4 Stakeholder Survey

3.1.5 Model of Freight Planning System

3.2 Pilot Proposal

3.2.1 Facilitating Frameworks: Ideation

3.2.2 Facilitating Framework: Extracting Opinions/Information &

Understanding Roadblocks

3.2.3 Political Considerations in Forecasting

4. Analysis and Results

4.1 Analysis of Project Implementation 4.1.1 Automated Systems Evaluation

4.1.2 Human Systems Evaluation

4.1.3 Development ofAdditional System Data 4.1.4 Stakeholder Survey

4.1.5 Model of Freight Planning System 4.1.6 Ideation Framework Results

4.1.7 Extraction and Roadblock Removal 4.1.8 Political Considerations 4.2 Results Analysis Page 6 7 7 8 10 11 11 12 13 13 14 15 18 21 25 26 26 27 28 28 29 29 30 31 33 36 37 37 37 37 38 38 39 40 42 45 45

5. Contributions and Conclusions 48

5.1 Contributions 48

5.2 Managerial Insights 51

5.2.1 Tilling the Soil 51

5.2.2 Development of HMI Paradigms 56

5.2.3 Proposed HMI Retrofit to Existing System 59

5.3 Future Research Opportunities 64

References 66

List of Figures

1. Parent Corporation Closed Loop Supply Chain 8

2. Manual v. Automated Planning Costs 9

3. Recursive Nested Behavior-based Control Diagram for Vehicle System 23

4. Methodology Overview 26

5. Five Part HMI Iterative Process 30

6. Swim Lane Diagram of Three Primary Systems and Manual Planning Team 40 7. Fishbone Diagram of Joint Cognitive System Causes of Manual Interventions 41 8. Fishbone Diagram of Stakeholder Questions about the Joint Cognitive System 42

9. Overview of Planning Methods by Count 46

10. Concept to Application Diagram for HMI Retrofit Implementation 49

1. Introduction

This master's thesis focuses on the application of human-machine interaction (HMI) design to an existing freight planning system used in a closed loop supply chain business. The system is analyzed along with opportunities for improvement. HMI principles are applied to model, retrofit, and improve the overall joint cognitive planning system consisting of a several-part automated system and human actors. This section introduces the client company's

("Company") business model, the current state of the Company's freight planning system, the research question, research overview and considerations, and proposed problem solution.

1.1 Company Business Model Overview

The multi-billion dollar umbrella corporation ("Parent Corporation") of the Company is an international logistics company and closed loop supply chain ("CLSC") ("Parent Corporation 2018 Annual Report"). The Parent Corporation provides a "pool of reusable pallets, crates, and

containers," which it aggregates and delivers for use by international leading companies ("Parent Corporation 2018 Annual Report", p.3). The Parent Corporation consists of various components defined by equipment type, including containers and pallets. "[The Parent Corporation's]

platforms form the invisible backbone of global supply chains, primarily serving the fast-moving consumer goods, fresh produce, beverage, retail and general manufacturing" ("Parent

Corporation 2018 Annual Report", p.3). Operating in over 60 countries, the Parent Corporation oversees hundreds of millions of assets and facilitates over 10,000 daily loads across nearly 500 service centers, hundreds of total pallet management locations (reduced capability service centers), and over half a million collection points ("Company MIT Presentation," 2018). The

primary assets for use are the 48" x 40" pallets, fluid containers, and reusable plastic containers ("Company MIT Presentation," 2018, p. 3).

Figure 1 - Parent Corporation Closed Loop Supply Chain ("Parent Corporation 2017 Annual Report")

As shown in Figure 1, the business model consists of a three-part iterative process: (1) Parent Corporation issues containers, pallets, and crates to manufacturers;

(2) Customers load equipment with goods and shuttle through the supply chain to end users;

(3) Parent Corporation collects, inspects, and (as needed) rehabilitates equipment for future use.

1.2 Company's Freight Planning System and Problem Description

A significant service component of the Company's business is meeting all customer

needs for pallets and other equipment within compressed time windows, which can be as short as a few hours. Customer responsiveness ensures unimpeded flow of partner goods to meet their demands. The Company uses an automated planning system to direct and assign the movement of pallets to client locations ("Company MIT Presentation," 2018). Pallet loads are assigned to

Parent Corporation

Company transportation assets and independent carriers, which deliver them to client manufacturers and producers ("Company MIT Presentation," 2018).

However, not all pallet requests can be handled through the automated process; the remainder are diverted to a manual planning process, which consists of fire-fighting with emphasis on demand fulfillment and speed over cost optimality ("Company MIT Presentation," 2018, p. 8). Currently, manual planning of pallet movements consumes the efforts of 35 logistics

coordinators and concerns approximately 45% total load volume (580,000 pallet loads annually in the United States) ("Company MIT Presentation," 2018). The loads planned include

issuances, collections, and relocations of pallets ("Company MIT Presentation," 2018).

Particularly, not all load planning decisions are reactive to customer returns; rather, nearly 48% of issuances are manually planned ("Company MIT Presentation," 2018). The high volume of loads diverted to the manual decision process results in cost increases of nearly 35% (over automated decisions) ("Company MIT Presentation," 2018). These facts are shown in Figure 2 below. Further, manually planned load costs are more variable than those of the automated system ("Company MIT Presentation," 2018).

Manual v. Auto

(Average Cost Per Unit)

$1,00 $0.94 5 0, $0.9 $0$.89 SO7 T 9(Y) 9 $ 6 5 4 o6 $ 0 .8 S 4 $0.86 $0 6 .4 0 . so._{SO. 70 05$.4} .. 6 5.3 $06 06 06 o& $0. O67 $0.69 60.64 S06 S06$064$.4906 06

- aua Man( ACU - Auto 'Plan 4(0

Figure 2 (Manual v. Automated Planning Costs) ("Company MIT Presentation, 2018")

Moreover, due to the load volume and need to move these loads immediately, logistics coordinators and senior logistics analysts (more experienced coordinators) (collectively, as "coordinators") are unable to improve the system by acting strategically. Rather, they spend their entire effort responding to short-fused problems. A consequence of dedicating logistics

coordinators nearly exclusively to fire-fighting is the risk they will lose a strategic level appreciation of the overall planning system.

1.3 Research Question and Hypothesis

Improving the percentage of loads planned through the automated planning process is critical to ensure identified optimality considerations are pursued, reduce costs, and to permit

logistics coordinators to prioritize system awareness and improvement over short-term problem solving. In measuring the overall load planning system (automated and manual), key

considerations included time spent planning and assigning loads; costs of planning, assignment, and transportation; and service fulfillment measurements (timely and correctly meeting customer equipment requests) ("Company MIT Presentation," 2018).

The research question then becomes: What process augmentation will incrementally decrease costs, reduce manual load planning, permit a system retrofit (rather than

rebuild), and improve the joint (manual and automated) cognitive system iteratively?

The hypothesis is that human-machine interaction ("HMI") principles, currently used to optimize tightly coupled systems including jet airplanes and air traffic control systems, can be applied to optimize the Company's joint cognitive load planning system. This application is novel both in its use with a freight planning system and so far as it serves to retrofit an existing planning system (rather than serve as a complete system rebuild paradigm). The specific thesis

objectives incorporate both conceptual and application goals, including to (1) characterize the current planning system under the HMI joint cognitive system framework, and (2) propose a solution that improves the system from cost (of the loads) and efficiency (manual interventions and focus) perspectives. Accordingly, this project serves to (1) characterize the current system and (2) propose a retrofit solution that iteratively improves the existing system, using HMI frameworks, whose application is novel.

1.4 Research Overview and Considerations

The two main blocks covered in this research section include transportation planning models and Systems Analysis: Human-Machine Interaction. The sections below provide the research overview and the key considerations to take into account in both cases.

1.4.1 Transportation Planning Models

Transportation planning models include both rules-based top-down centralized approaches (modeling resources, products, orders, hard and soft constraints) (e.g., Ropke & Pisinger, 2006) and cooperative agent models (providing incentives and behavioral descriptions to encourage inter-agent collaboration and freight matching) (e.g., Memon & Archimede, 2013). Regarding automated transportation planning systems, it is critical to consider the variety of problems addressed, computational complexity of the system, and level and depth of detail leading to the optimized solutions (Kozlak, 2008). Along this line, the speed, modularity, and flexibility of a multi-agent approach are promising, particularly from a conceptual design standpoint (Kozlak, 2008). System flexibility and integration potential are premium

paradigm as a fundamental concept (Nawarecki & Kozlak, 2008). Agent based models have been gaining ground due to the shifting nature of supply chains and variations in complexity and application (Chow, Siu, Chan, & Chan, 2013). The use of the agent model is instructive as the current critical system gap is between human actors and the computer planning system.

Additionally, a conceptual agent model permits expansion of an intermediate solution to a complete multi-agent model or further into a more developed cognitive computation system. In designing the current retrofit, staging for additional complexity and systems development is a key consideration.

1.4.2 Systems Analysis: Human-Machine Interaction

While this project addresses neither the development of the underlying load planning system nor the integration steps for automation, which involve substantial complexity across organizational silos (Ebner, 2015), a broad systems-based approach will be implemented. A systems approach is necessary to identify system parameters, stakeholders, reasons for manual drops from the automated system, and opportunities to work within the current structure to reduce manual interventions. Further, in moving towards more automation, a critical

consideration remains the nurturing and maintenance of essential skills by logistics coordinators; stated differently, automation can cause the lapsing of critical employee skills, which are

necessary for oversight of the service delivery (Albrecht, 2013). A human-machine collaborative model is ideal both from efficiency and systems development perspectives (Albrecht, 2013). Further, research validates improved outcomes by incorporating human action into the automated planning cycle (Kim, Banks, & Shah, 2017).

Automation is best considered an iterative process, wherein the technical and human systems interact repeatedly to smooth processes and improve workflow (Albrecht, 2013). This is

particularly true for the Company, which experiences uncertain and opaque micro-level (location and time) demand from customers and changing systems based on client relationships and freight flows. From a design standpoint, the HMI system should limit complexity and ambiguity to the extent possible (Albrecht, 2013). The modeling of human-machine planning systems is not novel (Ryan, 2011), but has been focused on tightly coupled systems requiring constant user-machine interaction. A critical hurdle is developing standards for the interaction between the automated process and the human exception management efforts (Ryan, 2011). Moreover, identifying the appropriate level of abstraction for the interaction is another key modeling step (Kim et al., 2017).

2. Literature Review

After evaluating the two predominant modeling and optimization schemas - top-down rules-based systems and agent-based models - the literature review considers the interaction between human and machine systems in general and specific contexts. Next, a novel application of HMI design frameworks to the integrated planning system is proposed. This HMI model's use is innovative both in its application to a freight planning system and its application to retrofit (rather than build from zero) such a system.

2.1 Supply Chain Planning Methodologies

The research question considers what process augmentation will promote human feedback in the automated process, improve the automation process iteratively, and

which fall into two broad categories: (1) centralized planning models ("CPMs")1 and (2) distributed agent-based models ("DABMs").

2.1.1 Centralized Planning Models

Historically, transportation planning problems have been solved using a combination of linear programming methods and heuristics, due to their complexity. Common planning efforts include vehicle routing and cargo loading problems, both of which have been expanded to address more nuanced scenarios. Transportation planning and scheduling problems

implementing CPMs span the logistics sector and include multiple site production environments with distributed transportation systems (Sauer & Appelrath, 2000), intermodal long-haul

transportation (Perez Rivera & Mes. 2017), dynamic vehicle allocation (Powell, 1987), pickup and delivery with time constraints (Ropke & Pisinger, 2006), full truckload routing

(Arunapuram, Mathur, & Solov, 2003), scheduling cross-dock inbound and outbound freight (Boysen, Fliedner, & Scholl, 2010), less than truckload load planning (Erera, Hewitt,

Savelsbergh, & Yang Zhang, 2013), real time truckload routing and load assignments (Yang, Jaillet, & Mahmassani, 2004), and inland container transportation (Zhang, Yun, & Kopfer, 2010), among many other applications.

As transportation systems have become more complex, so have the algorithmic and heuristic approaches used to understand and improve them. For example, in 2009, a multi-population genetic algorithm effectively found a globally optimal solution for sourcing and scheduling processes (Zegordi & Beheshti Nia, 2009). A year later, a multiple part solution consisting of a Monte Carlo scenario generator, transportation heuristic, and mixed integer

' While certain centralized planning models bear aspects of distributed decision-making approaches, the number of

program solved a "hierarchical decision problem" with probabilistic demand and multiple time periods (Klibi, Lasalle, Martel, & Ichoua, 2010, p. 225).

As the multiple objective and diverse agency nature of transportation has been realized and expanded operationally, algorithms have sought to keep pace. As mentioned, the integration of sourcing and scheduling concerns was addressed through a "multi-objective evolutionary algorithm" using different objective functions for different populations (Zegordi & Beheshti Nia, 2009, p. 952). While this algorithm starts to represent different vantage points, it does so only in

a centralized, top-down, and scripted manner (Zegordi & Beheshti Nia, 2009). Taking a step further towards distributed computation and analysis, researchers have recently used a modified ant algorithm, which echoes the tunneling of ants to find an optimal route, in making truckload routing decisions in Ukraine (Khalipova et al., 2018). Additionally, these CPMs have been applied to represent carrier cooperation over less than truckload freight (Xin Wang & Kopfer, 2014) and freight brokers in full truckload freight movements (Wang & Kopfer, 2015). In both these collaboration studies, modeling moved towards auction protocols to adequately protect agent information and goals, while actions were choreographed or instituted by system requests. These recent advances point towards the use of distributed problem-solving mechanisms to address growing problem complexity and the nature of the multi-party transportation systems.

2.1.2 Distributed Agent-Based Models

This section introduces the DABMs, discusses their application to planning systems, and finishes by providing several perspectives to encoding and instituting user preferences and interactions among designed agents. But first, this section considers the broader application of DAMBs to logistics problems. In 2008, a multi-agent computational approach was proposed in order to deal with (1) high transport problem variation, (2) significant complexity even for

simple transportation problems, and (3) the difficulty in prescribing the level and degree of detail impacting transportation decisions (Kozlak, 2008). Similarly, multi-agent based systems are considered applicable in the logistics realm because of the (1) distributed nature of logistics, (2) high degree of planning dynamics, (3) the importation of human knowledge to planning, (4) number of environmental factors, (5) cooperation requirements among various stakeholders, and (6) distributed and dynamic nature of resources (Baykasoglu & Kaplanoglu, 2015). Another

paradigm for judging DABM application suggests use where the system is (1) modular, (2) solvable via decentralized decision making, (3) "ill-structured," (4) likely to experience frequent changes, and (5) dynamic and complex (Chow et al., 2013). Under any of these tests, logistical applications suggest multi-agent system solutions.

In DABMs, agents have their personalities (behavior) and incentives (goals) specified, leading to modular, adaptable, and more easily conceived systems (Kozlak, 2008). Further, agent-based systems are more likely to display superior resiliency as compared to CPMs, which are fixed for optimality of a predefined system. Particularly, under the framework of measuring resiliency in terms of avoidance, robustness, recovery, and reconstitution (Pumpuni-Lenss, Blackburn, & Garstenauer, 2017), DABMs' modular autonomy, adaptability (both to market shocks and network design changes), and flexibility suggest that they will perform better under environment stress (Baykasoglu & Kaplanoglu, 2015).

In recent years, multi-agent solutions have addressed transportation problems including truck load planning (Baykasoglu & Kaplanoglu, 2015) and load consolidation decisions

(Baykasoglu & Kaplanoglu, 2011), along with other applications noted below. DABMs have been used to plan and schedule processes, including to permit greater speed and adaptability by small and medium size manufacturers (Denkena, Zwick, & Woelk, 2003), manufacturing

planning and scheduling across nested organizational levels (Karageorgos, Mehandjiev, Weichhart, & Hammerle, 2003), freight planning (Chow et al., 2013), and third party logistics distributed planning (Memon & Archimede, 2013), among other applications. Recently, researchers have outlined a comprehensive survey of cooperative multi-agent planning systems which details the consistent parts and problem solving considerations (Torrefio, Onaindia, Komenda, & Stolba, 2017).

DABMs are particularly useful in conceptualizing systems and encoding agent and system behaviors for computational study and refinement. Particularly promising is the development of multi-agent systems through evaluating human strategies and then encoding them into computational models. The DABM framework permits human augmentation and improvement, assigns tasks based on human-machine relative strengths, improves testing

strategies and interactions, and streamlines communications (Kim et al., 2017). DABMs are also useful in multi-party systems where different parties cooperate but have competing goals (Plinere & Borisov, 2011). In modeling and optimizing a manufacturing and procurement system, agents

were developed and interacted through a negotiation mechanism (Plinere & Borisov, 2011). More recently, a negotiation protocol has been used to balance truck parking spaces in Spain through a multi-agent system (Garcia, Tomis, Garcia, & Martinez, 2017). Beyond simple negotiation, recently efforts have focused on modeling the market interaction through multi-agent system, using an auction protocol to efficiently allocate freight movements (Robu, Noot, La Poutr6, & van Schijndel, 2011). In developing multi-agent systems for freight movements, a conceptual framework has been developed to address the behavioral aspects of different actors and business units in transportation systems (Roorda, Cavalcante, McCabe, & Kwan, 2010).

In sum, the recent use of DABMs to model and solve distributed multi-party

transportation problems is gaining steam. Further, the conceptual framework of DABMs feeds back into the entire system optimization (consistently of both human and automated

components), as discussed in the next section. Lastly, the DABM framework stages systems for pending technological advances, including cognitive computation systems. Accordingly, while the below-described HMI frameworks are newly applied to existing freight planning systems, the

DABM theory undergirds the approach as well.

2.2 Modeling Human-Machine Interaction in Planning Systems

The modeling of human-machine systems has typically been limited to tightly coupled systems, including nuclear plants and air traffic control (van Wezel, Cegarra, & Hoc, 2010); (Idris, 1994), flight control systems and driver assistance systems (Badreddin & Wagner, 2011), as well as space mission planning and scheduling and military applications (Gaudreault et al., 2017) such as unmanned vehicles (UVs) (Clare, 2010).

However, there is a need for understanding and effectively synchronizing both human and machine actors in other planning systems (van Wezel, Cegarra, & Hoc, 2010). Presently, the Company's opportunity involves the effective use of logistics coordinators, the reduction of manually planned freight movements, and the development of an integrated system that

maximizes the skills of both the human actors and the computer freight planning system. Based on the nature of the Company's systems and scope of the project, the methodology considers the improvement of only the static planning system and does not weigh whether to move the total system towards a dynamic function allocation system (van Wezel et al., 2010). While there is literature that addresses mixed-initiative systems (MIS) broadly applicable to various business

projects, these systems anticipate an entire system redesign where the human uses automated tools to perform expert planning (Gaudreault et al., 2017). As the Company is neither seeking an automated system redesign nor having human planners fine-tune automated planning decisions, MIS and similar systems are outside the scope of this project. Moreover, as this research project

focuses on how to retrofit, rather than redesign, an existing joint (automated and manual) freight planning system, its application is inevitably broader. How could a more limited solution be more broadly applicable? Due to capital limitations, labor considerations, and business focus reasons a complete system overhaul is risky and infrequently entertained. More often, logistics firms will more eagerly seek to augment existing legacy systems than to rebuild entire planning systems and accompanying business models.

Nonetheless, MIS foundational principles about human and machine relative strengths are worth mentioning (Gaudreault et al., 2017). Also, a recent article discusses factors bearing on Company's planning department, including the decision workload and time constraints in proposing top-level considerations for designing a human-machine system (Kucherov &

Artyushenko, 2017). However, the implementation considered by Kucherov and Artyushenko is a tightly coupled, constantly monitored system, which is distinct from the Company's freight planning system (Kucherov & Artyushenko, 2017). Despite this distinction, many of the design

considerations can be applied, as noted below.

Employee concerns of "lower job satisfaction, complacency, and performance degradation" result from function allocation systems that merely provide human actors with

"leftover" tasks (van Wezel et al., 2010, p. 342). The proposed HMI framework weighs the Company's goals of recalibrating logistics coordinators towards strategic views and actions (van Wezel et al., 2010, p. 342). Regarding how to split functions between the human and machine

systems, both the cognitive nature of the human worker and tasks themselves are weighed (van Wezel et al., 2010). Further, redundancy of effort, called "mutual control," can lead to better outcomes (van Wezel et al., 2010).

Contrary to the work performed by van Wezel et al., the opportunity facing the Company does not involve building the system from the ground up or having the human actor check or select options provided by the automated planning system. Further, differences in tightly coupled and loosely coupled joint cognition systems are defined in terms of time pressure, parallel process requirements, mission criticality and risk profiles, and constraint violability (van Wezel et al., 2010). However, no solution is proposed in the recurrent event where an existing automated system needs augmentation to make better use of human actors and possibly realign task responsibilities.

Despite this, these general themes impact how to assign tasks, including determining planning procedural components, breaking down procedures to sub-procedures until they reach a point of sole allocation, grouping sub-procedures by type of problem presented, and analyzing the "goals and constraints of each problem" (van Wezel et al., 2010). Regarding

sub-problems, four considerations are the "harm risk" (impact of less than optimal solution), the need for "situational awareness" (depending on extent of automation), timing considerations, and problem complexity (van Wezel et al., 2010, pp. 347-348). Further, care must be given to evaluate the human's trust of information and solutions provided by the system and to capture feedback from the human scheduler about impressions of the overall system (van Wezel et al., 2010). One particular strength of a joint cognitive system is the human-machine interplay that can evolve into "a common frame of reference (COFOR) between the agents" (van Wezel et al., 2010, p. 350). This common frame of reference permits iterative improvement of the joint

system and the development of the human actor as a strategic overseer of the machine

component. Concerning the development of a common frame of reference, one key step in the system design is identifying unconscious biases and other impacts on the human decision process (Arnold, 2018). Moreover, human actors possess latent implicit knowledge about a system that can be drawn out through a systems analysis and integration of the two systems (Gaudreault et al., 2017). Also, as recognized by the Company, human actors outperform machines on matters of "strategic thinking" (Gaudreault et al., 2017, p. 70).

In sum, while research does not squarely address the Company's system and opportunity, the broader themes do inform the proposed solution. Further, while an innovative application in the present case, HMI research presents a logical progression from centralized planning models, to distributed agent-based models, and now to joint cognitive system modeling using HMI design principles.

2.3 Proposed Application of Human-Machine Design Paradigm to Planning System The current application of the existing human-machine system design paradigm poses two novel solutions that are not addressed by the existing research on the topic. First, while the topic is generically discussed by several authors at a high level (Gaudreault et al., 2017; van Wezel et al., 2010), this solution goes a step forward by incorporating existing human-machine system frameworks to systematically improve the Company's freight planning system. Existing literature provides generic guidance for future applications, while the current proposed solution concerns a large-scale human-machine freight planning system that coordinates over 10,000 loads daily. Second, the solution is unique in that existing research proposes principles for designing systems from the ground up, whereas here the solution works with an existing

automated system and imposes design considerations around the existing system. Augmentation of existing systems has not been discussed and is broadly applicable given that many logistics companies and business units will not expend the capital for complete redesigns, opting rather for retrofits and augmentation. Several specific novel design applications are discussed as follows. Particularly, along with the top-level guidance detailed above, the five described

frameworks below combine to present an inventive system for modeling an existing joint human-machine planning system, identifying augmentation opportunities, and implementing a retrofit to drive automation and human actions towards their best usages.

First, using human-machine system design principles, the system outlines scenarios covering the set of exceptions managed routinely by logistics coordinators (Poklukar, Papa, & Novak, 2017). Scenarios permit an in-depth discussion on work flow, tasks and

sub-components, preferred and error states, human insights on the system, and human-machine allocation considerations (Poklukar et al., 2017). From the scenarios, analysis of the system proceeds to describing functional tasks (and sub-components) and identifying common

contextual elements, communication protocols, and allocation decisions (Poklukar et al., 2017). Second, the system appropriates the "Recursive Nested Behavior-based Control (RNBC)" structure, previously used for mobile robots, to map the human planning system and break down tasks for greater optimization (Badreddin & Wagner, 2011). The key aspects of this framework are decomposing human behaviors into multiple cascading control levels in a recursive fashion, paralleling machine and human frameworks to the extent possible, and designing interfaces between each operational level (Badreddin & Wagner, 2011). As shown in Figure 3 involving a semi-autonomous flying machine, recursion is an effective concept when dealing with nested tasks.

Task 8 Task Decomposition / Task

us( N Mission Planning y(t) status

Environ-sGround Monitoring

ad t 7 Reasoning y7(t)

14400 Model Building/

vi.o At 6 Action selection _{C y6(t)}

Cps 4 Homing / Position Control y.(t)

GPS

* 0 U 3 Collision Avoidance _YA)

U 2t 2 Velocity Control y2(t)

Attitude Stabilization /

/M

5 t Axis-Level Control yLLt)

Figure 3

Recursive Nested Behavior-based Control Diagram for Vehicle System (Badreddin & Wagner, 2011, p. 4835)

Recursion and control nesting are essential principles of this design, especially given the Company's desire to improve the strategic capabilities of its logistics coordinators and prevent duplication of effort by multiple logistics coordinators on overlapping tasks.

Third, this design mechanism implements related design considerations used in a tightly coupled spacecraft planning system (Kucherov & Artyushenko, 2017). In order to prevent cognitive overload while balancing the need for strategic awareness, system tasks will be funneled to groups of coordinators in order to batch collective responses. The batching of tasks prevents capacity conflicts and the reworking of tasks by logistics coordinators. Further, batching enhances a systems level awareness and promotes strategic action by the coordinators.

limiting the cognitive load of coordinators and the data requests to adjacent business units. Prioritization is a fundamental aspect of this system, given the decision to retrofit rather than rebuild the freight planning system.

Fourth, a key benefit of human planners is their ability to serve not just as system overseers but also as "soft data sensors" for automated systems (Lore, Sweet, Kumar, Ahmed, & Sarkar, 2015, p. 1). This aspect of the solution is critical as the goal is to improve the overall joint cognitive system (automated planning system + human logistics coordinators), requiring

that the human actors relay information back to the automated system. While this article noted involves application to robotics, the framework used can be applied in an innovative way to the Company's freight planning system. The paradigm of collecting data and feeding it back only if it meets the "value of information (VOI)" threshold, which balances data requests against

cognitive load constraints, is the critical takeaway from this research (Lore et al., 2015). Further, human feedback gains buy-in and trust by the human planners in the automated system.

Fifth, on the goal of improving logistic coordinator strategic awareness, one benefit of the Company's "leftover" system is that human and machine interactions are not tightly coupled. Here, a HMI feedback mechanism is modeled after the military style pre-briefing and post-mission debriefing practices (Miller, 2018). As increasing system transparency has been shown to improve overall performance and directly ties to the Company's goal of moving logistics

coordinators higher in the abstraction stack, this approach is preferable (Miller, 2018). Further, the concept of "displaced transparency," delaying information feedback loops to permit greater human attention and focus, tracks the conduct of high performing teams (Miller, 2018, p. 193). In addition to providing feedback from the system to human actors, after action reviews permit

greater understanding across human actors of the various mental models being used to solve similar problems (Miller, 2018).

In sum, the proposed joint cognitive system augmentation applies five HMI design

paradigms, namely scenario development, recursive nested behavior-based control structure, task batching, the use of logistics coordinators as soft-data sensors, and the application of displaced transparency through pre-briefing and post-mission debriefing practices. Each instance

represents a novel application to a distributed human-machine planning model, particularly in the freight transportation space. Further, the proposed innovation applies broadly to a process

augmentation (rather the complete redesign), which is likely to be preferable for many companies to a complete rebuild.

2.4 Literature Gaps

Three gaps in the literature bear on the opportunity facing the Company and the proposed innovative solution. First, research considering the application of HMI to planning systems focuses on tightly-coupled systems such as jet airplanes and nuclear systems. Where articles do address loosely integrated systems, they do so only from a top-level analysis of key principles to consider and do not reach detailed application. Second, no comprehensive framework is

proposed for loosely integrated planning systems, particularly freight planning systems. Third, research is consumed by ideal world build from zero scenarios, which represent the minority of systems changes. Here, the proposed solution impacts a broader swath of companies by focusing on retrofitting existing systems. On each vector, this thesis fills these existing research gaps.

3. Methodology

Considering the Company's freight planning process in light of the state of research and trend of solution implementations, the following methodology is proposed. The methodology consists of (1) Data Collection and Modeling, (2) Pilot Proposal, and (3) Results. Conclusions are considered in Section 4. Stages one and two are discussed below. Figure 4 below outlines the Methodology, including results.

-Au St " De Sy ' St

Methodology

Systems Analysis Modeling Initial Considerations Pilot Desin Results

tomated Systems Review Swim Lane Diagram * Need to Develop Stakeholder Survey(s) - Repetitive Data

Technical Manuals * Ishikawa Diagrams Additional System Data * Architect Actions Discrepancies Caused

Expert Interviews . Cost Impact Scope * HMI Frameworks Consistent Load

Field Observations - HMI Paradigm Selection Enabling Frameworks Reassignments

man Systems Review Scenario Modeling ' Political Monitoring - Low System Costs to

Interviews - RNNBC Prediction Analysis HMI Application

Field Observations Task B~tching . Employee Alignenat Communication Cost Far

velopment of Additional - iHman Sensing - Large Group Actions Less than Cost of Failure

stems Data Displaced Transparency Brainstorming to Communicate

akeholder Survey - Select and Test Enabling * 5 Why's Analysis - Alignment Issues for Frameworks Ishikawa Diagrams Data Providers

Predictions Small Group Actions

Execution Change Actions

Figure 4 - Methodology Overview

3.1 Data Collection and Modeling

"Seek first to understand." - Dr. Franklin Covey

The initial phase is composed of (1) modeling the automated and human systems to understand the system interactions and frameworks, (2) evaluating stakeholder viewpoints and staging for change management, (3) developing data on human coordinator actions, (4)

reviewing existing data maintained through the several automated systems, and (5) assessing

stakeholder sentiment at the outset of the change process. Central goals of this process are to (1) understand and map the joint cognitive planning system, (2) identify causes and effects of current system dynamics, including manual planning interventions, and (3) to represent current actor perspectives and change management considerations. Each of those goals is represented in the subsections below, after the initial fact gathering process is described.

The initial stage in this fact-gathering process is to understand the automated and manual components of the joint planning system. The automated system is comprised of three

interconnected computer systems, which (1) gather and maintain system information (pallet counts and locations, client requests, etc.) and execute various bookkeeping tasks, (2) perform optimization procedures to match client requests against pallet locations, and (3) assign transportation carriers to pallet movements.

Three goals were established for the initial measurement, modeling, and evaluation period, including to understand and map the system, identify causes and effects, and determine actor perspectives and change management considerations.

3.1.1 Automated Systems Evaluation

First, experts on each of the three computer systems were interviewed concerning the systems processes, inputs, outputs, system states, interactions, and error conditions. Materials including technical manuals for the three computer systems were reviewed. Further, the transportation and bookkeeping systems were observed first hand during operator field

observations and interviews. Coordinators do not work directly with the optimization system, though the particular system manual provides in depth information on its use. An expert on the system was interviewed several times and provided insight on system information which was reviewed during the course of the initial fact-finding stage and later during the pilot design

process. As part of this assessment, the automated system was evaluated to determine whether granular causes of manual interventions could be identified.

3.1.2 Human System Evaluation

Second, logistics coordinators, senior logistics analysts, and managers comprise the team responsible for manually planning loads that are not covered by the automated system. Members of each group were interviewed and observed over two separate periods in October and

December 2018 for a total of three days. Five interviews and field observations of logistics coordinators or senior logistics analysts were conducted, each for between one and three hours. During the field observations, coordinators and analysts explained the tasks they were

completing, the sources of their work, and the steps taken to resolve discrepancies and manually plan loads. At the beginning and end of each field observation, coordinators provided insights on their daily work and interaction with the various automated systems and other sections (both internal and external to the Company) with which they interact. Further, two managers were interviewed also, providing awareness of the role of human planners and opportunities for development of the joint cognitive system.

3.1.3 Development ofAdditional System Data

Third, as micro-level data was not available for each load including source of work and actions taken, a spreadsheet was developed allowing two logistics coordinators to chart their efforts each day, including the name and region of the planner, tracking number, date, number of loads impacted, source of work, tasks completed, other internal and external sections impacted, actions taken, time taken to fix the problem, and impact assessment (from a low of 1 to 5 based on impact of the system mistake).

3.1.4 Stakeholder Survey

Fourth, key stakeholders were identified, which include members of several planning groups, logistics coordinators and senior logistics analysts, automated optimization system team, and other personnel. Each person completed a survey shortly before a stakeholder meeting, which provided data on stakeholder beliefs about the current system, perspectives on other corporate change efforts, and insights concerning potential for system improvement. The survey served two purposes. First, it elicited employee opinions on the effectiveness of the current system from three perspectives (total joint cognitive system as well as human and automated components). The opinions prompted on the system were fine tuned to regard strengths,

frustrations, recommendations, and communication viewpoints. Additionally, participants were polled on their views on previous change efforts by the company (or elsewhere). Second, the survey began a continual process of explaining the purpose of the project, estimated benefits, and key goals. A sanitized copy of survey questions is contained in Appendix A, with company specific information removed.

3.1.5 Model of Freight Planning System

In modeling the freight planning system, a swim lane diagram was developed, as shown in Figure 4. A swim lane diagram is an effective conceptual tool for understanding processes and systems (Bursten, 2007). In developing the diagram, the relative roles, systems, information flows, and sequence of steps were considered and explored (Durugbo, Hutabarat, Tiwari, & Alcock, 2012). Further, several Ishikawa (fishbone) diagrams, as described below, were constructed to model different aspects of the system.

3.2 Pilot Proposal

With the benefit of understanding the system and stated stakeholder perspectives, the next step was considering the Company's data on the planned loads. For both systems, data is available from the Company to measure the efficiency, cost, and outcomes achieved by the separate manual and automated systems. However, as noted, data needed to be developed to identify the causes for manual interventions and cost in terms of time and focus (internal labor costs) and money (external transportation costs) of those interventions. Treble damages such as client welfare/loyalty and impact on the Company partners were not considered in this project.

After gaining an understanding of the joint planning system, the method shifted towards applying the HMI design standards (depicted in Figure 3) to innovate and augment the existing system. These five standards are at times overlapping and always complementary. Therefore, each HMI paradigm was applied both in sequence (one through five) but also in an iterative fashion, as each process provides feedback to each other HMI paradigm. As described in section 4, facilitating frameworks were necessary to implement the HMI process. The processes to be applied include the steps shown in Figure 5.

HMI Iterative Design Process

Scenario Modeling

Displaced Recursive Nested

Transparency: Behavior-Based

Pre and Post- Control

Briefings

Human Sensing Task Batching

Figure 5 - Five Part HMI Iterative Process

new retrofitted HMI system. Key considerations in the final pilot design process included minimizing disruption to the existing automated system, reducing non-strategic efforts by logistics coordinators, consolidating information requests to parallel business units, and creating a low-cost (focus and monetary) HMI overlay that continually improves itself over time.

3.2.1 Facilitating Frameworks: Ideation

Beneath the top-level HMI paradigm, several techniques were used to guide discussions and drive the HMI process. Particularly, methods evaluated and probed for information in order to determine cause and effects within the complex joint cognitive system. These cause and

analysis methods were used during weekly meetings with a joint cross-functional team overseeing the six-week pilot. These frameworks included (1) Five Whys analysis, (2) Brainstorming techniques, and (3) Ishikawa (Fishbone) diagramming as has previously been used in reduce errors in a manufacturing setting (Dziuba, Jarossovi, & Golebiecka, 2014). Root cause analysis can be an effective counterpoint, or substitute, for time consuming training endeavors to reduce causes of process errors (Murray, 2017). These methods are fairly easy to administer to novice users and can lead to important insights (Murray, 2017). Further, the collaborative nature of these approaches can lead to greater team buy-in for submitted solutions (Murray, 2017). As discussed more completely in section 4, achieving team buy-in was a critical consideration during the project.

The Five Whys method starts by asking the simple question, "Why (did this occur)?" when faced with a manual intervention (Dziuba et al., 2014). As it sounds, the process proceeds to ask "Why" again and again until a root cause or sufficiently deep cause is found; five is not a mandatory number but merely suggests a sufficient level of depth necessary to ferret out root causes (Dziuba et al., 2014; Murray, 2017). The Five Whys method can lead to splintering of

causes, which must be addressed sequentially to narrow again to root causes (Murray, 2017). Asking "why?" can result in trees that both contract (narrowing to systemic causes) or expand (identifying multiple parallel causes such as through a causal tree diagram) (Lyon, Popov, & Roberts, 2018). This method is broadly applicable to many company settings, is straightforward to follow, and requires the inclusion of a broad group of subject matter experts (with a sufficient collective vantage point) (Dziuba et al., 2014). During the pilot design phase, the Five Whys structure was implemented during one session and served primarily to expose the

cross-functional team to its use. Other methods were the primary frameworks used to further the HMI development process.

A Ishikawa or fishbone diagram is a chart that maps cause and effect dependencies

(Dziuba et al., 2014). Where quantitative sorting of causes is unavailable due to a dearth of data (as here), a fishbone diagram can be an effective tool for conceptualizing causes for a particular impact (Gawdzifska, 2011). Like brainstorming described below, the fishbone diagram is a structured technique permitting teams to explore and consider causes and impacts (Murray, 2017; Otitigbe, 2017). Fishbone diagrams can be particularly powerful to engage a diverse

group of stakeholders with different perspectives on the system (Otitigbe, 2017). A key feature is it promotes divergent and expansive thinking (no bad ideas) and visualizes a conceptual understanding of a system's process (Murray, 2017). Fishbone diagrams often sort causes based on type (Gawdzin'ska, 2011; Obudina & Hasanid, 2018). In terms of developing the diagram, one method is to identify all types of causes (segments), exploit each cause type (asking for more examples and follow-up why questions) until a cause type is exhausted by the group, complete the diagram, assess and modify the model, and circulate the diagram to other stakeholders for feedback (Otitigbe, 2017). As with other brainstorming techniques, team balance in terms of

contributions is critical (Otitigbe, 2017). In the service setting, which is comparable to the freight planning situation here, main causes are often grouped as Policies (decision rules) and Procedures (specific tasks) (Dziuba et al., 2014). An Ishikawa diagram was developed and is shown in Figure 6.

Brainstorming is a structured conversation method for considering solutions to problems, breaking down the conversation into sequential parts (Gawdzin'ska, 2011). First, group members practice divergent thinking, attempting to spawn as many ideas as possible without judging or limiting those ideas (Dziuba et al., 2014). During this phase, only clarifying questions are permitted, in order to not pre-judge or exclude ideas until all ideas and iterations have been exhausted. A key component of brainstorming is generating as many ideas as possible, without criticizing or limiting those options (Matsudaira, 2018). Second, the group proceeds to sort and evaluate ideas in terms of the cause and effect framework (Dziuba et al., 2014). Other methods of encouraging ideation include employing time constraints, mandating a question only period, assessing participant moods, and focusing on rapidly creating new questions (rather than insights and conclusions) (Gregersen, 2018). Question framing is a critical factor for idea creation such that short, open ended, descriptive, and increasingly complex questions are most effective (Gregersen, 2018). Potential roadblocks are lengthy, closed, speculative, simplistic, unrelated, and aggressive questions (Gregersen, 2018).

3.2.2 Facilitating Framework: Extracting Opinions/Information & Understanding Roadblocks

In order to address two critical considerations described below, a modified

polling/prediction market was implemented in a limited scope. The prediction method polled participants to determine their forecasts for potential risk and benefit as well as vote for whether to implement a request from a logistics coordinator to update the automated system based on a

manual intervention. At times, a manual intervention is unavoidable due to the automated planning policy being sound and preventing a larger systemic risk. However, at other times, a change is warranted as the automated system has poor data or is otherwise not finely tuned. The purpose of the prediction survey is to correctly categorize which system update requests should be granted (or denied) while also encouraging a maximum flow of information from coordinators to the automated system administrators. Balancing smart system decisions with encouraging ground level information flow are the key considerations. The iterative nature of the predictions across multiple change requests leads to scoring performance, permitting weighting of votes for future trading. In this way, the prediction method is a polling and prediction market amalgam.

Two fundamental considerations suggested the use of an early stage modified polling and prediction market to facilitate the top-level HMI design retrofit. First, an enticement is necessary to encourage logistics planners to relate their new information about the system back to the automated side of the house through change requests. Second, no one group (logistics, several planning groups, or automated system control) possesses a complete picture of the system, and each also further different local priorities. Encouraging a global outlook and preventing

parochialism also suggested a prediction market approach. Here, the modified polling/prediction market permits actors to make three decisions: (1) vote whether or not to implement a system change request, (2) predict the impact of the change (positive to negative benefit scale), and (3) predict the risk posed to the system (high to low risk scale). Participants are then evaluated against the system impacts from implemented changes and segmented based on decision type, benefit, and impact. Future iterations may weight participant votes based on past performance or vantage point, allowing trading of weighted votes amongst participants. Next, the contours of prediction markets are outlined, while each rationale for the modified prediction-polling tool is

discussed in the analysis section. Based on the project's scope, the prediction market/survey was not fully executed but was detailed to participants during discussions and a pilot design process.

Prediction markets (information markets) permit betting or forecasting future unknown events (Ahlstrom-Vij & Williams, 2018), and have been used at many major organizations including MGM, Yahoo!, Abbott, General Mills, and others (Buckley & O'Brien, 2017). Market signals have been proven to be accomplished predictors in a variety of fields, including sports, politics, business, and others (Ahlstrom-Vij & Williams, 2018). Prediction markets are also best suited to short term predictions (Ahlstrom-Vij & Williams, 2018), like the impact of a system change on upcoming manual interventions and transportation costs. While prediction markets may be subject to intentional manipulation, these impacts are quickly remedied by countervailing rational actors within the market (Buckley & O'Brien, 2017). Prediction markets have certain advantages over polling and expert panels, including fostering information searching incentives, revealing one's basis for beliefs, and the market's evaluative and communicative nature

(Buckley & O'Brien, 2017). Also, prediction markets are particularly helpful in incorporating real-time information, identifying risks, controlling monetary and attention costs, and resolving conflicting opinions from disparate actors (Kumar Varma, 2013).

In this case, the events predicted are the risk and potential impact of system changes. For instance, a logistics coordinator suggests changing the sourcing of one customer to a new plant. Stakeholders then vote publicly on a scale of -5 (high risk) to 5 (low risk) and -5 (low impact) to

5 (high impact), along with a "yes" or "no" for whether they support the change. Predictors are permitted to enter information about their assumptions, caveats, and rationale for each

prediction. Because no information is gained for denied requests (counterfactuals cannot be considered by prediction markets) (Ahlstrom-Vij & Williams, 2018), and logistics coordinators

are dissuaded from providing system information with too many denied requests, certain penalties can be used in the future to encourage system change "yes" votes. For instance, in considering type 1 (false positives - predicting high risk/low benefit, and opposite occurs) and type 2 (false negatives - predicting low risk/high benefit, and opposite occurs) errors, we can weight each error type differently in order to encourage greater infusion of changes in a systemic way. As the prediction market was only considered as part of a pilot design process, system feedback was gained but implementation data and successive iterations were not realized.

3.2.3 Political Considerations in Forecasting

As alluded to in the context of prediction markets, political considerations may bear considerably on forecasting efforts of companies comprised of multiple independent sections (Oliva & Watson, 2009). Forecasting biases may occur across sections due to group and individual incentives, information gaps, or procedural limitations (Oliva & Watson, 2009). Procedural refinement in terms of information exchanges and clarifying assumptions may alleviate these cross-sectional impediments (Oliva & Watson, 2009). Moreover, it may be necessary to impose an independent group upon the forecasting process in order to stabilize the political dimension of the process (Oliva & Watson, 2009). This group manages political conflict, information flows, and procedural gaps (Oliva & Watson, 2009). In considering the integration function of the independent body, four vectors should be considered - the

information flows, procedural efficiency, alignment across sections, and quality of cross-functional engagement (Oliva & Watson, 2011).

4. Analysis and Results

This section outlines the results of the project, including insights at each stage in the process. Section 4.1 addresses several conclusions developed the pilot design. Accordingly, this section maps to the methodology section, commenting on the results at each stage. Section 4.2 provides additional consideration of the project results.

4.1 Analysis of Project Implementation

4.1.1 Automated Systems Evaluation

After reviewing the automated system several times, it was determined that identifying micro-level causes for manual interventions through the automated planning system data was infeasible and extremely inefficient. Particularly, error codes and other information required significant additional research, systems expertise, and judgment determinations to pinpoint intervention causes. From this conclusion, the effort focused on determining specific causes of manual interventions from the logistics coordinators, who held the best and most up to date information on manual interventions. Here, the determination was to address manual interventions in a bottom-up approach, starting with the logistics coordinators.

4.1.2 Human System Evaluation

Next, several insights resulted from the human system evaluation. Field observations showed that manual planners perform asynchronous, parallel tasks that are extremely difficult to quantify. For example, at any one time, a logistics coordinator could have 10 or more tasks at various stages of completion going. Tasks could be for a variety of actions including cancelling loads, re-planning loads, initial planning of emergency loads, carrier follow-up, client follow-up,