Ph.D. Program
2008
Poster Session
ESS
Characterizing the Enterprise of
MIT
ESD
Military Systems Acquisition
Robb Wirthlin, Ph.D. Candidate
Thesis advisor: W. Seering, Professor
Committee: W. Seering, Professor; S. Widnall, Professor;
D. Lessard, Professor
Start date: September 2005
Research Group: Lean Product Development, Lean
Advancement Initiative
Chronic execution problems for large, complex, systems
acquisitions
•
Department of Defense Acquisition
•
A systemic issue: 30% -
40% cost and schedule overrun on
average (Source: Summary Selected Acquisition Reports,
last 30 years; Biery, Miller & Lessard, others)
Literature suggests Risk is part of the problem
•
Risk is underappreciated in large, complex projects (see
Megaprojects and Risk by Flyvbjerg)
•
The GAO, RAND, IDA and other groups have recently told DoD
this too.
Risk
•
For some, it means “almost anything that can go wrong”
•
Others see opportunities in Risk
•
Still others specify it further: political risk, technical risk,
organizational risk, technology risk, etc.
Is Risk at the Heart of Acquisition Problems?
>
Is risk really the problem? If so, we should manage risk
better, right?
>
And how do we manage risk better? What is the answer?
>
Would using portfolio theory help manage Enterprise risk?
A Representation of the Enterprise of “Cradle to
Grave”
Acquisition in the US Air Force
Conducting grounded research using Social Science Research methods to
characterize the AF Acquisition system. Over 50 interviews of key players
within system talking about risk and portfolios of systems
>
Borrowed ideas and concepts from Value Stream mapping
>
Using commonly accepted understandings of risk; probabilities and
occurrence
>
Represented activity in frame of reference understandable to target
audience: military and civil servants in AF
–
Main unit of measurement is a “program”
–
Restricted to ACAT I, II, and III programs; Limited to Milestone
C and
earlier in Acquisition phase parlance
Motivation / Problem
The Research
Methodology
Key Question(s)
•
Model tacitly accounts for portfolio “interdependencies”
–
a
problem identified in all interviews but deemed impossible to
quantify
•
Model reflects “things as they really are”, not theory
•
Model can be programmed and will lend itself to simulation
exercises and sensitivity analyses
•
Work represents first Enterprise Systems Analysis for Military
Systems Acquisition using this methodology
Preliminary Results
Remaining Research
Wrap Up
Robb Wirthlin
•
Finish the model Verification and Validation, program the model;
run and debug
•
Examine 2 or 3 key questions using the model with priority given
to questions dealing with portfolios and risk
•
For example: What is the overall process yield? What fraction
of time spent is value-added? What is the cost of waiting?
What interventions require the least disturbance to the
existing system?
•
Interview analyses suggests many root causes of Acquisition
anomalous behaviors originate outside of the formal Acquisition
system
•
Concept of managing through portfolios is immature and
portfolio risk understanding is primitive outside Acquisition as
well
Ev ent Hap pen sSc ope G rowth/T ech nic al Pro ble ms?YES T o "Prep are Cou rses of Actio n" i n Acq uis iti on Swim lan e Yes of NO Ta skCo mpl ete?
No
Yes T o Prototy pe Eng in eeri ng Mo de lsF ol low pa th at ap prop ria te stag e Con tra ct Start-up Perform work
Deve lo pme ntal testi ng (Sys tem), Li ve F i re T est, Op erati ona l Ass ess men t Inte grate d tes ting F ab rica tionAsse mbl y Comb ine d T es ting (DT &E, OT &E, LF T &E) Ch ang e Con tra ct/Resc ope e ffo rt (c ost/s che dul e i mpac t) Differen t Di strib utio n pro file and len gth d epe ndi ng u po n path ta ken Adj ust Prog ram an d Con tra ct (co st/sc hed ule esti mate s rev ise d)... Scop e an d Award Sys tem Dev elo pme nt an d Demo nstra tion con tra cts (o r Ao A)
Contrac t Man age men t Are Ob s &Exp OK?F u ndsRedi recte d Pre pare Co urse s of Actio n Pe rio dic Re vi ews
Intern all y or e xterna ll y ge nera ted mul tip le ti mes p er ye ar YesNo
Ye s
Se ek fun ds ? sc op e growth o r tec hni ca l pro ble ms
No Co ntract Cha ng e Re qui red? F u ndi ngSh ortfall PEM /o ther s ta ff fi nd m one ys Ye s
NoYes Obtai nF un ds?(tim ely )NoYes End l oo p Repe at "n" ye ars
Syste ms Eng ine eri ng Syste m Performa nceSpec ific ati on
Pro totyp es/Eng in eeri ngMo del s T o Req uire men ts
Acq ui siti on Pl ann ing Activ itie s Sou rce Sel ecti on Pl ans
Draft RF P Pre para tion RFP c oord ina tion proc ess Pre para tion for Acqu isi tio n Pane lsAc qui si tion Pa nel s MDA Ap prov al?
No
M il eston e C
Yes
RFP rel ea se a nd Sou rce Sel ec tion T o MAJ CO M PEM
Bu dg et in put To Budg et in puts
KPPs fro m Requ ire men ts Protes t?No
Del ay
Yes
Uphe ld?NoYes
Eng in eeri ngRe vi ews?Ye s
Rework a nd De lay
No
Ac qui siti on Pro gram Ba sel ine Afford abi li ty Asse ssm ent F ul l F und in g in FYDP
Acq uis iti on Pan el s Desi gn Re din essRe vi ew?/MDAAppro val F rom Req ui re men ts F rom Pl ann in g & Bu dge ting
No
Sys tems En gi nee ring Prototy pes /Eng ine erin gMod el s
T o Req ui re men ts Sys temVeri fica tionRevi ew?
Yes
Re wo rk an d Del ay
No
YES
In iti al Rate prod ucti onBase li ne Progra m Offic e Cost Esti mate Co ntrac to r Co st Esti mate Inde pen den t Co st Estim ate
T o Eve nt Happ ens step T o Sys te ms En gin eeri ng s tep
T o Eve nt Happ ens T o Sys tems En gin eeri ng Prep arati on for Acq ui siti on Pa nel s
T est Rea dei ne ss Re vie w Re wo rk an d Del ay
Ma ke T rad es? Re work a nd Del ay T o Req ui re men ts T o Requ ire ments
Yes
No
Yes
No
T o Eve nt Happ ens T o Sys tems En gin ee ring T o Ev ent Hap pen s T o Sy stem s Engi ne erin g T o Even t Ha ppe ns s tep
T o Syste ms Eng in eeri ng Step Adv an ced Co nce pts Bud ge t Req ue stMAJCO M PEMDec is ion to pu rsue
Retry? Arc hiv e
M AJCOM Pan elNex t s tep?MAJCO M Grou pNext Step ?M AJ COM Cou nci lNe xt Step?AF l ev el p roce ssNe xt Step?O SD Proc ess Rele ase mon eys to Acqu isi tio n Re pea t n+ 2 tim es
T o Ac qui si tionF a llo ut of Bud get Proc ess i mpa cti ng p lan ni ng
DoD POM?OMB BESCo ngre ssl aw PDM?PBD Y or N Y Go to "Prep are Cours es of Acti on " (Ac qu isi tion Ste p) Y or N
Y
N
Yes - in forma tion onl yYes cha nge s toBES? Y or N
Yes
CCD Cap abi li ty Road map "wish lis t" fo r bud get su bmi ss ionPre pa re Conc ept o f O pera tionT hres hol d Ob jec tiv e T ra de offs Deci si on to p ursu e req ui re men ts (i ndi vi dua l org ) Arc hi ve
No
Dra ft Bri efin g an d ma te ria ls (in di vid ual orga ni zati ons ) "Coo rd " and "Co ncu r" MAJCOM "A" Lette rs Deci de to purs ue Ind ivi du al o rg
No
Up date brie f and materi al s (In div id ual org)Y
N
Up da te and sch edu le cal end ar(i ndi vid ua l org ) Y Pre-RSRMAJCOM A8
N
F in al ize RSR & Cale nda r item s (Ind iv idu al o rg)Mr. Harry Dis browRSR
HQ USAF A8
Y
N
Y
Hi gh Perfo rman ce T ea m (MPT ) wo rk HQ USAF A8 o rg Draft Do cu ment ( CDD) Fo rm High Pe rforma nce T e am (HPT ) HQ USAF A8 org
Inform atio n to Acq uis iti on Sys tem for pre pato ry wo rk
Air Sta ff Pro ces sHQ AF A5RDCriti ca l Comm ents ? HQ AF A5 RD Comme nt Res olu tion MAJCOM org MAJCOM a ppro val ?
Y
Y
N
AF RO C p repa ratio nsMAJCOM o rg AFROC de ci sio n?AF ROCNJo in t inte re st?O SDN
Jo int Proc essJ8
Criti cal Comm ents?J8Y
J ROC Pre para tion s MAJCOM o rg N J ROC d ec isi on ?JROCYN
CPD?Ye s Wa it fortec hn ica l res ul ts
NoPre pare fo r Mil es to ne Re vie w In terac tion with Acq uis iti on
M ore Deta il ed res ul ts fro m Pro totyp es/Eng in eeri ng M ode ls Ac qui si tion Re sul ts Info to Ac qui si tion abo ut CONOPScap ab ili ty "wis h li sts"
Co ntract Start-up Perfo rm wo rk Ev ent Hap pe nsT ech ni cal Pro bl ems ?YESSc ope Growth/
T o "Pre pare Co urse s of Acti on" i n Acq ui siti on Swim lan e
NO
T as kCo mpl ete?
No
Yes
T o SRR Cha ng e Contrac t/Re sc ope e ffort (cos t/s ch edu le i mpa ct) Di ffe rent Dis tribu tion profi le a nd l eng th de pen di ng u pon p ath tak en Adj us t Pro gram a nd Con tract (c ost/sc he dul e es tima tes rev is ed)... Sco pe a nd Award T e chn olo gy De vel opm ent c ontrac ts
Co ntrac t M ana ge ment Are O bs &Ex p OK?Fu nd sRe di re cted Prep are Cou rses o f Ac tion Perio di c Revi ews
In terna ll y or ex terna ll y gen erate d mul tipl e tim es p er ye ar Ye sNo Ye s See k fun ds? s cop e gro wth or te chn ica l p ro bl ems No Contrac t Ch ang e Req ui red? Fu ndi ngSh ortfal l PEM/othe r s tafffind mon eys
Yes
NoYe s Obta inF u nds ?(ti mel y)NoYes End loo p Rep eat "n" y ears
Sys tems En gin ee ring Prefe rre dSys temConc ept
Sys tem Perfo rm anc eSpe cifi ca tionT o Req ui re men ts Acqu is itio n Pla nni ng Ac tivi ties
So urce Se le ctio n Pla ns Draft RF P Prep arati onRF P co ordi nati on p roce ss Prep arati on fo r Ac qui si tion Pa ne ls Acqu is itio n Pane ls
MDA Appro val ?
No
Mil esto ne B
Yes
RF P re lea se a nd So urce Se lec tion T o M AJ COM PEM
Budg et in put T o Bu dge t inp uts KPPs from Req uire men ts Pro test?No De lay Yes Up hel d?No Yes SRR? Yes Rework a nd Del ay No
Acqu is itio n Progra m Base li ne Afforda bil ity As ses sme nt F u ll F u ndi ng i n F YDP
T o Eve nt Ha pp ens T o Ev ent Hap pen s ste p Ad van ced Co nc epts Bu dge t Re que stMAJ CO M PEMDe cis io n to p ursue
Retry? Arc hiv e
MAJCOM Pan elNex t step ?MAJ CO M Grou pNex t Ste p?MAJCOM Cou nci lNext Step ?AF le ve l proc es sNex t Ste p?O SD Proc ess Rele as e mon eys to Acq ui siti on Re pea t n+ 2 tim es
To Ac qui si tionFa ll out of Bud ge t Proc ess imp acti ng p lan ni ng
Do D PO M?OM B BESCong ressl aw PDM?PBD Y or N Y Go to "Prep are Co urse s of Acti on" (Ac qui si tion Ste p) Y o r N
Y
N
Yes - i nforma tio n on ly Yescha ng es toBES?
Y o r N
Ye s
DraftCCD KPP Dev elo pme nt De ci sio n to p ursu e requ ire men ts (in di vid ua l org) Arch iv e
No
Draft Brie fing and mate rial s (ind iv idu al orga niz atio ns) "Coord " a nd "Con cur" MAJCOM "A" Le tters De ci de to p ursu eIndi vi dua l org
No
Up date b rie f a nd m ateri als (Ind ivi du al o rg)Y
N
Up date a nd s ch edu le c ale nd ar(in div id ual org) Y Pre-RSRM AJ COM A8
N
F in ali ze RSR & Ca le nda r i tems(Indi vi dua l o rg )RSRMr. Ha rry Disb row HQ USAF A8
Y
N
Y
Hig h Perform anc e T eam (MPT) work HQ USAF A8 o rg Draft Docu men t (CDD) F o rm High Pe rfo rman ce T ea m (HPT) HQ USAF A8 o rg
Info rmati on to Ac qui si tion Sy stem for p re pa to ry work Air Sta ff Pro ces s HQ AF A5RD Criti cal Comme nts?HQ AF A5RD
Co mme nt Reso lu tion MAJCOM o rg MAJCOM a ppro val ?
Y
Y
N
AFRO C p repa ra tio nsM AJ COM orgAF ROC de ci sio n?AF ROC
N
Jo int i ntere st?O SDN Joi nt Proc essJ8
Criti cal Comme nts?J 8Y
J RO C Pre parati on s M AJCOM org N J RO C d eci si on?J RO CYN
CDD?Ye s Wa it fortec hni ca l res ults
No
Pre pare fo r M il esto ne Rev ie w Prel imi nary Re sul ts from Ac qui si tionMo re Acq uis iti on Res ults
Pe rson ne l from Acq ui siti on o ffice wo rki ng o n this parti cul ar ac tiv ity
Co ntrac t Sta rt-u p Perform work Eve nt Happ ens
Sc ope Growth/ T e chn ica l Prob lem s?YES T o "Pre pare Co urse s of Acti on" i n Acq ui siti on Swim lan e YES an d NO
T askCom ple te?
No
Yes
To Pre ferred Sy stem Con ce pt Chan ge Co ntract/Res cop e effort (co st/sc hed ul e im pac t) Differe nt Distri buti on p rofil e an d le ngth dep end ing upo n pa th take n Adju st Progra m an d Contrac t (cos t/sch edu le e sti mates revi sed )... Pre pare for Ac qu isi tio n Pane ls
Ac qui si tion Pa ne l (DAB, MDA, ADM , e tc) Scop e an d Award Study con tracts (o r Ao A)
Con tra ct Ma nag eme nt Are Ob s &Exp O K?Red irec tedF un ds Prepa re Cours esof Acti on
Pe riod ic Re vie ws Inte rnal ly o r exte rn al ly g ene rated mu lti ple time s pe r y ear YesNo
Yes
Seek fu nds ? sc ope growth or tech ni cal prob lem s
No
Contrac t Ch an geRequ ired ? F un din gSho rtfa ll PEM/othe r staff fi nd mo ne ys
Ye s
NoYes
O btai nF und s?(time ly )NoYes End l oop Re pea t "n" yea rs
Sy stem s Engi ne erin g Be st Ma te ria lAp proa che s Data fromUser Stud ies Pre li min aryIn tegra tedArch etic ture
Preferre dSy stem Co nce pt T o Req ui re men ts
Ac qui si tion Plan nin g Activ iti es Sou rc e Sel ecti on Pl ans
Draft RF P Prep arati on RF P c oo rd in atio n pro ces s Prepa ratio n for Acq uis iti on Pan els Ac qui si tion Pa nel s
M DA App rova l?
No
Mil es tone A
Ye s
RF P re lea se a nd Sou rce Sel ec tion T o MAJCOM PEMBu dge t inp ut
T o Bud get i npu ts
T o Ev ent Hap pen s ste p Ap prov al?
Y
N
Pre pare Pro gram b udg et de cis io n (PBD) F rom Re qui reme nts s ystem : Dra ft d ocu men t fo r bas is o f wo rk
Adv anc ed Co nce pts Bud get Req ues tMAJCOM PEMDeci si on to pu rsue Re try ? Archi ve
M AJ COM Pan elNe xt step ?M AJCOM Gro upNext Step ?MAJCOM Co unc ilNe xt Step? AF le vel proc essNe xt Ste p?Pro gram min g Iss ues &
Bu dge ting He ari ngs DoD PO M?OSD
BESO MB & Con gres sla w
Re lea se m one ysto Ac qui si tion Re pea t n + 2 time s
T o Acqu is itio nF all ou t o f Bu dge t Pro ces s im pac ting pla nni ng
PDM?PBD Y or N
Y
Go to "Prep are Co urse s of Acti on" (Ac qui si tion Ste p) Y o r N
Y
N
Yes - info rmatio n on ly Yesc han ge s toBES?
Y o r N
Yes
Ye s
No
Dec is ion to pu rs ue re qui reme nts(ind iv idu al o rg) Archi ve
No
Dra ft Brie fing and mate rial s (ind iv idu al orga niz atio ns) "Coo rd " and "Co ncu r"MAJCOM "A" Le tte rs
De cid e to pu rsue In di vid ua l org
No
Up date b rief a nd m ateri als (Ind ivi du al o rg)Y
N
Up date and sch edu le c al end ar (in di vid ual org) Y Pre -RSRMAJCOM A8
N
F in ali ze RSR & Ca le nda r items (In di vid ual org) RSR Mr. Harry Di sbro wHQ USAF A8
Y
N
con du ct Ao A?(i ndi vi dua l org )
Y
No, bec au se o f wo rk alre ady don e Hig h Performa nce T eam (M PT ) workHQ USAF A8 orgDra ft Do cum ent (ICD o r draft CDD) F orm Hig h Performa nce T eam (HPT )HQ USAF A8 org De tail s of Prefe rre d Sys tem Con cep t (Pre lim ina ry) Air Staff Proce ss
HQ AF A5 RD Criti cal Comm ents?HQ AF A5 RD Comme nt Reso lu tionMAJ CO M org MAJ CO M app rova l?
Y
Y
N
AF ROC prep arati onsMAJCOM o rgAF RO C d eci si on?AFROC
N
Joi nt in teres t?OSDN J oin t Pro ces sJ8 Criti cal Comm ents? J 8Y
J ROC Pre para tion s MAJ CO M org N JROC dec is ion ?J ROCY
N
ICD?
Ye s
No
F und ing ava il abl e? (i ndi vi dua l o rg )Y Cond uct Stud y or Ana lys is(ind ivi du al o rg)
N
High Performa nce Te am (MPT ) workHQ USAF A8 orgDraft Doc ume nt (ICD) F orm Hig h Performa nce Te am (HPT )HQ USAF A8 org Ob tain fund ing from Ad va nce d Conc epts Bra nch
In forma tion to Ac qu isi tio n Syste mfor pre pato ry work De cis io n to pu rsue re qu irem ents
(i ndi vi dua l org ) Arch ive
No
Dra ft Bri efin g an d mate rial s (in div id ual orga niz ati ons ) "Coo rd" an d "Co nc ur" MAJCOM "A" Le tters De cid e to p ursu e Indi vi dua l org
No
Upda te bri ef an d mate rial s(Indi vi dua l org )Y
N
Up da te and sch edu le cal end ar (i ndi vid ua l org )
Y
Pre-RSRMAJCOM A8
N
F in al ize RSR & Cale nda r item s(In di vid ua l org) RSRMr. Harry Dis brow HQ USAF A8
Y
N
Re qui reme nts Proc ess Inp uts from vari ous sou rces F or e xis tin g prog ram? (Req uire men ts Poli cy sho p)
No
Extern al i npu t Ro ute to p rope r orga niz atio n(Requ ire men ts Po li cy Sh op) Ro ute to Adv anc ed c on cep ts (Re qu irem ents Po li cy Sho p) O ther in pu ts Pro ce ss Start
In sc ope of exi sti ng d ocu men ts? (in di vid ua l org)
N
Put in to mod erni za tion pla nni ng and sen d to Acq ui siti on(ind iv idu al org)
Y
Requirements Swimlane
Budgeting and Programming Swimlane
Acquisition
Contractors
to Acq ui siti on M ode rniz ati on/Sus tain men t Ac tivi tyTh resh ol d cros sed ?M DAP Co nti nue
No
Yes
Budgeting and Programming Swimlane
Acquisition
Contractors
Budgeting and Programming Swimlane
Acquisition
Contractors Requirements Swimlane Requirements Swimlane Funding available?
(individual org)
Conduct Study or Analysis (individual org) Y N RSR Mr. Harry Disbrow HQ USAF A8 N
