ValidatedRetrieval
inCase{BasedReasoning
Ev angelos Simoudis James Miller
DigitalEquipmentCorporationCambridgeResearchLabCRL90/2December12,1990
Abstract
Wecombinesimpleretrievalwithdomain-specicvalidationofretrievedcasestoproduceausefulpracticaltoolforcase-basedreasoning.Basedon200real-worldcases,weretrievebetweenthreeandsixcasesoverawiderangeofnewproblems.Thisrepresentsaselectivityrangingfrom1.5%to3%,com-paredtoanaverageselectivityofonly11%fromsimpleretrievalalone.cDigitalEquipmentCorporation1990.Allrightsreserved.
1
1 In tro duction
Wehavecombinedsimpleretrieval(basedonthesimilarityofsurfacefeatures)withdomain-specicvalidationofretrievedcasestoproduceausefulpracticaltoolforcase-basedreasoning.Startingwithacasebaseof200real-worldcases,wehavenarrowedourconsiderationtobetweenthreeandsixcasesoverawiderangeofnewproblems.Thisrepresentsaselectivityrangingfrom1.5%to3%,comparedtoanaverageselectivityofonly11%fromthissamecasebaseusingretrievalwithoutvalidation.Weareapplyingthesametechnologytoalargercasebaseinadierentdomain,andhavedeployedarelatedtoolwithamuchlargercasebaseforactualuseintheeld.Ourworkbeginswithareal-worldproblem:acomputermanufacturer'sdiagnosisofsystemsoftwarefailures.Inthisdomain,diagnosticknowledgeexistsinseveralforms:manuals,courses,productionrulesystems,andknowl-edgebases.Butthepredominantstartingpointincurrentuseisasetofdatabasescreatedbyrecordingsuccessfullydiagnosederrorconditions.Inordertodiagnoseanewfailure,non-expertspecialistsretrievefromadatabaseprevi-ouslysolvedcasesthatappearsuperciallysimilartothenewproblem.Theythenattempttoverifythesimilaritybyperformingtestsonthenewprob-lemandcomparingtheresultswiththoseofeachretrievedcase.Whentheybecomeconvincedthatapreviouscaseissubstantiallythesameasthenewproblem,theyexaminetheresolutionoftheoldcaseandreportit(possiblyamendedoreditedtomorecloselytthenewproblem)tothecustomer.Onlyinrarecasesareexpertsrequestedtoexamineproblems|mostareresolvedfromtheexistingdatabase|andthesolutionsarethenaddedtothedatabase.Thisexistinghumansystemisaconscioususeofcase-basedreasoning(CBR)techniqueswehaveimprovedthesystembyaddingtoitanautomatedtoolusingresultsfromAIcase-basedreasoningsystems.Inordertoproduceatoolofpracticalvaluewewereforcedtoexaminemorecloselythetaskofretrievalincase-basedreasoning.Basedonourexperienceweproposeanex-tensiontocurrentsystems,validatedretrieval,thatdramaticallyreducesthenumberofcasespresentedtothereasoningcomponent(humanorautomated)ofacase-basedsystem.Validatedretrievalreliesondomain-specicknowl-edgeabouttestsusedtocomparecasesretrievedfromthecasebasewithnewlypresentedproblemcases.Knowledgeabouttherelationshipsamongthevarioustestsiscapturedinavalidationmodelwhichweimplementasase-22RETRIEVALINCBR
manticnetwork8].Inordertobuildourvalidationmodelwearefacedwithaclassicknowledgeacquisitiontask.Byperusingexistingdatabasesusedbyspecialistsweareabletoacquirethisknowledgewithareasonableamountofeort|andwithonlyasmallinvestmentofspecialists'time.
2 Retriev al in CBR
CBRsystemsrstretrieveasetofcasesfromacasebaseandthenreasonfromthemtondasolutiontoanewlyposedproblem.Existingsystems(1],2],3],4],9]and10])maketwoassumptionsabouttheinitialretrievalofcasesfromthecasebase:1.Veryfewcaseswillberetrievedfromthecaselibrary.2.Theretrievedcasesarerelevanttotheproblembeingsolved.Inmanypracticalapplications,retrievalaloneissucienttosolvethedif-cultpartofatask.Forexample,inourdomainofdiagnosisofcomputersoftwarefailures,specialistscaneasilyrespondtocustomerproblemsiftheycanquicklylocateafewsimilarcasesfromtheircollectivepastexperience.Forthisreason,wehaveconcentratedontheretrievalaspectofcase-basedreason-ing.InMBRTalk10],also,theessentialtaskisretrievalthe\reasoning"componentconsistsofmerelypassingtheretrievedinformationdirectlytoanoutputunit.2.1 Related W ork
ClosesttoourownworkistheworkofKotononcasey5],aCBRsystemwhichhasbeenappliedinthedomainofmedicaldiagnosis.caseyhasajusticationcomponentwhosegoalistodeterminewhetherthecausalexpla-nationofaretrievedcaseappliestoanewproblem.Thisfrequentlyallowscaseytoavoidinvokingitscausalmodelwhencreatinganexplanationforanewcase.casey'sjusticationphaseissimilartoourvalidationphase.Butthereisanimportantdierencebetweenthesetwosystemsarisingfromdierentassumptionsabouttests.caseyreliesonpreciselytwotests(EKGandX-rays),bothofwhichareinexpensiveandnon-invasive.Bothofthesetestsareperformedpriortotheretrievalphaseandtheresultsareusedtoprovidesurfacefeaturesfortheretrievalalgorithm.Bycontrast,thereare2.2TwoPhases:RetrievalandValidation3
literallyhundredsofteststobeperformedinourdomainsanditisfartooexpensivetoperformalloftheminadvanceofinitialcaseretrieval.Asaresult,oursystemsdevoteattentiontominimizingthenumberofteststhatareperformed.Wenotonlyperformtestsincrementallyandcachetheresults,butalsoemployknowledgeabouttheteststhemselvestoreducethenumberoftestsperformed.TheCBRsystemchef2],whosedomainisChinesecooking,alsohasajusticationcomponent.Inordertojustifyeachretrievedcase,chefusesbackwardchainingrules.Whileourvalidationmodelisnotappropriatetothisdomain,wehaveexaminedandrejectedtheuseofrulesforourvalida-tionmodels.Thisdecisionisbasedonthedicultyofacquiringtheexpertknowledgeneededtocreatelargerulesets,especiallyincomparisontothesimplicityofconstructingourvalidationmodels.Thesemodelsarecapturedbyasemanticnetworkthatrepresentsgroupsoftestsandinformationabouttherelationshipsbetweenthegroups.Furthermore,aswithcasey,chefdoesnotusetheresultsofcomparisonswithearliercasestopruneitssearchforrelevantcases.Theswale6]systemconcentratesprimarilyonmodifyingtheexplanation(containedinanexplanationpacketorXP)ofaretrievedcasetomatchanewproblem.Nonetheless,ithasasubcomponent,xpaccepter,thatjustiestheapplicationofaretrievedXPtoacurrentsituation.TheaccepterveriesanXPbydeterminingifitcanbelievetheapplicabilitychecksthatarepackagedwitheachXP.Eachsuchtestissimilartothetestsassociatedwithourvalidationmodel.Becauseofthesmallnumberofcases(eight,byourcount)xpaccepterneveraddressedtheissuesofscalewhichareourmajorconcern.Thus,swaleneverdevelopedajusticationmodeltorelatethevariousapplicabilitycheckstooneanother.
2.2 Tw oPhases: Retriev al and Validation
Ourgoalistotakeasizablepre-existingcasebasealongwithanewproblemandproduceasmallnumberofrelevantcases.Likeourhumanspecialists,oursystemsperformdiagnosisintwophases:Retriev al:
itposesaquerytothecasebaseusingasubsetofthefeaturesthatdescribethenewproblem.43THEVALIDATIONMODEL
Validation:
itfollowsthevalidationprocedurefromeachretrievedcasetodetermineifitappliestotheproblemathand.Thegoaloftheretrievalphaseistoextractfromthecasebasethosecasesthatappeartoberelevanttothenewcase.Sincethecasebaseislarge,andwehavebeeninterestedprimarilyinsequentialimplementations,itisimportantthatthecasebasebeorganizedinawaythatpermitsecientsearchbasedonsurfacesimilarities.Forthisreason,weorganizethecasesintoageneralizationhierarchy(usingunimem7]).Theretrievalphaseconsistsoftraversingthegeneralizationhierarchytondaclosematchtothenewproblem.Theresultofthistraversaliseitheranindividualcase(aleafnode)orasetofcases(aninternalnodeinthehierarchy,returnedasallofthecasesindexedunderthatnode).Unlikethosesystemsthatrelyexclusivelyonunimemforcaseretrieval,wedon'tnetuneunimemtoreducethenumberofcasesretrieved.Thevalidationphasethenconsiderseachoftheretrievedcasesandat-temptstoshowthatthecaseisrelevanttotheproblemathand.Associatedwitheachcaseinthecasebaseisasetoftestsandtheirresultvaluesthatmustbemetforthestoredcasetobevalid.Wecallthesetoftestsandvaluesavalidationprocedure,andeachelementofthisset(i.e.asingletest/valuepair)iscalledavalidationstep.Thetestsareappliedtotheactualproblemandtheresultsarecomparedwiththeresultsinthecase.Basedonthiscomparisonboththecurrentcaseandotherretrievedcasescanberemovedfromfurtherconsideration.Onlywhenallofthetestsforagivencasearesuccessfullymatchedagainstthecurrentproblemisthecasereportedasacandidateforareasoningcomponent'sconsideration.(Inotherdomains,itmaybepossibletoassignweightstoindividualtestresultsanduseathresholdoraveragingschemefordecidingwhetherornottorejectthecase.)3 The Validation M odel
Thevalidationphaseofourmethodisstraightforwardiftheindividualvalida-tiontestsaresimpleandself-contained.Unfortunately,inourdomains,andprobablyinmostreal-worlddomains,thisisnotthecase.Ineachdomainwehavestudied,wehavefoundthatthetestsareinterrelatedinawaythatisnotevidentinadvance,andwehavebeenforcedtofacetheknowledgeacquisi-tiontaskhead-on.InSection3.2wedescribeamethodologyforacquiringthisknowledgeabouttests.Wehavesuccessfullyusedthismethodologytodevelop3.1Whatisavalidationmodel?5 validationmodels:structuresthatcapturemuchofanexpert'sknowledgeinawaythatmakesiteasyforthevalidationphasetoprocessthetestsitrequires.
3.1 What is avalidation mo del?
Ratherthanrequireacompleteandaccuratedescriptionofeachtestusedbythespecialist,wecapturetheoverallstructureofthetestspaceitself.Theresultingstructure,ourvalidationmodel,consistsofrelatedgroupsoftestsandinformationabouttherelationshipbetweenthegroups.Forexample,ifwewanttoknowwhyahouseishot(theproblem),wemayrstwanttoseeiftheairconditionerisworkingbutthisrequiresustondoutifthehousehasanairconditioner.Inthisexample,thedesiredtest(istheairconditionerworking?)isrelatedtoanothertest(isthereanairconditioner?).Thisknowledge,aswellasknowledgeabouttheimportantoutcomesandimplicationsofatest,iscapturedbythevalidationmodel.Wehavechosentorepresentthisknowledgeintheformofasemanticnetworkwhosenodescorrespondtosetsoftestsandwhosearcsindicaterelationshipsbetweenthesesets.3.2 Creating aV alidation M odel
Webuildourvalidationmodelsbyrstexaminingexistingdatabasesthatareusedbyhumanspecialists.Thesedatabasesmaybeeitherformalized(asinthecaseofourWPS-PLUS 1system)ormerelyinformalnotespreparedbythespecialistsfortheirownperusal(asinthecaseofourVAX/VMSsystem).Inourtwocase-basedsystems,theexistingdatacontainsatextualdescriptionofthestepsthatthespecialistsusedtoverifyahypotheticalexplanationoftheproblem.Inconstructingthevalidationmodel,itisourgoaltocapturetheinterrelationshipsbetweenthevalidationtests.Asaresult,wehavebuiltvalidationmodelsthatcorrespondtoaparticularcasebaseby:1.Readingthevalidationproceduresofeachcaseandbuildingalistofallthevalidationstepsusedintheentiredatabase.Intheprocessofreadingthedatabaseandpreparingthislist,theimplementordevelopsasenseoftheunderlying(butunstated)relationshipsbetweenteststhatarementionedinthedatabase.1DEC,VAX,VMSandWPS-PLUSareregisteredtrademarksofDigitalEquipmentCorporation.
64ANEXTENDEDEXAMPLE
2.Examiningtheresultinglist,lookingforgroupsofteststhatappeartoformrelatedsets.Organizingthelistprovidesabasisfordiscussionwithdomainexperts,whohelp\debug"theproposedorganization.3.Reningthestructureofthelistthroughknowledgeacquisitionsessionswithdomainexperts.Duringthesesessions,signicantrangesoftestresultsareidentied,asareinferencesfromtheseresultsthateliminatetheneedtoperformothertests.Thatis,adependencygraphbasedontestresultsisdeveloped.4.Iteratingtheabovetwostepsafterconsultingadditionalinformationsuchasmanualsandcodedocumentation.Thestructureofthedomainbecomesclearerateachiteration.(Wehavefoundthatthreeiterationsaresucienttoproduceausefulstructuring.)Thenalvalidationmodelconsistsprimarilyofentriescorrespondingdirectlytoinformationthatappearsintheoriginaldatabase.5.Integratingthetestsetsintothestructurederivedinthepreviousstep.Thisintegrationmakesexplicittheprerequisitesofeachtest,aswellasprovidingalternativewaysofobtaininginformationordinarilyprovidedbyaparticularcriticaltestincaseswherethattestcannotbeperformed.
4 An Extended Example
Inordertounderstandthevalidatedretrievalprocess,considerthefollowingexample.Ourdomainisautomobilediagnosisandrepair,andweassumeanexistingcasebasewithitsassociatedvalidationmodel.Wearegiventhefollowingcase:NEW CASE mak e
:MAZDAmo del
:626mo del year
:1985engine typ e
:2.0LEFImiles
:50,000problem
:enginedoesnotstart.Theretrievalphaseusesthemake,model,problem,andapproximateyearofmanufacturetosearchthroughacasebaseofpreviousautomobileproblems.Basedonthesesurfacefeatures,weretrievethreecasestobevalidatedbeforepresentationtoareasoningcomponent:7
CASE 1 mak e
:MAZDAmo del
:626mo del year
:1988engine typ e
:2.0LEFImiles
:10,000problem
:enginedoesnotstart.validation
:Thefuelinjectorwasclogged.Fuelwasnotdeliveredtothecombustionchamberfortheenginetoignite.Forthisreasontheenginecouldnotstart.solution
:cleanedthefuelinjector.CASE 2 mak e
:MAZDAmo del
:626mo del year
:1984engine typ e
:2.0Lmiles
:60,000problem
:enginedoesnotstart.validation
:Thecarhadafaultygaspump.Fuelcouldnotbede-liveredtothecombustionchamber.Forthisreasontheenginecouldnotstart.solution
:Replacedthegaspump.CASE 3 mak e
:MAZDAmo del
:626mo del year
:1987engine typ e
:1.8Lmiles
:20,000problem
:enginedoesnotstart.validation
:Aleakexistedinthegasline.Fuelcouldnotbedeliveredthroughthefuelline.Forthisreasontheenginecouldnotstart.solution
:Fixedtheleak.Thevalidationmodelcontains(atleast)thethreeteststhatarereferencedbythesecases:\checkifafuelinjectorisclogged",\checkifthegaspumpisworking",and\checkifthereisaleakinthefuelline".Therstoftheseisactuallycomposedoftwosimplertests:atestforfuelpresentinthereservoiroftheinjectorandatestforfuelexitingtheinjector'snozzle.Ifthereisnofuelintheinjectorthenwecandeducethattheinjectorisnotatfault.Rather,theproblemliesearlierinthefuelsystem|eitherinthepumporthefuelline.ThesystemrstattemptstovalidateCase1byrepeatingthevalidationstepsfromthatcase.Thatis,wewishtotestifthefuelinjectorisclogged.Intheprocessofperformingthistwo-steptestweactuallyacquireknowledgethatisrelevanttoCases2and3:ifthefuelreservoirisnotemptywecaneliminatebothcasesifitisempty,wecaneliminateCase1.Thisrelationship85RECENTRESULTS
isencodedinthesemanticnetworkthatrepresentsourvalidationmodelandisusedinthevalidationphase.Inthebestcase,thisvalidationmodelallowsustoreducetheworkrequiredtovalidatecasesfromfourteststotwotestsandsimultaneouslyreducesthenumberofcasestobeconsideredbythereasonerfromthreetoone(selectivityof33.3%).ThersttestisforanemptyfuelreservoirifthereservoirisfullthenCases2and3areeliminated.Wethentestthenozzleforfuelexiting.Ifnofuelleavesthenozzle,thenCase1ispresentedtothereasonerbutiffuelisleavingthenozzlewe,unfortunately,eliminateCase1aswellandleavethereasoningcomponenttoitsownresources.Theworstcaserequiresallfourtestsandprovideseitherzerooronecasetothereasoner.
5 Recen tR esults 5.1 An Op erating System: VMSTherstsystemwedevelopedisusedforthediagnosisofdevicedriverinducedcrashesofDigital'sVMSoperatingsystem.Theknowledgeaboutsurfacefea-tureswasobtainedprimarilyfromDECinternalpublicationsandwascom-plementedbyanexpertfromtheVMSsupportteamduringthreeknowledgeacquisitionsessions.Ittookatotalof84hourstoacquirethedomainspe-cicknowledgeaboutsurfacefeatures.Basedonthisinformation,thedomainknowledgeusedbyunimeminordertoorganizethecasesintoageneralizationhierarchywasimplementedinvedays.Ittookanadditionalfourdaysofreadingvalidationproceduresinthedatabasetodevelopavalidationmodelfordevicedrivers.Inaddition,fourmoreknowledgeacquisitionsessions,lasting40hours,wereneededtoreneandimprovethevalidationmodel.Encodingtheactualvalidationmodeltookabout80additionaldays.Thetotalnumberofdaysspentonknowledgeacquisitionanddevelopmentisshownbelow:ActivityPersonDaysKnowledgeacquisition20Development85Sincethiswasourrstattempttobuildacasebaseandvalidationmodel,thesenumbersaremuchlargerthanweexpectforsubsequentsystems.Our
5.2AWordProcessingSystem:WPS-PLUS9worktodateonthesystemdescribedinSection5.2appearstoconrmthisexpectation.Thesystemwasevaluatedusingacasebaseof200casesthatwereobtainedfromnoteswrittenbyspecialists.Thesurfacefeatureretrievalphaseofthesystemwasevaluatedbypresentingeachofthe200casestotheretriever(asnewproblems)andpreparingahistogramofthenumberofcasesretrieved.unimemprovidesamechanism,knownasretrievalweights,fortuningitsre-trievalcapabilities.Aftersomeexperimentation,wediscoveredthattheuseoflargerretrievalweights(i.e.morestringentmatchingcriteria)causedthere-trievertomissmanyrelevantcasesand,inmanyoccasions,tofailtoretrieveanycasesatall.Withlessstringentcriteriathisproblemwasrectied.How-ever,manyoftheretrievedcaseswerenotrelevanttotheproblem.Withtheoptimalweighting,wewereabletoretrieveonaverage22casesperretrieval(11%).Thevalidationphase,however,wasabletoreducethisnumberofcasestoanaverageof4.5casesoutof200(2.25%).Inaddition,wepresentedthreenewcasestothesystem.Basedonsurfacefeaturesalone,weretrieved20,25,and16cases(10,12.5,and8%selectivity).Thevalidationphasereducedthisto3,5and3cases,respectively(1.5,2.5,and1.5%selectivity).Ourexpertsconrmthatthesevalidatedcasesaretheonlyonesrelevanttotheproblemspresented.
5.2 AW ord Pro cessing System: W PS-PLUS
Thesecondsystemperformsdiagnosisofcustomerproblemswiththewordprocessingcomponentofanoceautomationproduct.During15hoursofknowledgeacquisitionsessions,theknowledgeaboutsurfacefeatureswasob-tainedfromasupportengineerfortheproduct.Itthentookanadditionalvedaystoencodethedomainknowledgeforusebyunimem.Thevalidationmodelwasobtainedfromthevalidationproceduresofthecasesinthedatabase,aninternalpublication,and10hoursofknowledgeacquisitionwiththesameengineer.Whiletheworkisnotyetcomplete(only50outof340caseshavebeenencoded),ithastakenonly10daystoimplementthevalidationmodel.Thissystemisstillunderdevelopment.However,thetimewespentonknowledgeacquisitionanddevelopmentisshownbelow:106SCOPEOFWORKActivityPersonDaysKnowledgeacquisition5Development10Thissystemwasevaluatedusingacasebaseof340cases.RepeatingthesameexperimentperformedwiththeVMScasebaseledtoanaverageof26casesperretrieval,or7.6%selectivity.Thevalidationphasereducedthistotwocases,or0.58%selectivity.Sincethevalidationmodelforthiscasebaseisnotyetfullyencoded,wehavenotpresentednewproblemstothesystem.
6 Scop eof W ork
Ourvalidatedretrievalmethodcanbeappliedinmanytypesoftasks.Thebasicrequirementsare:anexistingdatabaseofpreviouspracticalexperi-enceasetofquickteststhatservetoreducethesearchspaceatlowcostasetofmoreexpensiveteststhatcanfurtherreducethesearchspaceandanunderstandingoftherelationshipsbetweentheexpensivetests.Wehaveidentiedfourareasofpotentialinterest,butwehavelimitedourimplementationworktotherstofthese:Diagnostic tasks
.Asshownintheexample,weusethesymptomsofaproblemasthesurfacefeaturesfortheretrievalphase.Thevalidationproceduredescribeswhichteststoperforminordertodetermineifthecaseisrelevanttothenewproblem.Design tasks
.Thesurfacefeaturesarespecicationsthatadesignmustsatisfy.Thevalidationproceduresverifythataproposeddesignmeetsthespecication.Sales tasks
.Thetechniquecanbeusedtohelpidentifysalesprospectsforanewproduct.Thesurfacefeaturesarethecharacteristicsofacustomersuchas:sizeofbusiness,typeofbusiness,locationofbusiness.Eachvalidationproceduredescribesthecustomer'srequirementsthatweresatisedinaprevioussale.Thevalidationmodelincludesteststhatdeterminewhetherornotacustomerneedsaparticulartypeofproduct.11
Managemen ttasks
.Thesetasksincludeaccounting,creditanalysis,investmentdecisions,andinsuranceunderwriting.Ineachofthesear-eas,specialistscanidentifyeasilyrecognizedfeaturesintheirproblemdomain(typeofcompany,size,etc.)thatallowrapidretrievalofsimilarsituationsencounteredinthepast.Theythenhavemoredetailedteststhatcanbeapplied(debt/equityratio,paymenthistory,typeofclient,balancesheets,etc.).7 Conclusions
Ourworkhasconcentratedexclusivelyontheissueofcaseretrieval.Acare-fulstudyoftwoapplicationsinwhichpeopleconsciouslyusecaseretrievalhasshownthatretrievalbasedsolelyonsurfacefeaturesisnotsucientlydiscriminatingforusewithlargecasebases.Itresultsinlargenumbersofcasesreturnedtothereasoner,eachofwhichmustthenbefurtherexaminedatgreatexpense.Addingavalidationphasethatusesknowledgeofdomain-specicteststoprunetheretrievedcasesdramaticallyreducesthenumberofcasesthatmustbeexaminedbythereasoner.Wehavefoundthatacquiringknowledgeaboutdomain-specictestsisaidedbyaninitialperusaloftheexistingdatabaseusedbyspecialists.Withareasonableamountofeort,andwithonlyasmallinvestmentofspecial-ists'time,thisinformationcanbecapturedinavalidationmodelrepresentedasasemanticnetwork.Wehaveusedthismethodologytoproducetwosys-tems.Oneofthesesystemshasbeensuccessfulinpractice,andtheother(incomplete)systemislikelytobeequallyuseful.Whiletheburdenofknowledgeacquisitioninourmethodologyissmallcomparedwithothermethods,itisnotnegligible.AutomatingthisworkbycombininganaturallanguagesystemtoanalyzeexistingdatabaseswithAI-assistedstatisticalcomparisonofsurfacefeaturesprovidesafertileareaforfurtherinvestigation.Furthermore,wesuspectthatacarefulstudyofsuchasysteminpracticewillrevealvalidationteststhataresucientlycommonthatitmaybereasonabletopromotethemtosurfacefeatures,leadingtoasystemwithbetterretrievalcapabilities.Thisanalysis,whichmustrstbeperformedmanuallytovalidateourassumption,isitselfanexcellentareafortheapplicationAImethods.Ac kno wledgemen ts
127CONCLUSIONS
TheauthorswishtothankProf.DavidWaltzforhishelpwiththisresearch.Inaddition,wehavereceivedhelpfulcommentsfromMarkAdler,AndrewBlack,DaveHanssen,RoseHorner,andCandySidner.Wearealsoindebtedtotheengineerswhohavegiventheirtimeandknowledgetohelpusunderstandthetwodomainswehavestudied:theDigitalSupportEngineersatColoradoSprings,ColoradoandSpitbrook,NewHampshire.
REFERENCES13
References
1]RayBareiss,KarlBranting,andBrucePorter.Theroleofexplanationinexemplar-basedclassicationandlearning.InProceedingsofCase-BasedReasoningWorkshop,1988.2]KristianHammond.Case-BasedPlanning:AnIntegratedTheoryofPlan-ning,Learning,andMemory.PhDthesis,YaleUniversity,1986.3]JanetL.Kolodner.Reconstructivememory:Acomputermodel.CognitiveScienceJournal,7:281{328,1983.4]JanetL.Kolodner,Jr.RobertL.Simpson,andKatiaSycara-Cyranski.Aprocessmodelofcase-basedreasoninginproblemsolving.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,1985.5]PhyllisKoton.Usingexperienceinlearningandproblemsolving.PhDthesis,MassachussettsInstituteofTechnology,1988.6]DavidLeake.Evaluatingexplanations.InProceedingsoftheSeventhNationalConferenceonArticialIntelligence,1988.7]MichaelLebowitz.Experimentswithincrementalconceptformation:Unimem.MachineLearning,2:103{138,1987.8]M.R.Quillian.Semanticmemory.InMarvinMinsky,editor,SemanticInformationProcessing,pages227{270.MITPress,1968.9]EdwinaRisslandandKennethAshley.Hypotheticalsasheuristicdevice.InProceedingsoftheFifthNationalConferenceonArticialIntelligence,1986.10]CraigStanllandDavidWaltz.Towardmemory-basedreasoning.CACM,29:1213{1228,1986.14ACREATINGTHEVMSVALIDATIONMODEL
A Creating the VMS Validation Mo del
Thisappendixdescribes,indetail,themethodusedtoconstructthevalida-tionmodelfortheVAX/VMSdevicedriverdiagnosissystem.EachsubsectioncorrespondstooneofthestepsdescribedinSection3.2.Forexpositorypur-poses,ofcourse,wehaveremovedanumberofintermediatestepsandshowonlyselectedresults.A.1 Reading the Initial Data Base
WebeginwiththedatabaseincurrentusebyCustomerSupportSpecialists.Thisdatabase(actually,aninformaltextual\notesle")consistsofabout150entries.AtypicalentryisshowninFigure1.Atthisstagewenotonlyreadtheseentries,butwetidythemupabit.Forexample,someoftheseentrieswereincompleteandweremanuallyexpandedintomultiplecasesforourcasebase,resultingin200entriestobestudied.Theprimarypurposeofthisinitialreadingistocreatealistofalloftheteststhatareexplicitlymentionedbythespecialistsintheirexplanationsoftheresolvedcases.FromthecasepresentedinFigure1weextractthreetests:1.RetrieveProcessPCB2.CheckJIBAddress3.CheckCountForthisparticulardatabase,wederivedaninitialsetofabout100tests.A.2 Grouping the Tests
Readingthroughthisdatabaseledtoanaturaldecompositionofthetestsbygroupingthemaccordingtodatastructuresmentionedinthecases.Thus,wegroupedtestsrelatedtotheJIB(jobinformationblock)together,joiningthetests\CheckJIBAddress"and\CheckCount"fromthecaseshowninFigure1.Inthisparticulardatabase,wegroupedthetestsintoroughlysevensetsoftests.Bycontrast,inourworkwiththeWPS-PLUSdatabase,thenaturalgroup-ingwasalongfunctionallines.Whilewehavenormevidence,itseemslikelythatsuch\natural"groupingswilloccurinmostsizabledatabases.A.2GroupingtheTests15
Versionofsystem:VAX/VMSV4.2Reasonforbugcheckexception:ACCESSVIOLATIONProcesscurrentlyexecuting:
SP=>8018E7AC00000004
8018E7B07FFE7DE4
8018E7B4FFFFFFFD
8018E7B80000026A
8018E7BC00000000!THISSHOULDHAVEBEENON
PREVIOUSLINE
!ANDVISAVERSA
8018E7C000000001
8018E7C400000005!BEGINNINGOFSIGNALARRAY
8018E7C80000000C!ACCESVIOLATION
8018E7CC00000004!WRITEACCESSPROTECTION
8018E7D000000020!VA
8018E7D480004A4EIOC$BUFPOST+017
8018E7D804040000
8018E7DC80004A6DIOC$BUFPOST+036
8018E7E0800CD108DZDRIVER+118thereasonforthisbugcheckisbecausethepideldoftheIRPhasbeenzeroed.INpostprocessingthesystemtriestogivethebueredbytecountbacktotheprocessthatdidtheio.HEdoesthisbyusingthePIDeldoftheirptoindexintothepidtablethengetsthePCB(nullprocessinthiscase)andgetstheJIBaddress.THEJIBaddressforthenullprocessis0.THEsystemthengoestotheJIB$LBYTCNToset(20)fromthebaseoftheJIB(0inthiscase)andtriestoaddbackinthebytecount.INthiscasewegetanaccessviolation.haveresponsefromVMSdevelopementthatthereis/wasabugxedin4.2ithasbeenmentionedtotryandlowerthebaudrateonterminalsasaworkaround.Figure1:AnInitialDataBaseEntry
16ACREATINGTHEVMSVALIDATIONMODEL
A.3 Rening the Test-Group Structure
Armedwiththisgroupoftests,wereturnedtothespecialistswhohadpreparedthedatabase.Withtheirhelp,welearnedthemeaningofeachofthetestswehadidentied.Thespecialistspointedoutthatournaturalgroupingsdidn'tcorrespondtogroupingsthattheywouldhavemade|primarilybecausetheyhada(non-articulated)setofabstractionsthatourgroupingsdidnotmakeclear.Byreviewingourgroupingstheywereforcedtoverbalizetheabstractionsthattheytookforgranted,andwewereabletosortourtestsinto15groupingsthatreectedthespecialists'notionofthecorrectabstractionsfortheirdomain.InthecaseoftheVAX/VMSdatabase,thespecialiststypicallysubdividedourgroupingsintothreedistinctcategories:thosethattestfortheexistenceoftheexpectedlocationinmemory,thosethattestedtheinternalconsistencyofthedatastructure,andthosethatprobethedatastructuresforspecicvalue.InthecaseofthetestsextractedfromFigure1,\CheckJIBAddress"isintherstcategory,thespecialistspointedoutfortheneedforanadditionaltestwehadnotidentiedthatperformedtheconsistencytest,and\CheckCount"isinthethirdcategory.Thus,ourinitialmodelofsevenstructure-basedtestgroupswasrened,throughinteractionwithspecialists,into15groupswithalayeredstructure.Testingalwaysproceedsfromonelayertoanother,startingwithtestsforexis-tenceofthedatastructure,proceedingtotestsontheconsistencyofthedatastructure,andnallyontoprobingspecicvalueswithinthedatastructure.A.4 Acquiring Additional Kno wledge
Thisstepisprimarilyaniterationoftheearlierone,buttwounusualitemsaroseduringthisphasefortheVAX/VMSsystem.Therstwastherealiza-tionthatcertaintermsusedinthedescriptionswerenotreectedinanyofthetestsourgroupingswehadalreadyderived.Inthisparticularcase,therewascontinualuseofphraseslike\duringASTdelivery,"and\duringpost-processing."Examinationofadditionalmaterial(materialfromaninternalDECcoursesuggestedbythespecialistsasagoodstartingpoint)revealedthatthesereectedtheprocessingstagesofthesystemwewerediagnosing.Ourinitialbreakdownhadbeenalongdatastructureboundaries,butanor-thogonal,equallyimportantstructuringispossiblealongtemporallinesbasedonthesestages.ItisthesetwoalternativedecompositionsthatwereferredtoA.5IntegratingTestsintotheModel17
inSection3.2asthe\structureofthedomain."Furtheriterationwiththecoursematerialsandthespecialistsrevealsamuchricherstructure,consistingofmultiplelevelsofabstraction.Withthetemporaldecompositiontakenasprimary,Figures2and3showtwodierentlevelsofabstraction.Figure2istakenataveryabstractlevel,showingthefourmajorprocessingstagesandthedatastructuresusedateachstage.Figure3isa\close-up"ofthesingleprocessingstageknownas\ASTdelivery."
A.5 Integrating Tests into the M odel
Armedwithamuch-improvedmodelofthedomain,webuiltasemanticnet-workcapturingasmuchofthismodelasbearsdirectlyontheabilitytoselectandapplythetestsandtestgroups.IntheVAX/VMSsystem,ourvalidationmodelreectsbothdecompositionsalongdatastructuresandalongprocess-ingstages.Thesemanticnetworkhas15nodes,foreachtestgroup,77nodesthatrepresentknowledgeabouttheprocessingstagesofdevicedrivers,and22nodesthatrepresentknowledgeaboutdatastructures.18ACREATINGTHEVMSVALIDATIONMODEL
LEGEND:
data structuretest test groups IRPUCB PCB valid PCB?
processing devicedriver
postprocessing
AST Delivery preprocessing
startingaddress
startingaddress valid IRP? startingaddress valid UCB?
Figure2:DeviceDriverValidationModel:AbstractView
A.5IntegratingTestsintotheModel19
AST Delivery
Break Internal PID
retrieve process PCB
null process PCB?
PCB$PID PCB Vector
first ACB ACB$ASTQBL JIB
PCB$ASTQBL PCB$JIB JIB$BYTCNT Check ACB
type
get ASTQBL Check JIB Valid Return byte count Get JIB
address
JSB to AST routine REMQ AST
Enqueue ACB in AST queue
Table
Check
address JIB? byte count
Figure3:DeviceDriverValidationModel:ASTDelivery