Knowing what research organizations actually do, with whom, where, how and for what purpose: Monitoring research portfolios and collaborations

(1)

Knowing

what

research

organizations

actually

do,

with

whom,

where,

how

and

for

what

purpose:

Monitoring

research

portfolios

and

collaborations

Javier

Ekboir,

Genowefa

Blundo

Canto*,

Cristina

Sette

ImpactAssessmentspecialist,InternationalCenterforTropicalAgriculture(CIAT),Peru

ARTICLE INFO

Articlehistory: Received12April2016

Receivedinrevisedform1August2016 Accepted4December2016

Availableonline5December2016

1.Introduction

Managersandpolicymakershavestruggledtodevelopeffective

monitoringsystemstotracktheevolutionofresearch

organiza-tions. This paper presents the ﬁrst components of a novel

monitoring system for monitoring such organizations. These

componentscanbeused togeneratedetailed staticpictures of

theactualactivitiesandpartnershipsofalargeresearchprogram

ororganization,inotherwords,whattheorganizationisactually

doing,withwhom,where,howandforwhatpurpose.Itcanalso

identifywhethernewincentivesororganizationalstructureshave

animmediateeffectontheresearchers’activities.Oncedeveloped,

the full system will be able to monitor the evolution of the

organization’sactivitiesandassessmid-andlong-termeffectsof

speciﬁc incentives. Essentially, the system asks individual

researcherstolistall theimportant collaborationstheyengaged

in during the preceding 12 months and to provide some

information about these collaborations. The data are then

aggregated todescribe the organization’s portfolio of activities

andengagementwithotheractorsintheinnovationsystem.

The system presented here can show how an organization

actuallyallocatesitseffortswhichcanbedifferentfrombudgetary

allocations. This information is important for planning and

management of research. As Argyris and Schön (1974) have

shown,oftenthereisagapbetweenwhatanorganizationplans

andwhatitactuallydoes.Thedivergenceisparticularlyacutein

not-for-proﬁt research organizations because researchers are

expectedtoraiseanimportantshareoftheirfunds,meaningthat

managershavelimitedcontrolovertheresearchers’activities.

Also, the system maps researchactivities as they are being

conducted,whichis importantforresourceallocation. Research

organizations have struggled to generate information for this

purpose, reverting often toex-ante and ex-post impact

assess-ments.Whileex-anteassessmentscanprovidesomeguidancefor

decisionmaking,theyhavetoberevisedastheresearchprogresses

becausetherealityisusuallydifferentfromwhatwaspredicted.

The methodology presented in this paper can help in these

revisions.Ontheotherhand,ex-postimpactassessmentscannot

beusedforresourceallocationbecausetheycanonlybeconducted

afterenoughtimehaspassedfortheimpactstobemeasurable,in

otherwords,manyyearsafterthedecisionshavebeenmade.Since

oursystemcanbeusedatrelativelyshortintervals(ideallyevery

twoorthreeyears)andisbasedoncurrentactivities,itsresultscan

beusedwhiletheprojectsarestillbeingimplemented.

Three reasons justify analyzing the links established by

researchers. Firstly, research is increasingly implemented by

inter-disciplinary,multi-institutionalteamsthatnetworkformally

and informally both locally and globally (vom Brocke &Lippe,

2015;Adams2012;LieffBenderly2014;Stephan2012;Bennett, Gadlin, & Levine-Finley, 2010). Secondly, programs to foster

interdisciplinary, inter-institutional collaborations between

researchers and other actors in innovation systems have been

implementedin severalcountriesand policy-makersareasking

abouttheirimpacts(Trochimetal.,2011).Thirdly,collaborations

with researchersand non-researchers are important inﬂuences

thathelpresearcherstobettercontributetoinnovationprocesses

*Corresponding author at: Centro Internacional de Agricultura Tropical – InternationalCenterforTropicalAgriculture(CIAT),LimaOfﬁce(Perú).

E-mailaddresses:javierekboir@gmail.com(J.Ekboir),g.blundo@cgiar.org

(G.BlundoCanto),cris.sette@gmail.com(C.Sette).

http://dx.doi.org/10.1016/j.evalprogplan.2016.12.002

ContentslistsavailableatScienceDirect

Evaluation

and

Program

Planning

(2)

andtobecomemoreproductiveandcreative(Li,Liao,&Yen,2013;

Klenk,Hickey,&MacLellan,2010;Wagner,2008).

Finally,thereisastrongpressureonresearchorganizationsto

demonstratethesocial andeconomic impactsoftheiractivities

(Kraemer2006;Rusikeetal.,2014).Traditionally,theseimpacts

have been measured through rates of return estimated with

econometricmodels(see,forinstance,Alston,Norton,&Pardey,

1995).However,inrecentyearsthesemethodshavebeencriticized

because they depend on very strong assumptions that impose

simple, constant causalities on the data and do not take into

accountthecomplexityofresearch,wheremanycausesinteractin

waysthat changeover time (Patton 2010).Thus, thefocus has

progressivelyshiftedtotheanalysisoftherolesresearchplaysin

innovationandsocialprocesses,whichrequiresboththedeﬁnition

of the organization’s theories of change and how its actual

activitiesagreewithordeviatefromthetheoriesofchange(Mayne

&Stern2013).Whiletherearemanypublicationsonhowtobuild

theories of change for research activities (Davies, 2004; ISPC,

2012),fewworkshavebeenpublishedonhowtomaptheactual

researchactivitiesoflargeprogramsororganizations.Thispaper

contributestoﬁllthisgap.

Thesystemwasdevelopedinapilotprojectthatinvolvedthe

Roots,TubersandBananasCGIARResearchProgram(RTB),alarge

agricultural“researchfordevelopment”program(RTBisdescribed

inSection4).1_The_information_for_this_project_was_collected_only

ninemonthsafterRTBstartedoperating;therefore,itsnetworks

reﬂectmostlypre-existingactivities.However,initsshortlifeRTB

inducedimportant changes inthe wayresearchactivities were

conducted,fosteringgreaterinteractionamongCGIARcenters,and

refocusingpartnershipsaccordingtothepartners’capabilitiesand

RTB’sresearchpriorities.Thefactthatthesystemcouldidentify

these changes despite their incipient nature attests to its

effectiveness.

Section 2 presents the conceptualframework onwhich the

systemis based.Section 3reviewsrecent publicationsthat use

Social Network Analysis techniques (SNA) to analyze research

networks.Section4presentsRTB,whileSection5 discussesthe

methodologyusedinthestudy. Section6discusses thetypeof

informationthatthesystemgeneratesandSection7concludes.

2.Conceptualframework

The system is strongly anchored to complexity theories

(Axelrod & Cohen 1999) and evaluation frameworks based on them(Mayne&Stern2013;Patton2010),theinnovationsystems

framework(Edquist2005)andtherecentliteratureonresearch

systems (Stephan 2012; Wagner 2008). While several methods

havebeendevelopedformonitoringprograms(see,forexample,

Brandonetal.,2013),thereisadearthofresearchonmonitoring

complex programs and organizations such as large research

institutions.

Thesystempresentedinthispaperisbasedontheobservation

thatinteractionsamongresearchersandnon-researchactorsinan

innovation system can be represented as networks (Kratzer,

Gemuneden,&Lettl,2008)that forma complexsystem. These

systems are characterized by the interactions among different

types of actorsconstrained by the socioeconomicand physical

environmentinwhichtheyoperate(Axelrod&Cohen,1999).Due

tothelargenumber ofinteracting forces, complexsystemsare

essentiallyunpredictable.Planningcanreducetheuncertaintybut

cannoteliminateit.Therefore,rigidstrategicplanningdrivenby

ex-anteimpactassessmentisoflittleuseandactorsneedtoadapt

theirstrategiesastheycollectnewinformationabouttheevolving

stateofthesystem(Patton,2010).However,thisisnoeasytask.

Duetotheirlimitedresources,decisionmakersneedguidanceon

what information should be collected and how it should be

interpreted(Mayne&Stern,2013).Thisguidanceisprovidedby

whatthedecisionmakersknowabouttheprocessandhowthey

expecttheirinterventionstoinﬂuenceit,inotherwords,bytheir

theories of change. Also, in order to be adaptive, actors need

processindicatorsthatinformthemaboutthecurrentstateofthe

system.

Inthecaseofnot-for-pro_ﬁtresearchorganizationsthetheoryof

changeandtheprocessindicatorscanbebuiltfromtheinnovation

systemsframework,recentstudiesoftheorganizationofresearch

and a thorough knowledge of the organization that is being

analyzed. In these organizations, the theoryof changeplays a

criticalrolebecausetheylackaclearindicatorofsuccesssuchas

proﬁt.

Accordingtotheinnovationsystemsframework,researchhas

positivesocial andeconomicimpacts whenresearchersinteract

with different types of research and non-research partners in

knowledgeprocessesthatfeatureseveralfeedbackloops(Edquist,

2005).Therefore,thetheoryofchangepositsthatanagricultural

researchorganizationthatinteractsonlywithadvancedresearch

institutionsandafewextensionagentsislikelytohaveasmaller

impactthananorganizationthatalsointeractswithprivateﬁrms,

farmer organizations and innovative farmers. The literatureon

research has also found that the quality of research depends

criticallyontheresearchershaving activeinternational

connec-tions (Wagner, 2008), which indicates that a researcher in a

developing country that interacts only with colleagues in her

organization should be less productive and creative than a

researcher that has many international links. Therefore, the

process indicators can be constructed from the links the

organization establishes with other actors in the innovation

system, and can be analyzedwith simple tables and statistical

methods,andSNAtechniquesasisexplainedbelow.Itshouldbe

noted,though,thatthereareveryfewdetailed studiesthatlink

researchactivities,thestructureofnetworksandinnovations.For

example, it is not known how biotechnology networks should

differ fromanimal health networks.Therefore, theinformation

generatedwiththissystemcanalsobeusedtoanswerimportant

theoretical and empirical questions about the relationship

betweenresearchcollaborationsandinnovationprocesses.

Otherprocessindicatorscanbeconstructedbyanalyzingthe

organization’sportfolioofactivities.Forexample,theCGIARhas

deﬁnedthatitneedstostrengthenitsresearchonnutritionand

health,withspecialfocusonAfricaandtheleastdevelopedAsian

andLatinAmericancountries.Withthesystempresentedinthis

paper it is possible to calculate the share of collaborations

established for researchon a particular topic,the geographical

focus, the type of research that is being conducted and other

features.Theseresultscanthenbecomparedwiththe

organiza-tionalprioritiesandstudiedovertimetounderstandtheevolution

of the networks. The information can be used as an input in

managementdecisions.

Finally,sincetheresearcherscanidentifywhetheraparticular

collaborationwasestablishedasa resultofa speciﬁcincentive,

such as a new line of ﬁnancing, it is possible to identify the

immediate impactof the incentiveon thepatternsof research

activitiesandcollaborations.Themid-andlong-termeffectscanbe

identiﬁedbyrepeatingtheexerciseperiodically.

1

Researchfordevelopmentisdeﬁnedasscientiﬁcactivitiesthatareexpectedto havepositiveimpacts oneconomic and socialwellbeing andthe sustainable managementofnaturalresources.

(3)

3.PrevioususeofSNAintheanalysisofresearchnetworks

Anetworkisasetofnodes(inourcase,researchers)whoare

connected among them; the connections are also called links

(Newman,2003).Inmostsocialnetworks,thelinksarenotrandom

and form well deﬁned patterns of interaction; SNA uses

mathematicaltoolsandvisualizationtechniquestoanalyzethese

patterns,forinstance,whoarethemostconnectedresearchers,or

whethertheycollaboratewithnon-researchers.Important

dimen-sionsofthestructure ofresearchnetworkscanbestudiedwith

networkscenteredonindividualnodes;suchnetworksareknown

intheliteratureas_“egonetworks_”.Egonetworkscanidentify,for

instance,womencenterednetworksandtheirevolutioncanhelp

to assess programs that aim to increase the diversity of the

researchpool.Additionally,theanalysisofthenetworksthatwere

createdasaresultofRTBcanprovideevidenceontheconvergence

ofindividualresearchprojectstowardsanintegratedprogram.

Several methodologies have been used to analyze research

networks;hereweonlyreviewpublicationsthatuseSNAmethods.

Three main methods were identiﬁed in the literature review:

analysis of joint publications (co-authorship relationships),

analysisofproject documents (projectpartnerships) and direct

surveysofresearchers.

Studiesbasedonco-authorshiprelationshipsidentifypatterns

ofinteractionexploringdatabasesofscientiﬁcarticles.Ingeneral,

twoauthorsareconsideredtohaveadirectlinkiftheyco-authored

atleastonepaper. Whenalargenumberof co-authorshipsare

identiﬁed,amapofinteractionsemerges.Duetotheavailabilityof

largedatasets,thishasbeenthemostcommonapproachtostudy

researchnetworks;recentpapersincludeLi,LiaoandYen(2013),

Gonzalez-Brambila, Veloso and Krackhardt (2013), Klenk et al. (2010).Thisapproachdocumentscollaborationsthatresultedin

scientiﬁc publications but ignores connections for capacity

building and advocacy; it also does not include informal

collaborations,whichhavebeenrecognizedasimportant

compo-nentsof scientiﬁcworkeven if theydo not result in scientiﬁc

publications(Varga&Parag,2009;Kratzer,Gemuenden,&Lettl,

2008).Otherproblems withthis approach are that researchers

maybeincludedasco-authorsforsocialreasons(LaFollette,1992)

or because theyprovide data or equipment (Stokes &Hartley,

1989).Finally,scientiﬁcpapers oftentakeseveral monthstobe

published, by which time the collaboration may no longer be

active.

Studies of project partnerships analyze project documents,

mainly joint proposals and publications, to identify research

collaborations.Themaindifferencewithco-authorship

relation-shipsisthatprojectpartnershipsaremorelikelytoinclude

non-scientiﬁc collaborations. Several countries have implemented

projectspromotingresearchandinnovationnetworks,including

theEuropeanFrameworkProgramsandtheCanadianNetworkof

Centers of Excellence. Analyses of project partnerships include

BiggieroandAngelini(2014),KlenkandHickey(2013),Protogerou, Caloghirou and Siokas (2010) and Cassi et al. (2008). Two

drawbacks of this approach are that it overlooks informal

interactionsandonlycapturesinteractionsthat arerelevantfor

documentationof theproject, which mayhave notresulted in

effectivecollaborations.

Approachesbasedonco-authorshiprelationshipsandproject

partnershipscannotbeusedinresearchareaswithlowpropensity

topublish(such as engineering or developmentof agricultural

equipment)or where informal interactions are common.

Addi-tionally,theymayleadtotheinclusionofcollaborationsthatexist

onlyonpaper.Onewaytoovercometheseproblemsistoelicit

informationdirectlyfromtheresearchers.Dozieretal.(2014)and

BozemanandCorley(2004)usedastrategybasedonself-reported

information about the researchers’ networks to capture active

formal and informal collaborations as well as non-research

interactions. Klenk and Hickey (2013) used an internet-based

questionnaire to elicit respondents’ collaborations within two

Canadianresearchprograms.Kratzer,Gemuneden,&Lettl,(2008)

askedresearcherstoreportabouttheirperceptionofthestrength

ofinformalcollaborationsamonginstitutionsnamedinaroster.

4.WhatistheCGIARroot,tubersandbananasresearch

program?2

The CGIAR is a mission oriented global organization that

conducts researchthat contributes to solve problems affecting

poorruralhouseholdsindevelopingcountries,enhancehealthand

nutrition and improve the management of natural resources.3

CGIAR’s activities are mostly conducted by 15 international

researchcentersorganizedinresearchprograms,RTBbeingone

ofthem.4_In₂₀₀₈_CGIAR_embarked_in_a_major_{reorganization}_that

involvedthecreationoftheCGIARResearchProgramsasumbrella

organizationsforpre-existingprojectswiththeexpectationthat

overtimetheywouldreshapetheCGIAR’sresearchportfolioand

interactions with other actors in the agricultural innovation

system.

RTBwaslaunchedin2012tobringtogetherresearchonbanana,

plantain,cassava,potato,sweetpotato,yam,andotherrootand

tubercropstoimprovefoodsecurity,nutritionandlivelihoods.The

programfocusesoncutting-edgegenomicresearch,international

collaborationstoaddressprioritypestsanddiseases,andresearch

toincreaseharvestsandimprovepostharvestoptions.RTBisbased

onapartnershipofﬁveresearchcenters:theInternationalPotato

Center (CIP), which leads theprogram, Bioversity International

(Bioversity), the International Center for Tropical Agriculture

(CIAT),theInternationalInstituteofTropicalAgriculture(IITA)and

theCentredeCoopérationInternationaleenRecherche

Agrono-mique pour leDéveloppement (CIRAD). Currently,the program

collaborates with 366 partners, including national agricultural

researchorganizations,academicandadvancedresearch

institu-tions, non-governmentalorganizations,and privatesector

com-panies.Partnerships,communicationandknowledgesharingare

keystrategiesthattheprogramreliesonachieveimpacts.

5.Methods

Thesystemhasthreedistinctcomponents.Theﬁrstoneisthe

collection of information directly from the researchers on the

activities and collaborations they engage in. This information

capturesactualactivities andcollaborationsinsteadofactivities

andcollaborationsidentiﬁedfromwrittendocumentsthatmaynot

beactive.Thesecondcomponentistheanalysisofthedatawith

tablesandstatisticaltools.Throughthiscomponentitispossibleto

(a)composeacomprehensivepictureoftheorganization’sactual

portfolioofactivitiesandpartnerships,(b)identifytheallocation

oftheorganization’seffortsacrossactivitiesandpartnerships,and

(c) explore several dimensions of those activities and

collabo-rations,suchastheirrelativeimportance,genderdimensionsand

geographicordisciplinaryfocus.Finally,SNAtechniquesareused

todiscoverconﬁgurationsinthecollaborations(e.g.,thecollective

structureofcollaborations,reciprocity,orthetransdisciplinarityof

the organization’s activities). The three components make it

possible to identify aggregate patterns in the organization of

researchandpartnerships,areasofstrongintegrationorisolated

2

www.rtb.cgiar.org.

3

www.cgiar.org.

4

AcompletelistofallCGIARResearchProgramscanbefoundathttp://www. cgiar.org/our-research/cgiar-research-programs/

(4)

collaborations, and the inﬂuence newprograms have on these indicators.

Theinformationusedinthisprojectwascollectedwitha

user-friendly,web-basedsurveyforresearchersfullyorpartiallyfunded

byRTB.Thequestionnairewasdesignedsothatcompletionwould

not exceed 30min for researchers reporting up to 10

collabo-rations.The survey was conductedbetweenSeptember 25 and

November23,2012.Toinducemanagerstosupporttheanalysis

and researchers to provide accurate information, they were

informedthattheinformationwas tobekeptconﬁdential,that

the reports were not meant to be used for accountability (of

programsorindividuals)andthattheresultscouldnotbeusedto

comparedifferentresearchprograms.

Allresearcherswereaskedtolistthenamesofthecollaborators

theyengagedwithwhileconductingtheirresearch.Collaborations

weredeﬁned as a formal or informal sustainedrelationship in

whicharesearchereffectivelycooperatedwithotheractorsinthe

innovation system (including researchers, extension agents,

research managers, input suppliers, output buyers and

policy-makers); this deﬁnition is similar to the one proposed by

Sonnenwald (2007). Following published research (Bozeman andCorley,2004;Dozieret al.,2014),theinterpretationof this

statementwaslefttotherespondents.

Theresearcherswereaskedtoconsideralltypes of

collabo-rationsimportantfor theirworkintheRTB network, including

researchprojects,innovationplatforms,advocacynetworksand/or

capacity buildingactivities. It was speciﬁed that these

collabo-ratorsmayworkinthesame researchcenter,in anotherCGIAR

centerorinnon-CGIARorganizationsandthatthecollaborations

maybeformalorinformal(e.g.,acommunityofpractice)andnot

necessarilybeﬁnancedbyRTB.Theywerealsoaskedtoprovide

severalattributesofeachcollaboration,includingtypeandtopicof

research,genderofthecollaborator,geographicfocusandlocation

ofactivities(e.g.,desk,experimentalstationorfarmer’sﬁeld).Of

particularimportancewasthequestion ifthecollaborationhad

beeninducedbyRTB.Toavoidtheproblemofmemorylapseswith

regardtopastcollaborationsandtocaptureactiverelationships,

researcherswereaskedtoreportonlycollaborationsthathadbeen

activeinthetwelvemonthspriortothesurvey.

Theinformationfromtheindividualresearcherswas

aggregat-edintocategoriesdeﬁnedforeachofthevariablesofinterestand

tabulated.Insomecases,thecategoriesweredeﬁnedincardinal

numbers,inothers,inintervalsandtheanalysiswasconductedfor

proportions. In all cases, the results were calculated for the

collaborationsthatwereinducedbyRTBandthosethatwerenot.

By comparing both results, it was possible to identify the

immediateinﬂuenceoftheresearchprogram.Inthecaseofthe

proportions, a Chi square test was used to test whether the

differencebetweenthedistributionsofcollaborationsinducedby

RTBandthosenotinducedbyitwassigniﬁcantlydifferentfrom0.

TheSNAanalysiswasconductedwithUCINET6andNETDRAW

(Borgattiet al.,2002).Theinformationwas collectedonlyfrom

researchers funded by RTB but not from their collaborators.

Collecting information from the latter would have required a

massiveamountofresources.Sincethecollaboratorsarealsolikely

tocollaborateamongthemselves,thefullnetworkshouldbemore

connectedthantheonemapped.Totakeintoaccountthatmost

namedcollaboratorsdidnotrespondtothesurvey,twoseparate

SNA analyses were conducted, one with the full dataset (i.e.,

respondentsandcollaboratorsmentionedbytherespondents)and

theotherusinginformationonlyabouttherespondents.Thelarger

dataset was used to study the structure of RTB’s portfolio of

activitiesincludingtherangeofcontacts,geographicreach,main

research areas, and some patterns of interaction. The smaller

dataset was used to explore the network’s structure and

connectivity. Additionally, the questionnaire did not explicitly

statethat non-researchpartners shouldbeincluded.Thus, it is

possiblethat non-researchcollaborationsareunderrepresented,

especiallyiftherespondents’mainactivityisresearchandtheyare

not fullyawareof theotherdimensions of theirwork,suchas

capacitybuildingoradvocacy.Anecdotalevidenceindicatesthat

thisisacommonproblem.

The methodology has three limitations. Firstly, the

self-reportingapproachlacksoperationalprecisionaseachresearcher

decideswhich collaborationsheorshewillreport,andthere is

great variationamongresearchers’willingnesstoprovide

infor-mation but also different understanding of what important

collaborations are. Secondly, respondents could have forgotten

some collaborators. Thirdly, a signiﬁcant rate of response is

necessary to achieve representativeness of the organization’s

researchportfolioandactualcollaborations.Researchers’

motiva-tionsforreportingornotreportingcertainpartnershipsandthe

incentivestoachieveasigniﬁcantrateofresponseisbeyondthe

scopeofthisstudy,butshouldbetakenintoaccountwhenusing

this system. A very strong supportfrom the top authorities is

criticalfortheproject’ssuccess.Takentogether,theselimitations

mean that theestimatesof someparameters (e.g.,density)are

probablybiaseddownwardsdue totheexclusionofsomelinks

fromthecalculations.Solutionsforcorrectingthesebiaseshave

beenproposedintheliterature(Marschall,2012),buttheyrequire

theimpositionofstrongassumptionsaboutthetrue(unknown)

distributionoflinks.Thus,thecorrectionofthebiaswouldcomeat

the cost of using more prior information, which could cause

additionalbiases.Duetothelackofclearindicationsabouthowto

dealwiththeseproblems,theestimatesofthenetworkparameters

werenotcorrected.

Despitetheselimitations,thesystemcouldsuccessfullyelicit

fromtheresearchersinformationabouttheactivitiestheyengaged

inandtheircollaborators.Theinformationwasaccurateenoughto

provideaclearpictureoftheorganization’sportfolioofactivities

andoftheresearchers’immediatenetworks.

6.Results

Thequestionnairewassentto126researchersandthenumber

ofvalidresponseswas92,givinganeffectiveresponserateof75%.

Thesurveyidentiﬁed702links,ofwhich134(19%)wereenabled

by RTB.RTB’s research networkwas foundtobe quite diverse,

including624individualcollaboratorsfrom302distinct

organiza-tions. Some collaborators were mentioned by more than one

surveyrespondent.Thehighestnumberofcollaborationsreported

was21andthelowest1.Allrespondentsbut7workedforCGIAR

centers.Therespondentsincludedresearchersandseniorresearch

supportstaff.

AswasexplainedinSection2,thetablesandSNAresultsonly

havemeaningwheninterpretedthrough(a)thetheoryofchange,

(b) the literature on innovation systems and organization of

science,and(c)athoroughknowledgeoftheorganizationthatis

beinganalyzed.Thissectionpresentssomeexamplesofhowthese

threeelementscanbecombinedwiththequantitativeinformation

generatedbythesystemtounderstandthecurrentstructure of

researchactivities andcollaborationsandwhatthismeansfora

researchorganization.Itshouldberememberedthatforthetime

being thesystem is static; therefore,the resultsarea baseline

againstwhichitwillbepossibletoassessorganizationalchange.

TheanalysiswasconductedfortheRTBcase.

The resultsare presentedin threesubsections.The ﬁrstone

analysesRTB’sportfolioandtheinﬂuencethatthecreationofthe

researchprogramhadonthewholesetofactivitiesincludedunder

itsumbrella. Thesecondsubsectiondiscussesparticular

param-etersofthefullnetworkofRTB’sresearchers,includinggeographic

(5)

partners.Finally,thethirdsubsectionstudiesthenetworkformed

onlybythoseresearcherswhoﬁlledthequestionnaire;thisstudy

shedslight onparticular featuressuchasreciprocity,clustering

andconnectivity.

6.1.OrganizationofRTB’sactivities

Thesystem providedaclearpictureoftheportfolioof RTB’s

activitiesandhowtheychangedasaresultofthecreationofRTB.

Forexample,Table1showsthenumberofpartnershipsbytypeof

organizationandwhetherthecollaborationwasenabledbyRTB.

TheactivitiesthatwerenotenabledbyRTBeitherwereinherited

frompreviousprogramsorwerenotfundedbyRTBbutarestill

reportedaspartofit.ThecolumnonthefarrightinTable1shows

thedifferencebetweenthedistributionsofRTB-inducedand

non-RTB-inducedcollaborations.Morethanhalfofthecollaborations

induced by RTB (51%) were established among international

researchinstitutes(mainly CGIAR centers),compared with just

22%ofnon-RTB-inducedcollaborations.Meanwhile,the

propor-tionofinteractionswithadvancedresearchinstitutesandnational

researchorganizationswassubstantiallylowerascomparedwith

non-RTB-inducedcollaborations.Inshort,animportantresultof

RTBwastoinduceaproportionalincreaseofcollaborationsamong

CGIARcentersandaproportionaldecreaseincollaborationswith

advancedresearchorganizations and nationalresearchsystems

fromdeveloping countries. Also, the share of interactions with

non-research partners was slightly lower among RTB-induced

collaborations,indicating that RTB didnot immediately induce

strongerlinksofresearcherswithotheractorsintheinnovation

system,asitstheoryofchangeindicatesshouldhavehappened.

Thisshiftingpatternofcollaborationsreﬂectstheinitialneedto

bringtogether a diverse groupof projectsand researchersthat

previouslyoperatedunderadifferentorganizationalstructure.A

similaranalysisconductedatalaterdatecouldindicateifthese

patternsaremaintainedovertime.

Thechangesinthepatternsofinteractioncanalsobetrackedby

analyzing the collaborations by research area and type of

organization(Table2).TheﬁndingsindicatethatRTBhasinduced

a shift in partnerships according to the partners_’ strengths.

Collaborationsamong CGIARscientists formeda largershareof

the RTB-induced collaborations in areas where they have

traditionallyhadstrongcapabilities(e.g.,breedingandgermplasm

conservation),newareasthatarecritical inthechangeprocess

(e.g.,researchmanagement,impactassessmentandgenderissues)

and‘emerging’areasthatdonotrequiremajorinvestments(e.g.,

GIS and climate change). On the other hand, there was a

comparatively higher proportion of collaborations with other

research institutions in areas where the latter have stronger

capabilities, such as biotechnology in the case of advanced

researchinstitutes,andinnovationplatforms, seedsystemsand

post-harvestinthecaseofnationalresearchorganizations.Atthe

sametime,RTBhadasmallershareofcollaborationswiththese

partnersinareaswheretheylackaclearadvantage,forexample,

collaborations for breeding with advanced research institutes.

Moreover,RTBappearstohavehadaninﬂuenceonthenatureof

researchactivities:germplasmconservationandgenderissuesin

particular represented a larger proportion of the RTB-induced

collaborations,while biotechnology,valuechains,breeding, and

pestanddiseasemanagementhadasmallershare.

Mostofthereportedcollaborationshadmultiplepurposes:650

(93%) had research objectives, 444 (63%) included capacity

building activities and 200 (28%) incorporated advocacygoals.

However,RTBhasaclearresearchfocus:558collaborations(79%of

thetotal)includednationalresearchorganizations,CGIARcenters

andadvancedresearchinstitutions,whileonly75collaborations

(11%)wereestablishedwithdisseminatorsoftechnical

informa-tion,suchasNGOs,CBOsandprivateﬁrms.

As shown in Table 3, a higher proportion of RTB-induced

collaborationsinvolveddeskworkandresearchinfarmers’ﬁelds,

while a lower proportion took place in regular and advanced

laboratories,ascomparedtonon-RTB-inducedcollaborations.RTB

appears tohave inﬂuenced thetype of researchCGIAR centers

perform, increasing what is usually known as ‘downstream’

researchcomparedto‘upstream’research.5

6.2.AnalysisofthewholedatasetwithSNA

TheanalysisofthewholedatasetwithSNArevealsanetworkof

weakly connected actors, with no individual playing a clear

intermediaryrole.ThisstructurereﬂectsinpartthegenesisofRTB,

whichoriginatedasanumbrellaorganizationforalargenumberof

pre-existingprojectscreatedwithouta unifyingstrategy.Future

Table1

Typesoforganizationsmentioned,bytypeofcollaboration,andrelativeimportance. Typeoforganizationa

NotRTB-induced (1)

RTB-induced (2)

NotRTB-inducedin% (3)

RTB-inducedin% (4)

Differencein% (4)-(3)c

Internationalresearchinstitutes(mainlyCGIARcenters)b

124 68 22 51 29

NationalNGOs 10 6 2 4 2

Extensionagencies 0 1 0 1 1

Nationalprivateﬁrms 10 3 2 2 0

Independentconsultants 7 2 1 1 0

Ministryorotherpublicofﬁces(notpublicresearchorganizations) 12 3 2 2 0

Other 2 0 0 0 0

InternationalNGOs 12 2 2 1 1

FarmerorganizationsandCBOs 11 1 2 1 1

Multilateralorganizations(e.g.,FAO,GFARorWorldBank) 14 2 3 1 2

Multinationalﬁrms 10 0 2 0 2

InternationalCooperationAgencies(CIRADorGIZ) 23 0 4 0 4

Nationalresearchorganizationsornationaluniversities 182 36 33 27 6

Advancedresearchinstitutes(includinguniversitiesfromdeveloped countries)

138 10 25 7 18

Total 555 134 100 100

a

For13collaborations,thetypeoforganizationinvolvedwasnotspeciﬁed.

b

ThisincludestwointernationalresearchcentersthatdonotbelongtoCGIAR,whichwerementionedsixtimes.

c

Ax2

testindicatedthattheprobabilitythatcolumns3and4werederivedfromthesamedistributionwasalmost0(exactly1.28E-8).

5 _Downstream_research_is_expected_to_be_used_by_{non-researchers}_shortly_after

thereleaseoftheresearchresults/outputswhilethereisnosuchexpectationfor upstreamresearch.Upstreamresearchissimilarto(butnotexactlythesameas) basic research, while downstream research includes applied research and development.

(6)

surveyswillhelptodeterminewhetherRTBevolvesintoamore

coherentresearchprogram.Havingrelativelysparseconnectionsis

alsoaconsequenceofthenetwork’ssize;becausehumanshave

limitedtimetointeractamongthemselves,asnodesareaddedto

thenetworkthenumber ofactuallinkseachperson hasgrows

slowerthanthemaximumnumberofpossibleinteractions.The

analysis also revealed that most researchers were engaged in

multidisciplinarynetworks,andnoclustersdeﬁnedbydiscipline

orgeographicfocuswereidentiﬁed.Ontheotherhand,therewas

some grouping based on the speciﬁc location of the research

activities(e.g.,alaboratoryorfarmers’ﬁelds).

Theout-degrees(i.e.,thenumberofpeopleapersoncontacts)

ranged from0 to 21,while the in-degrees(i.e., the number of

peoplethatcontactaperson)rangedfrom0to5(Table4).The

differenceinthedistributionsofthein-andout-degreescanbe

explained because only 15% of the people mentioned by

respondentsalsoansweredthesurvey,and alsobecausepeople

have different commitments to relationships and value the

interactions differently(forexample,aresearcher thatallocates

80%ofhertime toaprojectvaluesthecollaborationdifferently

than a researcher that allocates only 5% of his time); similar

asymmetries arise in mentor and mentee relationships, in

Table2

Collaborationsbytypeoforganizationandresearcharea(asapercentageofcollaborationsbyresearcharea). Typeoforganization!

Researcharea# ARI ICA INGO IRI NARO NNGO

M-ﬁrm N-ﬁrm

Min Other IC CBO M-org

Pestanddiseasemanagement non-RTBinduced(N-R) 27 0 1 21 42 0 2 0 2 0 1 1 2

RTB-induced(R) 18 0 0 24 41 0 0 6 6 6 0 0 0

Breeding(plant,animal,ﬁsh) N-R 18 5 0 18 49 0 4 3 1 1 0 0 0

R 0 0 0 53 33 13 0 0 0 0 0 0 0 Germplasmconservation N-R 28 9 0 15 37 0 0 0 4 0 0 2 4 R 18 0 0 53 24 6 0 0 0 0 0 0 0 Biotechnology N-R 26 6 3 18 35 0 6 6 0 0 0 0 0 R 67 0 0 0 33 0 0 0 0 0 0 0 0 Seedsystems N-R 8 5 8 16 32 13 0 3 3 0 0 11 3 R 0 0 40 0 40 0 0 0 0 0 0 0 20 Impactassessment N-R 10 3 0 77 3 0 0 3 0 0 3 0 0 R 0 0 0 91 0 9 0 0 0 0 0 0 0 Post-harvest N-R 35 12 0 12 38 0 0 4 0 0 0 0 0 R 0 0 0 17 58 8 0 0 17 0 0 0 0 Innovationplatforms N-R 16 6 0 13 29 3 0 6 0 3 3 13 6 R 0 0 0 0 50 0 0 0 0 0 0 0 50

‘Local’naturalsystems(e.g.,agronomy,agroforestry) N-R 22 7 4 15 44 7 0 0 0 0 0 0 0

R 33 0 0 33 33 0 0 0 0 0 0 0 0

Humanhealthandnutrition N-R 33 4 8 21 29 0 0 0 4 0 0 0 0

R 20 0 0 0 60 20 0 0 0 0 0 0 0

Cropproduction N-R 5 0 0 36 50 0 0 0 0 0 9 0 0

R 0 0 0 33 33 0 0 33 0 0 0 0 0

Researchmanagement N-R 6 0 6 44 6 6 0 0 6 0 13 0 13

R 0 0 0 88 13 0 0 0 0 0 0 0 0

‘Large’naturalsystems(e.g.,climatechange) N-R 69 0 0 19 13 0 0 0 0 0 0 0 0

R 0 0 0 50 33 0 0 0 0 0 17 0 0 Policies N-R 23 0 8 46 8 0 0 8 0 0 0 0 8 R 0 0 0 67 0 0 0 0 0 0 33 0 0 GIS N-R 33 0 0 33 33 0 0 0 0 0 0 0 0 R 0 0 0 100 0 0 0 0 0 0 0 0 0 ICT N-R 20 0 20 20 0 0 20 0 0 0 0 0 20 R 0 0 0 50 0 0 0 0 0 0 0 50 0 Genderissues N-R 0 0 0 0 0 0 0 0 0 0 0 0 0 R 0 0 0 100 0 0 0 0 0 0 0 0 0

Key:ARI,advancedresearchinstitute;ICA,internationalcooperationagency;INGO,internationalNGO;IRI,internationalresearchinstitute(includingCGIARcenters);NARO, nationalagriculturalresearchinstitute;NNGO,nationalNGO;M-firm,multinationalfirm;N-firm,nationalfirm;Min,ministryorpublicorganization;IC,independent consultant;CBO,community-basedorganization;M-org,multilateralorganization;N-R,non-RTB-induced;R,RTB-induced.

Table3

Locationofthecollaborations.

Locationofcollaboration All

collaborations (1) NotRTB-induced (2) RTB-induced (3)

NotRTB-inducedin% (4) RTB-inducedin% (5) Differencein%(5)-(4)a Desk 165 118 47 21 35 14 Farmers’ﬁelds 157 117 40 21 30 9

Partner’slocation(e.g.,marketor ministry) 28 24 4 4 3 1 Experimentalstation 120 99 21 18 16 2 Regularlab 52 45 7 8 5 3 Advancedlab 164 150 14 27 11 16 Totala 686 553 133 100 100

a_For₁₆_{collaborations,}_the_location_was_not_speciﬁed.

(7)

Table4

Numberofrespondentsthatreportedaspeciﬁcnumberofcollaboratorsandwerementionedascollaboratorsbyothers.

Numberofcollaborations Out-degrees(mentionedothers) In-degrees(mentionedbyothers)

21 1 0 20 7 0 19 2 0 18 1 0 17 0 0 16 1 0 15 0 0 14 0 0 13 0 0 12 0 0 11 2 0 10 23 0 9 1 0 8 3 0 7 5 0 6 8 0 5 6 2 4 8 6 3 4 20 2 7 61 1 13 486 0 532 49

(8)

hierarchicalrelationships, and in situations where a researcher

provides crucial information for a project but receives little in

return.Thisinformationwouldbemoreusefulifthesystemwas

usedperiodicallybecauseitcouldtracewhethertheresearchers

becomemoreinterconnectedwhichwouldreﬂectamorecoherent

program.

Nonodewasfoundtohaveastronginﬂuenceasmeasuredby

the average number of degrees (i.e., a node’s number of

connectionsasa proportionof thetotal numberof nodes).The

maximumaverageout-degreewas0.034,meaningthateventhe

best-connected node contacted just 3.4% of the nodes in the

network. Themaximum averagein-degree was 0.008,meaning

thatthebest-connectednodewascontactedbyonly0.8%ofthe

nodes in the network. In other words, no researcher has a

dominantpositioninthenetwork;evenmore,noteventhe“most

important”researchersinaresearchareaoccupyacentralposition

inthenetwork.Thisobservationisconﬁrmedbyotherparameters

discussedbelowandtheimplicationsforRTBarediscussedinthe

conclusions.

TheconnectivityofthenetworkwasanalyzedwithFreeman’s

graphcentralizationindex,whichmeasuresthenumberofexisting

linksasaproportionofthelinksthatwouldbepresentinastar

graphofthesamedimension.Theout-degreecentralizationindex

is 3.20%; the small value indicates that no researcher had a

positionaladvantageinthenetwork.6_The_most_central_actors,_i.e.,

theresearcherswithhigherin-andout-degrees,werenotdirectly

connectedamongthemselves.

Acomponent isdeﬁnedasasubsetofconnected nodesthat

havenoconnectionsoutsidethis group.Asshownin Fig.1,the

wholenetworkcomprises onelargecomponentwith561nodes

(90%ofthetotal)and14smallercomponentsthatrangeinsize

from2to11nodes.Allthecollaborationswithaglobalfocuswere

partofthemaincomponent,indicatingthatinformationofglobal

importancecancirculatetomostnodesinthenetwork.Thesmall

components were largely made up of partners working at

experimental stations or in farmers’ ﬁeldsand witha regional

focus,indicating thatthesecomponentsmayberelativelylocal,

working in isolatedprojects. On the otherhand, these isolates

couldberesearcherswhohavedifﬁcultiesintegratingintothenew

structure.Withthisinformation,RTB’smanagementcanlookinto

thesesmallernetworkstoidentifyiftheyneedsupporttochange

theirworkpatterns.

The distance-based measuresarecalculated only withinthe

main component. The average density (0.002) is quite low,

reﬂectingthefactthat itisa relativelylargenetworkandmost

nodesdidnotparticipateinthesurvey.Themain componentis

relativelywellconnected;theaveragedistanceis4.093andthe

diameteris10(anynodeisnomorethantenstepsawayfromany

other).

Betweennesscentralityforthewholenetworkis73%,indicating

that a fewintermediariesnot onlyhad severalconnections but

werealsolinkedtopeoplewhowerewellconnected.Someofthe

nodes with high betweenness centrality have low degrees,

meaning that although these individuals did not have many

connections,theylinkeddifferentgroupsofresearchers.Asshown

inFig.2,thenodeswithhighestintermediarypower(betweenness

centrality) included breeders and agronomists. The most

con-nected researchersinteracted with collaborators fromdifferent

disciplines.However,theresearchersworkingonplantbreeding

Fig.2.FacilitationofcommunicationbydisciplineoftheresearchersinthemaincomponentofRTB’snetwork. Note:Thesizeofthenodeindicatesitsbetweenness.

6

Thein-degreecentralizationismeaninglessinanetworklikethiswherealarge proportionofnodesdidnotprovideinformation.

(9)

and plant genetics had less diverse networks. Few of the

collaborationsof thecentralnodesresultedfromRTB–aresult

thatwas anticipated sincethewell-connectedresearchershave

beeninthesystemforseveralyearsandRTBwasonlyninemonths

oldatthetimeofthesurvey.

Themaincomponentisquiterobust.Whenanetworkhaslow

densityitcanbefragileinthesensethattheremovalofanodemay

breakthenetworkintoseparateparts.Suchnodesareknownas

‘cut-points’andthenodesthatwouldbecomeseparatedarethe

‘blocks’.Themaincomponentinthisstudyhassixblocksandfour

cut-points.Theremovalofthecut-pointshaslittleeffectonthe

networkstructurebecauseonlythecollaboratorsoftheremoved

nodeswouldbelost.Thesmalleffectthattheremovalofa

cut-pointwouldhaveonthestructureofthenetworkimpliesthatthe

departure of particular researchers, even the most connected,

would not have major consequences in terms of network

connectivity,eventhoughitmaysigniﬁcantlyaffecttheefﬁciency

of information transmittal (lower betweenness centrality) and

researchcapacity(i.e.,ifhighlyspecializedresearcherswerelost).

Sub-networkofRTB-inducedcollaborations:

RTBinducedthecreationof134links,involvingatotalof131

nodes;this sub-network issplitinto 16components ofatleast

three people each (Fig. 3). The largest components have clear

geographicalfocus.Thelargestcomponentofthissub-networkhas

42nodes(31%oftheRTBnetwork),hasaglobalfocusandishighly

multidisciplinary,asrepresentedbythediversityofnumbersin

Fig.3.However,mostsocialscientists(rednodes)areconnectedto

thecomponentbyacentralnode.Plantpathologists,

entomolo-gistsandplantbiologists(lightblue)arealsomostlyconnectedby

one node. Reportedly, half of thecollaborations occurred on a

monthlybasis,about70%werebasedondeskworkand65%ofthe

linkshadaglobalfocus.

Thesecond-largestcomponenthas32nodes,withalessdiverse

mixofdisciplines.ThereisastrongconnectionwithLatinAmerica.

The third-largestcomponent hasan African focus. Theworkis

conductedmainlyinfarmers_’_ﬁelds,and abouthalfofit isona

contract basis, involving partners from several disciplines. The

componenthastwodistinctsub-groupsformedaroundindividual

nodes and linked by one single node. The relativelydispersed

structureoftheRTB-inducedsub-networkreﬂectsRTB’soriginsas

acombinationofpre-existingprojects.AsRTBreorientsresearch

accordingtoitspriorities,thesmallercomponentsshouldbecome

moreinter-connected.

6.3.Analysisofthenetworkof92surveyrespondents

The analysis of the full network did not allow for the

explorationofreciprocallinksandredundantpathsbecausemost

nodes were not asked to complete the survey, as they were

mentionedascollaboratorsbythesurveyrespondentsbuttheir

salarieswerenot,evenpartially,paidbyRTB. Toovercomethis

Fig.3.RTBinducedsub-networkandcomponents.

Note:Note:Thenumbersrepresentdisciplinesandafﬁliations,asfollows:1=socialscientists,2=agronomists,3=livestock,3=breeders,5agriculturesystemresearch, 6=researchmanagement,7 =communicationsspecialists,8=geneticists,9=plantpathologists,10=biologists,11=GIS/geography,12=donors,13=government/NGO, 14=nutritionists/health,15=forestry,16=post-harvest,17=privatesector

Table5

Numberofresearcherswithaspeciﬁcnumberoflinks.

Numberofcollaborations Out-degrees In-degrees

5 5 1 4 3 4 3 6 12 2 11 8 1 14 18 0 53 49

(10)

problem,asub-networkthatonlyincludedtheresearcherswho

providedinformation(92nodes)wasanalyzed.

Collaborationsinthereducednetworkarequitesparse.The

in-andout-degreesrangefrom0to5andtheirdistributionsarequite

similar.AsshowninTable5,ﬁveresearchershadﬁveout-linksand

only one researcher had ﬁve in-links. Meanwhile, the 53

respondentswhodidnotmentionanycollaborationswithother

surveyrespondentshad0 (Table5).

Thesub-network ofsurveyrespondentsis quitefragmented,

having a main component formed by 55 nodes (59% of the

network) and 37 isolates that did not interact even among

themselves(Fig.4).The main component includes88 links, of

whichonly11(13%)arereciprocal.Thesmallnumberofreciprocal

linksislikelytheresultofthreefactors.First,asRTBwasinitiallya

collection of existing disjointed projects, the researchers had

originallydevelopedtheirnetworksindependently.Second,there

isgreatvariationinthelevelofeffortdevotedbyresearchersto

particularcollaborations,andtheymayvaluetheirlinks

different-ly.Third,theresearcher’sroleintheorganizationdeﬁnesadefacto

hierarchy;forexample,aresearchmanagerhasmorepowerthana

juniorresearcherbecausetheformerdeterminestheallocationof

resources and may even have a say over the researcher’s

employment.Allreciprocalinteractionsbutoneinvolve

research-ers from different institutions and all but three of the nodes

involved are senior researchers. Seven of the 11 reciprocal

collaborationsinvolvethesocialsciences(i.e.,impactassessment,

policies,genderissuesorresearchmanagement).

Thecluster analysis identi_ﬁedtwo clusterswithin themain

component. Theﬁrst onehas27 nodes with34 links. While it

includes researchers from many disciplines, plant scientists

predominate.Mostlinks(53%)intheclusterinvolvedlaboratory

work followed by participatory research(18%). In terms of the

locationofactivities,44%ofthelinksinvolvedworkinadvanced

laboratories,26%atexperimentalstationsand20%atadesk.Two

thirdsofthelinkshadaglobalfocus,15%hadaLatinAmerican

perspective and18% focused onAfrica.RTB spurred62%of the

collaborations.

Thesecondclusterhas28nodeswith48links.Whileithasa

multidisciplinary composition, it is more oriented towards the

socialsciences(56%ofthecollaborations).Only23%ofthelinks

involvedresearchteams,11%werecontractedresearchand 23%

facilitated theaccess oflocalpartnerstointernationalscientiﬁc

networks. Asmuch as 85% of thecollaborations in this cluster

involveddeskwork,80%hadaglobalfocusand15%hadanAfrican

perspective.RTBinduced56%oftheinteractions.

The large proportion of RTB-induced interactions in both

clusters indicatesthat,injustoneyearofitsexistence,RTBhas

fosteredalargeincreaseincollaborationsamongacoregroupof

researchers.Anexpansionofthiscoregroupinthefuturewould

provideevidenceofhowRTBisreshapingresearchpatterns.

7.Conclusions

Thedemandforaccountability,efﬁciencyandeffectivenessin

publicorganizationshasincreasedsincethe1990s(Immonen&

Cooksy,2014;Kraemer,2006),spurringthedevelopmentofnew

approaches for planning research and measuring its impacts.

While these methods provide information for accountability

purposes,theyarenotusefulformanagingresearchorganizations

and programs which requires accurate and timely information

abouttheactivitiesthatareactuallybeingimplemented.

Thesystempresentedinthispapershowshowthisinformation

can be collected and analyzed, identifying emerging research

topicsandpartnerships,areasofstrongorweakcollaboration,and

howorganizationalchangesin_ﬂuencethem.Thesystemisbased

ontherecognition thatin thecourseof theirwork researchers

collaboratewithdifferenttypesofactorsintheinnovationsystem.

Byaggregatingtheinformationprovidedbyindividualresearchers,

Fig.4.Sub-networkof92respondents–maincomponent,reciprocityoflinks,andisolates.

Note:The55squarenodesbelongtothemaincomponentwhilethe37circularnodes(thestackontheleft)areisolatesnotconnectedtoothernodes.Reciprocallinksare shownwiththicklines,non-reciprocallinksinblue.

(11)

itispossibletoidentifyemergingpatternsintheorganization’s

allocationofefforts(forexample,bygeography,discipline,typeof

partnerandgender),conﬁgurationofitsresearchactivities (e.g.,

whether researchers engage in interdisciplinary research) and

participationininnovationprocesses(especially,engagementwith

non-researchactors).Thetimelyandaccurateavailabilityofthis

informationprovidesanimportantinputforadaptivemanagement

andorganizationallearning.

The system is descriptive, in other words, based on the

conceptual framework described in Section 2, it generates

information about the researchers’ activities. As such, it does

notsaywhetheraparticular structureis goodor bad;this is a

judgementthatcanonlybemadebythemanagersaftercomparing

thisinformationwiththeorganization’stheoryofchange,values

and management needs. Also, the system is not meant for

evaluation of individual researchers as this might suffer from

self-reportingbiasinthecasetheyknewthatitwasgoingtobe

usedtoassesstheirperformance.

The monitoring system was designed as an organizational

learning and adaptive management tool to enable the rapid

identiﬁcationofemergingresearchactivitiesandcollaborations,

but also of areas of low or isolated collaboration. In the case

presented, as a condition for approval, the CGIAR’s Research

Programs, must make explicit their theories of change, i.e.,

narrativesthatexplainhowtheproposedresearchactivitiesand

partnershipsareexpectedtocontributetoCGIAR’sdevelopment

objectives.ThetheoriesofchangeareanexampleofwhatArgyris

and Schön (1974) called the espoused theory, i.e., what the

organizationbelievesitisdoingandhowitsactionscontributeto

itsgoals. On theotherhand, in theirterminology,themapped

networks and activities are the theory in use because they

representwhattheorganizationisactuallydoing.Bycomparing

theespousedtheoriesandthetheoriesinuse,anorganizationcan

identifydifferencesinwhatitplannedtodoandwhatitisactually

doing. Thesedifferences signalemerging problems or

opportu-nitiesthattheorganizationcanaddressastheprogramsevolve.

Even more, by identifying whether (i) new partnerships are

created, (ii) existing collaborations are closed, (iii) interactions

withexternalpartnersarestrengthenedand(iv)researchactivities

arechanged,theorganizationcanmonitorwhetheritsresponseto

emergingissuesishavingtheexpectedresults.

At this stage of development of the methodology, it is not

possible to deﬁnitively identify the most effective set of

parameterstomonitor the evolution ofresearch programsand

networks.However,thespecializedliteratureindicatesthattheset

shouldincludesizeofthenetwork,distributionofdifferenttypes

ofcollaborations(especiallydisciplinesandtypesofnon-research

partners and geographic focus), gender dimensions, degree

distribution, connectivity (density, betweenness), analysis of

components,cut-pointsandblocks,reciprocity,compositionper

discipline,andtheshapeofthedistributionoflinks.

Currently,thesystemisbeingexpandedinthreedirections:(a)

studyingchanges in resource allocationsand engagement with

partnersintheinnovationsystem;(b)developingapproachesto

analyzeorganizationallearning,and (c)analyzinghowdifferent

types of research inﬂuence organizational structures, in other

words, how different is a network of biotechnologists from a

networkofsocialscientists?.

In addition to contributing to organizational learning, the

information generated by the system can be used to answer

important research questions related to the organization of

research.Forexample,dothenetworksforspeciﬁcresearchareas

differ among themselves? For instance, each CGIAR Research

Program(CRP)conductsspeciﬁctypesofresearch,suchasfocused

onacrop(forinstance,breeding,agronomicpracticesandpestand

disease management), natural resources management (e.g.,

managementofreservoirs),climatechangeorpolicies;

addition-ally,theCRPsareexpectedtocollaborateamongthemselvesand

withexternalpartners.ComparingacrossCRPsitwillbepossibleto

understand,forexample,howtheactivitiesandnetworksofcrop

speciﬁcCRPsdifferfromthoseofnaturalresourcemanagement

CRPs.Ofparticularinterestwillbetheanalysisoftheinteractions

between the different CRPs with non-research collaborators

becausethisinformationwillhelptounderstandtheactualimpact

pathways for different types of research. For instance, in the

analysisofRTB’sactivitiesitwasfoundthatmorethan80%ofits

collaborationsinvolvedresearchorganizations.However,fromthis

information it is not possibletoassertthat RTB doesnot have

effective channelsto diffuse itsoutputs. It is possible that the

outputsarediffusedbyotherCRPsthathavemorenon-research

collaborators (an indirect pathway) or that the current

non-researchinteractionsaresufﬁcienttodiffusetheoutputs.

Researchprogramsandthenetworkstheycreatechangeasthe

researchprocessmatures(Kratzer, GemuendenandLettl,2008;

Gay&Dousset,2005).Therefore,changesintheresearchnetworks

can inform how the research portfolio evolves in response to

managementinterventionsorautonomousdynamics.By

collect-ingtheinformationperiodically(ideallyeverytwotothreeyears)it

willbepossibletofollowchangesinthediversityandstabilityof

collaborations and teams, whether the different subnetworks

becomemoreintegratedovertime,andchangesintheresearch

portfoliosandtheresearchframeworks.However,developmentof

themethodologytoassesschangesovertimeintheactivitiesand

networks is not trivial; in particular, the networks should be

modeledasstochasticdynamicprocesseswheretheprobabilityof

twonodesinteractingisinﬂuencedbyasetofvariables,including

managementdecisions.

Acknowledgements

This project was jointly undertaken by the Institutional

LearningandChange(ILAC)InitiativeoftheCGIAR,andtheCGIAR

Research Program on Roots, Tubers and Bananas (RTB). ILAC

receivedspeciﬁcfundsfromtheInternationalFundforAgricultural

Development (IFAD), grant number 1165. RTB is supported by

CGIARFundDonors(completelistat

http://www.cgiar.org/about-us/governing-2010-june-2016/cgiar-fund/fund-donors-2/). References

Adams,J.(2012).Theriseofresearchnetworks.Nature,490(18/12/2012),335–336.

Alston,J.M.,Norton,G.W.,&Pardey,P.G.(1995).Scienceunderscarcity:Principles andpracticeofagriculturalresearchevaluationandprioritysetting.Ithaca:Cornell UniversityPress.

Argyris,M.,&Schön,D.(1974).Theoryinpractice:Increasingprofessional effectiveness.SanFrancisco,CA,USA:Jossey-Bass.

Axelrod,R.,&Cohen,M.D.(1999).Harnessingcomplexity.organizationalimplications ofascientificfrontier.NY:TheFreePress.

Bennett,L.M.,Gadlin,H.,&Levine-Finley,S.(2010).Collaborationandteamscience:A fieldguide.Bethesda,MD,USA:NationalInstitutesofHealth[Availableat: https://ccrod.cancer.gov/conﬂuence/download/attachments/47284665/ TeamScience_FieldGuide.pdf.].

Biggiero,L.,&Angelini,P.P.(2014).Huntingscale-freepropertiesinR&D collaborationnetworks:Self-organization,power-lawandpolicyissuesinthe Europeanaerospaceresearcharea.TechnologicalForecastingandSocialChange.

Borgatti,S.P.,Everett,M.G.,&Freeman,L.C.(2002).UCINET6forwindows.Harvard: Analytictechnologies.185.

Bozeman,B.,&Corley,E.(2004).Scientists’collaborationstrategies:Implicationsfor scientiﬁcandtechnicalhumancapital.ResearchPolicy,33,599–616.

Brandon,P.R.,Smith,N.L.,Oﬁr,Z.,&Noordeloos,M.(2013).Monitoringand evaluationofafricanwomeninagriculturalresearchanddevelopment (AWARD)anexemplarofmanagingforimpactindevelopmentevaluation. AmericanJournalofEvaluation[1098214013509876].

Cassi,L.,Corrocher,N.,Malerba,F.,&Vonortas,N.(2008).Researchnetworksas infrastructureforknowledgediffusioninEuropeanregions.Economicsof InnovationandNewTechnologies,17(7&8),663–676.

Davies,R.(2004).Scale,complexityandtherepresentationoftheoriesofchange. Evaluation,10(1),101–121.

(12)

Dozier,A.M.,Martina,C.A.,O’Dell,N.L.,Fogg,T.T.,Lurie,S.J.,Rubinstein,E.P.,etal. (2014).Identifyingemergingresearchcollaborationsandnetworks:Method development.Evaluation&theHealthProfessions,37(1),19–32.

Edquist,C.(2005).Systemsofinnovation:Perspectivesandchallenges.InJ. Fagerberg,D.C.Mowery,&R.R.Nelson(Eds.),Theoxfordhandbookof innovationGreatBritain:OxfordUniversityPress.

Gay,B.,&Dousset,B.(2005).Innovationandnetworkstructuraldynamics:Studyof thealliancenetworkofamajorsectorofthebiotechnologyindustry.Research Policy,34,1457–1475.

Gonzalez-Brambila,C.N.,Veloso,F.M.,&Krackhardt,D.(2013).Theimpactof networkembeddednessonresearchoutput.ResearchPolicy,42,1555–1567.

ISPC(IndependentScienceandPartnershipCouncil)(2012).Strategicoverviewof CGIARresearchprogramspartI.Theoriesofchangeandimpactpathways.Rome: ISPC.

Immonen,S.,&Cooksy,L.L.(2014).Usingperformancemeasurementtoassess research:Lessonslearnedfromtheinternationalagriculturalresearchcentres. Evaluation,20(1),96–114.

Klenk,N.L.,&Hickey,G.M.(2013).Howcanformalresearchnetworksproduce moresociallyrobustforestscience?ForestPolicyandEconomics,37,44–56.

Klenk,N.L.,Hickey,G.M.,&MacLellan,J.I.(2010).Evaluatingthesocialcapital accruedinlargeresearchnetworks:ThecaseoftheSustainableForest ManagementNetwork(1995–2009).SocialStudiesofScience,40(6),931–960.

Kraemer,S.(2006).ScienceandtechnologypolicyintheUnitedStates.New Brunswick,NJ,USA:RutgersUniversityPress.

Kratzer,J.,Gemuneden,H.G.,&Lettl,C.(2008).Revealingdynamicsand consequencesofﬁtandmisﬁtbetweenformalandinformalnetworksin multi-institutionalproductdevelopmentcollaborations.ResearchPolicy,37,1356– 1370.

LaFollette,M.C.(1992).Stealingintoprint.Berkeley,CA,USA:Universityof CaliforniaPress.

Li,E.Y.,Liao,C.H.,&Yen,H.R.(2013).Co-authorshipnetworksandresearchimpact: Asocialcapitalperspective.ResearchPolicy,42,1515–1530.

LieffBenderly,B.(2014).Thearcoftheacademicresearchcareer:Issuesand implicationsforU.S.scienceandengineeringleadership:Summaryofaworkshop. Washington,D.C:NationalAcademyPress.

Marschall,M.(2012).Methodologicalpitfallsinsocialnetworkanalysis:Whycurrent methodsproducequestionableresults.Germany:AVAkademikerverlag. MayneJ.andSternE.,(2013)Impactevaluationofnaturalresourcemanagement

researchprograms:abroaderview.ACIARImpactAssessmentSeriesReportNo. 84.AustralianCentreforInternationalAgriculturalResearch:Canberra.http:// aciar.gov.au/ﬁles/ias84.pdf.LastAccessedon4/4/2015.

Newman,M.E.J.(2003).Thestructureandfunctionofcomplexnetworks. arXiv:cond-mat/0303516[cond-mat.stat-mech].

Patton,M.Q.(2010).Developmentalevaluation:Applyingcomplexityconceptsto enhanceinnovationanduse.NewYork,NY,USA:GuilfordPress.

Protogerou,A.,Caloghirou,Y.,&Siokas,E.(2010).TheimpactofEUpolicy-driven researchnetworksonthediffusionanddeploymentofinnovationatthe nationallevel:ThecaseofGreece.ScienceandPublicPolicy,37(4),283–296.

Rusike,J.,Mahungu,N.M.,Lukombo,S.S.,Kendenga,T.,Bidiaka,S.M.,Alene,A.,etal. (2014).Doesacassavaresearch-for-developmentprogramhaveimpactatthe farmlevel?EvidencefromtheDemocraticRepublicofCongo.FoodPolicy,46, 193–204.

Sonnenwald,D.H.(2007).Scientiﬁccollaboration.AnnualReviewofInformation ScienceandTechnology,4,643–681.

Stephan,P.(2012).Howeconomicsshapescience.Cambridge,MA:HarvardUniversity Press.

Stokes,T.D.,&Hartley,J.A.(1989).Coauthorship,socialstructureandinﬂuence withinspecialties.SocialStudiesofScience,19,101–125.

Trochim,W.,Kane,C.,Graham,M.,&Pincus,H.A.(2011).Evaluatingtranslational research:Aprocessmarkermodel.Clin.Transl.Sci,4(3),153–162.

Varga,A.,&Parag,A.(2009).Academicknowledgetransfersandthestructureof internationalresearchnetworks.Workingpaperintangibleassetsandregional economicgrowth(IAREG)1.3d. [Availableat:http://www.iareg.org/ﬁleadmin/ iareg/media/papers/WP1_03d.pdf].

Wagner,C.S.(2008).Thenewinvisiblecollege:Sciencefordevelopment.Washington, DC,USA:BrookingsInstitutionPress.

vomBrocke,J.,&Lippe,S.(2015).Managingcollaborativeresearchprojects:A synthesisofprojectmanagementliteratureanddirectivesforfutureresearch. InternationalJournalofProjectManagement,33(5),1022–1039.