Assessing various software development methodologies and matching software development methodologies with projects

ASSESSING VARIOUS SOFTWARE DEVELOPMENT METHODOLOGIES AND MATCHING SOFTWARE DEVELOPMENT METHODOLOGIES WITH PROJECTS

by Yuqing Ke

M.S. Computer System Engineering (2008) Boston University

Submitted to the System Design and Management Program in Partial Fulfillment of the Requirements for the Degrees of

Master of Science in System Design and Management at the

Massachusetts Institute of Technology June 2019

C 2019 Yuqing Ke All rights reserved

The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or

hereafter created. 1

Signature redacted

Signature of Author

~

/

Ke

Signature redacted

JUN 27

Z019

Signature redacted

Joan Rubin Executive Director, System Design & Management Program

Assessing Various Software Development Methodologies and

Matching Software Development Methodologies with Projects

by Yuqing Ke

Submitted to the System Design and Management Program in Feb 2019 in Partial Fulfillment of the

Requirements for the Degree of Master of Science in Engineering and Management

Abstract

As the software industry evolves, various software development methodologies have become widely used in the industry. Most commonly used methodologies are Waterfall and Agile, along with less known alternatives, such as spiral and hybrid methodologies. When deciding on the methodologies, program managers tend to choose one based on the team preference or historical pattern. However, each software project is unique in its own way and has characteristics that are distinct from the previous projects the team has worked on. For each project, it is crucial to adopt a suitable methodology that help teams to produce the software that meets customer needs within schedule and budget constraints. Therefore, a practical question for every program manager is "How to find a suitable methodology for a specific project?"

This thesis is aimed to help program managers answer this question. We first explore how to evaluate each software development methodology based on the two-level decomposition of software development methodology, then analyze the project characteristics based on the

situational inputs in three categories: scope, schedule and budget. Thereafter, the thesis proposes a framework to match software development methodology with a specific project.

This thesis extends West's work in [1] by introducing a systems approach to assess a software project and a framework to determine the degree of compatibility between a methodology and a software project. The benefits of leveraging the systems approach are:

* The decomposition of a project enables a program manager to evaluate the input elements of a project and gain a systems view on the project characteristics.

The framework allows program managers to compare several candidate methodologies and choose the most compatible one using the mismatch scores, weighted summations that indicate the incompatibilities between the candidate methodologies and the project based on the ilities

ranking decided by the program managers. To demonstrate how to use this framework for a real world project, an example project is given. The detailed steps of calculating the mismatch scores

between three methodologies and the project are shown. The proposed framework can be used as a guideline for program managers to find methodologies for different projects with the

information gathered from project stakeholders.

This framework has some limitations. A major one is that, since the framework is quantitative based, induvial experience is used to evaluate the elements of methodologies and factors of projects. Further work can be done to improve the objectivity of the evaluation through the

surveys of industrial experts and members of teams adopting this framework.

Thesis Supervisor: Bryan Moser

Title: MIT SDM, Academic Director and Sr. Lecturer; MIT SERG, Associate Director U Tokyo GSFS, Project Associate Professor; U Tokyo GTL, Director

Acknowledgement

My MIT journey has definitely been a challenging but rewarding one. It not only pushed me in many ways that I could not imagine before, but also taught me how to be resilient when facing seemingly unsurmountable challenges. This thesis is one of the challenges that defined my MIT journey. I want to thank my parents, partner (C.B. Lim) and friends for their unwavering support through my entire MIT journey.

I want to thank my MIT classmates (Rajaram Srinivasan, Apoorva Parikh, Robin Jha and Ray Vetter) who encouraged and helped me to think about how to apply systems thinking to software development process. The countless discussions with them encouraged me to discover solutions in this thesis. The introduction of APH methodology made by Andrew Tsang helped me to form the framework introduced in the thesis.

Being a full time employee while pursuing my MIT master degree is not an easy feat. Without the support of my company, Akamai Technologies, I cannot even imagine how hard this journey would have been. I want to say thank you to all my team members for allowing me to reschedule numerous meetings with them to accommodate my busy school schedule. I also appreciate HR partner and my manager, who allowed me to have a flexible work schedule while managing a team of dedicated software developers.

I'd like to thank all SDM staff who helped me along the way. Their diligent work helped me navigate through the complex MIT system. I want to extend my special thanks to Professor Bryan Moser, who guided me through this thesis. His teaching and passion in program management inspired me of this thesis.

Table of CDOntetsL

Chapter 2 : Literature Review... 17

2.1 Software Development M ethodologies ... 17

2.2 Current Software Development M ethodologies ... 18

2 .2 .1 W ate rfall ... 18

2 .2 .2 A g ile ... 2 1 2 .2 .3 S p ira l ... 2 5 2.3 Comparison of Software Development M ethodologies ... 27

2.4 Hybrid Approaches... 31

2.5 Summary ... 33

Chapter 3 : Research M ethods ... 35

3.1 Stakeholders of software projects... 35

3.2 "Ilities" of software development methodologies ... 38

3.3 Characteristics of software projects ... 39

3.4 Connecting Development M ethodology to Software Projects... 41

Chapter 4: System Architecture of Software Development Methodologies ... 43

4.1 System decomposition principle... 43

4.2 M ethodology decomposition ... 43

4.2.1 W ork management... 44

4.2.2 Release Management ... 45

4.2.3 Team Management ... 45

4.3 Architectural Decisions... 46

4.4 W aterfall M ethodology with Architectural Decisions: ... 52

4.5 Agile M ethodology with Architectural Decisions: ... 54

Chapter 5: Factors that affect software project characteristics... 58

5.1 Iron Triangle... 58

5.2 Scope Factors ... 58

5.2.1 Pro posed C hanges ... 59

5 .2 .2 C u sto m e r ... 6 1 5 .2 .3 T e ch n o lo gy ... 6 3 5.3 Budget Factors ... 66 5 .3 .1 Fu n ding ... 6 6 5 .3 .2 T ea m reso urce ... 67 5.4 Schedule Factors ... 68

5.4.1 Schedule commitment level:... 68

5.5 Summary ... 69

Chapter 6: M atching Projects with M ethodologies... 71

6.1 Mapping software development methodology elements to ilities ... 71

6.1.1 Mapping Work Management Elements to Ilities:... 71

6.1.2 Mapping Release and Team Management Elements to Methodology Ilities... 72

6.2 Scoring Methodology based on Ilities... 73

6 .2 .1 W ate rfall ... 7 3 6 .2 .2 A g ile ... 7 4 6.2.3 Spiral Methodology with Waterfall Adaptation... 75

6.2.4 Comparison of scores for three methodologies ... 76

6.3 Mapping factor options to project characteristics ... 77

6.3.1 Scope factors on project characteristics:... 77

6.3.2 Budget and Schedule factors on Project Characteristics ... 79

6 .3 .3 Exam p le Project ... 80

6.3.4 Match Project with Methodology... 82

6.3.5 Weight ilities based on priority ranking... 84

6.3.6 Mismatch Calculation Method: ... 86

6.3.7 W eight Sensitivity Test ... 88

6 .3 .8 In sig hts... 9 0 6 .3 .9 Lim itatio n s ... 9 0 Chapter 7: Conclusions ... 92

Chapter 8: Future W ork... 95

Appendix I: Rationale of Values Assigned for the Factors Affecting Project Characteristics.... 97

Appendix I/ Rationale of Values Assigned for the Elements Affecting Software Development M ethodology iities... 128

Chapter 1: Introduction

The topic of this thesis is to propose and demonstrate a methodology that can meet the constraints of a software projects in scope, schedule and budget so that resources can be best utilized to produce a software product that meets customer needs.

1.1 Motivation

I started my software journey 12 years ago at Boston University as a research assistant on sensor network algorithms for indoor location detection. Afterwards, I started my career as a software developer at Akamai Technologies, which is specialized on the content delivery

network and cyber security. My first task was to develop and test a software component that can help Akamai monitor the status of thousands of regions around the globe. The goal of this component is to quickly identify the regions that either are slow to respond or lost connection completely. The output of the component is then analyzed by downstream components so that the problematic regions will be removed from serving traffic to Internet users. The software component is internal facing and has a strict requirement on the performance and scalability. The small team we were in was able to handle the development and set our own pace to finish the initial version of the software in 3 months.

Several months after the software component was deployed to the production network, the first incident happened. Due to a design oversight, the component starts to identify close to

50% of regions as bad. This resulted in the loss of half of Akamai's global capacity and the subsequent denial of services to major tier 1 customers, including eBay and Apple. Due to the severity of the incident, my team scrambled to find a quick but comprehensive solution to fix the issue. The stressful time I spent on debugging and applying the fixes taught me how important an initial robust design is when the software is used by large commercial customers. After the incident, we went through a thorough review with technical architects and designed a safeguard mechanism to prevent incidents like this from happening. The review process was something our team should have done before the implementation of the function. However, my team did not staff architects with the deep system level knowledge that was needed in the development. In order to ensure the service quality and avoid similar incident, the design review process was added in the entire development for products that have high customer impact.

As our system becomes more complex, my team was tasked with a user interface product that could help customers easily configure arrays of functions of the system. We followed a

similar procedure to develop the user interface product. We initially obtained the requests

through interviews and discussions with customers, followed by design review meetings, aiming to draft detailed requirements on how the interface should function. Later, we found out what we

designed were different from what the customers really imagined or hoped.

One cause was that some customers have limited understanding of the system, leading them to an inaccurate mental simulation of how the function should behave. Second cause was that the effectiveness or usability of UI related functions could not be properly evaluated before the implementation was presented. Only after the implementation was presented, the users realized that the implementation was not effective or simply opposite of what they had imagined.

Our release cycle, including design, implementation and testing, was around six weeks. This relatively long development cycle, combined with the lack of accuracy in customer requests and

inefficiency in the communication between the customers and my team, rendered a large portion of our work wasteful. To achieve higher effectiveness in meeting real needs from customers, we started to follow faster development process and higher frequency of releases with less time on the design phase. In this way, we achieved a fast feedback cycle with customers and constantly

improved our design based on customer feedback. After all, for customer facing projects, customer feedback should be the driving force of our design.

Our improved development process reduced our product development cycle to two to three weeks. That significantly reduced the duration of the feedback cycles of our user interface product. Beta customers were also much more engaged because their requests could be met in a timely manner.

However, the fast and design-light process was not well received by developers who focused on the development of backend API and infrastructures. For them, the customer requests were not their highest priorities, instead, the efficiency, scalability and performance were more crucial.

Pressing them to release function updates every 2-3 weeks was counterproductive based on the following reasons:

1. Back end development requires careful specification & architecture design. Architectural design typically requires the involvement of technical architects from different key functional areas. Unless the team is specially formed for architecture design and review,

it is difficult to gather all architects frequently due to their roles in other projects. If the frequent meetings are not conducted, the fast development cycle becomes less effective. 2. Most of the back end development solely benefits the internal teams. Without the external

customer commitments, having a longer release cycle can save lots of release overhead, such as testing environment setup, source code building, system level testing and compliance checks.

3. The effect of the back end changes can be system wide and propagative to the rest of the system. The frequent releases can cause unnecessary interruptions to other parts of the system.

4. The impact of back end code changes can cause high severity incidents in the production. Frequent releases will increase the chances of having such incidents.

As a lead who manages both front end and back end teams, I was tasked to find a solution to satisfy developers from both ends. The initial solution was to find a reasonable process that lasted around 4 weeks, a compromise that was made between 2 weeks (initial user interface development cycle) and 6 weeks (initial back end development cycle). This method was poorly received due to the following reasons:

1. The user interface development team had to come up with a list of features changes worth of two previous cycles (2 weeks) and fit them into a 4 week release cycle. However, due to the infrequency of release, the change could not be evaluated by the beta customers till

after 4 weeks. This rendered some work done without proper customer feedback, increasing risk of misjudging customer requests.

2. The back end development team was forced to reduce the time spent on design and testing phase, rushing to release the software without proper architectural design or

system testing, therefore, resulting a problematic release and causing issues for customers. To fix these issues, the entire development team had to drop out of regular release cycle, causing cascading delays to the release pipeline. It also cost effort for the operation team to revert back the code line. The end result was losing valuable

development time and revenue from customers.

3. Testing was the last stage of the development cycle. Therefore, testing duration was squeezed by the tight release. Additionally, QA/testing team was unfairly blamed for not properly testing the software when the algorithm or feature was not properly executed in a tight cycle. This unfortunately created the tension between the developer and QA team.

After the initial failure, we discovered that 4 week cycle was a compromise unable to satisfy either team. Admittedly, with some improvements, such as setting up staging release to

demonstrate the beta customers and limiting the number of change requests in the back end development, the proposed 4 weeks cycle could have been more effective. However, what we discovered was that the fundamental difference in requirements of software projects should play a role in determining the software development processes.

My experience at work led to believe that many program managers ran into similar issues when choosing a development methodology for a complex commercial software project, which has its own unique characteristics than previous projects. Applying one development

methodology for all software projects equals to making a compromise that does not bode well with any project. It is especially true when developing cloud-based software. For the front end, user interface needs to be simple but capable enough to allow even the most tech-savvy

customers to customize the options for their changing business needs. For the back end, the software ought to be capable of handling a large number of requests from end users that are globally distributed. Moreover, the service provided by the software needs to achieve high availability indicated by the service level agreement (SLA).

Is there a single innovative product development methodology that produces software that shows the flexibility in meeting customer's changing needs while ensuring high level of performance and availability at a large scale? Or utilizing different software methodologies for

different types of software projects is a better way to take advantages of the benefits provided with various software development methodologies?

In established firms, the software projects vary, but the team size or dynamics tends to stay stable. Is there a way to map software projects to development methodologies so that the established firm can take on changing software projects with a stable team organization?

Developing complex commercial software faces budget and deadline constraints. The high quality software that is overbudget or misses the deadline can be detrimental to its commercial success. If multiple software development methodologies are used for different software components, will the entire project still meet the budget and deadline restraints?

All these questions lead to this thesis, which is to help software teams and program managers discover software discover methodologies for their various projects and understand what

elements are important to keep the project on time and within budget while ensuring the software products meet the requirements.

1.2 Purpose

The purpose of this thesis is to understand what crucial elements should be analyzed for the software development methodologies. Based on the elements analyzed and the projects at hands, we can determine if there is one methodology that matches the project better than the others.

In most software companies, software projects vary based on the market needs and customer requests. To ensure the products are competitive in the market, the software product manager tracks the market trend and manages the expectations from the customers. As

companies update the products to cater the changing market, the architectures of their software systems become important foundations to their successes. For example, Amazon web service's micro service architecture, Akamai's globally distributed network architecture and Google's unparalleled data collection/analyzing platform are all the core competences of these companies. Services, such as Amazon's storage cloud, Akamai's content delivery service, Google's

advertisement platform are built on their unique architectures.

For cloud-based services like these, there can be different projects that develop different parts of the system, including front end UI, high level services, supporting platform and

infrastructure software. In this thesis, we will examine what factors differentiate these

development projects. Then the thesis will explore if any specific methodology is better suited for a project that shows certain characteristics.

1.3 Industry Trend

The incremental and iterative nature of Agile methodology enables organizations to dynamically change the functions of their products based on the flexible customer requirements. Due to this, Agile gains popularity in the software industry as many organizations seek ways to improve the release agility to meet customers' changing business needs.

The success story of the Agile methodology drove a large number of businesses to give up its original methodology and adopt this new methodology. There are two major reasons behind the success of Agile. One reason is that the popularity of open source libraries and lightweight development empowered the software developers to build core functions iteratively,

even without too much consideration on the design patterns, core algorithms or data structures, since most of those are hidden and well packaged in the open source libraries. The other reason is that Agile is effective when the project requirements are not clear initially, and the sprint based development methodology quickly helps all stakeholders to reveal more information to reach clearer requirements.

This Agile adoption trend however is a double edge sword to the software industry. Although Agile methodology speeds up the release upgrade cycles of many projects, especially the consumer facing ones, such as phone Apps, the negative effects of blind adoption are also evident.

The obvious one is that many program managers blindly follow the industry trend without considering the idiosyncrasies of the software projects. The program managers might also be instructed to adopt Agile methodology from the upper management, and the explanation given by the upper management is that the company has to become more agile and release the software in a more frequent cadence to meet the customer requests.

Another downside is that adopting Agile without considering the aspects of the team formation or dynamics is counterproductive. Agile works well in teams that already established a collaborative relationship. Forming a new team with Agile can be challenging. Additionally, the makeup of the team is important because of collective expertise plays a big role of the quality of the final products. Due to the frequency of the meetings and communication required in an Agile team, it is difficult to dedicate a high level software architect in a single team, limiting the collective expertise level of the team.

Industry-wide Agile adoption becomes more damaging when teams simply adopt this methodology to avoid the requirement collection and the design debate phase. The main reason cited by these teams of adopting Agile is that 'customer feedback is the key, so build fast and let the customer decide'. While the customer requests should serve as the core drivers of the feature development, the standardizing of codes or libraries need to be determined during the design phase to avoid repetitive work and ensure the code quality. To meet constant changes requested by customers, software development should ensure that the software has a solid foundation to build on. This is more attainable if the requirement collection or the design phase yields an effective architect and predictive functional roadmap of the potential upgrades.

My experience at several software developer teams in the past decade led me to believe that most program managers tend to follow the popular methodology in the software industry rather than examining the individual projects with a set of reasonable metrics. This thesis can cast some lights on what metrics program managers should consider for a software project and if the entire project should follow only one methodology.

1.4 Problem Statement

To find a suitable software development methodology for a software project By searching for the methodology that mismatches the project the least

Using a framework that calculate the total mismatch scores between ilities of various software development methodologies and software project characteristics

Chapter 2: Literature Review

2.1 Software Development Methodologies

What are software development methodologies? In [2], the author listed the following definition: Software development methodology is a framework that is used to structure, plan, and control the process of developing an information system. The framework includes but is not limited to:

" How the development team is formed

" How the software development phase is planned and executed

" How the communications within the team and outside the team (to the customers and external teams) are organized

The software development methodology emerges after 1960s. According to Elliott, the main target of this methodology in the 1960s, a period when the computing systems were building sized mainframe computing machine, was "to develop large scale functional business systems in an age of large scale business conglomerates. Information systems activities revolved around heavy data processing and number crunching routines" [3].

Software development methodologies have since evolved together with the advancements of the software systems. The need of software development methodology becomes evident when the software systems became more difficult for developers to add features and fix bugs. Software development methodology imposes a disciplined process upon software development with the aim of making software development more manageable and more efficient.

For years, several methodologies emerged as widely adopted methods in the industry. According to [4], these methodologies can be categorized into two: heavyweight and lightweight development methodologies. Waterfall, as one of the heavyweight methodologies, have heavy aspects such as elicitation and documentation of a complete set of requirements, architectural and high level design development and inspection. Agile, as one of the lightweight methodologies, does not impose formal procedures such as formal requirement documentation or architectural level design and inspection. Instead, it places more emphasis on team members, interaction, customer collaboration, and changes.

The critics of the heavyweight methodologies argue that such methodologies are overly regulated, planned and micro-managed. They state that such methodologies are not suitable to volatile business environments where the changes are rampant and hard to predict. In [5],

Cockburn pointed out that "people are the most important factor in software development and heavy weight methodologies ignore the creativity, treating the developers as predictable components similar to what they treat their processes".

The critics for the lightweight methodologies however argue that 'not every problem can be sliced and diced into the right pieces for speedy incremental refinement.' [6, pp. 67-69] One major disadvantage of Agile and other lightweight methodologies is that they are difficult to

scale for a large team. In [5], Cockburn believes that for Agile, the face-to-face communication breaks down and becomes more and more difficult and complex with a team sized of 20 or more. Another concern regarding to Agile is that its focus on early results for customers. In [7] , Clifton

states that "Over focus on early results in large systems can lead to a major rework when the architecture doesn't scale up".

2.2 Current Software Development Methodologies

2.2.1 Waterfall

The waterfall methodology is a relatively linear sequential design approach for software development. In [8], Waterfall was explained as following:

Waterfall methodology "tends to be among the less iterative and flexible approaches, as progress flows in largely one direction ("downwards" like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, deployment and maintenance."

The methodology was first introduced by Winston W. Royce. In his model, he defined seven stages in the waterfall model:

" Systems requirements " Software requirements " Analysis " Program design " Coding " Testing, and

* Operation

The basic model is illustrated below:

SYETEU SOE1WARE RIEWUREWLNT6 MOGRAMI

D~LFILN

_E

Figure 2-1 Royce's Basic Waterfall Model

Royce foresaw various the shortcomings of the basic model in [9]:

"The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from

analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software

requirements upon which the design is based and which provides the rationale for everything are violated. "

To fix the issues, he proposed several measures:

" Insert a preliminary program design phase between the software requirements generation phase and the analysis phase

* Do it twice, create the first result that produce the simulation result for the final product * Emphasize on testing phase by planning, controlling and monitoring testing

* Involve customer in an early stage so the customer is committed early before the final delivery

The final model is illustrated below:

Figure Watefamu 2- Royce's Mode Lnam

ONTSMS *

s f t w a r e r a"r s c ont

leadng t anexpesiverewrk ad wate f reourcs fr th devlopent eam

T a r s d g . t p t f r .r sIgn _{at th} * NLn4knm 044 % _u0w464~ 4 LEJ] ANN4 =014 WT11W* calyx S4.44 I F we"TAIN VOA4

Figure 2-2 Royce 's Final Waterfall Model

As Waterfall methodology is designed for large and complex projects, the basic waterfall

model was initially used by the Department of Defense for their standard for working with the

software contractors.

The critiques of Waterfall model are:

The clients might not know the specification or requirements at the initial phase of the

project. When more information is presented, clients might change the project requirements,

leading to an expensive rework and waste of resources for the development team.

The architect or software developer might not initially predict the future requirements or

initial stage. In Waterfall model, the design is based on a set of fixed requirements, leaving little room for requirement changes during the development cycle.

2.2.2 Agile

The Agile methodology was described by the "Manifesto for Agile Software Development" in [10]. The core principles of Agile are:

" "Our highest priority is to satisfy the customer through early and continuous delivery of valuable software." Agile teams focus on the constant communication with customers either through product manager or developers directly and present the prototype or beta products to the customers for feedback.

" "Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage." This is closely related to the first principle. After the demo of beta products, the customer feedback is received. Based on the customer feedback, the development team embrace the changing requirements. " "Deliver working software frequently, from a couple of weeks to a couple of months,

with a preference to the shorter timescale." To ensure the product meets the customer changing requirements, the frequent delivery of the product is needed. Fast delivery also reduces the feedback cycles of beta products.

* "Business people and developers must work together daily throughout the project." Typically in Agile team, product manager, product architect and development team work closely to meet the customer requests.

" "Build projects around motivated individuals." This requires the team members are fully engaged in both discussions and development.

" "Give them the environment and support they need, and trust them to get the job done." The daily standup meetings are aimed for the developer to discuss ongoing issues at the development phases, enabling the team to work on the impasse together.

" "The most efficient and effective method of conveying information to and within a development team is face-to-face conversation." This is done through daily meetings. Due to the frequent face-to-face communication, the team dynamics is key to the success of Agile teams.

" "Working software is the primary measure of progress." At end of each sprint, a working

software is expected for the customer to test.

" "Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely." This means that only the

development team determines how many items can be taken from the product backlog and brought into the sprint backlog. Sprint backlog scope can be modified during the sprint by the development team through coordination with the product owner/manager and customers.

* "Continuous attention to technical excellence and good design enhances agility."

" "Simplicity--the art of maximizing the amount of work not done--is essential." Although listed as one of the principles of Agile, this one is not unique for Agile methodology. The simplicity for software development generally means that the software is easy to

maintain, easy to understand and easy to verify and test.

* "The best architectures, requirements, and designs emerge from self-organizing teams." This is one key reason why the composition of Agile team is crucial. The accumulative expertise in the team directly affect the quality of the final product. Also in sprints, the iterative change requests (story points) are assigned to the engineers voluntarily, rather than by the managers.

" "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly." This is done through sprint review meetings at the end of each sprint.

As Cockburn stated in [5], the claimed advantages of Agile are:

" reduces the cost of moving information between people through colocation and person to person communication, and

" reduces the elapsed time between making a decision to seeing the consequences of that decision through customer involvement and frequent delivery of incremental updates " focuses on the talents and skills of individuals, molding the process to specific people and

teams. In this way, the methodology is designed to capitalize on each individual and each team's unique strengths

In [11]. the authors surveyed a group of 20 Agile development experts working in and with large companies about the most challenging organizational impediments of adopting Agile. The authors listed out ten most common impediments. One major impediment is the shift from culture of individual workers to culture of real teams and teamwork. Since Agile requires the team work to collaborate on basic feature implementations and technical issue resolution, the engineers who are used to the individual work in traditional methodology might find intensive teamwork challenging.

According to the survey, the second common impediment is that some organizations treat Agile methodology as "silver bullet solution" and conduct "superficial adoption" of this

methodology. The misunderstanding of equating agility and productivity (rather than for

adaptability), and the lack of educated executives lead to belief that meaningful problems can be solved by saying "we do agile" and going through the motions, with no deep understanding or change by the leadership team.

Another concern about Agile is that the methodology is its customer driven development. If the software product is customer facing, the constant involvement of the customers can be positive to the final product. However, for internal projects or projects that build up the

foundation of a complex software system where the customer feedback is not needed, the frequent releases of the software might cause unnecessary overhead in testing and release engineering.

In [12, p. 46], Karlstr6m conducts case studies on three large commercial software projects that adopts Agile and compiled a list of the positives and negatives, which are useful to understand the benefits and challenges of adopting this methodology. I updated the table to include additional points from the materials I have read and experiences I had with Agile methodology:

A

Table 2-1Pros and Cons ofAgile Methodology

Area Agile feature Effect

Planning and Most important feature first + Early feedback on customer impacting

prioritization features

+ No delays of important features

+ Easier to fulfil budget requirements without compromising on the key features

Micro planning + Avoidance of requirements' cramming + Fixed plans avoided

- Little support for long-term plans Communication Coherent teams + Good internal communication

and follow-up _{-Potential isolation of agile team}

-Require motivated and similarly high quality developers

-Hard to implement with a newly formed team

Continuous testing + Fast verification of iterative changes + Better feedback of quality

-Requires dedicated QA resource

Small, manageable tasks + Give developers better control on tasks Continuous integration _{+ Avoid the release day issues through}

frequent integration

+ Progress measure for management and customers

-Need clear roadmap to maintain focus on large scale and long lead time projects

Process model Customer involvement + Continuous feedback

and roles _{+ Relevant features}

+ Dedicated resource to ensure customer feedback

-Important but not customer facing function might get lower priority, leaving expensive technical debt down the road

Project Engineering-level + Engineers feel motivated and less resistant

management empowerment to adoption

- Management training needed

Focus + Engineers focus on past and current release, managers on current and future release

+ (-) Technical issues raised (too) early for management

2.2.3 Spiral

Spiral model was first introduced by Barry Boehm in 1986 via his paper [13]. This model has been evolved based on Boehm's experience with Waterfall in large projects. In this model, the author describes the software development process as an iteration on four phases of activity and could be fitted to various approaches and methods. The four phases include:

" Determine objectives, alternatives and constraints " Evaluate alternatives, identify and resolve risks " Development and Test next level products " Plan the next iteration

According to Boehm [14], spiral model is characterized by repeatedly iterating a set of elemental development processes and managing risk so it is actively being reduced. The author also shows how the model can be used for a more cost-effective incremental commitment of funds. The following diagram illustrates how this iteration evolves for a project:

PLAN

NEXT PHASES

Figure 2-3 Boehm's diagram of Spiral Development [13 p2]

As illustrated in Figure 2-3 Boehm's diagram of Spiral Development [13 p2], each iteration involves development, prototyping, risk analysis and planning based on the analysis of the previous prototype. Boehm stated the essence of the model:

"The spiral development model is a risk-driven process model generator. It is used to guide multi-stakeholder concurrent engineering of software intensive systems. It has two main distinguishing features. One is a cyclic approach for incrementally growing a system's degree of definition and implementation while decreasing its degree of risk. The other is a set of anchor

REVIEW COMMITMENT PARTITION am PLAN Lff C"M mom PL#A CUMULATIVE COST PROGRESS THROUGH STEPS EVALUATE ALTERNATIVES;

IDENTIFY. RESOLVE RISKS

RISK ANALYSIS I RISK ANALYSIS RISK w-ol ANALYSIS _000P OEAI DE .,,. VERAFY DETAILED WOUWAOM SFMAWDESIGN VEXT-LEV PRO UC NE VES, TIVIES, ANTS DETERMI OBJECTI ALTERA CONSTFR

point milestones for ensuring stakeholder commitment to feasible and mutually satisfactory system solutions."

Spiral can be seen as a method that sits between the Waterfall and Agile because of its iterative updates on the original waterfall methodology. It serves as a model generator in a way that at the start of a cycle, all project's critical stakeholders must participate to review the project's risks and choose a working process model accordingly.

Spiral model is beneficial for projects that costs and risk evaluations are important. When a project exhibits medium to high risk level, spiral model can be utilized to reduce the risk and assign funding through iterations. Another use case for spiral model is that the users are not certain about the requirements, having iterative prototypes will be able to guide the users to gain better understanding of the requirements of the project.

2.3 Comparison of Software Development Methodologies

To understand what methodology to choose for a particular project, it is crucial to understand the differences of various software development methodologies. Many of the comparisons are based on the anecdotal evidence and survey results. In [4], Awad solicited feedback from software industry practitioners to evaluate which methodology has a better

success rate for different sizes of software development. His conclusion was that the Agile methodologies can provide good benefits for small scaled and medium scaled projects, but for large scaled projects, traditional methods seem dominant. This conclusion is consistent with the my experience in the software development industry.

To dive deeper into the characteristic differences among heavyweight and lightweight methodologies, Khan listed out differentiating characters in areas, such as customers,

requirement, architecture, size, primary objective, developers, release cycles, development life cycle, style of development, documentation, team members, client involvement, market and measure of success [15]. Furthermore, Khan suggested ideal situations for particular

methodologies:

* Developers with expertise and the project is small: Agile methodologies

* Developers not highlight skilled or the project is of large scale: Waterfall or Spiral methodologies

" Customer have defined business goal but the requirements are not freeze: Agile methodologies

" Reuse of existing software components: Component/Feature based methodologies However, the situation faced by most program managers hardly fall straight into these categories. Program managers tend to face combination of mentioned situations and need to make judgement calls on which characters of the project play key roles in determining the final success of the project. For example, if the project has customer requirements that are flexible, and the team is newly formed and do not have enough expertise, should program managers choose Agile methodologies because of the flexibility and potential change of requirements, or Waterfall methodologies to compensate the inexperience developers and low level of

collaboration in the team?

To guide information system design and development in medical field, Center for Medicare and Medicaid Services (CMMS) listed out the pros and cons in development methodologies, such as Waterfall, Spiral, Incremental, Rapid Application Development,

Prototyping [16]. The purpose of this guideline is to establish a standard for development of the medical field information system. To guide the project managers quickly decide on which methodologies to employ, the guideline also listed out the most and least suitable situations for each development methodology. When discussing Rapid Application Development (RAD), an iterative method, the author stated that the iterative method has the following principles [16, pp. 8-9]:

* Attempts to providing more ease-of-change during the development process. " Aims to produce high Prototyping (at any stage of development), active user

involvement, and computerized development tools. These tools may include Graphical User Inter Computer Aided Software Engineering (CASE) tools, Database Management

System(DBMS), fourth-generation programming languages, code generators, and object-orient techniques.

" Key emphasis is on fulfilling the business need, while technological or engineering excellence is of lesser importance.

" Active user involvement is imperative.

CMMS agrees that iterative processes are aimed at projects where user experience and business needs are valuable and the technological or engineering excellence is less important.

The author also stated Waterfall, a relatively linear development methodology, is difficult to respond to changes. The most appropriate application in for the situations where the projects such as mainframe-based and transaction-based batch system, where the system is large, complex and expensive and the project has clear objective and solutions.

Another interesting comparison method is proposed by Kitchenham in [17]: DESEMT. Although this method was created to evaluate the tools/methods developed by software tool developers amid a lack of standard and adoption of emerging developer tools, this method can be useful in evaluating the existing software development methodologies. In DESEMT, authors proposed nine evaluation methods, including three quantitative evaluation methods, four

qualitative evaluation methods and two hybrid methods. With some updates, these can be applied to evaluate different methodologies for software development

" Quantitative experiment: An investigation of the quantitative impact of methodologies organized as a formal experiment.

* Quantitative case study: An investigation of the quantitative impact of methodologies organized as a case study.

* Quantitative survey: An investigation of the quantitative impact of methodologies organized as a survey.

" Qualitative screening: A feature-based evaluation done by a single team who not only determines the methodology ilities to be assessed and their rating scale but also does the assessment. For initial screening, the evaluations are usually based on literature

describing the software methodologies rather than actual use of the methodologies. " Qualitative experiment: A feature-based evaluation done by a group of potential teams

who are expected to try out the methodologies on typical tasks before making their evaluations.

* Qualitative case study: A feature-based evaluation performed by someone/team who has used the methodologies on a real project.

* Qualitative survey: A feature-based evaluation done by people/teams who have had experience of using the methodology, or have studied the methodology. The difference between a survey and an experiment is that participation in a survey is at the discretion of the subject.

" Hybrid method 1: Qualitative effects analysis: A subjective assessment of the quantitative

effect of methodologies, based on expert opinion.

* Hybrid method 2: Benchmarking: A process of running a number of standard tests using alternative methodologies and assessing the relative performance of the methodologies against those tests.

There are limitations for those methods:

" To gain comprehensive comparison results, a large number of representative projects need to be evaluated due to the complex situational inputs of each project

* Benchmarking requires a same team working on the same project with different methodologies, a highly unlikely scenario for any sensible business unit

" Survey objects tend to have bias and the subject selection is crucial in obtaining a fair comparison

* It is difficult to come up with a fair assessment for the relative performance of the methodologies since each project values different aspects of final deliverables Utilizing a systems approach in [1], West viewed the entire software development methodology as a socio-technical system and reducing it into basic elements. To compare different methodologies, each element of a methodology was assigned with a value based on author's prior experience. The combination of each element contribute to the differences in life cycle ilities of the methodology. This decomposition is useful in a way that it presents the project manager or product owner a way to look at a project as an intuitive integration of elements that they can make separate decisions on.

Furthermore, West analyzes how each basic element affects the life cycle properties of the methodology, enabling program owners to understand the trickle-down effect of each decision they made on the elements. West's work casts lights on how each project can be designed based on the elements project manager chosen to achieve an optimal tradeoff between life cycle ilities. By deciding on what ilities to focus on, the program managers can choose a process to maximize these ilities. This process naturally turns the designed methodology into a hybrid one that potentially takes advantages of the benefits provided by each methodology. However, the limitation of the method is that the tradeoff is between two ilities. In reality, most project managers need to trade off more than two ilities. Nonetheless, West's work paved ways

to a flexible project design through adopting favorable elements of the canonical software development methodologies.

2.4 Hybrid Approaches

Although Agile development methodology gains support in the industry, traditional development methodologies, such as Waterfall, still hold ground in many software development organizations. Other than projects that are designed to take advantages of the benefits that Waterfall methodology provided, many organizations are in the process of transitioning to Agile methodology, hence both traditional methodologies and Agile methodology coexist in some organizations. During this transition, it is imperative to find a workable model to incorporate both methodologies.

In [18], based on his extensive experience, Flahiff stated that project manager can successfully integrate Agile in a Waterfall environment to improve project predictability, cost effectiveness, and ultimately success. He proposed a three-layer model in which a project

manager can use the understanding of project and product management combined with deliberate methodology selection to manage an Agile project in a Waterfall context. The model is graphed below:

In this model introduced in [18], a series of successive layers or envelopes are used to insulate the Agile team from other layers. For example, in the Agile iterations, the agile team works to execute the software development, unit testing, component testing and system level integration testing. The Agile team lead is the first line of issue resolution. The Agile team lead is responsible for intra-team communication, coaching the developer team, and helping prioritize work when the team is stuck. Generally, the Agile team lead is responsible for the health and progress of the entire team. Impediments that the Agile team lead cannot resolve are escalated to the project/program manager. The project/program manager also acts as a protector for the team keeping the complexities of the outside organization from impeding their progress.

The predictive elements are developed through traditional methodologies, such as

Waterfall. These elements are suitable for Waterfall because the requirements are clearly defined and the lead is long. They may include hardware deployment and enterprise level acceptance testing. The interface between the hardware and software should be clearly defined to ensure the compatibility of the two. The third layer is where the entire project interfaces with the rest of the enterprise portfolio. At this layer, the project manager coordinates with external projects, with the enterprise release planning and enterprise reporting.

Utilizing this model, the author proposed a workable scheme that keeps the Agile team protected, especially during the iterations. No external (Waterfall) team should interrupt the progress or the delivery of the final delivery of the iteration. To coordinate the teams operated using different methodologies, the author proposed several formats of the communication forums:

All the forums, except for 2, require the participation of all teams. The details of each forums is explained in the followings:

1. Joint iteration planning: Planning meeting done in a unified team on the same day, in the same room if possible

2. Iteration coordination: Identify the issues that require cross-team collaboration. Project/Program managers and Agile team leads come together daily to discuss the identified cross-project dependencies and any new items that the Agile team leads may have heard from their attendance at the key team meetings

3. Joint sprint demos: At the end of each iteration, the working software is demonstrated to the project sponsors. This demo is done as a joint team.

4. Joint retrospectives: After the iteration, each team should review their processes and work on improvement.

This model presents a way that Waterfall and Agile methodology can coexist in an organization. The author proposed this as a way to help the organization enable the flexibility of Agile during software development process while ensuring the rest of the company, which are operated under Waterfall, function well with Agile teams. This is one proposed model to apply Agile methodology in a large and complex project or organization, creating a hybrid approach that utilize the Agile methodology for software development and Waterfall for hardware deployment and enterprise project integration.

Another hybrid approach was proposed in [19]. To facilitate the mobile software development, Rahimian suggested a model that combines Agile with RUP (Rational Unified Process). The core of this approach is to run Scrums for all five stages that were inherited from RUP. Those five stages include Business Modeling Analysis & Design, Implement, Testing, Deployment and Configuration. The goal of this method is that the Rational Unified Process is used as a skeleton in the hybrid method while Scrum is embedded into the Rational Unified process to offer project management and tracking mechanisms through structured ceremonies (daily stand up meetings, sprint planning meetings), roles, and artifacts (story points and project backlogs). However, the author did not use specific cases to showcase how this approach is helping the mobile software development.

2.5 Summary

The literature review section discussed several software development methodologies, including waterfall, Agile, spiral and hybrid methodologies. It presented other researchers' views on these methodologies and the comparisons among these methodologies. Most of the researches mainly focused on generic discussions of the methodologies based on the project experience and/or surveys of numerous software development experts. A few comparison frameworks have also been introduced, but many lack the depth in explaining which elements contribute to the differences in various software development methodologies. Most importantly, these research work, except for West's thesis, failed to address a fundamental question: how to match a methodology with a software development project?

West's work in adopting systems approach to compare different methodologies is effective in two folds:

1. Decomposing a methodology into basic elements provides a way to map these elements to methodology ilities, enabling program managers to understand how each element affects the methodology ilities.

2. Moreover, West's systems approach shows how a decision on each element can change the desired properties of a methodology.

His work sets a solid foundation in exploring how to match a project with a development methodology. One limitation of West's work is that his model lacks the consideration of some important factors that determine the project characteristics. These Factors, such as the type of proposed changes, the technology adopted and the team resource available, are decided either by customers or other stakeholders and serve as the inputs to the program managers. Together, these factors determine the characteristics of a project, which play crucial roles in determining what methodology to adopt. Another limitation of this work is that the trade space analysis is based on the pairwise tradeoff between ilities. In real world, program managers trade off multiple ilities at the same time. To do this, program managers commonly rank desired ilities based on the project analysis. To help program managers decide on a methodology, a framework should consider how the ilities ranking impact which methodology to adopt.

This thesis is going to address both limitations by: (1) introducing the factors that determine the project characteristics and (2) proposing a framework that takes multiple ilities ranking into consideration.

Chapter 3: Research Methods

In this chapter, we will first discuss what "ilities" (defined in 3.2) are considered to differentiate software development methodologies. Then we will discuss what characteristics should be weighed when assessing a software project. After introduction of both methodology ilities and project characteristics, we will explore if the relationships can be established between them.

3.1 Stakeholders of software projects

Meeting the needs of stakeholders are the goals of software projects. For a better understanding on how the stakeholders needs affect the software projects, it is beneficial to analyze the stakeholders in a commercial software project.

The diagram below illustrates the major stakeholders and the communications among them:

Demo & Product

T Ch stomea Resource allocation

Technicaa _{prioritizaton}

solutions _ect

Upper Management issues &

progress-Technical

Feedback & Needs ,issues , Proj cost

Product Owener/Manager Engineering Manager/Scrum Master

easibillty

Progress Budget Allocalon

& issues fare

Cap ate Financial Customer Needs _{Technical Architecture}

Product Architect

Guidance & Resource Alocati,

External Teams Software Engineers

Testing Results ALCoo rdi natfi

and issue reporting & Progres

, olutos _{Pors ors}

Testing Engneers/QA r Ma

land Progress Release Engineers

As indicated in the diagram, all stakeholders play key parts in a commercial software development project. The product owner/manager and engineering managers interact directly with customers to understand their needs and solve their technical issues.

Product Owner/Manager: The major stakeholder that interacts with customers. The main role of product manager is to research the market, design the product business user case, collect the customer needs and conduct demos of new products to customers.

Product Architect: An intermediate stakeholder to bridge the product manager and engineering manager. The role of product architect is to turn the business needs of a product to executable technical projects by translating customer needs to technical requirements and

software architectures.

Engineering Manager: Manage software developers and guide the software engineers to solve the technical issues.

Software Engineers: The key participants of software developments and major technical contributor to software projects. Their roles include: generate codes, solve technical issues and conduct unit tests and code reviews to ensure the quality of the code.

Testing Engineers/QA: The key stakeholder to ensure the system level quality, performance and avoid functional regression of the software. Work closely with software engineers to ensure the code lines are properly tested in component and system level.

Program Manager: Interact internally to ensure all parties that the project is on time and within budget. Communicate externally on cross-team collaboration to ensure the various teams that are involved are on track for their deliveries. Program manager makes decisions on which software development process is used and which teams should be allocated for a project.

Upper management: Decision maker on the project viability and budget allocation. They make decisions on if the product should be developed or prioritized and how much total resource should be allocated.

In Agile methodology, the scrum master role can be assigned to either engineering manager or individual software engineer in round-robin manner. To achieve the agility, software engineers, testing engineers, release engineers and even product architect and product manager are assigned to a single team. The cross functional team ensures the issues can be resolved quickly and comprehensively with views and expertise from major key stakeholders. The

program manager's internal facing role is reduced in Agile methodology since release

management work is partially executed by the scrum master. However, program managers are key in cross team collaboration project because they are responsible to coordinate with the external teams to allocate enough resource to facilitate the project development.

Depending on the nature of the projects, the stakeholders will change. For instance, if a team develops a software that is aimed to be used by a large group of external users. The product manager's role is important in determining the features and usability of the product, and the entire development should include constant customer feedback. Product architect will work closely with both product manager and engineering manager to design a technical feasible architect for the product. Other major stakeholders will ensure the project is developed to meet needs within budget and schedule constraints. However, if a team develops a software that is aimed to provide supporting function for internal teams, there is no need to collect the feedback from the external customers. The roles of product manager and product architect are not evident. In this case, we can exclude these two stakeholders. The engineering manager, or a technical lead in software engineering team, can be an excellent resource to gather requirement from internal customers. In this case, the stakeholder network will be reduced to following:

Resource allocation and priorttization

Prolect Uper Managemen PrqjcostJ Engineering Manager/Scrum Master

-rjtcs

Pro ess quirements Budget Allocation

I Software _{Corporate Financial} Cap

Guidance &

Resource Allocation _{Interfaces PI External Teams}

Software Engineers & function

Testing Results Coordinati

and Issue reporting & Progres

Iolution s Progress #rogress

Testing Engineers/QA _{PormMngr suc}

-Ures nd Progress Release Engineers