• Aucun résultat trouvé

Crystal ball planning for analytics implementation in Singapore

N/A
N/A
Protected

Academic year: 2021

Partager "Crystal ball planning for analytics implementation in Singapore"

Copied!
66
0
0

Texte intégral

(1)

Crystal Ball Planning for Analytics Implementation in

Singapore

By Cong Fang

Ph.D. Molecular and Medical Pharmacology, University of California, Los Angeles, 2010 B.S. Biological Science, Wuhan University, 2003

Submitted to MIT Sloan School of Management and Department of Biological Engineering in partial fulfillment of the requirements for the degrees of

Master of Business Administration and

Master of Science in Biological Engineering MASSACUE OF TE

in conjunction with the Leaders for Global Operations Program at the

JUN

MASSACHUSETTS INSTITUTE OF TECHNOLOGY

LIB

June 2014

K 2014 Cong Fang. All rights reserved.

The author hereby grants to MIT permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part in any medium now known or hereafter created.

Signature redacted

A utho r

...-Department of Biological Engineering and the MIT Sloan School of Management

Signature redacted

May 9, 2014

18

2014

RLARIES

Certified by... C-IrIfIIU Vy Acce te~d b V ...

Peter C. Dedon, Thesis Supervisor Professor of Biological Engineering

Signature redacted

...

Roy E. Welsch, Thesis Supervisor Professor of Management, MIT Sloan School of Management Professor of Statistics and Engineerjpg Systemrs, MlTfngineering Systems Division

Signature redacted

p y...,--'~i~ ...

Accepted by...

...

.. ... ...

Forest M. White, Committee Chair for Graduate Program Professor of Biological Engineering, Department of Biological Engineering

Signature redacted

iviaura rerson, Director for IVIMA Progra

(2)
(3)

Crystal Ball Planning for Analytics Implementation in Singapore

by Cong Fang

Submitted to MIT Sloan School of Management and Department of Biological Engineering on May 9, 2014 in partial fulfillment of the requirements for the degrees of

Master of Business Administration and

Master of Science in Biological Engineering

Abstract

Amgen is building a new drug substance manufacturing site in Singapore (ASM). This project identified and mitigated the risks associated with implementing analytical technologies to facilitate the design and implementation of the quality control process in the new plant. Here, a systematic risk evaluation model was established to identify sources of high risks from implementing the analytical technologies in ASM, evaluated business cases and proposed technical strategies for risk mitigation. This project also included a case study about the cross-functional initiatives at Amgen, and made recommendations regarding how to bridge the gaps between the technology development in R&D and the technology implementation in manufacturing sites.

Thesis Supervisor: Peter C. Dedon

Title: Professor of Biological Engineering

Thesis Supervisor: Roy E. Welsch

(4)
(5)

Acknowledgements

I wish to acknowledge the Leaders for Global Operations program, Amgen, Inc., my thesis

advisors Peter Dedon and Roy Welsch for their support of this work.

Special thanks go to my internship advisors Jared Byrne and Bryan Steadman, for their mentorship and guidance. They took considerable time to contribute ideas and discuss the fundamentals and nuances of biopharmaceutical manufacturing processes, quality control processes, operations, project management, and regulations. I would like to also thank Kimball Hall for sharing her leadership experience with me and facilitating interactions that greatly contributed to this work. Thanks also go to the MoF Leadership team, the Process and Product Development group, the Raw Material Group, the CMC Life Cycle Management Group, the Operation Risk Management Group, the Regulatory Affairs department, the AoF group, the Quality Control department at ARI, the CP2 manufacturing group at AML and the Global Analytical Science group for their highly productive cross-functional collaboration and technical support. While many people from these groups helped with this research, I am particularly grateful to Colin Cao, Dave Meriage, Dan Zhou, Becca McCoy, David Kolwyck and Tony Wang for providing media samples, acquiring and analyzing data.

My final thanks are reserved for my husband, Jin Zhang, and my parents, Caijin Fang and

Shenghua Xiong, for their unconditional love, generous support, amazing patience and rational optimism. Without their company, I could have not gone so far. When I received the PhD degree,

(6)
(7)

Contents

1 Introduction 11

1.1 Project M otivation... 11

1.2 Problem Statem ent... 11

1.3 Project G oals... . 12

1.4 Project A pproaches...12

1.5 T hesis O verview ... 13

2 Amgen Inc and the Biopharmaceutical Manufacturing Process 16 2.1 C om pany B ackground... 16

2.1.1 C om pany H istory ... 16

2.1.2 C om m ercial Products ... 16

2.2 The Past and Future in Large-scale Production of Monoclonal Antibodies ... 18

2.2.1 Current Practice of Biopharmaceutical Manufacturing Processes ... 18

2.2.2 Recent Advances in Manufacturing and Analytical Technologies ... 22

2.2.3 Drug Substance Manufacturing Site in Singapore ... 25

3 Problems Identified in the Analytics Implementation at ASM 26 3.1 Current State Analysis Identified Major Challenges and Gaps ... 26

3.1.1 Key Stakeholders and Sources of Information ... 26

3.1.2 Q ualitative Interview s ... 27

3.1.3 Current Gaps and Challenges ... 28

3.2 K ey Problem Solving Strategies ... 30

3.2.1 Optimized Decision Making Process ... 30

3.2.2 Quantitative Risk Assessment Model ... 31

4 Risk Mitigation of Advanced Analytics Implementation at ASM 36 4.1 Identification of Advanced Analytics with Different Risk Levels...36

(8)

4.1.1 Application of the Risk Assessment M odel ... 36

4.1.2 Risk Contributors to Technologies with High Implementation Risks...36

4.1.3 Raw Material ID Technology with Highest Risks...39

4.2 Feasibility Evaluation of Using Raman Technology for Liquid Media ID ... 42

4.2.1 Advantages of Replacing Solid Media with Liquid Media...42

4.2.2 Challenges in Liquid M edia ID ... 44

4.2.3 Root Causes A nalysis ... 46

4.2.4 Experimental Design for Feasibility Assessment...47

4.3 Evaluation Results for Liquid M edia ID... 49

4.3.1 Seven Liquid Media Samples Were Identified by Optimized Raman Technology...49

4.3.2 Product Identification Achieved by Optimized Raman Technology...51

4.4 Implementation Challenges and Plans for Raman Technology at ASM...53

5 Conclusions, Lessons Learned, and Recommendations 56 5.1 Summary of Raman Technology Development at Amgen...56

5.2 Key Operational Lessons Learned from Developing Raman Technology at Amgen...57

5.2.1 Dynamic Project Management in Development Processes...58

5.2.2 Reduce Unnecessary Delay ... 60

5.2.3 Enhance Collaboration Among Stakeholders...61

(9)

List of Figures

Figure 1. Upstream process for mAb production at commercial scale ... 21

Figure 2. Platform downstream process for mAb production ... 22

Figure 3. Stakeholders play different but connected roles in the analytics development ... 30

Figure 4. The optimized decision making process ensures the involvement of all stakeholders..32 Figure 5. Analytics implementation readiness is evaluated by risk management ... 33

Figure 6. Total score rank list of all analytical technologies for ASM ... 38

Figure 7. Demonstration of the principle for Raman Spectroscopy (upper panel) and raw Raman spectra of seven media samples used at ASM (lower panel) ... 42

Figure 8. Benefit analysis of liquid media versus powder media ... 44

Figure 9. An optical picture of a sample powder media under 50X microscope ... 45

Figure 10. A poor signal-to-noise ratio in the liquid media sample detected by current Raman tech n o lo g y ... 4 6

Figure 11. Seven liquid media and distilled water were tested by current Raman technology under optimized settings and then analyzed by Principle Component Analysis (PCA) ... 47

Figure 12. Raw Raman spectra for seven liquid media samples with decent quality ... 51

Figure 13. PCA clustering results of seven liquid media ... 52

Figure 14. Flowchart of Net Present Value calculation for replacing AAA to Raman technology for raw m aterial identification ... 55

Figure 15. The timeline and milestone events of Raman technology development at Amgen

... 5 8

Figure 16. A process map and task inventory model for Raman technology development ... 60 Figure 17. The expected, observed and optimized timeline for the Raman project development

... 6 2

Figure 18. Flow chart of optimized decision making process to connect stakeholders to g e th er...6 3

(10)

List of Tables

Table 1. List of interview groups, interviewees and interview content...28

Table 2. Risk causing factors and their corresponding weighted score...34

Table 3. Criteria and risk scores for various risk-causing factors...35

Table 4. High, medium and low risk levels are categorized based on total risk scores...36

Table 5. A partial list of analytics for ASM with detailed implementation risk scores...38

Table 6. Media types and properties for all seven liquid media used in ASM...49

Table 7. Liquid Media Identification Based on Discriminant Function Analysis...53

(11)

Chapter 1

Introduction

1.1 Project Motivation

As the leading biotechnology company in the world, Amgen "strives to serve patients by transforming the promise of science and biotechnology into therapies that have the power to restore health or even save lives". However, pressures for cost-effective, efficient and flexible manufacture of biopharmaceuticals are growing given their high doses and increasing potential that have resulted in significant increase in total manufacturing site capacities without physically expanding the site. To better serve patients, Amgen is looking for ways to drive efficiencies throughout its manufacturing processes by increasingly integrating new innovative technologies in these processes. Unfortunately, the rate of technology adoption and the ability to use the technology to add value on the manufacturing floor remain to be improved, primarily for two reasons. Firstly, technological innovation in a biotech company is usually a long-term and highly cross-functional process, which has not been particularly optimized to increase project management efficiency. Secondly, a disconnection exists between technology validation at early stages and technology implementation at late stages, causing further delay in the already tedious process. Therefore, this project is trying to explore solutions to increase the successful rate of incorporating innovative technologies at manufacturing sites by addressing challenges for advanced analytics implementation in a new drug substance manufacturing site at Amgen.

1.2 Problem Statement

The drug substance manufacturing plant in Singapore implements an advanced manufacturing network that is highly productive, cost effective, modular and flexible. This will require advanced quality control systems that allow for flexibility and efficiency. Instead of relying on quality assessment at the end of production, the new quality control system will incorporate in-process quality check during development and manufacturing. However, in the in-process of designing and implementing these new quality control technologies, several gaps were identified, including gaps between the independently established manufacturing process and the in-progress

(12)

analytics initiative, gaps between the technical advancement achieved in the analytics initiative and the implementation feasibility in the current drug commercialization process, and gaps between the long-term technical development process and the short-term window for technology implementation in Singapore.

1.3 Project Goals

This project aims to bridge these gaps and specifically to achieve four goals.

(1) Connect stakeholders and sources of information to establish efficient communication channels, and ultimately reach consensus on the list of analytical technologies to be implemented in Singapore manufacturing site.

(2) Evaluate implementation risks associated with each analytical technology and identify root causes.

(3) Propose risk mitigation plans for the technology with highest risk, including overcoming

technical limitations and building implementation plans.

(4) Learn lessons from business cases in analytical technology development to apply to future projects from a perspective of dynamic project management.

1.4 Project Approaches

This project proceeds in four phases: current status analysis, systematic risk evaluation, mitigation of the highest risks and lessons learned to be applied to other cross functional initiatives at Amgen.

The current status analysis phase begins with connecting with major stakeholders involved in the project, including the operation group, the analytics group, the commercialization group, the filing group and the technical development group. This phase includes interviewing those stakeholders, learning the manufacturing process and the quality control process for a specific drug that will be manufactured in Singapore and understanding the analytical technologies in the current practice and in the development phase. Based on the information collected in the first phase, the project proceeds with building a risk evaluation model by analyzing risk causing factors, defining risk levels, and applying this model to the analytical technologies that will be implemented in Singapore. After identifying high risks by the model, the project will follow up

(13)

with those risky technologies, analyze root causes and propose corresponding solutions. The project moves into the third phase with a risk mitigation plan for analytical technologies with high risks, which will be integrated into the implementation process in the Singapore site. Specifically, alternatives will be evaluated for liquid media identification by searching for advanced technical options, conducting preliminary assessment of utility, documenting costs, benefits or values and eventually making a recommendation on how to mitigate risks. Lastly but not the least, the project will be finalized with a case study regarding how to improve the performance of cross-functional initiatives at Amgen. Using this project as an example, lessons learned will be discussed and recommendations will be made, focusing on how to avoid unnecessary time delay, how to improve collaboration among different stakeholders and how to use a dynamic project management perspective to improve business decision making processes. Due to the time constraints, this project will only include analytical technologies that are going to be implemented in the Singapore site once the construction finishes in 2014. It will not include analytical technologies that are still in the early-phase development at Amgen, or the in-detail analysis of filing strategies.

The project is driven by a literature review of relevant research in biotech/pharmaceutical industries, by consideration of Amgen standards, and by data and insights learned during the internship. Throughout the project, qualitative questions are addressed by quantitative methods, such as the risk evaluation model to define and identify analytical technologies with high risks, feasibility experiments to make data-driven decisions on technical strategies for raw material identification, and the cost-benefit analysis to screen the best options for ASM. Specifically, the data for the first phase is mainly obtained through literature search and stakeholder interviews. In the second phase, the risk model is built and validated based on the data from the operation and analytics groups. In the third phase, the technical data is generated through a series of scientific experiments, in close collaboration with the technical development group. In the final phase, extensive discussions with all of the main stakeholders will be carried out to assemble the final recommendations that best represent the opinions from all perspectives.

1.5 Thesis Overview

(14)

Chapter 1 gives a complete introduction about the project, including why this project is conducted, what problems the project aims to solve and the project goals. In particular, chapter 1 details the different phases as the project rolls out and the various approaches used in each phase. Furthermore, it outlines the scope and the stakeholders involved in the project.

Chapter 2 gives a brief background introduction, including Amgen's history, the commercial products, and the conventional and advanced biopharmaceutical manufacturing procedures. Two company-wide initiatives to improve the operational performance are introduced: Manufacture of the Future (MoF) and Analytics of the Future (AoF). Lastly, this chapter introduces the new drug substance manufacturing site in Singapore, the first MoF site at Amgen.

Chapter 3 describes the research methodology and the problems identified in this study. It started with the current state analysis to identify main stakeholders and sources of information, followed by conducting qualitative interviews to define the current gaps and the challenges in developing and implementing advanced analytical technologies, with a particular emphasis on the technology used for liquid media identification. Based on the problems identified, both qualitative and quantitative problem-solving strategies are proposed, including building the optimized decision making process and establishing risk evaluation model for a series of AoF technologies under development at Amgen.

With the list of analytics to be implemented in Singapore and the risk assessment model defined in chapter 3, chapter 4 applies the model to identify high risks and conducts a case study for risk mitigation. Basically, it is going to apply the model to evaluate implementation readiness for each technology on the list, identify high risks, investigate root causes and propose risk mitigation plans. In particular, this study will focus on addressing technical and operational challenges of the technology identified with the highest risk, presenting the technical breakthrough for overcoming the technical limitations, and planning for implementation of this technology at Singapore.

(15)

Chapter 5 draws conclusions and lessons on what it takes to evaluate and implement an advanced technology in the biotech/pharmaceutical manufacturing process. This chapter gives a full picture of what it takes to develop Raman technology at Amgen. Using Raman technology development as a case study, this chapter draws lessons learned in the development process and makes suggestions for future projects of advanced technology development in terms of dynamic project management.

(16)

Chapter 2

Amgen Inc and the Biopharmaceutical Manufacturing

Process

2.1 Company Background

With over $15 billion annual sales and over 17,000 employee worldwide, Amgen (Applied Molecular Genetics) is the world's largest independent biotechnology firm, headquartered in Thousand Oaks, California. [1]

2.1.1 Company History

Amgen was founded in 1980 as Applied Molecular Genetics by William Bowes, who was the financial manager and board member of the world's first biotechnology company, Cetus Corporation. The company recruited a formidable scientific advisory board. With influential advisory board members in place, Bowes was able to raise $200K in seed capital from six venture capitalists. With this funding, Amgen recruited the former chief of Abbott Laboratories' diagnostics division, George Rathman, to be CEO of the new company. Rathmann was able to raise a large round of private equity funding to begin operations, despite hesitation from venture investors, because at the time, biotechnology was an uncharted scientific frontier and insulin was the only genetically engineered substance approved for human use. Hence the company recruited big-name researchers from leading pharmaceutical companies in order to create a panel qualified to evaluate different potential directions for research and development. The company eventually identified five therapeutic products that showed promise and then narrowed its focus down to one, the erythropoietin (EPO) gene responsible for stimulating red blood cell production. Then Amgen survived the lengthy FDA approval process and in 1989, was granted the right to begin selling Epogen, their first drug.[2] Between 1994 and 2013, the company had made at least five major corporate acquisitions, including the recent $10.4 billion deal of acquiring Onyx Pharmaceuticals, a company developing and marketing medicines for the treatment of cancer. [1]

(17)

Amgen began as a little-known drug developer based in the outskirts of Los Angeles. Thirty years later, the company earns US$15 billion in annual revenues and employs 18,000 across North America, Europe, Asia, Australia and the Middle East.

2.1.2 Commercial Products

As of December 2013, Amgen had twelve approved drugs or therapeutic biologicals as well as 11 drugs in phase III clinical trials. [1] The names and the indications for the 12 approved drugs are described as follows.

(1) Aranesp (darbepoetin alfa) is a synthetic form of erythropoietin. It stimulates erythropoiesis

that increases red blood cell levels and is used to treat anemia, commonly associated with chronic renal failure and cancer chemotherapy.

(2) Enbrel (Etanercept) is a biopharmaceutical that treats autoimmune diseases by interfering with tumor necrosis factor (a soluble inflammatory cytokine) by acting as a TNF inhibitor. It is approved to treat rheumatoid, juvenile rheumatoid and psoriatic arthritis, plaque psoriasis and ankylosing spondylitis.

(3) Erythropoietin or EPO is a glycoprotein hormone that controls erythropoiesis, or red blood

cell production. It is a cytokine (protein signaling molecule) for erythrocyte (red blood cell) precursors in the bone marrow.

(4) Pegfilgrastim is a PEGylated form of the recombinant human granulocyte

colony-stimulating factor (GCSF) analog filgrastim. It serves to stimulate the level of white blood cells (neutrophils).

(5) Granulocyte stimulating factor (G-CSF or GCSF), also known as

colony-stimulating factor 3 (CSF 3), is a glycoprotein that stimulates the bone marrow to produce granulocytes and stem cells and release them into the bloodstream. G-CSF also stimulates the survival, proliferation, differentiation, and function of neutrophil precursors and mature neutrophils.

(6) Romiplostim (rINN, USAN) is a fusion protein analog of thrombopoietin, a hormone that

regulates platelet production. Romiplostin is indicated as a potential treatment for chronic idiopathic (immune) thrombocytopenic purpura (ITP).

(7) Denosumab is a fully human monoclonal antibody for the treatment of osteoporosis,

(18)

(8) Cinacalcet (INN) is a drug that acts as a calcimimetic (i.e. it mimics the action of calcium on

tissues) by allosteric activation of the calcium-sensing receptor that is expressed in various human organ tissues. Cinacalcet is used to treat secondary hyperparathyroidism (elevated parathyroid hormone levels), a consequence of end-stage renal disease. Cinacalcet is also indicated for the treatment of hypercalcemia in patients with parathyroid carcinoma.

(9) Panitumumab (INN), formerly ABX-EGF, is a fully human monoclonal antibody specific to

the epidermal growth factor receptor (also known as EGF receptor, EGFR, ErbB-1 and HERI in humans). It is for the treatment of EGFR-expressing metastatic colorectal cancer with disease progression.

(10) XGEVA is for the prevention of skeletal-related events (SREs) (pathological fracture,

radiation to bone, spinal cord compression or surgery to bone in adults with bone metastases from solid tumors.

(11) Palifermin (trade name Kepivance, marketed by Biovitrum) is a truncated human

recombinant keratinocyte growth factor (KGF) produced in Escherichia coli. KGF stimulates the growth of cells that line the surface of the mouth and intestinal tract. When patients with blood cancers (leukemia and lymphoma) receive high dose chemotherapy and radiation therapy to undergo bone marrow transplantation, they usually get severe oral mucositis. Palifermin reduces the incidence and duration of severe oral mucositis by protecting those cells and stimulating the growth of new epithelial cells to build up the mucosal barrier. Palifermin is also being studied in the prevention and treatment of oral mucositis and dysphagia (difficulty swallowing) in other types of cancer.

(12) Anakinra (brand name Kineret) is a drug used to treat rheumatoid arthritis. It is an interleukin- 1 (IL-1) receptor antagonist.

2.2 The Past and Future in Large-scale Production of Monoclonal Antibodies 2.2.1 Current Practice of Biopharmaceutical Manufacturing Processes

Monoclonal antibodies are widely used in the treatment of infectious diseases, cancer and autoimmune diseases. [3] In the manufacturing process of monoclonal antibodies, product quality is defined by the process, such as equipment parameters, the sequence of unit operations and operation parameters, because no complete analysis of these complex molecules is possible.[4] However, process development can often be the rate-limiting step in the introduction of

(19)

biopharmaceuticals into clinical trials, as it is impossible to have a generic process that could be employed for all mAb candidates since significant physicochemical differences exist among mAbs.[5] Amgen has developed a flexible, generic platform for mAb processing, serving as a guidance document that defines the overall scheme of downstream processes and brackets the operating conditions for individual unit operations, thus limiting the scope of experimentation required to reach a solution for a given molecule. [6]

The entire platform consists of upstream and downstream procedures. Upstream processes refer to process steps associated with the production of a recombinant protein by propagation of the host cells (Figure 1) and downstream processes refer to process steps associated with the purification of a recombinant protein and removal of impurities (Figure 1).

In the upstream processes, highly productive cell lines that have both the right growth characteristics and the high specific productivity are selected and grow in chemically defined media that can achieve high cell density and sustained viability over the course of the bioreactor run. Cells are expanded through a series of seed bioreactors before being transferred to the production bioreactor where mAb is secreted to the medium. The early stages of cell expansion are conducted in shake flasks or spinner flasks, which progressively increase in size and volume. The cell mass is then scaled up through several bioreactor stages prior to transfer to the production bioreactor. Fed-batch production is the most prevalent and involves the addition of small volumes of feed to supplement the nutrients present in the bioreactor.[7]

(20)

kHocuIum C CUR"

Vial thaw and Inoculum Seed Production

expansion bioreacior bloreactor

Figure 1: Upstream process for mAb production at commercial scale. [7]

The downstream platform incorporated processes by which most of the approved monoclonal antibodies are manufactured: centrifugation and purification steps that relay primarily on chromatography with intermediate filtration and viral clearance operations (Figure 2). The first step in the manufacturing process is to separate mAbs that are secreted into the cell culture medium during mammalian cell culture from cells and cell debris. This is accomplished by large-scale centrifugation, followed by depth filtration steps to remove residual cellular debris. The cell culture supernatant is then directly loaded on columns for protein A chromatography to capture mAbs. This process has proved to be highly selective and can in many cases yield >99% purity. Thus, the product stream is concentrated from a relatively dilute cell culture supernatant to the eluted solution, whose volume is significantly reduced with concentration of more than

(21)

Coll

culture

harvest

Protein A

chromatography Viral

Inactivation

Chromatographic polishing steps (2) Viral

filtration

UF/DF

Figure 2: Platform downstream process for mAb production. [6]

The viral reduction step is required by the FDA to assure safety of products produced by mammalian cell culture, as indicated in the FDA Q5A guidance document. Viral inactivation is achieved by a low pH incubation step after the Protein A elution, which is at low pH already. Following acid inactivation, the solution is then neutralized to move the product into a more stable pH range. During viral inactivation and neutralization, strong acid or strong base are avoided as they may denature the products at localized region or cause corrosion issues for steel vessels. After the viral inactivation, at least two additional chromatographic steps are involved in mAb downstream processes to reduce host cell protein impurities, high molecular weight

(22)

aggregates, low molecular weight clipped species, DNA and leached Protein A to acceptable low levels by the FDA. Cation-exchange chromatography (CEX), anion-exchange chromatography

(AEX), hydrophobic interaction chromatography (HIC) and hydroxyapatite are the typical

technical options. The viral filtration operated at constant pressure is employed following either one of the polishing chromatographic steps, to complement the low pH viral inactivation step. The last step to complete the downstream purification is the ultrafiltration/diafiltration setup, and the product is then buffer-exchanged into the formulation buffer, ready to be filling or packaged into drug products. [6]

During the entire production process, a set of quality control (QC) activities is employed to ensure that products are consistently produced and controlled to the quality standards appropriate to their intended use and as required by the marketing authorization. QC attributes covered in this study include but not limited to endotoxin, bio-burden, protein quantification, impurity purification, monitoring facility and inventory/logistics management.

2.2.2 Recent Advances in Manufacturing and Analytical Technologies

As shown in section 2.1, the majority of the drug products made by Amgen are monoclonal antibodies. However, they are amongst the most expensive of all drugs where the annual cost per patient can reach as high as $35,000 for antibodies treating cancer conditions. This is due to two facts. The first fact is that antibodies are now marketed for chronic diseases and are prescribed for repeated and long-term treatments. The second fact is the need for high cumulative doses (grams rather than milligrams) because of their relatively low potency. Therefore, large-scale production capacity is required to fulfill market demand and produce 10-100s kg/year.[8] These trends have triggered the renewed interest in innovative manufacturing technologies to improve yield efficiency and reduce financial burden, ultimately to deliver more affordable treatments to patients.

The continuous evolution of platform technologies has made it possible to make biopharmaceuticals faster, more efficient and cheaper, and Amgen has brought those advanced technologies to its biotech manufacturing processes.

(23)

In the upstream processes, two major improvements were achieved to increase the productivity of fed-batch mammalian cell culture. First is the creation of a cell line with rapid growth and high specific productivity. Chinese hamster ovary (CHO) cells have been chosen as the host mammalian cell lines due to its high level of product synthesis, which is linked to the increased gene copy number and effective transcription.[9] The second improvement is the understanding of chemically defined media and feeding strategies to achieve high cell density and sustained viability over the course of the bioreactor run. One example of the innovative technologies in this area is the real-time control of dissolved oxygen, pH, temperature and mass transfer of oxygen and C02 in the production bioreactor, thus new media is fed continuously with maintaining sterility over long periods. Another example of innovation is to substitute stainless bioreactors for disposables, and achieve the same yield with no need to transfer cell culture media from small bioreactors to larger ones. Thus, Amgen is able to deliver the medicine to patients while consuming fewer resources, respond to patients' demand more rapidly and operate the manufacturing system more efficiently.

In the downstream processes, batch chromatography is traditionally used as the core purification method. Chromatography is a physical method of separation that distributes components to separate between two phases, one stationary (stationary phase), and the other (the mobile phase) moving in a definite direction. The batch chromatography has long been labeled as a slow process that requires large volumes of mobile phase and results in very large pool volumes. Recent advances in the design of chromatographic stationary phases available for purification of biomolecules have provided some solutions to the throughput dilemma, and have facilitated reasonably productive platforms for industrial protein purification. [7] However, the throughput for conventional preparative chromatography is inherently limited as a result of the batch nature of the process and the limitations in fluid velocity for medium-sized particle beads (40-120 pm) that are used in these columns. Therefore, significant debate has been made for non-chromatographic processing steps. But these changes are slow to be adopted, as they require significant capital investments or modifications to existing facilities.

Another technological improvement in the downstream manufacturing process is the incorporation of the single-use technology at bioreactors. During the mammalian cell culture

(24)

process, instead of a culture vessel made from stainless steel or glass, a single-use bioreactor is equipped with a disposable bag. The disposable bag is usually made of a three-layer plastic foil. One layer is made from Polyethylene terephthalate or LDPE to provide mechanical stability. A second layer made using Polyvinyl acetate (PVA) or Polyvinyl chloride (PVC) acts as a gas barrier. Finally, a contact layer is made from PVA or Polypropylene (PP). [10] The main advantages of the single-use solution are process simplification and cost reduction. Applications of single-use technologies reduce cleaning and sterilization demands in pharmaceutical production. Complex qualification and validation procedures can be made easier and will finally lead to significant cost reductions. It is estimated that single use systems save more than 60% compared to fixed stainless steel bioreactors. [11] Since single-use bioreactors contain fewer parts compared with conventional bioreactors, the initial and maintenance costs are reduced as well. Additionally, the application of single-use bioreactors reduces the risk of cross contamination and enhances the biological and process safety.

In the meantime, the quality control processes has to be aligned with the improved manufacturing processes. The new control strategies and quality control systems should allow for flexibility and efficiency, such as fast changeover between manufacturing batches using real-time disposition. A fundamental shift for process development will be to design in-process quality checks during development and manufacturing rather than relying on quality assessment at the end of production. For example, in controlling the protein concentration in cell culture processes, instead of taking samples out of the bioreactor periodically, a real-time technology is implemented inside the bioreactor to monitor the protein concentration. This new technology significantly reduces the risk of contamination and costs of conventional QC assays. Another advanced technology has been brought into the sterility and bio-burden assays. Compared with the conventional assays that require weeks to read results, new assays can report contamination cases in three days by incorporating a highly sensitive UV detector. Despite those advances, the development for analytical technologies started at Amgen much later than the advancement in manufacturing technologies and many new technologies are still in the validation or feasibility

(25)

2.2.3 Drug Substance Manufacturing Site in Singapore

On 3rd June 2013, Amgen broke ground on its new manufacturing facility at the Tuas Biomedical Park. This marked Amgen's first major expansion into Asia. The new facility will be Amgen's first commercial-ready "Manufacturing of the Future" site, using many newly developed production technologies. Amgen anticipates investing approximately US$200 million over the next several years to build this new innovative facility. When fully operational, the site is expected to employ 200 employees.

Amgen's Singapore manufacturing site is expected to complete construction in 2015, with licensure anticipated in 2016. When completed, the site will be capable of manufacturing both clinical and commercial products. The innovative facility will initially focus on expanding Amgen's manufacturing capability for monoclonal antibodies. Although the size of the Singapore site is only one-third of that of a conventional manufacturing site, it is expected to achieve the same level of annual yield.

(26)

Chapter 3

Problems Identified in the Analytics Implementation at

ASM

3.1 Current State Analysis Identified Major Challenges and Gaps 3.1.1 Key Stakeholders and Sources of Information

Developing an analytical technology used in pharmaceutical manufacturing process involves multiple stakeholders, each of whom play different roles in various stages of technology development. In the beginning, according to the literature search and the industry knowledge review, the Analytics of the Future (AoF) group decides what advanced analytical technologies have potential in cost reduction, efficiency improvement or performance enhancement. Next, the Process and Product Development (PPD) group needs to investigate the feasibility of bringing the new technology into the pharmaceutical industry, explore the possible applications in the manufacturing process, research for the optimal parameter settings and establish the recommended protocols. Protocols for new technologies or any changes to established protocols need to be approved by the FDA, therefore pharmaceutical companies have a regulatory affairs department dedicated to handle this issue. Once approved, the Global Analytical Science (GAS) group is responsible for bringing this technology to one or more manufacturing sites and is in charge of monitoring the performance of the technology at high level. At the same time, each individual manufacturing site has its own quality group, to collect and analyze quality control data, following the standardized operational protocol that has been issued by the PPD group and approved by the FDA. In a nutshell, in order to thoroughly understand a new analytical technology that may be used in ASM, researchers need input from the following stakeholders or sources of information: the AoF group, the PPD group, the Regulatory Affairs group, the GAS group and manufacturing sites that operate facilities with traditional analytical technologies or in the process of testing new analytical technologies. It turned out that the manufacturing sites in Puerto Rico (AML) and Rhode Island (ARI) covered all the analytical technologies under this study.

(27)

3.1.2 Qualitative Interviews

Qualitative interviews were scheduled with executives and/or group leaders of those key stakeholders and sources of information. Interviews were designed to collect qualitative information in two areas: (1) the analytics in the traditional manufacturing process and those in the MoF process and (2) the entire process for a new technology from R&D development to implementation on the manufacturing floor. Interviewees were introduced to the researcher by project advisors. Each interview was conducted for at least 30min, in the form of teleconference or face-to-face meeting.

Interviewees and main information gathered from each interview are listed in Table 1.

Table I List of interview groups, interviewees and interview content Stakeholder or Source Interviewee Interview Content

MoF group Quality head of ASM Requirement of analytics at ASM

AoF group Group leader Current state of advanced analytics under development at Amgen PPD group Director, Experimental processes required

Principal Engineer to assess technical feasibility Regulatory Affairs group Manager FDA approval process;

Assessment of filing challenges for advanced analytics

GAS group Director Applications of traditional analytics and new analytics across different sites AML Principal Engineer Current manufacturing and analytical

technologies at AML

ARI Principal Engineer Current manufacturing and analytical technologies at ARI

(28)

3.1.3 Current Gaps and Challenges

According to the analysis of the comprehensive interviews with key stakeholders and sources of information, two major challenges that ASM team faces to successfully implement advanced analytics were identified.

(1) Lack of sufficient and efficient communication among stakeholders

Close collaboration across different functions is necessary in developing a new analytical technology or making changes to an existing technology. As addressed in 3.1.2, the development for advanced analytical technologies at ASM requires several steps including a) demand identification (the MoF group), b) technology discovery (the AoF group), c) feasibility assessment (the PPD group), d) FDA approval (the Regulatory Affairs group), e) high-level monitoring (the GAS group) and f) local practice (manufacturing sites in AML and ARI). During the technology development phase, the AoF group or GAS group usually oversees multiple technologies or manufacturing sites and makes decisions based on experimental data or quality control data generated by the PPD group or the local sites. On the contrary, technical teams from the PPD group are in charge of developing one or a few technologies to meet the needs of the AoF group. When implementing those technologies, ASM, AML, ARI and other manufacturing sites run their complete manufacturing and quality control systems and conduct daily operations that other stakeholders rely on to collect first-hand information. Those relationships are shown in

(29)

AoF: oversee RGAS: monitor multiple Affairsgtoup

QC

of multiple

technologies group sites

PPD: investigate Local Sites:

single FDA comprehensive

technology systems

Figure 3: Stakeholders play different but connected roles in the analytics development.

Analytics development progresses from feasibility assessment (led by the AoF group and the PPD group) to possibility assessment (the regulatory affairs group works with the FDA) and ends with implementation in the manufacturing sites that are operated by local sites and monitored by the GAS group.

Due to those interconnected relationships, stakeholders require effective and frequent communication among each other. For example, without knowing what advanced analytics needed in MoF facilities, the AoF group cannot target right technologies when performing industry-wide research. Take the relationship between the PPD group and the AoF/MoF groups as another example. Mutual communication among those groups is necessary to keep both parties on the same page. The PPD group needs to update the AoF group and the MoF group with the progress of assessing feasibility of each technology to help the latter to prepare implementation plans. On the other hand, the AoF/MoF groups should keep the PPD group aware of the demand and limitations in terms of pharmaceutical manufacturing processes to let the PPD group design the R&D experiments well to fit the manufacturing needs.

Therefore, it is critical to connect all the stakeholders together and enhance the understanding of each other through effective communication, which was missing at the beginning of this project.

(30)

(2) Timeline and funding differences between the MoF and AoF initiative

One aspect of a well-functioning manufacturing site is the alignment between the manufacturing process and the quality control system. However, there is significant misalignment between establishing the manufacturing process and developing the quality control system at ASM.

The design of the manufacturing process at ASM incorporated the majority of MoF technologies, which have been validated at Amgen years ago, therefore, transferring those MoF technologies to ASM has a clear schedule with expected dates for budget planning, construction completion, initial lots production and commercialized manufacturing. To match the well-established MoF process, Amgen started the AoF initiative one and half years ago. But the AoF initiative still lags far behind the MoF development, causing uncertainty about the technical development timeline. Many advanced analytics explored by Amgen are still at an early assessment stage (conducted by the PPD group), and very few new analytical technologies have been implemented on the manufacturing floor. As discussed above, bringing advanced analytics from the feasibility assessment stage under R&D context to the implementation stage on the manufacturing floor is time-consuming and risky. On the other hand, the defined MoF schedule requires all the analytics to be implemented by the end of site construction. Besides timeline differences, advanced analytics implementation also carries budget uncertainty as most of the technologies have not figured out the implementation details yet, including vendor selection, equipment models, software purchase, etc. Thus, how to bridge the gaps between the MoF initiative and the AoF initiative becomes the second challenge that ASM needs to tackle.

3.2 Key Problem Solving Strategies

3.2.1 Optimized Decision Making Process

To build up the connection among stakeholders, the study established an optimized decision making process, in which everyone's opinion could be considered and ultimately the teams would reach consensus on the list of analytics to be implemented in Singapore.

The decision making process started with the MoF group submitting the request for implementing advanced analytics to the AoF group. The AoF group reviewed the request and

(31)

made recommendations based on the current portfolio of analytics development at Amgen. With the recommended technologies, the MoF group communicated with the AML site and the Regulatory Affairs department to get feedback on (1) whether these technologies match the manufacturing process for the specific drug that is going to be made in Singapore, (2) whether these technologies have significant advantages over the current analytical technology and (3) the estimate of the difficulty to be approved by the FDA. The list was shortened after this screening process and was then internally reviewed by the quality head of the ASM site to confirm that they would match the advanced manufacturing process at Singapore. Finally, the PPD group went over the list to assess the likelihood of completing the technical validation by the required time points. This decision making process is demonstrated in Figure 4.

MoF

Fiing AoF+ A

MoF

,

jMoFh

MoF

Mo- MoF

Figure 4: The optimized decision making process ensures the involvement of all stakeholders. Stakeholders were grey if not connected and were highlighted in different colors after connection. The arrows represent the communication directions.

3.2.2 Quantitative Risk Assessment Model

A quantitative risk management approach was used to assess the qualitative gaps between the

MoF and the AoF initiatives. Specifically, risk-causing factors for each analytical technology were identified, and then a risk assessment model was built to rank technologies with

(32)

implementation risks from high to low. With high implementation risks, root causes were analyzed and risk mitigation plans were proposed. With this risk mitigation plan, the advanced analytical process would catch up with the mature status of the manufacturing system at ASM (Figure 5).

Manufacture

Target Status

Propose risk mitigation plans Identify high risks

Build a risk assessment model

Define risk causing factors

Analytics

Current Status AMGEN

Figure 5: Analytics implementation readiness is evaluated by risk management. Risk management involves four steps: define risk causing factors, build a risk assessment model, identify high risks and propose risk mitigation plans. The ultimate purpose of reducing implementation risks is to bridge the gaps between well-established MoF and early-stage AoF initiatives.

In the process of defining risk-causing factors, the following five issues were considered and assigned with weighted scores according to their relative difficulty and significance during technology development. The total score of the five issues is 10.

(1) Stages of technology development: developing a technology is a cross-functional, long-term

process, with certain degree of unpredictability. Therefore, it was assigned with a high weighted score, 3 out of 10.

(33)

(2) The Plan B: whether there exists a reliable alternative technology to switch to if the new technology does not work. Because the size of ASM is only one third of that of a traditional manufacturing site, and the sizes for warehouse, QC lab and other functions are proportionally smaller than regular ones, the ASM site has less inventory-holding space, faster QC assay turn-around and less safety stock for raw materials. Therefore, if one manufacturing step stops working due to no technical backup, it may hold up the entire manufacturing process. This factor was assigned 3 out of 10 to emphasize its importance.

(3) Filing challenges: needless to repeat, how difficult to convince the FDA to approve new

technology or changes in current technologies literally determines whether or not the particular technology can be incorporated into the commercial manufacturing facilities. The regulatory affairs department at Amgen has accumulated years of experience at working with the FDA and could give a confident estimate on the level of approval difficulty and provide detailed tactics to resolve potential issues. This factor was given a score of 2 out of 10.

(4) The budget and (5) The floor space: Two questions are often asked when technical implementation plans are discussed: how much it costs and how much floor space it takes up. Although the capital budget for the Singapore site is fixed and the entire floor space for the facility is limited, the MoF group can relocate the budget and move the equipment around to fit specific technical needs. Hence each of these factors had a score of 1 out of 10. Risk-causing factors and their weighted scores were summarized in Table 2.

Table 2 Risk causing factors and their corresponding weighted score

(34)

With the defined risk-causing factors, each technology from the confirmed list was evaluated by giving risk scores to represent high (score 60), medium (score 20) or low (score 1) implementation risks. Risk evaluation was performed based on qualitative interviews with related stakeholders. For example, the technology development can be grouped to three different risk levels according to different development stages. Low risks are assigned for those that have been evaluated, validated or implemented, medium risks for those in the middle to late stage of development and high risks for those in early evaluation phases. Similarly, for plan B and filing challenges, the ones easy to switch to existing technologies or sell to the FDA were considered as low risks, those that need extra efforts or money were assigned with medium risks and high risks were given to those difficult or costly to switch or sell. For the budget and the floor space, whether the new technology stays within the original budget/floor plan separates low risks and high risks. Medium risks refer to those that are not included in the original budget/floor plan but additional funding or space could be secured. Lastly, the technologies not planned for Singapore site were given the score of zero.

Criteria for risk scores on various risk-causing factors are summarized in Table 3. This risk assessment model was reviewed and endorsed by the risk management group at Amgen.

Table 3 Criteria and risk scores for various risk-causing factors

Technology Validated, Evaluated In the middle to late Not started yet or in early phase Development Implemented stage of development

Plan B Easy to switch Costly or need some No reliable plan B or very costly efforts to switch

Filing Easy to sell Need some efforts Hard to sell Costly to file Budget In the budget Over the budget but No budget planned

secure $$ from other sources

Floor Space In the original plan Newly add, but have No space at all additional space

(35)

Total risk scores are the sum of the weighted risk score of each risk-causing factor, whereas the weighted risk score is the product of the individual risk score and the weighted score of a particular risk-causing factor. The individual score numbers (the low risk with score 1, the medium risk with score 20 and the high risk with score 60) were designed for two purposes. (1) They could clearly separate low risks from medium/high risks. The total risk score for a technology bearing low risks for every single risk factor is 10 while the least score for a technology bearing at least one medium risk is 20, therefore, any risk factors bearing medium or high risks could be distinguished. (2) Risk scores should reflect the accumulative effect of the risks. For example, if a technology has a medium risk on the technology development (the risk score is calculated as 20*3) or has medium risks on the filing challenge and the budget (the risk score is calculated as 20*2 + 20*1), this technology is considered as risky as the one that carries

one high risk on the budget or the floor space (the risk score is calculated as 60*1).

With calculated total risk scores for each technology, the model needs to set implementation risk levels from high to low. The MoF leadership team wanted to focus on identifying technologies with at least one high risk or two important medium risks, therefore, the model set the criteria of a high risk level as the total risk score of 67 and above (Table 4). If a technology, for instance, carries medium risks for the budget and the floor space (the total risk score is 48), it would not be considered posing significant implementation challenges. However, if it bears medium risks in the filing process and the budget (the total risk score is 67) or one high risk in the floor space (the total score is 69), this technology will need further investigation.

Table 4 High, medium and low risk levels are categorized based on total risk scores

Total

Overall Risk Level Score Cmet

One important medium risk or two low weighted

(36)

Chapter 4

Risk Mitigation of Advanced Analytics Implementation at

ASM

4.1 Identification of Advanced Analytics with Different Risk Levels 4.1.1 Application of the Risk Assessment Model

Based on the risk assessment model discussed in chapter 3, a rank list of technologies with total risk scores from high to low was generated (Figure 6). Out of 17 analytical technologies that cover QC assays for the entire manufacturing process at ASM, four technologies are identified with high risks, one with medium risks, seven with low risks and five are eventually excluded from the current plan due to various reasons. The MoF leadership team decided to focus on addressing the high implementation risks associated with those four analytical technologies.

4.1.2 Risk Contributors to Technologies with High Implementation Risks

In order to further understand the causes of those high risks, we need to look into the risk scores of each individual risk-causing factor. Table 5 listed detailed risk scores for 14 technologies in different QC applications, including four technologies with high risks, one with medium risks, seven with low risks and two that are not planned. This study focused on investigating risk contributors to those four high-risk technologies.

(37)

450 400

4

0 15_ 350 0 0 07 ~200 150 100 Medium Risk Low Risk Not Planned

AoF Technologies for ASM

Figure 6: Total score rank list of all analytical technologies for ASM. Seventeen advanced analytical technologies were evaluated by the risk assessment model and were ranked according to the total risk scores from high to low. Technologies with high risks were shadowed in red, the one with medium risks was shadowed in yellow and the ones with low risks were shadowed in grey.

Table 5 A partial list of analytics for ASM with detailed implementation risk scores

Technological Floor

QC Assavs Development Budget Space Plan B Filing Total

Multi-attribute Method N/A N/A N/A N/A N/A N/A

(38)

(1) Micro ID assays

In September 2004, the FDA issued a strong guidance recommending genotypic identification of microbes as part of the routine environmental monitoring program for aseptic processing under GMP guidelines. A quote from Pharmaceutical Current Good Manufacturing Practices as "Genotypic methods have been shown to be more accurate and precise than traditional biochemical and phenotypic techniques...Advanced identification methods (e.g., nucleic-acid based) are valuable for investigational purposes" (FDA Draft Guidelines 9/04).

Although the technology for Micro ID is well developed, ASM is the first site at Amgen to apply it for microbiology identification. Therefore, no existing monitoring system could be used as an alternative (the risk score for plan B is 20*3 = 60). In addition, the financial investment for

implementing this technology was not included in the original budget plan but could be secured from the savings elsewhere (the risk score for the budget is 20*1 = 20)

(2) Water viability assays

In cell-culture-based processes, water viability is reflected by important parameters such as osmolality and pH, key elements that are examined and optimized to improve productivity. Osmolality, the concentration of a solution expressed as the total number of solute particles per kilogram, affects cell viability and growth by regulating the transport of water and nutrients through cell membranes. The appropriate pH value maintains the isotonicity of a culture. The current practice for monitoring osmolality and pH values is to withdraw samples from manufacturing systems and then pass them through multifunction analyzers. Withdrawing samples during a bioreactor operation can introduce contamination and lead to a failed run of an entire batch.

To minimize the risk of cell culture contamination, Amgen decided to implement a real-time monitoring system for water viability. Although Amgen just purchased the system and had not done any internal validation when the project was conducted, the real-time monitoring system proved to be working well in the industry, so it is scored as a medium risk for the technological development (20*3 = 60). Likewise, it may require extra efforts in securing financial support,

(39)

floor space and persuading the FDA for approval, but those challenges are manageable (medium risk scores for the budget, the floor space and the filing challenge). And it is very easy to switch back to the current practice to monitor water viability (a low risk score for the plan B).

(3) Sterility and Bio-burden assays

Products or components used in the pharmaceutical manufacturing procedures require the control of microbial levels during any material processing and handling. Sterility and Bio-burden testing on these products proves that these requirements have been met. According to the standard

ISO/TS 11139:2006 (Sterilization of health care products - Vocabulary), sterility refers to the state of being free from viable micro-organisms and bio-burden means the population of viable micro-organisms on a product and/or sterile barrier system. Currently, sterility and bio-burden testing used in quality control processes at Amgen takes up to several weeks per sample. This time delay prevents Amgen from discovering contamination incidents on a timely fashion and may cause significant waste on material, labor, time and money. To this end, the risk score of the Plan B is 60 (20*3). Similar to the real-time monitoring system for water viability, the new technology for sterility/bio-burden testing has been validated by some players in the biotech/pharmaceutical industry and Amgen has started its own validation process. ASM does need to relocate some budget to support an implementation plan and to figure out the optimal floor plan for implementation. The new technology uses the same principal for testing as that of the traditional assay, hence it would not be impossible to get the FDA approval. Thus, all the risk-causing factors bear medium scores.

For the three technologies above, Amgen has staffed several cross-functional teams dedicated to addressing the challenges, and this study has focused on raw material identification (especially liquid media ID), which has the highest implementation risks.

4.1.3 Raw Material ID Technology with Highest Risks

Pharmaceutical companies often deal with a large number of raw materials and, without carefully monitoring, confusion between similar or mislabeled substances can easily occur. In addition, before they can be used in the production process, incoming raw materials may also have to be tested to see that they meet certain specifications, such as moisture content, purity, etc. Therefore

(40)

the manufacture of high quality pharmaceutical products requires identification of the incoming raw materials.

Amgen decided to apply Raman technology to increase efficiency and reduce the cost in the process of raw material identification, as Raman has proven to be an easy-to-use and powerful tool.[12] Raman spectroscopy, a molecular spectroscopy observed as inelastically scattered light, allows for the interrogation and identification of vibrational states of molecules.[12] As illustrated in figure 7, samples are illuminated by monochromatic light, usually from a laser in the visible, near infrared, or near ultraviolet range. After interacting with molecule vibrations, phonons or other excitations in the system, the laser photons shift energy level up or down. The shift in energy corresponds to the unique sample information in the system.

Références

Documents relatifs

Dans sa thèse de doctorat d’Etat consacrée aux structures communautaires traditionnelles altogovéennes (de l’Ogooué Supérieur), le professeur de sociologie,

Against these backdrops, the current paper can be considered as a first attempt to address the call and fulfil the mentioned void by proposing an analytical framework built

These first results suggest that organizational learning, as defined in our study, has positive influence on financial performance and innovation performance in our sample

We will show in Proposition 4 that the critical value S c decreases when r e increases, when the productivity A e or the total number of skilled workers increase, when the price of

In this paper, we develop a model of credit rationing with ex-ante and ex-post moral hazard to clarify when and how parametric weather insurance can relax collateral constraints

From the software programming perspective, in quantum programming all is new and, although already having quantum hardware, quantum programming languages and development

Average Slug or Snail expression levels in IDC associated with lymph node invasion was found to be significantly higher (p< 0.05) than the levels found in IDC not associated

Whenever human teachers are being compared with some alternative, we should immediately ask ourselves "What kind of teacher?" The answer to this question, and others, has