SOME PROBLEMS WITH COLLECTION, ANALYSIS AND USE OF RELIABILITY DATA

J. KUBIE

Central Electricity Generating Board, London, United Kingdom

Abstract

Typical problems with the collection, analysis and use of reliability data are discussed.

It is argued that the collection of reliability data has to be selective, and that insufficient attention to this selectiveness is

responsible for the majority of problems with the collection of data. The collection of reliability data must be carefully planned and undertaken by dedicated, well-trained and we11-motivated staff.

The reliability data must be analyzed, tested and used as carefully and cautiously, and under the same discipline, as other engineering parameters.

1. INTRODUCTION

In this paper I will highlight some of the common problems associated with the collection, analysis and use of reliability data. This is not a comprehensive survey; rather this is a

selection of the various problems I have encountered during my work in safety assessment.

Reliability data are generally required for three different, but related purposes:

(i) to learn from the past, i.e. to ensure that past problems are not repeated,

(ii) to choose at the present, i.e. to ensure that adequately reliable components and systems are used, and

(iii) to forecast the future, i.e. to develop models of component and system failures and to assess their reliability and the risk they impose.

I believe that because of the different requirements there cannot be

I will discuss the various problems under three different headings:

- problems due to collection of data,

- problems due to analysis and evaluation of data, - problems due to retrieval and use of data.

2. COLLECTION OF DATA

It must be appreciated that we cannot collect all the data all the time. Such an approach would not be practicable; it would place an unbearable strain on the collection system. It is inevitable that we will have to be selective, and it is the insufficient attention to this selectiveness which is responsible for the majority of problems with the collection of data. Since it is impracticable to have an all-embracing, universal collection scheme, we have to select - we have to decide on which aspects to concentrate: which plant to collect the data from, which components and systems to

include, how to define failures, which non-failures to report, etc.

This selectiveness must not be ad-hoc or considered only as an afterthought. It must be regarded of fundamental importance in devising and designing adequate collection systems.

Thus, I do not believe that the objective of data collection is to collect the maximum amount of information, but rather the objective must be to collect the relevant information. Hence, we have to decide at the beginning why we are collecting the data and for what purpose. Obviously, computerisation enables more and more data to be handled, but it should be appreciated that data collection is much less ameanable to computerisation than, for example, data analysis and retrieval.

Insufficient consideration of what to collect and why can lead to the following problems with the collection of data:

(i) insufficient information collected, (ii) inconsistent information collected, and (111) unreliable information collected.

2.1. Insufficient information collected

This follows practically always from a badly executed preliminary analysis of the need for the data. Typical omissions, which may make a particular data collection less than useful, are as follows:

- the underlying causes of the failures and the failure mechanisms,

- the consequences of the failures,

- the operating and environmental conditions,

- the period over which the data have been collected and the behaviour and history of the component or system in question.

2.2. Inconsistent information collected

This, once again, usually follows from a less than thorough

preliminary analysis. Comprehensive, but inconsistent collections may appear superficially adequate, but a more detailed analysis of the data (which is invariably undertaken much later) then reveals many hidden shortcomings. Unfortunately, it may then be too late to amend the collection system.

Typical problems are as follows:

- inconsistent definition of components and systems (especially in the definition of the boundary),

- inconsistent definition of component failures. For example, what constitutes a failure - a pump not starting on the

start-signal or not starting within 30 seconds of initiation.

Problems can also arise if a particular collection system is based on a physically incorrect model. The collection requirements may be so strongly driven by the demands of this particular model, that if the model is then shown inadequate the collected data may not be appropriate for any other purpose. This is particularly true when data are collected on rare events, such as dependent failures, etc.

2.3. Unreliable information collected

We have to know how reliable and error free is our particular collection system. This can be the most difficult problem of data collection. It can be partially dealt with by having an adequate in-built QA scheme, but this in itself may not be sufficient. What is of primary importance is to ensure that the data are collected by dedicated, well trained and well motivated staff.

3. ANALYSIS AND EVALUATION OF DATA

The problems in this area can be conveniently discussed under the following headings:

(i) insufficient understanding of the failure mechanisms, (ii) insufficient distinction between the various sources of

data,

(iii) statistical shortcomings, and (iv) bias in evaluation.

3.1. Insufficient understanding of the failure mechanisms

Many problems have been observed in this area. For example, in the case of dependent failures the analysis may be particularly strongly model-driven. The scarcity of the data may then make the

3.2. Insufficient distinction between various sources of data

There are many sources of reliability data and they all should be considered. However, the limitations and the benefits of the various data sources must be taken into account. The typical sources of reliability data are :

- operational data, - field trials,

- laboratory testing,

- generic published information, - expert opinion.

All these sources can provide useful information, but only if used correctly. It is generally accepted that operational data are most appropriate. The advantage of field trials and laboratory testing is that various parameters can be varied, but the limitation is that important operational factors may be missed. This again shows that field trials and laboratory testing must be carefully designed and controlled.

The advantage of the generic published information is that the data are usually conveniently available, and some of them may have been endorsed by virtue of being used by reputable organisations.

However, the main disadvantage is that the primary sources of the data are not always given and thus not open to scrutiny. Hence the status of the generic published information may be uncertain, and the use of the information for purposes different than those initially envisaged may be inappropriate and possibly misleading.

The use of expert opinion can be contentious. First, some people find the whole philosophy of the Bayesian approach flawed. However, I do not think that this is the major problem. I believe that the second aspect of this approach causes much greater difficulties -the credibility and -the expertise of -the experts.

It can be dangerous to use either anonymous experts or well known experts with expertise which is irrelevant to our problem. We must always ensure that we know who the experts are, what are their credentials and what are the bases for their opinion. We must not forget that experts frequently over-estimate their knowledge and that, because of their background and contacts, they may not be giving independent advice. Expert opinion should be tested as any other source of reliability data, and not accepted uncritically.

3.3. Statistical shortcomings

One of the most common examples of this problem is the use of the median of a distribution instead of the mean. Since for certain distributions the numerical difference between the two can be

without too many questions what we consider normal, without perhaps appreciating that our assumption of normality may not be justified.

4. RETRIEVAL AND USE OF DATA

After collection and analysis the reliability data become available for retrieval and use. Since the data are commonly used by groups different from those responsible for collection and analysis, the design of the appropriate retrieval system also deserves careful considerations. It is not sufficient to give just the reliability data; the range of validity, the operating conditions and the limitation of the data must be given too.

It was suggested in Section 2 that the objective of the data

collection is to collect the relevant information (as opposed to the maximum amount of information). However, the design of the

retrieval system must follow a different philosophy: all the information collected must be available for retrieval. Hence, computerization of the retrieval system is required.

If the collection system is designed to give detailed descriptions of the causes and the mechanisms of component and system failures, the data can be used purely qualitatively. Such data can be used most effectively in design development. Thus, for example, we can use the data to re-design a particular component or system to ensure that particular failures are eliminated or at least made acceptable.

It must be stressed that to be suitable for this purpose the database must be carefully designed. For example, it is not

sufficient to give numerical values of reliability; good qualitative description of the failures must be also given.

As suggested in Section 1, the reliability data are mainly used quantitatively - either in the design stage or the evaluation stage.

There are some important differences between the two applications, but they are in many respects similar and inter-related.

The most important common problem is to decide which reliability data should be used. Do we use the data from historical databases, or do we take into account technological progress. There are good arguments for both approaches.

For example, the use of the historical databases will indicate how well we have used proven technology. This will give us a direct comparison with other designs based on the same or similar

technological developments.

On the other hand technology has advanced and we do learn from our past mistakes. Thus some improvement in availability and

- this observed and documented improvement, rather than a postulated (or a hoped for) improvement, is cautiously taken into account,

- the targets for improved performance, reliability and availability are challenging, but realistic.

It cannot be over-emphasized how important the above conditions are.

If they are not followed and if the targets for various improvements become divorced from the reality, the credibility of the whole

approach is lost.

This implies that, once again, only appropriate reliability data should be used. As in the collection and the evaluation of data, the use of the reliability data must also be planned. Using the data in isolation and in an ad-hoc manner wastes much of the

considerable effort put in their collection and evaluation and can lead to distorted and uneconomic designs.

5. CONCLUSIONS

The collection of reliability data must be carefully planned and undertaken by dedicated, well-trained and well-motivated staff.

The reliability data must be analyzed, tested and used as carefully and cautiously, and under the same disciline, as other engineering parameters.

Acknowledgement

This work is published by permission of the Central Electricity Board. However, the views expressed are those of the author alone.

PACS PROGRAMME — PROJECT FOR ANALYSIS OF