ANALYSIS RESULTS

Oracle DB 11g Rel. 2 Ent Oracle DB 11g Rel. 2 Stan DB MS SQL Server 2008 R IBM DB2 V 9.1 for z/OS V DB MS SQL Server 2008 E Netezza Performance Serve Oracle DB 11g Ent Edition Teradata DB 13.0 Teradata DB 12.0 IBM DB2 V 9.7 Ent Server Oracle DB 11g Standard Ed Oracle Ent Manager 10g Gr Oracle DB 11g Ent Edition 0

10 20 30 40 50 60

SFRs

Figure 4.2: SFR variation in Database Category for EAL4+ (Each shade of the bar represents products that claim conformance to a particular Protec-tion Profile)

Wind River Linux Secur MS Win 7, MS Win Serve IBM AIX 7 for POWER V7 IBM zOS, V 1, Rel. 13 IBM z/OS V 1 Rel. 12 IBM z/OS V 1 Rel. 11 BAE Systems STOP OS? v Red Hat Ent Linux Ver. Red Hat Ent Linux V 5. Hewlett−Packard HP−UX IBM z/OS V 1 Rel. 10 Red Hat Ent Linux V 5. Unisys Stealth Solutio VMware? ESX 4.0 1 an VMware? ESXi 4.0 1 a MS Win Server 2008 R2 VMware? ESXi Server 3. VMware? ESX Server 3.5 JBoss Ent Application MS Win Mobile 6.1 MS Win Mobile 6.5 0

20 40 60 80 100

SFRs

Figure 4.3: SFR variation in OS category for EAL4+ (Each shade of the bar represents products that claim conformance to a particular Protection Profile)

51

ICs, Smart Cards & Smart Multi−Function Devs Other Devs & Sys Network & Network−Related Boundary Protection Devs Operating Sys Access Control Devs & Sys Data Protection Products for Digital Sign Databases Key Management Sys Detection Devs & Sys Biometric Sys & Devs Trusted Computing PP availability

# Certficates

# PPs

0 10 20 30 40 50 60

0120270420570

Figure 4.4: Protection Profile Availability and Certified Products across cat-egories

same category. It can be noted that the availability of protection profiles is rather low across all categories of products except theIC’s and Smart Card category.

Figure 4.5 shows the percentage of certified products conforming to at least one protection profile. The average PP conformance rate among cer-tified products of our data set is 14%, with standard deviation around 10

% (see Fig 4.5, rightmost column). This indicates that a relatively low number of certified product use PPs with relevant differences among cate-gories. Indeed, a closer inspection reveals that the products broadly related tohardware or firmware show higher conformance than products that fall under the software-only category. This low conformance could also be due to vendors finding it difficult to build products that conform to a particu-lar protection profile, while the products themselves are targeted for the general commercial market. And so, to conform to a particular protection profile, which is produced by specific consumer or a consumer group, does not provide any competitive advantage in the general market.

On the other hand, the low PP conformance makes it difficult to com-pare and contrast the security requirements addressed by the certified prod-ucts. In fact, the non-conformance to a standardized PP allow vendors to customize the scope of certification to features that are very different from other certified products. As an example, a product in a certain category

4.3. ANALYSIS RESULTS

ICs, Smart Cards & Smart Multi−Function Devs Other Devs & Sys Network & Network−Related Boundary Protection Devs Operating Sys Access Control Devs & Sys Data Protection Products for Digital Sign Databases Key Management Sys Detection Devs & Sys Average

0 10 20 30 40 50 60

●

●

% Products

Figure 4.5: Protection Profile Conformance among certified products (Right-most bar shows the average conformance to PP for our dataset, 14%, and corresponding standard deviation, 11%)Note that categories with less than 10 products are not shown

could make claims that it addresses more SFRs related to data protection, while another certified product in the same category may have claims ad-dressing SFRs in the access control class. Furthermore, they can make dif-ferent assumptions about the threat agents and their capabilities, thereby affecting the nature of the security assurance that can be gained by the con-sumers. Hence, comparison of certified products in such cases can become rather labour intensive and a very time consuming process.

In order to better understand the variation of Security Functionalities addressed in each certified product, it is not simply enough to look at the number of SFRs addressed. Hence, we looked more closely at the usage of each SFR in various products. The variation in the usage of SFRs for prod-ucts in the database category, certified at EAL4+ is shown in Figure 4.6. It can be clearly noted, that there are not a lot of SFRs that are commonly used by the various products, even though they are in the same category of products. This further illustrates that comparison of certified products is not a trivial aspect.

53

Figure 4.6: SFRs usage among different products in Database category (EAL4+)

4.3.2 Point in time Certification

The CC scheme certifies products at a point in time, that is, certification applies to a particular version of the product and in a certain set of config-urations. But products do need to evolve - either to provide new function-alities or to fix problems or both. And in such cases, the CC certification does not apply to the new version and the whole product has to undergo the certification all over again which is once again a very time consuming and expensive process, especially when the changes made to the product are very minor. In order to avoid such situations, the CC scheme allows products to be under the CC Maintenance Agreement (CCMA) where only the changes made to the product are evaluated and certified. This aspect of the CC scheme would allow the products to be certified over aperiod of timeinstead of apoint in time.

We verified this aspect in practice and Figure 4.7 shows the certified products that are under the maintenance agreement across the various product categories. It can be observed that the number of products under the CCMA scheme is high among IC’s and Smart Card category when com-pared to the other categories. And indeed the total percentage of products that are under the maintenance agreement is just 22% of all the certified products. And in fact, excluding the IC’s and Smart Cards category, the

4.3. ANALYSIS RESULTS

Access Control Devs & Sys Boundary Protection Devs Data Protection Databases Detection Devs & Sys ICs, Smart Cards & Smart Key Management Sys Multi−Function Devs Network & Network−Related Operating Sys Other Devs & Sys Products for Digital Sign

0 50 100 150 200

# Products

Figure 4.7: Products under CCMA

percentage of products that are under the CCMA scheme comes down to approximately 15%.

Such low numbers of products under maintenance raise an important question on the product’s lifecycle, especially when vulnerabilities are found in the product which need to be fixed - can a product vendor issue a fix and technically lose the certification or keep selling the vulnerable version of the product to claim the certification?

In order to better understand this question, we have used the National Vulnerability Database (NVD), to cross-reference the certified products with products that have known vulnerabilities. Since we could not automate this step, we limited our analysis to the Database and Operating System categories certified at assurance level EAL4+. In the Operating System category (shown in Figure 4.8), we found 22% of the products under the maintenance agreement have disclosed vulnerabilities. And in the database category we found only 25% products under the maintenance agreements are shown to have a known vulnerability. To contrast this, we cross refer-ence products (in the database category at EAL4+) that are not under the maintenance agreement and 85% of the products have shown to have a known vulnerability (Figure 4.8) while 72% of products not under CCMA in the Operating systems have reported vulnerabilities.

Though we do not claim that the vulnerability is in the certified “golden”

configuration, these figures show that in practical usage of the products the 55

Figure 4.8: Vulnerabilities in products under CCMA compared to regular CC certified products

issue of addressing new vulnerabilities must be discussed. And clearly, a point in time certification does not cope well with the dynamic landscape of a product’s lifecycle.