Code Coverage - The Mac

Code coverage is used to determine which lines of code in an application have been executed. This has been used for years by testers and quality-control engi-neers to fi nd which code has been tested and which hasn’t. Security researchers can take advantage of it, too. Consider the case of code coverage used in con-junction with dynamic analysis, i.e., fuzzing. After fuzzing the system under test, code-coverage information can be obtained. This information can be used to fi nd which portions of the code have not been tested yet with the fuzzing.

(It cannot determine, in a meaningful way, whether a given executed line has been well tested, but it can determine which lines have not been tested). Such information can be used in refi ning the fuzzed inputs to improve their quality and execute additional code. Furthermore, fi nding the untested lines means they can be analyzed more carefully statically, or the dynamic analysis can be suitably improved to test those sections. Either way, code coverage can be a useful metric to analyze dynamic testing.

Therefore, one thing you can do with the Apple source code, besides read it, is to collect code-coverage information on it. For example, the WebKit regres-sion-testing page (http://webkit.org/quality/testing.html) states the following:

If you are making changes to JavaScriptCore, there is an additional test suite you must run before landing changes. This is the Mozilla JavaScript test suite.

95363c05.indd 116

95363c05.indd 116 1/25/09 4:41:06 PM1/25/09 4:41:06 PM

Chapter 5 ^■ Finding Bugs 117 Since WebKit is a very big project to look through for bugs, it might help to

focus on the areas that are not well tested with these regression tests. That is to say, some code is not as well tested as others and the code that is not well tested probably has more bugs to fi nd. To collect code-coverage information, WebKit needs to be built with the proper fl ags.

$ WebKit/WebKitTools/Scripts/build-webkit –coverage

This should build the whole package with code-coverage information built in, i.e., with the GCC fl ags -fprofi le-arcs and -ftest-coverage. The build will likely fail at one point with an error complaining that warnings are treated as errors. In that case, you have to fi nd and remove the -Werror fl ag from the compilation. For example, open the Xcode project fi le JavaScriptGlue.xcodeproj. Select Project ➯ Edit Project Settings and unclick the box by Treat Warnings as Errors. Make sure Confi guration is set to All Confi gurations. Then quit Xcode and rebuild the WebKit project. It should build all the way through without errors. The build succeeds if you see a message like the following:

===========================================================

WebKit is now built. To run Safari with this newly-built code, use the “WebKitTools/Scripts/run-safari” script.

NOTE: WebKit has been built with SVG support enabled.

Safari will have SVG viewing capabilities.

Your build supports the following (optional) SVG features:

* Basic SVG animation.

* SVG foreign object.

* SVG fonts.

* SVG as image.

* SVG <use> support.

===========================================================

If the code is really instrumented to do code coverage, it should have created a bunch of .gcno fi les that contain information about the code, such as basic block and control-fl ow information.

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackConstructor.gcno

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackFunction.gcno

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackObject.gcno

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSClassRef.gcno

95363c05.indd 117

95363c05.indd 117 1/25/09 4:41:07 PM1/25/09 4:41:07 PM

To test that the coverage data is being generated when executed, run a test program.

$ ./WebKitBuild/Release/testkjs

Usage: testkjs -f file1 [-f file2…][-p][-- arguments…]

See if .gcda fi les are produced in response to the program being run. These fi les contain the dynamic code-coverage information—in particular, which lines of code have been executed.

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackConstructor.gcda

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackFunction.gcda

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSCallbackObject.gcda

WebKitBuild/JavaScriptCore.build/Release/JavaScriptCore.build/Objects-normal/i386/JSClassRef.gcda

Since these fi les show up, we know it is working! Now run the JavaScript regression tests and see what code they cover.

$ WebKitTools/Scripts/run-webkit-test

This will generate a whole bunch of .gcda fi les, one for each source fi le (plus headers if they contain code). At this point, we could use gcov to view the results on a fi le-by-fi le basis, but a better way is to use lcov (http://ltp.sourceforge.

net/coverage/lcov.php) which is a graphical front-end for gcov. The fi rst thing lcov does is combine all the testing data (.gcda fi les) into one single fi le. WebKit is pretty complicated and lcov won’t work on it out of the box. To set things up for lcov, run the following commands:

$ cp Release/DerivedSources/JavaScriptCore/grammar.* JavaScriptCore/

mkdir JavaScriptCore/JavaScriptCore cd JavaScriptCore/JavaScriptCore ln -s ../kjs kjs

Then run lcov:

$ lcov -o javascriptcore.lcov -d WebKitBuild/JavaScriptCore.build -c -b JavaScriptCore

This command will generate a single fi le, in this case javascriptcore.lcov, which contains all the code-coverage information from the regression-test suite. lcov comes with a tool called genhtml that makes pretty HTML docu-ments of this data.

$ genhtml -o javascriptcore-html -f javascriptcore.lcov

95363c05.indd 118

95363c05.indd 118 1/25/09 4:41:07 PM1/25/09 4:41:07 PM

Chapter 5 ^■ Finding Bugs 119 These HTML documents show code coverage per directory, fi le, and line, as

well as overall program statistics; see Figure 5-1.

Figure 5.1: The main lcov file that describes the code coverage obtained by the JavaScriptCore regression tests

As you can see, overall 81 percent of the lines have been executed. There is a lot of useful data here for the bug fi nder. These HTML fi les (as well as the binary lcov fi les) can be easily searched to identify lines that were executed and not executed and those that contain certain source-code constructs. For example, a quick grep will fi nd all the “copies” that have never been executed during testing.

$ grep -i cpy * | grep lineNoCov

DateMath.h.gcov.html: 112 <span class=”lineNoCov”> 0 : strncpy(timeZone, inTm.tm_zone, inZoneSize);

DateMath.h.gcov.html:<a name=”157”> 157

0 : strncpy(timeZone, rhs.timeZone, inZoneSize);</a>

number_object.cpp.gcov.html: 94 <span class=”lineNoCov”> 0 : strncpy(buf.data(), result, decimalPoint);

number_object.cpp.gcov.html:<a name=”285”> 285

 0 : strncpy(buf + i, result + 1, fractionalDigits);</a>

number_object.cpp.gcov.html: 366 <span

95363c05.indd 119

95363c05.indd 119 1/25/09 4:41:07 PM1/25/09 4:41:07 PM

class=”lineNoCov”> 0 : strcpy(buf + i, result);

ustring.cpp.gcov.html: 86 <span class=”lineNoCov”> 0 : memcpy(data, c, length + 1);</

span>

ustring.cpp.gcov.html: 102 <span class=”lineNoCov”> 0 : memcpy(data, b.data, length + 1);

ustring.cpp.gcov.html: 127 <span class=”lineNoCov”> 0 : memcpy(n, data, length);

ustring.cpp.gcov.html:<a name=”129”> 129

 0 : memcpy(n+length, t.data, t.length);</a>

ustring.cpp.gcov.html:<a name=”145”> 145

 0 : memcpy(data, c, length + 1);</a>

ustring.cpp.gcov.html: 160 <span class=”lineNoCov”> 0 : memcpy(data, str.data, length + 1);

ustring.cpp.gcov.html: 743 <span class=”lineNoCov”> 0 :

memcpy(const_cast<UChar*>(data() + thisSize), t.data(), tSize * sizeof(UChar));

ustring.cpp.gcov.html: 854 <span class=”lineNoCov”> 0 : memcpy(d, data(), length * sizeof(UChar));

Looking at one of these in more detail shows that the entire function has never been called; see Figure 5-2.

Figure 5.2: Code coverage for one particular source file

95363c05.indd 120

95363c05.indd 120 1/25/09 4:41:07 PM1/25/09 4:41:07 PM

Chapter 5 ^■ Finding Bugs 121 Notice in Figure 5-2 that some functions containing memory copies were

never executed by the regression suite. How the code coverage of this test suite changes over time can often be very telling. For example, during this test from April 2008, 83.8 perecent of the kjs directory (which contains the main JavaScript parsing code) was executed and 91.1 perecent of the PCRE code was executed.

One year earlier, 79.3 perecent of the kjs directory was tested and 54.7 perecent of the PCRE library was tested. This discrepancy between the kjs and PCRE directories in 2007 is what led us to pick so heavily on PCRE, since it was so much less tested than the JavaScript code. The authors of the JavaScript regression tests have greatly increased the effectiveness of the PCRE test cases since then.

Dans le document The Mac (Page 134-139)