8.6 Tools Comparison
8.6.2 Performances of the Proof Engines
8.6.2.1 Quick Reminder of the Tools Available, and the Way we Use Them
A detailed explanation of the mechanisms of the tools we are using has been given earlier. We give here some precisions on the way we used them, so that the benchmark be reproducible.
8.6.2.1.1 LESAR. LESARis Verimag’s original LUSTREmodel-checker. It was developed by Pascal Raymond. Its default strategy is the enumerative one, but it also provides a symbolic strategy (forward, with the option-forwardor backward, with the option-backward). We use the current version as of July 21, 2005.
8.6.2.1.2 GBAC. GBACis a prototype of model-checker also developed by Pascal Raymond in Verimag.
It uses a better BDD library and better algorithms when used with the-fb(for “Forward Bis”) option. The key point is the computation of BDDs at each step of verification, which is simpler than computing one big, generic BDD at the beginning of the proof. The-siftoption enables BDD dynamic variable reordering, which usually fasts up the proof.
It doesn’t use LUSTRE as its input, but the BA format, which is obtained from LUSTRE with the commandlus2ba. The version of GBACused is the current as of July 21, 2005.
8.6.2.1.3 NBAC. NBAC is the abstract interpreter developed by Bertrand Jeannet. We use the version current as of January 6, 2004. Some improvements have been made since that time, but are not sufficient to compete with the Boolean model-checkers for what we try to do anyway, so, we haven’t investigated further. We use it without any option.
Matthieu Moy Ph.D Thesis 139/190
Chapter 8. Back-Ends: ConnectingHPIOMto Verification Tools
8.6.2.1.4 SCADETM. SCADETM is the commercial and graphical environment for LUSTRE, devel-oped by Esterel Technologies. It can be provided with a proof engine called PROVERPLUG-INTM for SCADETM. We use the version provided with SCADETM v5.0. The prover itself is not available as a separate component to check LUSTREprograms, and has to be used through the integrated development environment. We used a somewhat dirty hack to import our generated LUSTREfiles in SCADETM: first, the LUSTREprogram is compiled into the EC format (flattened LUSTRE). Then, we have to modify it slightly to conform to the SCADETMsyntax. The perl script doing this modification is quite trivial:
s/ˆlet$/let equa well_scade_needs_this_so___[ , ]/;
s/ˆtel.$/tel;/;
We create a template project using SCADETMitself, with a sample program, and the property stating that the outputOKmust remain true and save it. The SCADETMproject is actually a directory containing several files, among which the file containing the LUSTREsample program (Node2.saofdin our case) plus many annotations in comments. We can simply replace this file with the EC file modified as described above, and reload the project in SCADETM.
8.6.2.1.5 SMV. SMVis a symbolic model checker, in the same family as LESARand GBAC, except that it does not use LUSTREas an input language. We use the free version, v10-11-02p46, which is limited in terms of functionalities. For example, it does not provide a SAT engine, whereas the commercial version does. We usedSMVwithout command-line option.
8.6.2.2 Benchmarking the Tools on a Simple Platform
8.6.2.2.1 The Test Platform. We use a simple SystemC platform to test the performances of the dif-ferent proof engines. The platform contains one BASIC channel, one master that will simply launch one transaction on the channel (Figure8.10), one slave that will raise an error whenever it receives a transi-tion (Figure 8.12) and a variable number nof transmitter (Figure8.11). When a transmitter receives a transaction, the processing of the transaction notifies an event that wakes up a thread that will itself send a transaction to the next module. The master talks to the first transmitter, theithtransmitter (i < n) talks to thei+ 1thone, and thenthtransmitter talks to the slave.
1 void compute(void) { 2 while(true) {
3 initiator_port.write(m_addr, m_data);
4 while(true) { 5 wait(20, SC_NS);
6 }
7 }
8 }
Figure 8.10: Benchmark platform: Master Process
8.6.2.2.2 The Results. The results are given in table8.1. “NT” means “not tested”, “OM” means the prover went out of memory before the end of the proof. Results of the form “>X” means the prover was killed at time “X”, with no success. The bold numbers correspond to the lastnfor which the proof engine answered in reasonable time.
The results are given in terms of user time, on a dual AMD Athlon(TM) MP 2000+ with 512Mb of RAM, running Linux.
140/190 Verimag/STMicroelectronics — 9 d´ecembre 2005 Matthieu Moy
8.6. Tools Comparison
1 tlm_status write(ADDRESS_TYPE addr_in,
2 DATA_TYPE * source,
3 int number,
4 int port_id ,
5 basic_metadata& metadata) { 6 tlm_status response;
7 e.notify();
8 response.set_ok();
9 return response;
10 }
11
12 void compute() { 13 tlm_status status;
14 while(true) { 15 wait();
16 status = initiator_port.write(m_addr, m_data);
17 }
18 }
Figure 8.11: Benchmark platform: Transmitter
1 tlm_status write(ADDRESS_TYPE addr_in,
2 DATA_TYPE * source,
3 int number,
4 int port_id ,
5 basic_metadata& metadata) { 6 ASSERT(false);
7 tlm_status response;
8 response.set_ok();
9 return response;
10 }
Figure 8.12: Benchmark platform: Slave Method
Matthieu Moy Ph.D Thesis 141/190
Chapter 8. Back-Ends: ConnectingHPIOMto Verification Tools
number of transmitters 1 2 3 4 5 6 7
PROVERtime (prove) >2h NT NT NT NT NT NT
NBACtime (no AA) 3.51 81 OM NT NT NT NT
PROVERtime (debug) 11.9 2,295 12,797 >5h NT NT NT
GBACtime 0.13 0.64 4.76 120 >1h NT NT
NBACtime 1.96 13 76 296 >1h30 NT NT
LESARtime 0.2 0.4 3.15 105 4,647 NT NT
LESAR-forward time 0.26 0.79 4.63 113 4,783 NT NT
GBAC-fb -sift time 0.2 0.76 1.95 4.4 10.5 36 147
GBAC-fb time 0.2 0.76 1.97 4.4 10.5 36 191
SMVtime 0.46 1.46 4.82 16.6 84 360.82 41.5
counterexample size 34 60 86 112 138 164 190
SMVBDD nodes 74,888 138,103 294,448 483,183 977,629 2,588,187 1,183,562
LESARBDD nodes 20,110 57,576 192,592 722,259 2,633,568 NT NT
number of transmitters 8 9 10 11 12 13
PROVERtime (prove) NT NT NT NT NT NT
NBACtime (no AA) NT NT NT NT NT NT
PROVERtime (debug) NT NT NT NT NT NT
GBACtime NT NT NT NT NT NT
NBACtime NT NT NT NT NT NT
LESARtime NT NT NT NT NT NT
LESAR-forward time NT NT NT NT NT NT
GBAC-fb -sift time OM OM OM OM NT NT
GBAC-fb time OM OM NT NT NT NT
SMVtime 67.75 150.21 697 1,648 8,951 13,691
counterexample size 216 242 268 294 320 246
SMVBDD nodes 1,396,510 1,863,109 2,839,374 4,243,537 7,433,346 14,964,410
LESARBDD nodes NT NT NT NT NT NT
Table 8.1: Comparison of Proof Engines: Results of the Benchmark
142/190 Verimag/STMicroelectronics — 9 d´ecembre 2005 Matthieu Moy
8.6. Tools Comparison
8.6.2.2.3 Conclusions About the Proof Engines. Let’s first get rid of the hopeless solutions.
PROVERPLUG-INTMfor SCADETM gives surprisingly bad performances. It is known as a good SAT solver, and wouldn’t have been chosen by Esterel Technologies if it gave so bad results in the general cases, but we are clearly in a case where it is awfully bad. In proof mode, we killed it after one hour and a half for the smallest example on which all other tools answered in less than 4 seconds! In Debug mode, it gave a few results, but several orders of magnitude worse than all other tools, and it would anyway not be applicable on correct platforms, since the only termination condition for the debug algorithm is when a counter-example is found.
When used with the abstract address encoding in LUSSY, NBACgives bad but acceptable results. Note that the platform we use does not contain integer values, so, we do not take advantage of NBAC’s strength, but use it in a case for which hasn’t been optimized. No big surprise, then . . . If we do not use the abstract addresses encoding, then, NBACwill perform its own abstractions on the addresses. Since the abstraction used (based on polyhedra) is more precise and more costly than our encoding, the bad results obtained are not surprising either. This solution is therefore more precise but clearly hopeless in the case of realistic platforms (even the biggest platform tried in this benchmark, with n = 13, can still be considered as
“small” compared to real-life designs).
Then comes LESAR. The enumerative strategy (the default) give results comparable to the results of the symbolic forward strategy (-forwardoption). LESARallowed us to go further than all the above-mentioned tools. GBAC, with the -f option, uses algorithms similar to LESAR and gives comparable results.
Remark:
We didn’t mention the results usinglesar -backwardsince-forward is obviously better: ourHPIOMencoding results in a LUSTREprogram with a small set of initial sets, a huge amount of by-construction unreachable states (any state where more than one variable encoding a control point is true for example), and therefore a huge amount of trivially unreachable bad states. The backward strategy will therefore start with a large set of bad states, and iterate backward mainly in the set of unreachable states. The forward strategy will start with a smaller set of states, and iterate forward inside a very constrained set of reachable states. Even on the smallest example,lesar -backward doesn’t find the counterexample after running more than an hour, i.e. 15,000 times the time it took for forward verification.
The bad performance for the SAT engine may come from the same problem, since SAT has a global view of the state-space, and does not necessarily start from the initial states.
GBAC, when used with the option-fb, gives better results. The fact that BDDs are recalculated at each step in a simpler way dramatically improves the performances over LESARandgbac -f. The results in terms of execution time are satisfying, but it quickly explodes in terms of memory. It would be interesting check GBAC’s memory usage, to see whether the algorithm actually needs so much memory or GBAChas a memory leak problem. Using the-siftoption slightly improves the performance, but did not allow us to prove a larger platform.
SMVgives the best results. It is slightly slower than the other tools for small systems, but is able to find the counter-example withn= 11in less than one hour, and we even pushed it ton = 12although the verification is really long in this case. Looking at the number of BDD nodes allocated, we can see that
SMVseem to manage its BDD in a better way: forn≥3, the number of BDD nodes allocated by LESAR
explodes quickly whileSMV manages to keep the progression relatively linear. What’s surprising is that the problem withn= 7is solved faster, and allocating less BDD nodes than the one withn= 6.
Globally, we can say thatSMVis the most adapted proof engine in our case.
Matthieu Moy Ph.D Thesis 143/190
Chapter 8. Back-Ends: ConnectingHPIOMto Verification Tools