Speed Comparison

The violet curve plots the FCP2 and FCP3 accuracy variation. As discussed above, FCP2 and FCP3 explore exactly the same portion of the search space thus their accuracy is the same, they only differ in the speed comparison as we will see later. We notice that the violet curve is generally below the blue, but it is still close to the x axis. As we expected, FCP2/3 perform a little bit worst than SCP and FCP1, since FCP2/3 explore a smaller portion of the search space.

From beam size = 1000 we notice that FCP2/3 occasionally performs better than SCP and FCP1. This shows that, for beams bigger than 1000, FCP2/3 are to be considered converged to SCP and FCP1, and the quality of its output should not considered significantly different. FCP2/3 variation has a lower bound of

≈ −3.5%.

The red curve plots the LCP accuracy variation. LCP accuracy is clearly below the O(n log n) algorithms, and does not show signs of convergence before beam = 10000. Anyway the accuracy is reasonably good and LCP variation has a lower bound of ≈ 7% that is touched with a beam in the range [100,1000].

After beam 1000, LCP starts to slowly move toward convergence. LCP accuracy is to be considered pretty good for a linear time algorithm, and possibly it is the best that can be achieved in linear time. The gain over the Naive linear time CP algorithm is evident. NCP accuracy variation is plotted with a green line.

It reaches the peak of the relative loss with beam in the range [10,100], losing

≈40% of the probability mass, then starting slowly to move toward convergence.

At beam 10000, NCP is still far from the quality of other algorithms. NCP greatly deteriorates the accuracy, showing that finding a reasonable linear time approximation algorithm is not a trivial task.

2.8.2 Speed Comparison

The speed comparison is done in terms of algorithm run-time. Table 2.3 reports the speed measures for beam = 30, comparing Standard Cube Pruning with the algorithms we propose. In the second column we report the average time spent

avg. sent. t. total t. var. CP t. CP t. var.

Table 2.3: Evaluation and comparison of the speed performance with beam size

= 30. For each of the algorithms are reported: the average time for each sentence, the speed gain with respect to SCP, the average time spent on the Cube Pruning problem, the speed gain on CP with respect to SCP.

to translate a sentence. And in the third column we report the average time variation with respect to SCP.

As described in Dyer et al. (2010), HMT systems can be decomposed into a variety of subproblems and can be designed as a pipeline; the Cube Pruning is just a single module of the sequence of operations that compose the Machine Translation decoding. To have a precise idea of the speedup obtained using the proposed algorithms, we measured the average running time for the single module that executes the CP algorithms. We report this measure in the fourth column of Table 2.3. And in the fifth column we report the variation with respect to SCP.

SCP is the algorithm exploring the biggest portion of the search space, and is the slowest one. By comparing the overall runtime with the CP runtime, we notice that more than 2/3 of the time of the MT decoder is spent running the CP module. This shows that solving the CP problem is time consuming for a HMT model, and a significant reduction in runtime for the CP algorithm will result in an overall speedup for the MT decoder. FCP1 is significantly faster than SCP.

FCP1 speeds up the CP module by 11.86%, and speeds up the overall translation by 5.58%. FCP2/3 are faster than SCP too. FCP2 speeds up the CP module by 8.02%, while FCP3 speeds up by 10.19%. From these results we can conclude that FCP3 is faster than FCP2 even if it explores the same portion of search space. This is due to the fact that FCP3 uses a simple boolean vector instead of a hash-map to check candidate collisions, as explained in section 2.6.3. On the other hand, FCP2/3 are not faster than FCP1, even if FCP1 explores a portion of the search space significantly bigger. This is due to the fact that FCP1 does not need any extra data-structure to check candidate collisions, as explained in

section 2.6.1. The NCP reaches a 16.98% speedup. This can be considered as a higher bound on the speedup that can be reached with a linear time algorithm, since NCP does not use additional data structures and is the simplest linear time algorithm solving the CP problem. LCP speedup is 14.97%, significantly better than the FCP algorithm and close to the NCP performance.

avg. sent. t. total t. var. CP t. CP t. var

Table 2.4: Evaluation and comparison of the speed performance with beam size = 300. For each of the algorithms are reported: the average time for each sentence, the speed gain with respect to SCP, the average time spent on the Cube Pruning problem, the speed gain on CP with respect to SCP.

We repeat the same series of experiments with beam 300, and report the results in Table 2.4. By comparing the overall runtime with the time spent for the CP module, we notice that the portion of the time spent on the CP module grows with respect to the measure taken with beam 30. And it is easy to generalize this result and state that with higher beam a bigger portion of the time is spent on the CP module, and the overall execution time increases. We also notice that the relative variations increase, while the ranking of the algorithms is invariate.

To further investigate the speed comparison we plot the run time variation for all the algorithms for beam sizes in the range [1,10000] in Figure 2.11. We can confirm that the trends observed for beams 30 and 300 are consistent. All the presented algorithm are faster than SCP, and the relative speedup grows with the beam size. FCP1 is the faster among the FCP algorithms even if it explores a bigger portion of the search space. The linear time algorithms are considerably faster. While Figure 2.11 compares the speed variations with respect to SCP, Figure 2.12 plots relative speed gain with respect to LCP. This allow us to clearly show the asymptotic speed advantage of the linear time algorithms (LCP and NCP) over the O(k log k) algorithms (SCP and FCPs). Given the log-scale used to plot the beam size on thex-axis, the linear shape of the speed gain over FCP and SCP empirically confirms that LCP has alog(k) asymptotic advantage.

-5

Figure 2.11: Run time variation relative to SCP for all the proposed CP algo-rithms, at a different beam sizes in the range [1,10000].

While, in contrast, the NCP speed gain converge to a horizontal line.