Quality of Experience Evaluation

We have conducted the handover field-test multiple times with two main types of scenario : in the first one, the mobile device requests a streaming service from a server over the Internet. While receiving the video, the device gets out of the current cell’s coverage and the handover happens as the way it should be ; In the second scenario, all conditions remain the same as they were in the previous test except that we put a delay into the control-plane in the handover procedure so that a disruption will be created in user-plane with an amount corresponding to the one we get from the emulation. For example, with the round trip delay of 600 ms to the Control-plane, the interruption time is about 1 second (approximately equal to the disruption given by OAI emulator in case our solution is not applied and the MCCH repetition period is 256 radio frames)

With both experiment scenarios, we identify the media stream disrupted time by mea-suring the user throughput at the mobile device. TEMS Investigation [36], the industry-leading tool for troubleshooting, verification, optimization, and maintenance of wireless networks, is used for the measurement. Integrated with VSQI method, this tool is also able to estimate the service quality and generate the QoE value for streaming service in real time. This value is comfort with MOS (i.e. it takes the value from 1 to 5). Some experiment results are given in the following figures.

Fig. 6.7 indicates the throughput at the mobile device in the first experiment. We can observer that, during the Handover, there is a moment where the throughput at physical layer of the receiver is dropped to zero, which means that the user does not receive any data from the source at that time. The interval where the throughput stays at zero corresponds to the stream interruption in user-plane. Following the record in TEMS, this disruption of the video stream in the first test is about 90-100 ms and it is hardly recognized in the video player. This amount of interruption is corresponding to the case when our method is applied with the repetition period of MSI equals to 16 radio frames.

The QoE value of the service during the handover procedure in this experiment is displayed in Fig. 6.8. As we can see, the QoE, or to be more exact, the VSQI value of the service during the handover declined a little but still at the acceptable level (from 4 to 3.4) and it goes back to the level as it was before the handover in 12 seconds.

In the second scenario, when we add a delay to the control-plane, the interruption time in the user-plane increases to one second as shown in Fig. 6.9. And as expected, with this disruption, the VSQI value dropped dramatical from 4 to 1, which is a very bad quality.

We have done the experiment for this scenario several times and receive the similar results in stream disruption time and QoE value as given in Fig. 6.10 and Fig. 6.11

(a) Diruption Starts

(b) Diruption Ends

Figure 6.7 – U-plane Interruption Time in normal HO Procedure.

(a)

(b)

(c)

Figure6.8 – QoE of the Service during normal HO Procedure.

(a) Time Diruption Start

(b) Time Diruption End

Figure 6.9 – U-plane Interruption Time with the delay in C-plane.

(a)

(b)

(c)

Figure 6.10 – QoE of the Service in case delay is added to the C-plane (first trial).

(a)

(b)

(c)

Figure 6.11 – QoE of the Service in case delay is added to the C-plane (second trial).

6.5 Conclusion

In this chapter, we have done the emulation with different scenarios to validate the implementation of eMBMS in OpenAirInterface platform. The in-lab emulations in both link level and system level have been carried out to evaluate the BLER and user throughput of the eMBMS system. Moreover, we have tested the eMBMS transmission in both TDD and FDD system using the traffic generator tool OTG as well as the real VLC video streaming application for creating the source data. The emulation results have validated our eMBMS implementation in a real time system and it is ready to be embedded into hardware for a real environment test. The emulation also allows identifying the time for retrieving every eMBMS signaling information. From the analysis, we know that the total time to collect all control information is one of the main factors contributing to the service interruption time during the handover procedure. Therefore, our method that helps to transfer a large part of these signaling messages to the user during handover period can reduce considerably the interruption time and thus improve the service quality.

In another effort, we have conducted a field-test in real environment to identify the impact of service disruption on the QoE. The outcome of real experiments has confirmed the influence of disruption time on the quality perceived by the users (i.e. the VSQI metric).

Particularly, in case the user-plane interruption when not using our method is about one second (which is roughly equal to the disruption in eMBMS service during the handover with the MCCH period of 32 radio frames), the service quality will change from good to very bad level. While if our mechanism is applied, the quality is only dropped a bit, to accepted level. This result has allowed us to conclude that our proposed solution can enhance the quality of eMBMS service perceived by the human users in handover situation.