HAL Id: hal-02128381
https://hal.archives-ouvertes.fr/hal-02128381
Submitted on 14 May 2019
HAL is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
for 3D Pose Estimation
Kohei Toyoda, Michinari Kono, Jun Rekimoto
To cite this version:
Kohei Toyoda, Michinari Kono, Jun Rekimoto. Shooting Swimmers Using Aerial and Underwater
Drone for 3D Pose Estimation. 1st International Workshop on Human-Drone Interaction, Ecole Na-
tionale de l’Aviation Civile [ENAC], May 2019, Glasgow, United Kingdom. �hal-02128381�
and Underwater Drone for 3D Pose Estimation
Kohei Toyoda
toyoda-kohei648@g.ecc.u-tokyo.ac.jp The University of Tokyo
Bunkyo-ku, Tokyo
Michinari Kono mchkono@acm.org The University of Tokyo Bunkyo-ku, Tokyo Jun Rekimoto
rekimoto@acm.org The University of Tokyo Bunkyo-ku, Tokyo
Sony Computer Science Laboratories, Inc.
Shinagawa-ku, Tokyo
ABSTRACT
Methods of 3D pose estimation using RGB cameras have been studied in recent years. They are used for sports to improve athletes’ abilities and techniques by providing visual feedback. However, they still can be difficult to be used for swimming. This results from several problems, such as the difficulty of camera installation or tracking and the disturbance of bubbles and other optical issues due to the characteristics of water. To address these issues, we propose a method for shooting videos of swimmers using multiple drones. We aim to realize 3D pose estimation by using videos shot from the top and under the water with aerial and underwater drones.
This paper is published under the Creative Commons Attribution 4.0 International (CC-BY 4.0) license. Authors reserve their rights to disseminate the work on their personal and corporate Web sites with the appropriate attribution.
iHDI ’19 - International workshop on Human-Drone Interaction, CHI ’19 Extended Abstracts, May 5, 2019, Glasgow, Scotland, UK, http:// hdi.famnit.upr.si
© 2019 Creative Commons CC-BY 4.0 License.
CCS CONCEPTS
•Human-centered computing→Human computer interaction (HCI); •Applied computing
→Computers in other domains.
KEYWORDS
Sports Assistant, Pose Estimation, Aerial Drone, Underwater Drone
ACM Reference Format:
Kohei Toyoda, Michinari Kono, and Jun Rekimoto. 2019. Shooting Swimmers Using Aerial and Underwater Drone for 3D Pose Estimation. In.ACM, New York, NY, USA, 4 pages.
INTRODUCTION
3D pose estimation is an important approach and technique for analyzing human motions.
Methods of 3D pose estimation with RGB cameras have been studied in recent years and presented remarkable results by using monocular cameras [2, 3].
Figure 1: The system to shoot a swimmer using aerial and underwater drones.
Although these techniques for 3d pose estimation using RGB cameras have been used in the field of sports, using them for swimming may still be difficult. This results from several problems, such as the difficulty of camera installation or tracking and the disturbance of bubbles and other optical issues due to the characteristics of water. Therefore, motion capture techniques using sensors or markers have been used for 3D pose estimation of swimmers [4]. However, when a swimmer wears sensors or markers on their body, it may produce resistance of water. Furthermore, preparing such as motion capturing equipments are difficult in general (e.g. expensice or requires large equipments to be installed). To address these issues, we propose a method for taking videos of swimmers using multiple drones and to estimate the 3D pose.
Applications of using a drone for sports and pose estimation have been investigated in recent years.
Higuchi et al. proposed a system for a drone to autonomously track the target, and to capture athletes’ external visual imagery to support soccer and other sports [1]. Flycon is a method for environment-independent estimation of human poses in real-time with aerial vehicles [5]. Swimoid is an underwater buddy robot to support a swimmer by following them and to present visual feedback with a display [6].
We aim to take videos of the swimmer from the top and from under the water using the aerial and underwater drones. Figure 1 shows the overview of the proposed system we propose. By using these videos, we can acquire 3D pose for swimming activities.
METHOD
As a pilot study for using two drones to take videos of swimmers, we took videos of the swimmer from the top using a drone and from under the water using an action camera installed at the bottom of the swimming pool. We used a DJI Spark and its integrated camera to take videos from the top and GoPro HERO 6 installed at the bottom of a pool to take videos from under the water. Figure 2 shows an image shot from the top of a swimmer using DJI Spark and Figure 3 shows an image shot from under the water using GoPro HERO 6.
Figure 2: An image of a swimmer from the top using DJI Spark.
DISCUSSION
Swimmers can observe themselves from external point of views by using drones for video shooting.
Moreover, drones allow swimmers to see themselves from an unusual perspective such as from directly above them. This may help swimmers to perceive some amendable flaws which are not detected in the ordinary workout. On the other hand, there are some problems to shoot videos using drones. The sensors on the drone may be disturbed due to the fluctuation of the water face.
Figure 3: An image of a swimmer from un- der the water using GoPro HERO 6.
FUTURE WORK Tracking the Swimmer
Currently, we control the drone manually to trace the swimmer. For future work, by using a swim cap or a swimsuit as a marker for the tracking, we can track a swimmer without specific equipment.
Using this system, swimmers can shoot their swimming style by themselves.
Using an Underwater Drone
In our pilot study, we use an action camera installed at the bottom of the pool to shoot the video from under the water. However, a fixed camera may only shoot a swimmer for a short duration when they pass through the camera. Therefore, we consider to use an underwater drone to shoot the video from under the water so that they can trace a swimmer moving through a pool.
Feedback to the Swimmer
We also plan to apply 3D pose estimation to videos shot by aerial and underwater drones. By using 3D poses, we can present and visualize the differences between themselves and experts. Accordingly, swimmers can understand how they should fix their swimming styles.
CONCLUSION
We propose an approach to shoot videos of a swimmer from the top using an aerial drone and from the bottom using an underwater drone. By using drones to shoot a swimmer, they can observe themselves
from an external point of view and check their swimming styles. Moreover, multi-view videos shot by aerial and underwater drone may be used for 3D pose estimation and this allows us to estimate 3D pose in swimming using RGB cameras without specific equipment such as sensors or markers.
REFERENCES
[1] Keita Higuchi, Tetsuro Shimada, and Jun Rekimoto. 2011. Flying Sports Assistant: External Visual Imagery Representation for Sports Training. InProceedings of the 2Nd Augmented Human International Conference (AH ’11). ACM, New York, NY, USA, Article 7, 4 pages. https://doi.org/10.1145/1959826.1959833
[2] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik. 2017. End-to-end Recovery of Human Shape and Pose.CoRRabs/1712.06584 (2017). arXiv:1712.06584 http://arxiv.org/abs/1712.06584
[3] Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. 2017. A simple yet effective baseline for 3d human pose estimation.CoRRabs/1705.03098 (2017). arXiv:1705.03098 http://arxiv.org/abs/1705.03098
[4] Tony Monnet, Mathias Samson, Anthony Bernard, Laurent David, and Patrick Lacouture. 2014. Measurement of three- dimensional hand kinematics during swimming with a motion capture system: A feasibility study.Sports Engineering17 (03 2014), 171–181. https://doi.org/10.1007/s12283-014-0152-4
[5] Tobias NÃČâĆňgeli, Samuel Oberholzer, Silvan PlÃČÅŠss, Javier Alonso-Mora, and Otmar Hilliges. 2018. Flycon: real-time environment-independent multi-view human pose estimation with aerial vehicles. 1–14. https://doi.org/10.1145/3272127.
3275022
[6] Yu Ukai and Jun Rekimoto. 2013. Swimoid: A swim support system using an underwater buddy robot.ACM International Conference Proceeding Series, 170–177. https://doi.org/10.1145/2459236.2459265
