HAL Id: hal-01385375
https://hal.inria.fr/hal-01385375
Submitted on 21 Oct 2016
HAL
is a multi-disciplinary open access archive for the deposit and dissemination of sci- entific research documents, whether they are pub- lished or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.
L’archive ouverte pluridisciplinaire
HAL, estdestinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d’enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.
Distributed under a Creative Commons
Attribution| 4.0 International LicenseSimulation and Visual Analysis of Neuromusculoskeletal Models and Data
Dimitar Stanev, Panagiotis Moschonas, Konstantinos Votis, Dimitrios Tzovaras, Konstantinos Moustakas
To cite this version:
Dimitar Stanev, Panagiotis Moschonas, Konstantinos Votis, Dimitrios Tzovaras, Konstantinos Mous-
takas. Simulation and Visual Analysis of Neuromusculoskeletal Models and Data. 11th IFIP Inter-
national Conference on Artificial Intelligence Applications and Innovations (AIAI 2015), Sep 2015,
Bayonne, France. pp.411-420, �10.1007/978-3-319-23868-5_29�. �hal-01385375�
Simulation and Visual Analysis of Neuromusculoskeletal Models and Data
Dimitar Stanev1, Panagiotis Moschonas2, Konstantinos Votis2, Dimitrios Tzovaras2, and Konstantinos Moustakas1
1 University of Patras, Patra 26500, Greece, stanev@ece.upatras.gr, moustakas@ece.upatras.gr,
WWW home page:http://www.vvr.ece.upatras.gr/
2 Center for Research and Technology Hellas (CERTH), Thessaloniki 57001, Greece moschona@iti.gr, kvotis@iti.gr, Dimitrios.Tzovaras@iti.gr
WWW home page:http://www.iti.gr/
Abstract. This work presents a novel medical decision support system for diseases related to the upper body neuromusculature. The backbone of the system is a simulation engine able to perform both forward and inverse simulation of upper limb motions. In forward mode neural sig- nals are fed to the muscles that perform the corresponding motion. In the inverse mode, a specified motion trajectory is used as input and the neural signals that are the root cause of this particular motion are es- timated and investigated. Due to the vast amount of information that results from even simple simulations, the results are presented to the expert using visual analytics metaphors and in particular both embod- ied and symbolic visualizations. Several use cases are presented so as to demonstrate the analytics potential of the proposed system.
Keywords: neuromusculoskeletal simulation, big data, forward simula- tion, inverse simulation
1 Introduction
Neuromusculoskeletal modelling and simulation has emerged as a potential tool for understanding how different neurological diseases have a direct impact on motor behaviour [1]. Because of the diversity, complexity and the massive amount of data, presenting in studying the aforementioned problem ether measured or produced by the simulation schemes presents a grand challenge. New tools and methods must be developed both for the production of reliable results and means to organize and process them especially in a personalized and patient driven health care [2, 3]. By introducing these technologies in health care, will have a direct impact on the patients’ quality of life, better management of the treatment and support in taking the best possible decisions for personalized diagnosis, reduction of the overall health cost, important contribution to cope with the growing demand of health services, new market areas and many benefits [4].
There is a great challenge when studying the massive amount of data handled by the biomechanics simulation process. Visual Analytics provide the suitable
tools to cope with this matter as it is defined by the union of information visuali- sation and scientific visualisation that focuses on analytical reasoning facilitated by interactive visual interfaces [5]. Visual Analytics have been assembled suc- cessfully in the domains of brain activity simulation, as well as in the biomechan- ical motion analysis [6]. Spurlock et al.[7] presented a framework for combining automated and interactive visual analysis techniques for use in high-resolution biomechanical data, analysing the complex motion of animals. Additionally, the study of human factors and ergonomics have benefited from advanced visualisa- tion techniques in order to extract high level rules from the complex data space [8]. Moreover, Visual Analytics provide the proper algorithms for biomedicine by analysing multi-scale biomedical data, especially when they derive from a variety of sources [9]. Finally, it is worth mentioning that Visual Analytics are used extensively in drug discovery frameworks [10].
2 Analysis and Simulation
2.1 General Architecture
Fig. 1.The general architecture of the proposed system: the input data (colored or- ange), the model and simulation stages and the presentation of the results using visual analytics.
Figure 1 presents the general architecture of the system. The input to the model are the highly dimensional data, such as MRI, CT or ultrasound record- ings, where model parameters are assessed and tuned per patient. Namely, each individual patient presents different geometric characteristics of the skeletal and muscular system, physical quantities: mass, inertia, length of the segments and muscular: muscle geometry, force capacity, length, point of action and many others. After the model is parametrized, simulation methods can be applied for studying different factors that have a direct impact on the motor behaviour. For the simulation, additional data are required, such as 3D motion recording, exter- nal forces, attached sensing devices and electromyography (EMG). The output produced by the simulation can be the state of the neuromuskuloskeletal system,
e.g. joint torques, reaction forces, muscle lengthening, moment arm, individual forces, neural activation signals and many model dependent variables. Finally, the aforementioned results must be presented to the clinicians in an intuitive way, emphasizing the most important factors. The variations of the presented data and the quantity are an essential factor in utilizing big data techniques.
To understand the size and the volume of the data that are processed and produced, let’s first assume that MRI, CT, Ultrasound and other recording used for model parametrization are roughly 2GB and approximately over 500x100 entries. 3D Motion capture and external measurements, can be at least 200M B for each recorded trial with the number of records depending on devices’ fre- quency and the duration of the activity, typically 3000x50 entries. A single simulation can produce around 200M B with at least over 300 different vari- ables and more than 10000 records, depending on the model and the hypoth- esis under study. So roughly speaking the volume of data is approximately O(p·(2000 + 200·n+ 200·m)) MB, where p is the number of patients, n is the number of recorded trials and mis the number of simulation executions with different settings. As for the number of entries that must be processed O(p·(50000 + 150000·n+ 3000000·m)) entries. Of course the above analysis can vary with respect to the model, the available data and the simulation resolution (e.g. microscopic of macroscopic).
Currently, we are employing visual analytics metaphors, such as scatter plots, 2D or 3D signal representations, and different visualization techniques for the assessment and the interpretation of the simulation results. Robust decision sup- port systems, based on machine learning algorithms and statistical methods can be applied to discover different criteria for classifying what is the best treatment strategy per individual. Many different models exist with a variety of experimen- tal recordings and simulation results, but unfortunately we lack the ability to organize and share them [11]. The correct assessment of patient specific disease can be improved if there are available data sets (”Golden Standards”) for the modelling and simulation tools to be enhanced and used for the improvement of health care [12].
2.2 Neuromusculoskeletal Dynamics
Multibody mechanics is the field studying the classical mechanical properties, especially motion, of systems of bodies interconnected by joints, influenced by forces, and restricted by constraints. The key feature of a system that makes it suitable for multibody treatment is the observation that its motion is localized, that is, it is well-described as a set of independently identifiable parts which undergo large motion with respect to one another, but are themselves rigid or nearly rigid. Multibody mechanics, which has been used effectively to model skeletal motion and gait and many other systems relevant to a wide variety of scientific and engineering disciplines.
Movement in animals and humans is a result of a cascade of neurological and muscle physiological processes that lead to forces on bones that generate reaction forces and accelerate joints. Multibody dynamics plays a key role by providing
a physical basis for transforming physiological forces to movement according to Newtons laws of motion. The dominant forces driving the skeletal system are musclulotendinous forces. Muscle forces arise from protein interactions that cause muscle-fibers to contract in response to the electrical state of the muscle- fiber. The electrical state of a muscle (its activation) is modulated by neural inputs from the central nervous system (CNS) also known as muscle excitations.
We are interested in understanding the control of musculoskeletal dynamics to produce coordinated movement.
The musculoskeletal model is composed of the skeletal and the muscular dy- namics. The skeletal dynamics can be successfully described from the equations of multibody theory, by modeling the degrees of freedom (DOFs) and the mass properties of the bony segments [13]. The equation of motion is defined as:
M(q)·q¨=Fa+V(q,q) +˙ G(q) +F(q,q)˙ (1) whereM∈RN xN is the mass matrix of the system,N are the number of DOFs, {q,q,˙ q} ∈¨ RN are the general coordinates position, velocity and acceleration respectively, Fa∈RN are the general forces actuating the system,V∈RN are the centrifugal and Coriolis forces,G∈RN are the internal gravitational forces andF∈RN are the external forces.
Muscles are the main actuators of the body, their characteristics are highly non-linear, with many parameters because of the variety of the presented anatom- ical types. The muscle can be described by its activation and contraction dy- namics [14]. The activation dynamics relate the calcium ion release from the sar- coplasmic reticulum with the firing of motor units which innervate the muscle.
As for the contraction dynamics, they relate the force production in accordance with the state of the muscle (length, velocity). The muscle dynamics can be expressed by the following equations:
˙
a=f(u, a, tact, tdact) (2) fM =g(a, lM, vM, Fmax, θ) (3) where, u ∈ [0,1] is the excitation of the muscle, a ∈ [0,1] is the activation level, tact, tdact are activation and deactivation time constants, fM is the force produced by the muscle, which in terms is a function of the activation level, the muscle lengthlM, the muscle velocityvM, the maximum muscle forceFmaxand other parameters θdepending on the muscle’s type.
In order for the muscle forces to be coupled into the equation of motion (1), a description of the geometrical insertion sites of the muscles on the skeleton and their joint of action must be specified [15]. The equation of motion is transformed into the following form (4) by substitution ofFa:
M(q)·q¨=R(q)·FM +V(q,q) +˙ G(q) +F(q,q)˙ (4) where, R∈ RN xM is the moment arm matrix, which relates the muscle forces FM ∈RM to the acceleration of the degrees of freedom. It should be emphasized that based on the equation (4) a muscle can contribute to the acceleration of multiple joints [13].
2.3 Forward Simulation Pipeline
Forward dynamics simulation is a valuable scheme for estimating the impact of different neural commands to the motor behavior [1]. During forward simulation the analysis begins with the cause and ends with the effect which is a motion trajectory Figure 2. The cause of the resultant motion can be a neural command orchestrating a group of muscles, activating them, thus producing forces which accelerate the joints of the model. The aforementioned process can be described mathematically by the equation of motion (4) which was derived previously, so accurate predictions can be made by solving for the general coordinatesq.
¨
q=M−1(q)· {R(q)·FM +V(q,q) +˙ G(q) +F(q,q)}˙ (5) The equation (5) can be numerically integrated twice and the coordinates of the system are predicted as a consequence of the forcesFM generated by the neural command.
Fig. 2.The forward simulation pipeline where the cause is a neural excitation command to the muscles and the effect is an observed motion trajectory.
2.4 Inverse Simulation Pipeline
In contrast with the forward, the inverse simulation begins with the effect, which usually is a recorded motion Figure 3. The goal of these methods is to estimate the internal forces, muscle activations or neural commands that govern the mus- culoskeletal model to reproduce the recorded motion trajectories, since these signals are typically difficult to be measured or observed.
Usually, the analysis begins with a general model which is personalized and scaled accordingly to the experimental subject. Along with the experimentally measured kinematics the Inverse Kinematics (IK) method is used to translate the recorded motion to model space coordinates (typically joint angles). IK can be expressed as an optimization problem (6) of finding the general coordinates q of the model, so that for every instance of time the position of the virtual
markersp(qj) placed on the model to be as close as possible to the experimentally measured landmarkspj, subject to the mobility constraints bj of the joints.
minimize
q N
X
j=1
wj· kpj−p(qj)k2 s.t.cj(qj)≤bj, i= 1, . . . , N.
(6)
At this point, is worth mentioning that Inverse Dynamics (ID) can be used to estimate the joints’ general forces using the result of the IK along with the experimentally measured external forces (e.g. ground reaction forces during gait activity) by solving the equation of motion (1) forFa.
Fig. 3.The inverse simulation pipeline for computing the muscle excitation (IK: Inverse Kinematics, RRA: Residual Reduction Algorithm, CMC: Computed Muscle Controls).
Alternatively, the muscles’ neural excitations can be assessed by using dif- ferent methods such as Computed Muscle Controls (CMC) [16]. CMC solves the problem by tracking the experimental kinematics using a PD controller in a forward dynamic manner and an optimization scheme such as Static Optimiza- tion (SO) for estimating the redundant muscle forces [17]. Figure 3 presents a method called Residual Reduction Algorithm (RRA), which can improve the results’ quality by reducing the overall error introduced by noisy measurements, model assumptions and inconsistencies [18].
3 Visual Analytics
A simulation engine has been constructed for generating the motion of a 3D vir- tual upper limb arm. The motion characteristics are derived from the proposed neuromusculoskeletal methodology. The simulation engine receives as input gy- roscopic data measured from real users and computes the activation signals of
the respective muscles. The graphical user interface of the simulation engine is depicted in Figure 4. The tool can handle multimodal data information, such as gyroscopic, accelerometer measurements, forces, activation signals and simula- tion states. The computed data timelines are fully interactive and bound to the 3D model, so any user interaction with them is automatically transformed into a resultant motion of the model, thus allowing the researcher to better under- stand the motion factor characteristics. This tool has been created for studying the upper limb kinematics and dynamics of subjects with Parkinson’s disease.
Fig. 4.Graphical user interface of the musculoskeletal upper limb model simulation.
The user can study the motion by interacting with interface components. Muscle colour- ing is applied dynamically based on activation signal profiles.
An upper limb model Figure 5 of the right arm was developed based on the previ- ous work of [19, 20] and is adjusted accordingly for the specific needs. The initial model has 15 degrees of freedom, but during simulations some of the DOFs were disabled in order to reduce the redundancy. The muscles were also reduced to 24 to simplify the number of control units (actuators) that the brain has to coordi- nate. It should be emphasized that the model is purposely simplified in order to study the main characteristics of coupled simulation between brain, peripheral nervous and the musculoskeletal systems for simple motion trajectories.
4 Preliminary Results
The potential of the proposed framework is examined by producing some pre- liminary results by employing inverse simulation techniques. More specifically, Figure 6 presents a comparison of the activation signals to the muscle for two sub- jects that were instructed to perform similar motion. First the recorded motion
Fig. 5. The developed right arm model with 15 DOFs and 24 muscles used during simulation.
trajectory was translated to model space using IK, and then CMC was utilized for the assessment of the muscle activations. Figure 7 presents a comparison be- tween the muscular force of the triceps for the two subjects. The results do not emphasise the usage of big data techniques, but if additional experimental trials were presented with more muscles and different motion trajectories the quantity of data will be significantly higher. Thus new methods must be applied for the assessment of qualitative results that can be used in decision support systems, such as machine learning or statistical analysis.
Fig. 6.Comparing the muscle activations between two subjects for the same motion.
Each figure depicts the activation signals for seven arm muscles. The user may interact with the signals and apply the motion state to the virtual humanoid.
Fig. 7.Comparing the forces generated by the same muscle (triceps) of two subjects;
although the two subjects were instructed to perform the same motion.
5 Conclusion
Due to the increased needs for patient specific health care, ICT approaches are expected to be used as a clinical decision support tool. Neuromusculoskeletal modelling and simulation has shown promising results in both accuracy and qualitative assessment of different observations that are hard to be measured or observed. The proposed approach integrates different simulation strategies in a unique medical decision support tool that also uses successful visual analytics metaphors to provide all the necessary information to the clinician in an intuitive manner. The proposed framework will be further developed so as to include brain computational models that will drive the musculoskeletal simulation. Thus, brain pathology will be directly linked to motor behaviour and vice versa.
Acknowledgement Partial funding for this work was provided by the EU co-funded project NoTremor: Virtual, Physiological and Computational Neuro- muscular Models for the Predictive Treatment of Parkinson’s Disease (EC FP7 Grant agreement no 610391).
References
1. Buchanan T., Lloyd D., Manal K., Besier T.: Neuromusculoskeletal Modeling: Es- timation of Muscle Forces and Joint Moments and Movements From Measurements of Neural Command. Journal of Applied Biomechanics. 20(4), 367–95, (2006) 2. Swan M.: Emerging Patient-Driven Health Care Models: An Examination of Health
Social Networks, Consumer Personalized Medicine and Quantified Self-Tracking.
Int. J. Environ. Res. Public Health. 6(2), 492–525, (2009)
3. Nitesh C., Darcy D.: Bringing Big Data to Personalized Healthcare: A Patient- Centered Framework. Journal of General Internal Medicine. (2013)
4. Chen H., Chiang R., Storey V.:Business intelligence and analytics from big data to big impact. Special Issue: Business Intelligence Research. 36(4), 1165–88, (2012)
5. Wong P. and Thomas J.: Visual Analytics. IEEE Computer Graphics and Applica- tions. 24(5), 20-21 (2004)
6. Keefe F.: Integrating Visualisation and interaction research to improve scientific workflows. Computer Graphics and Applications IEEE. 30(2), 8–13 (2010) 7. Spurlock S., Chang R., Wang X., Arceneaux G., Keefe F., Souvenir R.: Combining
automated and interactive visual analysis of biomechanical motion data. Advances in Visual Computing. 564–73 (2010)
8. Drury G.: Human factors/ergonomics implications of big data analytics: Chartered Institute of Ergonomics and Human Factors annual lecture. Ergonomics (ahead-of- print). 1–15 (2015)
9. Vaquero M., Rzepecki J., Friese I., Wolter E.: Visualisation and user interaction methods for multiscale biomedical data. 3D Multiscale Physiological Human. 107–
33 (2014)
10. Huan T., Wu X., Chen Y.: Systems biology Visualisation tools for drug target discovery. Expert opinion on drug discovery. 5(5), 425–39 (2010)
11. Hicks J., Uchida T.,Seth A., Rajagopal A., Delp S.: Is my model good enough? Best practices for verification and validation of musculoskeletal models and simulations of human movement. Journal of Biomechanical Engineering. 137(2), 1-24, (2014) 12. Fregly B., Besier T., Lloyd D., Delp S., Banks S., Pandy M., D’Lima D.: Grand
challenge competition to predict in vivo knee loads. Journal of Orthopaedic Research 30(4), 503-13, (2012)
13. Pandy M.: Computer Modeling and Simulation of Human Movement. Annals of Biomedical Engineering. 3, 245–73 (2001)
14. Millard M., Uchida T., Seth A., Delp S.: Flexing Computational Muscle: Modeling and Simulation of Musculotendon Dynamics. Journal of Biomechanical Engineering.
135(2), 1–12 (2013)
15. Erdemir A., Lean S., Herzog W., Bogert A.: Model-based estimation of muscle forces exerted during movements. Clinical Biomechanics. 22(2), 131–54 (2007) 16. Thelen D., Anderson F.: Using computed muscle control to generate forward dy-
namic simulations of human walking from experimental data. Journal of Biome- chanics. 39(6), 1107-15, (2006)
17. Anderson, F., Pandy M.: Static and dynamic optimization solutions for gait are practically equivalent. Journal of Biomechanics. 34(2), 153–61, (2001)
18. Delp S., Anderson F., Arnold A., Loan P., Habib A., John C., Guendelman E., The- len D.: OpenSim: Open-Source Software to Create and Analyze Dynamic Simula- tions of Movement. IEEE Transactions on Biomedical Engineering. 54(11), 1940–50, (2007)
19. Holzbaur K., Murray W., Delp S.: A model of the upper extremity for simulating musculoskeletal surgery and analyzing neuromuscular control. Annals of Biomedical Engineering. 33(6), 829-40, (2005)
20. Saul K., Hu X., Goehler C., Vidt M., Daly M., Velisar A., Murray W.: Computer methods in biomechanics and biomedical engineering. Computer methods in biome- chanics and biomedical engineering. 1-14 (2014)
