A Telepresence Environment for the Organization of Distributed Musical Rehearsals

(1)

Book Chapter

Reference

A Telepresence Environment for the Organization of Distributed Musical Rehearsals

KONSTANTAS, Dimitri

Abstract

A Distributed Musical Rehearsal is one of the most demanding telepresence applications, in terms of video and audio quality and transmission latency. In the frame of the EU ACTS project Distributed Video Production (DVP) we developed an ATM based environment for the organization of Distributed Musical Rehearsals. In this paper we describe the technical specifications of the installations required for the organization and studio set-up for a distributed musical rehearsal, present our implementation of the environment and give the first results obtained from the organized distributed rehearsal trials.

KONSTANTAS, Dimitri. A Telepresence Environment for the Organization of Distributed Musical Rehearsals. In: Tsichritzis, Dionysios. Objects at large = Objets en liberté . Genève : Centre universitaire d'informatique, 1997. p. 127-142

Available at:

http://archive-ouverte.unige.ch/unige:155516

Disclaimer: layout of this document may differ from the published version.

1 / 1

(2)

Organization of

Distributed Musical Rehearsals ¹

Dimitri Konstantas

Abstract

A DisLributed Musical Rehearsal is one of fhc most demnndins tcleprcsence applica- 11ons. in tcm1$ of video and audio quality and transmission latem;y. In the frmnc of rhe EU ACTS project Dl~trihuced Video Prod11ccio11 (Df/PJ we developed an A TM basi:d environmenr for the organizarion of Oisrributed Musical Rehearsals. In this paper we de- sc.ribe the ttchnical specilications of 1he installations requ ired for the organization and studio ser-up for a distributed musical rehearsal, present our implementation of the environment ond give the first results obtained from the organized distributed rehearsal trials.

1 Introduction

The recent technological advances in digital media technology and communication networks are rapidly changing the media production environments. Media producers are now able to remotely collaborate in the production of different media productions using digital video technology and broadband communication networks. The EU ACTS Distributed Video Production (DVP) project [I] aims to take existing broadband technology, mainly ATM, as starting point and to develop services (applications) on top of ATM that directly address the needs of the television industry. Several broadcasters are included in the consortium and are providing user input and participating in field trials.

One of the four pilot applications of the DVP project is the Distributed Rehearsal which aims in developing a studio-based teleconferencing service enabling small groups of geograph- ically separated actors and musicians to conduct rehearsals as if face-to-face. The target is to pro- duce a natural immersive teleconference environment allowing several people to work together as if present in the same (rehearsal) room. Musical Rehearsal, being one of the most demanding teleconference applications, was chosen as the pilot application for the development of the teleconferencing service. Toward this target a two site set-up was implemented. The first site is installed at the German Research Center for Information Technologies - GMD at Sankt Augustin, just outside Bonn, and the second site at the Centre Universitaire d 'Jnformatique of the Univer- sity of Geneva. A number of tests and rehearsal trials were performed (2] aiming in testing and fine tuning the equipment, and measuring the qualitative and quantitative performance of the system based on a methodology developed within the project.

1. This work was supported by the Swiss Federal Government with the OFES project "DVP" (project number OFES 95.0493)

127

(3)

128 Organization of Distributed Musical Rehearsals

In this paper we present an overview ot'the distributed rehearsal environment and describe the trials perfonnecl. Tn thP. SP.~tion 2 we give an overview of the functional requirements for the organization of a distributed rehearsal, while in section 3 we present the environment. Section 4 describes the rehearsal trials we organized and discuss related issues and problems. Finally section 5 presents our conclusions on the ongoing work and our future directions.

2 Functional Specifications for a Distributed Musical Rehearsal

In order to define in detail the technical requirements of an environment that will allow the organization of distributed rehearsals we need to first analyze how a (localized) rehearsal is con- ducted.

2.1 Organization of Musical Rehearsals

TI1e target of a musical rehearsal is to prepare the musicians and lhe conductor for the final per- fonnance.The musical rehearsals of modern music leading from the first meeting of the musicians and conductor, to the public performance of the musical piece, are organized, in general, into three major phases:

l . During the first phase, which we will call the protocol phase, the musicians and the conductor get acquainted with the musical piece. The conductor explains to the musicians the different notations of the musical piece and the gestures that he will be using in the direction of the orchestra.

2. During the second phase, which we will call the rhythm phase, the musical piece is played in successive fragments with the tempo slowed down in the delicate parts. The musical piece is played "out of time'', linearly, so that the musicians can have a first notion of the complete musical piece. Once this notion has been obtained it is possible to start a more detailed work on the different fragments

3. Finally the third phase, which we will call the sound phase, deals with the sound consid- erations of the rehearsed musical piece. The sound level and quality is fine-tuned and possible problems are resolved.

The protocol phase is in general short (30 to 60 minutes), while the rhythm and sound phases are very long (hundreds of hours). It must be noted that the three phases can be (and in general are) intermixed and musicians and conductor can go from one phase to another according to the needs and problems faced during the rehearsal. Also it is common for the musical play to be par- titioned in smaller pieces, each of which is rehearsed independently with alternating protocol, rhythm and sound phases. Other types of musical rehearsals (classical, jazz etc.) might have different organizational phases but the requirements are practically the same for all of them.

During a rehearsal (and of course a performance) the conductor coordinates the musicians using body language. That is, in addition of the hand movements for keeping the rhythm, the conductor expresses the strength of the music with the way he is breathing and/or the expression of his face, gives the starting or preparation signal to a musician with a fast glance and/or a movement of his head etc. The musicians constantly observe the conductor and follow his sig-

(4)

nals while in parallel they listen to the music from the other musicians synchronizing and con- trolling their own tempo. From his part the conductor also observes constantly the musicians verifying, for example, if a musician is ready (has taken the appropriate position) to start play- ing, and listens to the intensity and rhythm of the individual instruments and of the overall orchestra. In synchronizing the musicians the conductor takes imo consideralion the layou1 (distance, position) of the musical instruments in the room as well as the speed of the reflexes of the musician.

2.2 Two-site Distributed Musical Rehearsal

In a localized rehearsal the musicians and the conductor are physically in the same room. In a distributed rehearsal the musicians and the conductor will be distributed in two or more sites.

The most simple distribution of the rehearsal is when we have only two sites. In this case we have two alternatives for the distribution of the musicians and the conductor: first we can have the musicians at one site and the conductor at the other site and second we can have part of the musicians at one site with the conductor and the rest of the musicians at the other site. Since the experiment for a distributed rehearsal using telepresence techniques is novel, we will first con- sider for the definition of the functional requirements, the two-site case where the conductor is at one site and the musicians are at the other site.

2.2.1 Distributed Rehearsal Requirements

In general the requirements for organizing a distributed musical rehearsal are these of a telepresence session. That means that the basic need is the existence of (at least) a video wall with minimal dimensions 2x3 meters and high resolution video projection, and at least a hi-fi audio system. The goal is to give the impression to the rehearsal participants that they are physically in the some room. However a musical rehearsal needs not only a higher quality of audio (at least 20Hz to 22KHz) but in addition it requires an accurate 3D restitution of the sound space and very low transmission latency. This is due to the fact that the conductor needs to be able to also identify the exact position of each musical instrument and synchronize as accurately as possible with the musicians.

It is clear that the requirements for the three rehearsal phases, in terms of audio and image quality as well as synchronization and delays, are different. We thus need to define up to which point the three phases of the localized rehearsal can be distributed, and under what environment characteristics each rehearsal phase is possible.

• During the protocol phase the conductor and the musicians communicate orally explaining gestures and going over the musical piece. Thus the requirements for audio and video are relatively low, to the level of simple teleconference. No special needs for high synchronization or high quality of audio and video exist.

• During the rhythm phase the musical piece is traversed in "slow motion" so that the musicians and the conductor can be familiarized to its tempo and divisions. Thus good quality of sound and image is required as well as relatively good synchronization of audio and video. However the transmission delays can be rather high (at the level of200 ms) since the conductor syn- chronizes the musicians in an rough way.

(5)

During the s·ou11d phase lhe musical piece is perfonned as it is originally composed for. The requirem~nrs in 1wdiC1 quality and audio-video synchronization ore increasingly high. 111 <u.!Ji- lion the transmission delays must be as smaJI possible and in no case grea1er than 80 to I OOms.

This delay corre:.-ponds to a distance of 30 meters between the conductor and musicians.

which is not uncommon for large orchestras.

IL is clear that tbe rehearsal phases have increasing technical requirements, with the sound phase having the Strictest ones. Tims the stricter the requirements a distributed rehearsal environment fulfil ls. the more use!i.il it will be for the organization of distributed rehearsals. Since on the other hand the three rehearsal phases will be in genernl intermixed. an environment satisfying the requirements of only the two first phases might not be useful for more that one rehearsals. For this reason we target an environment that can satisfy the requirements of the sound phase.

Transmission Delay Requirements

During Lhe rehearsal the cond1,1ctor and the musicians must be able to perceiw l!ach other as if U1ey were in the same room in 1enns of vision and sou11d. TI11s means that the conductorshou.ld be able to see/hear U1e reaction to his gestures from the musicians with a maximum delay of about I 60ms. This corresponds 10 a one way transmission delay of 80ms.

Video Quality Requirements

Since conducting an orchestra is done using body language. th~ video quality should be high enough to allow musicians ro correctly see and interpret t·he conductor's signals. Thal means tha!

we must have high resolution video and that Lhe musicians should even be able to see where the conductor is watching. The conductor on the other hand should be able to identify correctly the position of each musician, as if they were physically i.n Lhe same room. Thus Lbe projected video should preserve the natural dimensions and perspective of the orchestra and conductor. Fortu- nately the conductor and musicians do nol change posit.ion during the rehearsal, meaning that we can statically calculate 1he right perspective for the projected video.

Needless to say we must have al least 25 frames per second for the transmitted video and at least a 50Hz projection refresh rate.

Audio Quality Requirements

The exact reproduction of the music is of major importance for the evolution of the rehearsaL Furthermore due to tbe faot that Lhe conductor needs to be able to identify the exact location of an instrument, the so.und space needs also reproduced. This means that the microphones not only need to be ofa high quality but also placed in well srudied points so that the reproduction of the sound space can be as accurate as possible.

2.2.2 Rehearsal Studio

Of major importance for the disliibuted rehearsal application is lhe organization of the rehearsal studios and the layout of the required equipment for the capture and reproduction of image and sound. The target is that at the remote site the image and sound is reproduced as accurate as possible.

(6)

yA ^' ' ^, ,

ii

~ .8- ; ,

/:•JI:. _{·r'· ...}

^I-. ^I ^'I^{'i ....}

^, , , , _,

_\

""-• .

\_ ^...

..

;N-- ^~

I I",

^{, ,} ^, , , _/\

'~

,

·~A,_; ,

' , , !

Figure 1 Conceptual view of the rehearsal

---

__

^.,..._

I

\ 0

_{~ I}

l

I

_.

1~''1·

_J

=

_'

z~ ~

_:

/\! l I

A ^! ^, ^{. ,}

I ^U ^' ^-,

¹

^I

I ' : \, I

l ,

^·~

I

Figure 2 Physical partitioning and layout of the rehearsal Video Capture and Reproduction

In a localized rehearsal the musicians and the conductor are physically in the same room. In the distributed rehearsal the musicians and the conductor will be logically in the same room. That means that the video images will be projected on a large screen (2x3m) (video wall) which, for the two peers, will be like if a frame was placed between them in the rehearsal studio (Figure I, Figure 2). Needless to say that the luminosity of the video wall should strong enough so that both sites can have sufficient ambient light.

Since the perspective of the positions of the musicians and the conductor should be pre- served, the video capture should be made from a position between the screen and the musicians/

conductor. This means that we cannot place large cameras which will hide the screen itself, but we must use non-intrusive miniature cameras. Furthermore a perspective correction of the video might be required, depending on the characteristics of the camera, so that the physical propor- tions of the participants are accurately reproduced.

(7)

132 Organization of Discributed Musical Rehearsals

Audio Capture and Reproductwn

For a distributed rehearsal the audio quality is Lbe prime factor for its success We must not only reproduce the sound in high quality but also presllrve the depth and direction infom1a1ion [3][4J.

The conductor should be able to understand the direction of eacb sound produced by the musicians. Thus the major question is bow to capture the sound so that it can be reproduced as accurately as possible.

Onemt:thod for capturing the round for a hi-fi recording is by placing a micmphone in from of each musician. However this suppresses all infonnat.ion regarding the placement of i:he musicians. In order Lo correctly reproduce the music for the conductor we must in this case use spe- ciaJized virtual reality sotlware. like for example the RSS-10 (5), le Spwialisate11r r6] or r.he ln- terac:tor [1), that recreates the spa rial infomaiion of the sound.

A second mt:ihod is 10 use a specific number of microphones (2 10 4, depending on the con- figurnrion) that will capture the sound along with all its spatial info1mation. The position and 0rien1ation oflhe microphones can be defined according to specific guidel ines as for example is described in [8]. For lhe case where Lbe conductor is at one site and the musicians at the ocher site. the calculations are rela1ivdy simple and two microphones placed al the right position will be sufficient. For more "complex·· configurat10ns tmore sites, the conductor shares a site with part of the musicians) this solution might not be adequate. Virtual reality 1echniqLit:s might need to be employed. However the study of these techniques is out of1l1e scope of this project and we will opt for simpler solutions.

A Lhird method is Lo use a dummy head, placed approximately at the place where the conductor's head would be ifhc was physically present t11e room, equipped with microphones capturing the sound at exactly tbe position of !lie ears with all the vibrations and interferences as would have been captured by a. persons ears.

Tbe sound space needs to be reproduced accurately only for the conductor. For the musicians the only sound source is the conductor who is at o specific place in front of them and urns a speaker behind the screen will be sufficient. For the reproduction of 1he sou.nd at the conductor's site we have two possibilities. The first is to use headphones and the second to use loud- speakers. Using 'headphones has the advantage that with the use of vinual reality technology [5][6][7), we can correctly reproduce the sound with all the its spatial information [9]. However this solution requi.res the installation of sensors Lhal will be able to capture the position and ori- entation of the head of the conductor.

The use of loud-speakers simplify the audio installations and provides a non-intrusive solution but it bas lhe disadvantage U1at, if no virtual reality techniques are used (like tracking of the conductor) the spatial infomlation is correct.ly reproduced at only a speclfic point in space and the conductor will have a very reduced localization capability.

2.3

Multi-site distributed rehearsal

In a multi-site distributed rehearsa"I or in the two-site sei. up with one group of the musicians and the conductor at one site and a second group of musicians at the other site, the requirements be- come more complex. First the logical layout oflhe orchestra need"s to be defined in a way so that

(8)

a physical representation is possible. That means that if for example we have a three site set-up, we will need for the two sites of the musicians 10 have two projection screens, one in front of them for the conductor and one on the side for the second group of musicians. For the conductor the image of the two sites will be merged into one in order to give the impression of a complete orchestra. However the most important challenge in multi-site set-ups is the synchronization of the music played by the musicians. Assuming that we have even a very small delay in the transmission of the video and audio, like for example 20 ms, it is impossible for the music coming from all sites to be synchronized to more than one site. For the three site example, if the conductor is listening to the synchronized music from the two musician sites, then each musician site will be listening the other site with a delay of20ms. Thus the musicians will not be able ¹⁰synchronize between themselves. This of course does not means that it is not possible to conduct a rehearsal, but that a different way of organizing and conducting the rehearsal will be needed.

3 Overview of a Distributed RehearsaJ studio instaUation

Based on the above requirements we installed two distributed rehearsal studios, one in the Uni- versity of Geneva and one at GMD. Although the two studios are not identical, the basic technology used (encoding, video/audio capture and reproduction etc.) is the same. The major ^dif~

ference of the two studios is that the GMD studio has a far greater processing power and can integrate virtual smdio techniques in the video production. In the next sections we will concen- trate at the infrastructure that is directly related to the distributed rehearsal environment. More details regarding the virtual studio installations at GMD can be found in [I O][ 11 ]. A diagram of the studio set up and network installations is given in Figure 3.

Video capture and reproduction

For the video capture non-intrusive micro-cameras (Panasonic) as well as small consumer digital cameras (SONY DCR-VX 1 OOOE) were used. The projection was made using 2 tri-tube projectors low luminosity projectors (SONY - 230 ANSI Lumen) at GMD and a high luminosity

light-valve projector at CU[ (BARCO 8 l 00 - 800 ANS! lumen). The video wall used standard medium quality screens with a size 2x2.6 meters.

Audio capture and reproduction

To accurately reproduce the sound of the orchestra at the conductor's site, two different sound capn1re systems are installed. A dummy head (Figure 4) and a dual microphone set up. The dual microphone was placed next to the video wall. For both systems a matrix was used for the correct three dimensional reproduction of the sound.

Video and audio encoding and transmission

The video and audio were digitally encoded and transmitted using ATM lines. The codecs used were the FORE AVA-300 and ATV-300 (Former name Nemesys AVA-300, ATV-300). The video was encoded in an MJPEG stream and the audio was digitized in DAT quality (48KHz sampling). The bandwidth used for the transmission of the video (non-interlaced, PAL 25 fps) was between 12 and 14 Mbps (depending on the image complexity) and for the audio (DAT ste-

(9)

Dummy head

Tom~ns'

speakers CUI

AVA-300

24 Mbps' ATM

GMD Video from

othec 'lies

Non-intrusive

•I

~ ^'th

x64

^V ^{SWl C} ^- ^.

~ -~f.,, v;deowall \ ATM Fore switch ~;;"~~ ~ ,,.,~~ I

Audio

m1xe "- ~

Microphone ----...

~

PAL video

---1 .. ~

Analogue audio Head-set

___

^.,~

A TM (15 5 Mbps Interface)

---1 .. ~

A TM (34 Mbps Interface)

Figure 3 Distributed rehearsal studio Installation

(10)

Figure 4 Dummy head

reo) 1.5 Mbps. lniLially the video encoding-decoding delay \vas 70 ms while the transmission delay was 9 ms (which makes a totaJ delay of79 ms). However further experimentation with the codecs allowed us to reduce the vidoo encoding-decoding delay lo 46 ms (by sending even fields). The audio encoding-decoding delay on the other hand was 6 ms and we introduced a 20ms buffe.ring in order 10 eliminate some buffer underflow which produced an annoying click- ing in the audio. Thus in total the audio delay was about 35 ms.

To be noted that che total bandwidth required for interlaced PAL video at 25 fps was around 29Mbps, while the available ATM bandwidth we had was 24 Mbps. It was for this reason that we Lransmiued non-interlaced video (that is, one .field per frame) which required half the bandwidth. In total (audio+ video) we used a bandwidLh of approximately 15.5 Mbps.

The A TM switching was done using a DEC GigaSwitch and Fo.re switches while the con- Lrol and ATM traffic analysis was done using a SUN Spare station and an HP ATM traffic ana- lyzer. The A TM connection from the University of Geneva to GMD was done using PVCs. It is interesting to note that we were unable to use SVCs due incompatibilities in the UN13 signaling protocols between the DEC and Fore switches.

4 Distributed Rehearsal Trials

From the several tests and trials performed the most characteristic ones were a distributed singing rehearsal and a distributed musical rehearsal [12]. While the singing rehearsal was an early trial and it did not make use of the video wall at CUI but rather a large ( l l 7cm) television screen, the musical rehearsal was a full scale trial using the complete studio installations.

(11)

4.1 Distributed Singing Kehearsal

At the 30th of May 1996 we organized a distributed singing rehearsal trial of a duet with a piano.

The pianist (Luc Baghdassarian) and one singer (Laurent Dami) were in CUI - Geneva while the second singer (Frederic Mey Jan) was in GMO - Bonn. The songs played were extracts from Hen- del's "Israel in Egypt" and Britten's "Abraham and Isaac". The total duration of the rehearsal was 2 hours.

Because no spatial sound information was needed for this rehearsal, monophonic audio channels were used for the audio transmission. The audio sent to GMO from CUI was the mixed audio signals of the piano and the singer, while the audio from GMO to CUl was the audio signal from the singer. To be noted that the used early version of the codecs' control software did not allow a control over the audio delay and buffering. As a result the audio delay was the same are the video delay and namely around 80 ms.

Figure 5 Singing Rereasal trial - CUI set-up

The video sent from CUI to GMO included both the pianist and the singer at CUI (Figure 5). However the video send from GMO to CUI was a mixing of the CUI video signal and of the GMO singer video (Figure 6). That is, the singer at GMO was placed at the left side of the image and the singer at CUI was placed at the right side ifthe image. This way each singer saw on the screen himself and the remote singer.

4.1.1 Evaluation and issues of the distributed singing rehearsal

The goal of the distributed singing rehearsal was, on the one hand, to give us a first idea of the technical problems and issues related to the organization and setup of the rehearsal and, on the other hand, to give a first subjective appreciation of the feasibility and limitations ofa distributed rehearsal.

(12)

Figure 6 Singing Rereasal trial -Video from GMD

The most important issue in the singing rehearsal trial was how to synchronize the singers.

The problem was that we had a delay of about 80 ms (one way) in the transmission of sound and image and thus it was impossible to be synchronized at both sites. lf at one site the local singer was synchronized with the remote singer, then the remote singer would perceive a 160 ms delay of his peer. The way to resolve this problem is to use the notion ofa central point where singing is synchronized. This can be either of the two sites or even a third site. In fact singing under this type of situation, that is, long delays and synchronization of the audio at a central point only, appeared and mastered during the Renaissance at the Cori Spezzati, where in a large church the conductor was in the middle of the church and the singers were distributed all around at the church's balconies. Due to the size of the church the sound delay between the singers could reach 200 ms or more (60 m distance) and the only point where the sound was synchronized was at the center of the church where the conductor was standing. Of course the difference of the Cori Spezzati and the distributed rehearsal is that in the later we not only have a delay in the sound but also a delay in the image. Thus the singers must anticipate the conductor (or each other). Nevertheless a similar case has been faced in operas when the singers sing behind the stage where they are invisible to the conductor. The singers behind the scene have to anticipate the conductor in order to synchronize with the singers on the scene.

The degree up to which synchronization can be achieved in the presence of delays also de- pends in the musical piece played. In the distributed singing rehearsal two different songs were rehearsed. One by Hendel (from Israel in Egypt) and one by Britten (from Abraham and Isaac).

The Hendel piece having a regular rhythm was easier for anticipating the remote singer. The Britten piece on the other hand did not have a regular rhythm, being free time music, and thus it

(13)

138 Organization of Distributed Musicu/ Rehearsals w:is more di fficult to anticipate the rcmult:: siugt::1 <111c.J synchronize. However m both cases after some test and trial the singers managed to synchronize their singing.

In the distributed singing rehearsal seLUp al CUI the image received Md displayed was a mix of the GMO singer and the ofU1e CUl singer (half screen each). The two images were mixed at GMO from their locaJ video and the signal transmitted from CU I, and then were send to CU1.

As a resu.11 the image oft.he CUI singer that was project at CUI had a delay of about I 60ms. This confused the CUJ singer since he was seeing himself with a quit.e long delay.

An important technical problem we faced in the distribmed singing reht:arsal was the fine tuning of the audio signals in order lo fit the acoustics of che studios. In the distributed singing rehearsal setup the sound from the remote site was reproduced using loud-speakers. This way the microphone of the singer also captured the reproduced sound from the remote singer which was then transmiued back Lo the remote site. As a result each singer heard the remote singer a11d his own voice delayed by approximately 160 ms. These artifacts and echo were extremely con- fusing for the singers. Since echo cancelers could not be used in this case we adjusted the gain of the microphones and the volume of"the loud-speakers in order to eliminate the effect.

4.2 Distributed Musical Rehearsal

A first fu ll scale distribured musical rehearsal trial was organized in November 15th 1996 between the Universi ty of Geneva and the GMO by the GRAME EOC. The total duration of the rehearsal was 6 hours. More-details on the organization and evaluation methodology of the Dis- tributed Musical Rehearsal can be found in (2) and [ 13].

4.2.1 Overview of the Distributed Musical Rehearsal

The piece retained for the distributed musical rehearsal trial. .. Derive". was composed in 1984 by Pierre Boulez for six lnstniments: piano, vibrapllone. violin, cello, flute and clarinet. The musicians where installed in Geneva while the conductor was in Gennany. The dummy head was placed approx imately where the conductor's head would be if he was physicall y present in the rehearsal (Figure 7).

The distributed musical rehearsal trial was organized into four phases. The first phase con- sisted of the tuning and equipment for the correct capture and reproduction of the sound. The second phase was dedicated in the quantitative quality control oftbe installation. The thfrd phase was the main part of the trial where the orchestra rehearsed the retained musical piece (Figure 8). Finally in the fourth phase the musicians and the conductor were interviewed independently in order to obtain a subjective measurement of the distributed rehearsal environment.

4.2.2 Evaluation Methodology

A very important issue for the distributed rehearsal system is the measurement of its quality and in consequence the limits of its usability. The quality oftbe system can be measured subjectively and objecti vely. Although subjective measurements are very important for tbe users of the system, we needed objective measurements in order to evaluate different options and technology choices. For this reason a methodology was developed for the objective measurement of tbe sys-

(14)

Figure 8 Distributed Musical Rehearsal - CUI set-up

tern quality [13]. The methodology is based on the fact that in a musical rehearsal the conductor must control, identify and possibly modify notes that are corning wrong in the musicians· scores.

During \he second phase of the trial speci fie scores were given to the musicians and the conductor. The scores given to the musicians contained various errors (like time errors, pitch errors,

Figure 7 Distributed Musical Rehearsal - Conductor's View

(15)

140 Organization ofDistributed Musical Rehearsals

dy11ami1.- e11u1s t:lc.) when compared to the score given to the conductor. The errors ranged from very easy to detect to very difficult ones. The conductor was then asked to detect the <:rrurs in the musicians scores. By reproducing the same test in a local situation with different but equiv- alent test scores, and comparing the errors found in the distributed rehearsal and in the local rehearsal we will be able to establish a concrete measurement oflhe system quality. To be noted that this is the first time that a musical rehearsal is studied from the psychocognitive point of view.

The complete evaluation requires at least two sessions, one in a local set up and one in a distributed set up, so that a comparison can be made. Although only the distributed part of the evaluation has been done to time of writing of this paper, some first results can be extracted. First of aJI the major reservation of the musicians in the distributed rehearsal, the effect of the transmission delay, was proven to be acceptable. On the other hand eye to eye communication and gestural designation between the musicians and conductor were nearly impossible. This was due to the video quality which was considered as low. An interesting psychological result was the fact that the musicians acted more as a group and felt to have increased musical responsibility, something which is not valid for normal rehearsals.

In summary we can say that the distributed musical rehearsal performed was successful and aJlowed the musicians and conductor to bypass their reservations. Interestingly the rehearsed piece "Derive" was later performed in concert without additional rehearsals, a fact which indi- cates that the distributed rehearsal did prepared the musicians for the concert as good as a local rehearsal would have done.

5 Conclusions and future plans

In the frame of the ACTS DVP project an advanced immersive teleconference environment was set up between the University of Geneva and the Gennan National Research Center for Infor- mation Technologies. The pilot application being a distributed musical rehearsal. The system was based in A TM connection and MJPEG video encoding, using video waJls and 30 sound capture and recreation. A number of trials were organized for testing and evaluating the DR system. First results of the performed distributed rehearsals are very encouraging and indicate that the system can be useful to professional musicians for conducting rehearsals over long distances.

The major technical issues identified during the trials concern primarily the importance and problems in sound localization for the conductor and the video reproduction for the musicians.

Transmission delays were not the major problem, as it was originaJly expected, as long as they did not exceeded 80 ms for the video and 40 ms for the audio.

It is obvious that in addition to the purely technical characteristics of a rehearsal there are other elements that cannot be measured, like the relation between the conductor and the musicians, the "feeling" of the rehearsal room etc. It is for these reasons that we cannot expect, at the time being, that a distributed rehearsal system can replace completely a localized rehearsal.

However the advantages of such an environment can be numerous, both in gaining time and money and in bringing together artists and ideas from around the world. For example the preparation of a musical piece for a performance requires a number of sessions (rehearsals), depend-

(16)

ing on the music to be performed, the expertise of the musicians and the conductor, etc. Bringing together a group of 10 to 20 musicians and the conductor or conductors for even 5 rehearsals requires advance planning (8 to 12 months or even more) and an important budget in order to cover travel and subsidiary expenses. Even if just half of the rehearsals can be perfonned in a distributed environment the gain in time and money is considerable, since most of the times musicians and conductor can rehearse a piece in the morning and perform another at the local orchestra in the evening. Furthermore a distributed rehearsal system can also give the possibility to musicians to work together with other musicians and conductors from other distant countries, allow music students to follow courses given by famous composers and conductors without the need to travel to the other side of the world and even allow a first level selection of musicians for an empty place in an orchestra.

Our work for the distributed rehearsal environment will continue with target the improve- ment of the system. In order to improve the sound localization we plan to install a tracking system for the conductor's head and use the tracking infonnation for eiiher driving via servo mech- anisms the dummy head or modifying the sound perspective using a spatializcr, In order to improve the video perspective for the musicians we will first try to improve tbe video quality and then investigate 3D projection technology and panoramic screens. Furthem1ore we will also experiment in the direction of the reduction of the ATM bandwidth and the video/audio compres- sion-decompression delay, testing different MJPEG and MPEG codecs. Finally we will organize a number of musical rehearsal trials where we will test the improvements and obtain more pre- cise measurements concerning the quality of the distributed rehearsal environment.

References

[ l] Distributed Video Production - DVP, ACTS project AC 089, lrttp:l!www.gmd.de/DVPI

[2] Dimitri Konsmnias. Yann Orlarcy, Simon Gibbs and Olivier Carbone!. "Distributed Musical Rehearsal'', proceeding of the lmemotiona/ Computer Music Cof!(eronce 97. September 25-30 1997, Thesssaloniki, Greece, (also in this report).

[3] Georges Canevet, "La locnllsation auditive des sons dans l'cspace", Proceedings oftheRencomres M11sirnles P/uridiscipli11aires /11for11101ique et M11~·ique-LeSm1 & 1'£.fpoce, Ed. ORAME. March 31-AprilI.1995.

Lyon, France.

[ 4] Marc Emcri1 and Jaques Mortin, "La locnlisa1ion de sources sonon:s - Appon de l'aualysc en coniposnnte indcpendantcs". Proceedings of the Re11co111res ,'vfusicales Pluridlscipli11oires l11/on1101iq1H!. et Musique -Le Son & /'£space. Ed. GRAME, March 31 - April !, 1995, Lyon, France.

[5] Christian Snles, "Processcur d'espacc sonorc Roland RSS- l O", Proceedings oflhe Rencomre.r Musicales Plu- ridfscip/111aircs lnformntique et Musique -Le Son & /'£sp11ce, Ed. GRAME, March 31 ·April I, 1995, Lyon, France.

[6] Jean-Marc Jornnd Olivier Warusfel, "La Spatialisatcur", Proceedings of the Rencontres Musfc:11/es .Pl11rfdis- cip/i11aires hiformotique et Musiq11e -Le Son & l '.Espace, Ed. GRM.!E, March 31 - April I, 1.995, Lyon, France.

[7] Pierre Alnin Jaffrennou, "De la Scenographie Sonore", Proceedings of the Rencontres Muslcoles Pl11ridisci- pli11aires hiformotiqueet Musique - LeSon & /'Espoce. Ed. GRAME, March 31 -April I, 1995, Lyon, France.

[8] Michael WiUinms, "Enregistrcmcntcl Reproduction d.e !'Environnement Sonorc Na1urcl", Proceedings of the Re11co111res Musicales Pl11ridiscipl/1111il"l!S /liformotiqul! e1 Musfque -le Son & I '£space, Ed. ORAME, March 31 -April I, 1995, Lyon, France.

(17)

\91 Xavier Marnauh, "I .a simulation des ambianoco ~onorcs rc41ite vi1luuU\!~. Proceeding,,; ot the llencon11·es Mu- s/1.:ule.< Pl11rltl/,<1;lp//11aires Injimnatique et Musiqut! -l.e Son & /'£space. Ed_ GRJ\ME. March 31 - J\pril I.

1995. I.yon. France.

(10] Chistian Brciicneder. Simon Gibb~and Con.cnantin Aranis. "TELEPORT-An Augmemcd Reality Telec6n·

fcrcncing Envlronmen1''. 3rd Eurographics Workshop oa Virrual Environments. Monte Carlo. Monaco, Feb- ruary 1996.

(11] Simon Gibbs, Constantin Arapis. Christian Breitcneder. Vali Lalioti, S. Mostafawy, Joseph Speier, "Virtual Studio_s: The Sw1c of the Art''. Eurogmphics'96 ST AR Reports-

( 12] DVP Work Package 4.3 Distributed Rehearsal, http:/lcuiwww.unige.ch/OSG/projects!DVP/wp434/

( 13] Yann Orlarey & Olivier Carbone!. "Evaluation d'un dispositif de repetition

a

distance: rapport intermediare".

GRAME Technical Report, 1997.

A Telepresence Environment for the Organization of Distributed Musical Rehearsals

Book Chapter

Reference