SmartReality: Integrating the Web into Augmented Reality
Lyndon Nixon1, Jens Grubert2, Gerhard Reitmayr2, and James Scicluna3
1 STI International, Neubaugasse 10/15, 1070 Vienna, Austria lyndon.nixon@sti2.org
2 Graz University of Technology, Inffeldgasse 16c, 2. Floor, 8010 Graz, Austria {grubert, reitmayr}@icg.tugraz.at
3 Seekda GmbH, Grabenweg 68, 6020 Innsbruck, Austria james.scicluna@seekda.com
Abstract. This poster and accompanying demo shows how Semantic Web and Linked Data technologies are incorporated into an Augmented Reality platform in the SmartReality project and form the basis for enhanced Augmented Reality mobile applications in which information and content in the user’s surroundings can be presented in a more meaningful and useful manner. We describe how things of interest are described semantically and linked into the Linked Open Data cloud. Relevant metadata about things is collected and processed in order to identify and retrieve related content and services on the Web. Finally, the us- er sees this information intuitively in their Smart Reality view of the reality around them.
Keywords. Augmented Reality, mobile, Semantic Web, Linked Data, Web ser- vices, Web APIs, Web content, Things of Interest.
1 Introduction
SmartReality is a project which began in October 2010 as a nationally funded project in Austria. The participating partners are STI International, Technical University of Graz, Seekda GmbH and play.fm GmbH. Together they represent expertise and inno- vation in Augmented Reality (AR), semantic technology, Web services and online media. In the context of the project, they explore how people may be able to access relevant information, content and media about things of interest in their vicinity via AR. The AR is enhanced dynamically by the Web of data and the Web of services.
This enhancement is made possible by a SmartReality platform which mediates be- tween the client device and the Web-based data and services, focused on using avail- able metadata to select the most appropriate content and services for display to the user. This poster and accompanying demonstrator will present the first results and prototypes of the project. We will explore and demonstrate how semantic technology and Linked Data can be integrated with AR in order to make a user’s reality “smart- er”, based on a scenario with street club posters.
2 Sm The value reflected searchers plored us informati GPS posi make use objects in of points found in selection flexible A
For ou common likely to b typically Together cess to m our goal around u Internet w tReality u
martRealit e of the ideas in recent pos s [2]. A mobi se of Linked ion and cultur itioning and a e of new data s n the AR focu s of interest ( Linked Data.
and presentat Augmented Re
ur initial appli and increasin be early adop
interested in with our part more than 18 0
is to use sema s in our curre which enhanc use case, for e
ty Vision an of Semantic W sition papers,
ile application Data and fac ral artefact me are designed to sources or ada us. The vision (POIs) interlin
This forms th tion at the AR eality experien
Fig. 1. Sm ication domai ngly shared e ters of Smart listening to m tner Play.fm G 000 DJ mixes antics and Lin ent reality to v ce our experie
example, we c
nd Scenario Web and Link not only from n, ‘Mobile cu ceted browsin etadata. These o work in spe apt the presen of SmartReal nked with the he basis of en R client. This nce.
martReality post in, SmartReal experience via
Reality soluti music, discov GmbH, whose s and live rec nked Data to d virtual inform ence of music consider a mu o
ked Data to th m SmartRealit
ultural heritag ng in combina e apps make u cific domains tation of resul lity is to lever e broad sourc nabling dynam s results in a m
ter scenario lity is focusin a the Internet ions are young vering artists a e web site and cordings to 15 dynamically l mation, conten c in our curren usic-conscious
he AR domain ty [1] but also ge guide’ [3]
ation with de use of (often i s, with limited lting content a age a richer d ce of concept mic and relevan more useful, a
g on music. M t. The user gr
g professional and attending d mobile apps 50 000+ users ink references t and services nt reality. In s person wand
n has been o other re-
, also ex- evice GPS
imprecise) d ability to around the description t metadata ant content
aware and
Music is a roup most ls who are g concerts.
s offer ac- s a month,
s to music s from the the Smar- dering city
streets an Now, at b and can b supposes link toget find out a venue wh for the co poster wi events, ve
3 Sm
A SmartR and servi low (Fig.
image rec This iden tore we te in the W services f display in
11
nd seeing the best, if this pe begin to searc they can iden ther the inform about the artis here the artist oncert. Howev ith related con enues and oth
martRealit Reality server ices. A simpli . 2). First, the cognition serv ntifier is linke erm a “TOI R eb of Data, is from the Web n the AR view
ubiquitous st erson is intere ch on aspects ntify these asp mation they a t, find some a is playing, fin ver, with Smar ntent automati
er things. Thi
ty impleme handles the i ified illustratio e object in the vice (we use K d with a desc Repository”. T s processed in b. The resulti w.
Fig. 2. Smart
treet posters a ested in the co like the artist pects from the are interested audio to listen nd out how to rt Reality they ically overlaid is is illustrated
ntation interaction bet
on of the step e mobile devi Kooaba1), whic
cription of a “ The TOI descr n order to sele ing content is
tReality workflo
advertising co ontent they ma
t, event or ven poster and th in across vari to from the ar get there, fin y can view an d over the pos d in Figure 1.
tween the clie ps taken in Sm
ces camera v ch returns an i
“Thing of Inte ription, enrich ect the most a packaged an
ow illustration
oncerts and clu ay have mobil nue. However hey still must g
ious Web sear rtist, check wh d out how to g enriched vers ter references
ent and the W martReality is iew is identifi identifier for t rest” (TOI) i hed by links to
appropriate co nd sent to the
lub nights.
le Internet r, this pre- gather and
rches, e.g.
here is the get tickets sion of the s to artists,
Web of data given be- fied via an
the object.
in a datas- o concepts ontent and client for
3.1 An To ease t we imple ports sele regions fo display th the selec from an e for play.f entry and concept).
When ed notation f event po runtime. T and is pub
nnotation the process of emented a We
ecting street p for triggering
he content ov ted regions, u existing conce fm (artists, ev d Ajax-based
The user is a diting is finish
for the poster ster is upload The TOI data blished at http
f generating th eb based anno posters from
the appearanc er the poster.
users rather s eptual scheme vents, clubs) concept selec also free to us hed, the annota and stores it i ded to Kooab
model has be p://smartreality
Fig. 3. Sm
he initial meta otation tool (F
play.fm’s im ce of content Instead of ad select a Link e. In this cas and support ction (automa se a full URI ation tool gen in a TOI repo ba to make i een created in y.at/rdf/toi.
martReality anno
adata about Th Fig. 3). The t
age database.
as well as reg dding a concre
edData URI e we use the their addition atically filling
from any oth nerates a Thin sitory. Additio it possible to
RDF/S specif
otation tool.
hings of Inter ool currently Here, the us gions where t ete link to con
representing Linked Data i n by allowing in the full U her Linked Da ng of Interest onally, the im identify the fically for Sm
rest (TOI), only sup- ser selects to actually ntent from a concept identifiers g free text URI of the ata source.
(TOI) an- mage of the
poster at martReality
3.2 Server
The server is developed as a set of components which interchange messages to realize the SmartReality functionality expressed in the above workflow (Fig. 2). The platform has been developed in Ruby on Rails and exposes a RESTful API to clients. The re- positories and APIs used by the components to retrieve data into the workflow are separated from the code of the core components so that different storage and remote API solutions (including cloud) could be used as required. After parsing the TOI’s metadata (the TOI being identified via the Kooaba identifier which is included in its description in the repository), it provides an initial response to the client which identi- fies the TOI’s regions of interest for display in the AR view (see below, Fig. 4 left).
Two further functional steps are realized on the server to provide the content bundle for the enrichment of the TOI’s regions of interest in the AR view with links to con- tent:
• Linked Data consumption. The (Linked Data) concepts used to annotate the TOI’s regions and extracted from the TOI’s metadata are crawled and further, related concepts extracted as defined in a set of Linked Data crawling rules. The rule syntax makes use of LDPath2. As a result, a local repository of relevant structured metadata about the concepts of interest in the TOI has been created. We use mainly play.fm Linked Data3 in the current demo, while the approach is vocabulary-independent, i.e.
any Linked Data could be used in the annotation and supported in this step. For the Linked Data step, we make use of the Linked Media Framework (LMF4), which pro- vides on top of a RDF repository the means to directly cache Linked Data resources and synchronize their metadata with the description on the Web. This means for a new annotation the LMF will automatically use the locally cached resource metadata in available rather than repeatedly retrieve it from the Web which can lead to latency in the platform response.
• Service selection and execution for content retrieval. Based on this local metada- ta, a service description repository is queried. Services or APIs are described in terms of their conceptual inputs and outputs so that, for the given class of a concept in the annotation, appropriate services can be found to provide content for the enrichments of the TOI in the AR view. In the current demo, we use an API provided by play.fm to access audio streams of recordings by artists as well as an API provided by Seekda to link an event to a booking interface for nearby hotels with rooms available on the night of the event. Service execution will require querying the concept descriptions to extract the necessary input values – e.g. for the hotel booking interface, the service API needs the event’s data and its location’s longitude and latitude to be passed in the input request. Likewise, the service response needs to be parsed to return to the Smar- tReality platform the content inks which can be used to enrich the TOI with respect to the original concept. For this, we use the concept of “lowering” and “lifting” in the Linked Services approach [4] where the semantic concept is ‘lowered’ to datatype values for the input request to the service, and the datatype values from the service
2 http://code.google.com/p/ldpath/wiki/PathLanguage
3 http://data.play.fm
4
output re Maps of t
The “l JSON.
3.3 Cl We built Augment tered on smartpho initializat Augment now poin right). Th object in The Sm ent on An selves.
Fig. 4. Sm are indicat
the artis booking a user is at
5 http://ww
sponse are ‘li the place).
lifted” respon
lient
t a client app ted Reality in
the physica one on the phy tion, segment ted Reality int nt towards ind he rendering o 3D space is b martReality d ndroid smartp
martReality vie ted. Middle: on st Fabio Almeri a hotel room for t a hotel room b
ww.opensceneg
ifted’ to new s nses are collec
plication proto nterface visual al object via ysical thing of
ts containing terface on the dividual segme of content that
ased on Open demo will use phones to give
w in the client.
n pressing a con ia is associated r after his next booking screen ev
raph.org
semantic conc cted and sent
otype for An lizing relevan Natural Fea f interest to ini relevant info e physical obj ents and obta t is spatially re nSceneGraph5
two real club e visitors the e
Left: poster is ntent region, the with an audio concert from S (date and locat vent’s descriptio
cepts (e.g. fro as a content
ndroid smartp nt information ature Trackin itialize a TOI ormation are h
ject (Fig.4 be ain detailed in egistered to se .
b event poster experience of
recognized and e available cont stream from pla eekda. Right: fo tion is not input on)
m a Place to bundle to the
hones. It lev n that is spatia ng. A user p query. After s highlighted th low left). The formation (Fi egments on th rs with the ins
SmartReality
d the regions wi ent items are sh ay.fm and a web following the we t, as it is known
images of e client in
verages an ally regis- points her
successful hrough an e user can ig.4 below he physical stalled cli- y for them-
ith content hown. Here
b link to eb link the n from the
4 Future Work
The SmartReality project has focused on a proof of concept with club event posters and enrichment via LOD from mainly the play.fm database. The infrastructure devel- oped has been deliberately designed to separate distinct data and content sources from the workflow which realizes a SmartReality experience, i.e. the use of other objects as
“Things of Interest”, the annotation with other LOD sources, or the linkage to content from other providers for display in the AR view, should be feasible as a configuration issue and not require any changes to the SmartReality platform or client.
5 Acknowledgements
This work has been performed in the Austrian project SmartReality (http://www.smartreality.at) which is funded by the FFG.
6 References
1. Nixon, L., Grubert, J. and Reithmayer, G.: Smart Reality and AR Standards, at the 2nd in- ternational Augmented Reality Standards meeting, Barcelona, Spain, 2011.
2. Reynolds, V., Hausenblas, M., Polleres, A., Hauswirth, M., Hegde, V.: Exploiting Linked Open Data for Mobile Augmented Reality. In: W3C Workshop on Augmented Reality on the Web, Barcelona, Spain, 2010.
3. Van Aart, C., Wielinga, B. and van Hage, W.: Mobile cultural heritage guide: location- aware semantic search. In: Proceedings of the 17th International Conference on Knowledge Engineering and Knowledge Management (EKAW ‘10), Lisbon, Portugal, 2010.
4. Pedrinaci, C., Liu, D., Maleshkova, M., Lambert, D., Kopecky, J., Domingue, J.: iServe: a linked services publishing platform. In: Ontology Repositories and Editors for the Seman- tic Web Workshop, Heraklion, Greece, 2010.
