Sensor Observa,on Streams within Cloud-based
IoT Pla:orms: Challenges and Direc,ons
Antoine Auger
,
Ernesto Exposito, Emmanuel Lochin
ISAE-SUPAERO, Université de Toulouse, France
MoMvaMon
•
Internet of Things (IoT) and data explosion
•
Many “IoT plaTorms” for many use cases providing different
services and/or APIs
•
“Black box approach” so far
Which requirements? Which so[ware soluMons? Why?
•
Common challenges when building so[ware plaTorms, need
to explain:
– Concepts, technical issues – How to cope with them, good pracMcesMoMvaMon
•
We focus on:
– IoT pla:orms: ingest informaMon from Things, create value from data, services exposed to users or applicaMons – Cloud-based: available on demand via the Internet, So[ware as a Service model (SaaS), other non-funcMonal requirements – Sensor observa,on streams: physical-observed phenomena or virtual-occurred events•
ContribuMons of the paper:
– Insights for researchers who are interested in IoT plaTorms capable of handling sensor observaMon streams – Takeaways for developersMoMvaMon
•
“Lessons learned” coming from:
– State of the Art – Study of available tools/components for IoT plaTorms – Acended conferences (in parMcular, see [Kamienski2016]) – Own experience: development of an IntegraMon plaTorm for QoI Assessment as a Service (iQAS) [Auger2017]•
Disclaimer:
– Not exhausMve! – May contain subjecMve thoughts…Agenda
1.
ObservaMon-related challenges
What? ObservaMon streams, levels, quality
2.
ImplementaMon consideraMons
What solu5on? How? ObservaMon collecMon, processing and consumpMon
3.
Non-funcMonal requirements
With what features? AdaptaMon, scalability, availability
ObservaMon-related challenges
•
We assume that an observaMon is
something
produced by a
sensor (either physical or virtual)
•
IoT plaTorms are asked to create value from received
observaMons
– Data-centric systems•
Heterogeneous observaMons:
– Level – Quality – Sensing rate – Coverage – …ObservaMon-related challenges
•
Taxonomies to denote different observaMon levels
– Useful to esMmate the level of complexity required to process and “understand” observaMons by final consumers Context annotation Semantic annotationAnalysis and processing
Wisdom Information Raw Data Knowledge An augmented version of the “DIKW ladder” of Sheth [Sheth2016]
Observa,on levels
ObservaMon-related challenges
Observa,on quality
•
Several quality dimensions:
– Quality of Service (QoS) – Quality of InformaMon (QoI) [Bisdikian2013] – Context, Quality of Context (QoC) – Quality of Knowledge (QoK)In short:
ü All dimensions not always needed ü Acributes to consider may vary according to use cases (delay, obs. life5me, accuracy, trust, …) ü Recent works consider mainly QoS-QoI (e.g., FP7 CityPulse)ObservaMon-related challenges
•
OperaMons on streams ≠ databases
•
ObservaMon streams
– Unbounded – Heterogeneous – Real-Mme•
ReacMve Streams
– hcp://www.reacMve-streams.org – Asynchronous stream processing with non- blocking back pressure – Java APIsUnbounded observa,on streams
5 4 2 -1 9 8 8 4.5 0.5 ?? 5 4 9 8 8 Time Sliding window Incoming stream "Sliding" mean Filtering (obs > 3) 8ImplementaMon consideraMons
•
A jungle of IoT plaTorms
– Open-source soluMons, projects: FP7 INSIGHT, FP7 OpenIoT, FP7 CityPulse, H2020 OrganiCity, … – Commercial or proprietary soluMons (SaaS): AWS IoT, IBM Watson IoT, …•
Researchers may want to add new features:
– By extending exisMng soluMons: requires a learning phase – By developing new prototypes from scratch (preferred)•
Most popular data architectures:
– Lambda Architecture hcp://lambda-architecture.net – Kappa Architecture hcp://www.kappa-architecture.comImplementaMon consideraMons
Speed (real-time) layer
Batch (offline) layer Observation
ingestion Observation
sources Observation consumers
Distributed storage
Cloud-based IoT platform
Observation collection Observation processing Observation consumption Serving layer Unified Stream Processing layer Observation ingestion Observation
sources Observation consumers
Distributed storage Cloud-based IoT platform
Serving layer Job version n Job version n+1 Result n Result n+1 Kappa Architecture Lambda Architecture
ImplementaMon consideraMons
•
Virtual Sensor Containers (VSCs)
– For preliminary tests – To assess plaTorm scalability – APIs to modify sensor behavior on the fly•
Apache Nifi
– hcps://nifi.apache.org – Open source, web-based GUI – Run in local/cluster mode – Supports directed graphs of data rouMng, transformaMon, and system mediaMon logic Sensor Observations Virtual Sensor’s APIObserva,on collec,on
ImplementaMon consideraMons
•
Event Stream Processing (ESP)
•
Pick one framework according to your needs
– Final choice may depends on many features [ESPcomparison]Observa,on processing
ESP / CEP Applications Sensors File Systems Message logs Applications Databases File Systems Message logsImplementaMon consideraMons
Observa,on consump,on
•
Message Brokers
– “Shock absorbing” technology – Compliant with ReacMve Streams iniMaMve•
Trend: Kapa for the IoT
– Developed by LinkedIn, open sourced in 2011•
Benchmarks for
comparison but biases
•
Real boclenecks: final
consumers
Non-funcMonal requirements
•
FuncMonal requirements depends mainly on use cases
•
Some non-funcMonal (NF) requirements are recurrent
regarding Cloud-based IoT plaTorms
•
Among them, some NF requirements may be fulfill by relying
on the Cloud CompuMng paradigm
Non-funcMonal requirements
•
Autonomic CompuMng [Kephart2003]
– IBM MAPE-K control loop • Monitor • Analyze • Plan • Execute•
IoT plaTorms act as mediators
– Sensor capabiliMes – Consumer needs – Service Level Agreements (SLAs)•
AdaptaMon is needed
Pla:orm adapta,on
Autonomic manager Managed element Managed element touchpointSensors Effectors
Monitor Execute
Plan Analyze
Knowledge
