Report
Reference
Requirements and specifications for the development of BSC-OS Portal
GORGAN, Dorian, et al.
Abstract
This document describes the requirements and specifications of the enviroGRIDS BSC-OS Portal. It concerns mainly with user, functional, technological, and usability requirements of user applications. The functionality and data repositories required in Web and Grid infrastructures are analyzed together with the main issues of the interoperability between functional modules as well as between the geospatial and Grid infrastructures. The document explores the possible architectural and technological solutions, and details and recommends solutions for the development of the BSC-OS portal.
GORGAN, Dorian, et al . Requirements and specifications for the development of BSC-OS Portal . 2010, 75 p.
Available at:
http://archive-ouverte.unige.ch/unige:79002
Disclaimer: layout of this document may differ from the published version.
Requirements and specifications for the development of BSC-OS Portal
Title Requirements and specifications for the development of
BSC-OS Portal
Creator Dorian Gorgan (UTC)
Creation date 10.02.2010
Date of last revision 01.06.2010
Subject Requirements and specifications of the enviroGRIDS BSC-
OS Portal. User, functional, technological, and usability requirements. Architectural solutions and software components.
Status Finalized
Type Word document
Description Describes the requirements and specifications of the
enviroGRIDS BSC-OS Portal. It concerns mainly with user, functional, technological, and usability requirements.
The document explores as well the possible architectural and technological solutions, and details and recommends solutions for the development of the portal.
Contributor(s) Dorian Gorgan, Pierluigi Cau, Karel Charvat, Denisa Rodila, Victor Bacu, Andreja Jonoski, Ann van Griensven, Petr Horak, Karim Abbaspour, Simone Manca, Gregory Giuliani, Nicolas Ray, Anthony Lehmann
Rights Public
Identifier enviroGRIDS_D61
Language English
Relation 1. Interoperability Guideline (enviroGRIDS_D21)
2. Data Storage Guideline (enviroGRIDS_D22) 3. Remote Sensing Data Use and Integration Guideline
(enviroGRIDS_D24)
4. Infrastructure Sustainability Guidelines (enviroGRIDS_D25)
Abstract:
This document describes the requirements and specifications of the enviroGRIDS BSC-OS Portal. It concerns mainly with user, functional, technological, and usability requirements of user applications. The functionality and data repositories required in Web and Grid infrastructures are analyzed together with the main issues of the interoperability between functional modules as well as between the geospatial and Grid infrastructures. The document explores the possible architectural and technological solutions, and details and recommends solutions for the development of the BSC-OS portal.
Executive Summary
EnviroGRIDS (Building Capacity for a Black Sea Catchment Observation and Assessment System supporting Sustainable Development) is a 4-years project funded under the EC Seventh Framework Programme, aiming to address the subjects of ecologically unsustainable development and inadequate resource management in the Black Sea Catchment area. The project will develop a Spatial Data Infrastructure (SDI) targeting this region and linked to the EGEE infrastructure. A large catalogue of environmental data sets (e.g. land use, hydrology, and climate) will be gathered and used to perform distributed spatially-explicit simulations to build scenarios of key environmental changes.
The enviroGRIDS system resources are accessible to the large community of users through the BSC-OS Portal that provides Web applications for data management, hydrologic models calibration and execution, satellite image processing, report generation and visualization.
There are five categories of users such as data providers, earth science specialists, decision makers, citizens, and system administrators. The citizens and the decision makers visualize the reports generated by the specialists as results of executing environmental scenarios. The input data for the reports are built up by the specialists by running hydrological models of the Black Sea catchment area and by processing related satellite data. All data sets required building up the hydrological models, environmental scenarios, and spatial models are provided and entered into the system by the data providers.
The portal publishes through the Web applications the geospatial functionality provided through Web technologies and the high power computation supported by the Grid technologies. One great challenge of the enviroGRIDS project is the interoperability between the Geospatial and Grid platforms in order to scale and combine efficiently the complex specialized functionality and the computation potential of these platforms.
The purpose of this document is to identify the requirements and to define the specifications of the enviroGRIDS BSC-OS Portal. The document describes the user, functional, technological, and usability requirements, and explores the possible solutions for the portal development. Finally, a set of recommendations concludes on this technical analysis.
Contents
1 INTRODUCTION... 7
1.1 PURPOSE AND SCOPE... 7
1.2 DOCUMENT STRUCTURE... 7
2 ENVIROGRIDSSYSTEM ARCHITECTURE... 8
2.1 MAIN OBJECTIVES... 8
2.2 ENVIROGRIDSLAYERED FUNCTIONALITY... 8
2.2.1 Data Repositories ... 9
2.2.2 Resource Management and Data Processing Services ... 9
2.2.3 EnviroGRIDS Applications... 10
3 ENVIROGRIDSSYSTEM INTEROPERABILITY... 11
3.1 ARCHITECTURAL LAYERS... 11
3.2 GRID ORIENTED ARCHITECTURE... 11
3.3 GEOSPATIAL AND GRID INTEROPERABILITY IN ENVIROGRIDS... 12
3.3.1 Overview and Related Achievements ... 12
3.3.2 OGC Web Services ... 13
3.3.3 Specific Case of Interoperability in EnviroGRIDS ... 13
3.3.3.1 Portal Level Interoperability ... 14
3.3.3.2 Execution Level Interoperability... 14
3.3.3.3 Data Level Interoperability ... 14
3.3.3.4 Web and Grid Service Classification ... 14
4 ENVIROGRIDSPORTAL REQUIREMENTS... 19
4.1 PORTAL OVERVIEW... 19
4.2 USER REQUIREMENTS... 19
4.2.1 User Categories ... 19
4.2.2 Functional User Requirements ... 19
4.2.2.1 Spatial Data Management ... 19
4.2.2.2 Hydrologic Model... 21
4.2.2.3 Satellite Image Processing ... 24
4.2.2.4 Data Visualization and Report Generation... 26
4.2.2.5 Report Visualization for Decision Makers and Citizens ... 28
4.2.3 Usability Requirements ... 28
4.3 TECHNOLOGY REQUIREMENTS... 29
5 PORTAL SPECIFICATIONS... 30
5.1 PORTAL ARCHITECTURE... 30
5.1.1 BSC-OS Portal ... 30
5.1.2 Web Infrastructure ... 31
5.1.3 Grid Infrastructure... 31
5.2 DATA FLOW... 31
5.2.1 Spatial Data Management ... 31
5.2.2 Hydrological Model Data Flow... 31
5.2.3 Satellite Data Processing... 33
5.2.4 Data Visualization and Report Generation... 33
5.2.5 Report Visualization ... 33
5.3 FLOW CONTROL... 34
5.4 TECHNOLOGICAL PLATFORMS BASED ARCHITECTURE... 34
6 SPATIAL DATA MANAGEMENT SOLUTION... 35
6.1 URM INTRODUCTION... 35
6.2 GEOPORTAL AND AUTHORIZATION... 37
6.3 SIMPLE CMS ... 37
6.4 MICKA-METADATA EDITOR... 38
6.4.1 Functions... 38
6.4.2 MIcKA System Requirements ... 39
6.4.3 Key Words Based Working Concept ... 39
6.4.4 Support of the INSPIRE Project... 39
6.5 DISCOVERY SERVICES -CATALOGUE... 39
6.5.1 Functionality Description... 40
6.6 GEOHOSTING... 40
6.6.1 DataMan... 42
6.6.2 MapMan... 42
6.7 VISUALISATION HSLAYERS... 43
6.7.1 HSLayers Components... 43
6.7.1.1 Server Scripts... 45
6.8 PYWPS ... 45
6.9 SENSOR INTERFACE IMPLEMENTATION... 46
7 HYDROLOGIC MODEL MANAGEMENT SOLUTION... 48
7.1 SWATHYDROLOGIC MODEL OVERVIEW... 48
7.2 HEC-HMSHYDROLOGIC MODELING SYSTEM... 48
7.3 GANGA IN ENVIROGRIDS... 49
8 SATELLITE DATA PROCESSING SOLUTION... 52
8.1 ESIP-ENVIRONMENT ORIENTED SATELLITE DATA PROCESSING PLATFORM... 52
8.2 GPROCESSANDESIPPLATFORM... 53
8.2.1 gProcess Architecture Description ... 54
8.2.2 ESIP Operators... 54
8.2.2.1 Basic Operators... 54
8.2.2.2 Logical Operators ... 55
8.2.2.3 Spatial Domain Transformation Operators ... 55
8.2.2.4 Application Oriented Operators ... 55
8.3 ESIP BASED GRID APPLICATION DEVELOPMENT METHODOLOGY... 56
8.4 PROCESS DESCRIPTION GRAPHS... 56
8.4.1 gProcess Graphs... 56
8.4.2 Resource Nodes... 57
8.4.3 Operator Nodes ... 58
8.4.4 Subgraphs... 58
8.4.5 Service Nodes ... 58
8.5 MONITORING SOLUTION... 59
9 DATA AND REPORT AUTHORING AND VISUALIZATION... 60
9.1 COLLABORATIVE WORKING ENVIRONMENT... 60
9.2 SCRIPT BASED REPORT AUTHORING... 62
9.3 ROLE BASED AUTHENTICATION AND AUTHORIZATION... 62
9.4 INTEROPERABILITY BETWEEN CWEPORTAL AND GRID... 63
9.5 CWEDEVELOPMENT PHASES... 63
10 ON-LINE TRAINING SYSTEM... 65
10.1 TRAINING SYSTEM REQUIREMENTS... 65
10.2 USE REQUIREMENTS OF E-LEARNING APPLICATIONS... 65
10.3 EGLE E-LEARNING ENVIRONMENT... 66
10.3.1 eGLE Overview ... 66
10.3.2 Lesson Scenarios... 67
10.3.3 Lesson Development Phases ... 68
11 CONCLUSIONS AND RECOMMENDATIONS... 71
11.1 CONCLUSIONS... 71
11.2 RECOMMENDATIONS... 72
ABBREVIATIONS AND ACRONYMS... 73
List of Figures
Figure 1 EnviroGRIDS functional layers. 9
Figure 2 EnviroGRIDS system conceptual architecture. 11
Figure 3 Gridification process based on Mediator Component. 15
Figure 4 Case1: Web Geospatial database, no Grid workers, Case2: Web Geospatial database, Grid workers. 16 Figure 5 Case3: Grid Geospatial database, Grid workers, Case4: Grid Geospatial database, no Grid workers. 17
Figure 6 General Case: Web and Grid Geospatial database, Grid workers. 18
Figure 7 GUI prototype: (a) User describes the report content by editing the script code; (b) The information is shown on the portal using widgets like maps, charts or tables. They can be organized using HTML and JavaScript.
26
Figure 8 GUI prototype: Decision makers and Citizens may visualize the reports by fixed and interactive manner.
27
Figure 9 BSC-OS Portal related architecture. 30
Figure 10 Data flow within the BSC-OS Portal related architecture. 32
Figure 11 Service oriented EnviroGRIDS Functional Architecture. 33
Figure 12 URM – Uniform Resource Management architecture. 41
Figure 13 Project Editor and its connectors to data sources and publication functionality. 42
Figure 14 PyWPS based architecture. 45
Figure 15 Screenshots of Ganga Graphical User Interface. 49
Figure 16 Job components in Ganga. 50
Figure 17 Functional levels of the ESIP platform related architecture. 52
Figure 18 gProcess platform based architecture. 53
Figure 19 GreenView application – satellite image classification by vegetation index computation. 55
Figure 20 Process description hypergraph involves data (Input), operators (Op1-3), services (S1, S2) and subgraphs (SG).
56
Figure 21 iPDG description and the resulted image of the Enhanced Vegetation Index (EVI) algorithm. 57
Figure 22 The CWE functionality should support the development of reports. 60
Figure 23 Reports generated by CWE includes maps and results of SWAT model execution. 61
Figure 24 Interoperability between the CWE Portal and the Grid infrastructure. 63
Figure 25 Functional levels in the eGLE related architecture. 66
Figure 26 Lesson authoring in eGLE. 68
Figure 27 Grid processing based lesson execution. 69
Figure 28 Samples of the Grid based lesson execution. 70
1 Introduction
1.1 Purpose and Scope
EnviroGRIDS (Building Capacity for a Black Sea Catchment Observation and Assessment System supporting Sustainable Development) is a 4-years project funded under the EC Seventh Framework Programme, aiming to address the subjects of ecologically unsustainable development and inadequate resource management in the Black Sea Catchment area. The project will develop a Spatial Data Infrastructure (SDI) targeting this region and linked to the EGEE infrastructure. A large catalogue of environmental data sets (e.g. land use, hydrology, and climate) will be gathered and used to perform distributed spatially-explicit simulations to build scenarios of key environmental changes.
The BSC-OS Portal is the single way the users access the enviroGRIDS applications and spatial data. The portal provides Web applications for data management, hydrologic models calibration and execution, satellite image processing, report generation and visualization. The portal allows the users the access to geospatial functionality given by Web infrastructure, and to high power computation resources given by the Grid infrastructure.
The purpose of this document is to describe the requirements and specifications of the enviroGRIDS BSC- OS Portal. It concerns mainly with user, functional, technological, and usability requirements. The document explores as well the possible architectural solutions, and details and recommends one solution for the portal development.
This document is related with other technical guidelines for the enviroGRIDS project that have been published in the previous phases of the project:
1. Interoperability Guideline (enviroGRIDS_D21) 2. Data Storage Guideline (enviroGRIDS_D22)
3. Remote Sensing Data Use and Integration Guideline (enviroGRIDS_D24) 4. Infrastructure Sustainability Guidelines (enviroGRIDS_D25)
This document has to be considered as a reference document as we expect continuous development of the proposed portal while we gain experience with practical implementations, feedback from our users and partners, and evolution of the requirements and technology.
1.2 Document Structure
This document is structured in eleven chapters. Chapter 1 (this chapter) is a short presentation of the enviroGRIDS project and the purpose of the document. Chapter 2 is an introduction to the enviroGRIDS project describing the objectives, and the layered functional architecture of the enviroGRIDS system. It includes a short description of data repository, main services, and applications. Chapter 3 describes the main issues of interoperability between geospatial and Grid infrastructures. Chapter 4 presents the user requirements of the applications provided through the portal. Chapter 5 analyzes an architectural solution by the specification of the data and control flow throughout the BSC-OS Portal. Chapter 6 details the solution proposed for spatial data management. Chapter 7 analyzes the solution for the hydrologic model calibration and execution. Chapter 8 details the solution given by the gProcess and ESIP platforms for the satellite image processing. Chapter 9 describes the solution offered by CWE environment for the report authoring and visualization. Chapter 10 presents the training system included into the BSC-OS Portal. The final chapter 11, sketches the conclusions and the main recommendations of this document.
2 EnviroGRIDS System Architecture
The Black Sea Catchment is recognized for its ecologically unsustainable development and inadequate resource management leading to severe environmental, social and economical problems. The FP7-funded enviroGRIDS project (April 2009 – March 2013) will address these issues by developing a Spatial Data Infrastructure (SDI) targeting this region and linked to the EGEE infrastructure. A large catalogue of environmental data sets (e.g. land use, hydrology, and climate) will be gathered and used to perform distributed spatially-explicit simulations to build scenarios of key environmental changes.
2.1 Main Objectives
During its 4-year timeframe, the enviroGRIDS project aims at several objectives for which the EGEE infrastructure will be instrumental:
• A high resolution (sub-catchment spatial and daily temporal resolution) water balance model will be applied to the entire Black Sea catchment using the Soil Water Assessment Tool (SWAT). This tool will first be gridified to be used on the EGEE infrastructure.
• Access to real time data from sensors and satellites will provide early warning and decision support tools to policy-makers and citizens. These data need to be streamlined into the grid-enabled enviroGRIDS SDI to ensure fast computation and dissemination of results.
• Because spatial data is very heterogeneous in format and quality across the European community, urgent efforts are needed to organize and standardize spatial data to improve its interoperability. The enviroGRIDS SDI will rely on the development of policies, technologies, data, common standards, standard practices, protocols and specifications such as those of the Open GIS Consortium (OGC).
Through cataloguing, the grid infrastructure will help implementing and sharing standardized data sets.
By taking a watershed approach involving 15 eastern-European countries, enviroGRIDS is the first truly trans-national effort to address many societal issues in the region. This system will incorporate a shared information system that operates on the boundary of scientific/technical partners, stakeholders and the public. It will contain an early warning system that will inform in advance decision makers and the public about risks to human health, biodiversity and ecosystems integrity, agriculture production or energy supply provoked by climatic, demographic and land cover changes on a 50 year time horizon. It will significantly build local, national and regional capacity on Earth Observation Systems through active contribution to the GEOSS and INSPIRE initiatives.
The strong grid component of the project will foster data interoperability, and will most certainly trigger new directions of research or alternative ways of analyzing high resolution data sets. The large user community of SWAT may greatly benefit from a gridified version of the software, and a dedicated Virtual Organization will be created in this regard.
2.2 EnviroGRIDS Layered Functionality
EnviroGRIDS is a distributed system built up on Service Oriented Architecture (SOA) that allows the flexible usage of services over heterogeneous architectural components and technologies. The functionality provided by services could be used anywhere over the computing infrastructure by open standards and communication protocols.
The functionality and the layered architecture of the enviroGRIDS system are presented in Figure 1. The lower level is the data level. The following levels are Grid Infrastructure and Middleware. The Grid Infrastructure is provided by EGEE network, on which the gLite middleware is running. The upper levels
consist of services and Web Portal. Actually the services implement the basic functionality and processing that is combined in scenarios and interactive applications.
2.2.1 Data Repositories
Data Repositories consist of stored data and functionality required to manage the repository. The repository stores raw data (e.g. spatial data) as well as processed data (e.g. maps, tables). They store application data, which are specific for each application type and instance (e.g. hydrology, climate, soil, etc). The register stores metadata catalogues that support the searching, discovering and using of distributed data by the user applications, and processing services.
2.2.2 Resource Management and Data Processing Services
This level provides the basic services available over the Grid as secure and persistent services and over the Web as stateless services. The services encapsulate the basic functionality provided to user applications.
Such service categories are:
Figure 1 EnviroGRIDS functional layers.
Web Portal Applications/
SWAT Scenarios
Decision Maker/
Citizen Tools Data Management
Tools
Web and Grid Services Data Management Data access, transfer, replication, storage metadata, catalogues
Security and User Management VOMS, authentication, authorization, credential management
Scheduling, Monitoring
SWAT Management and
Execution Workflow Management
Edit, service composition, Grid mapping, execution, fault recovering
Spatial Data Acquisition, processing, Visualization, mapping
gLite Middleware Grid Infrastructure (EGEE)
Data Repositories
- Spatial data, catalogues, maps
- Application data (hydrology, clime, soil, etc.) - Scenarios
- Results of processing
− Data Management: provides the basic operation on data repositories (e.g. data access, transfer, replication, storage metadata). The catalogues oriented operations are supported by AMGA Metadata Catalogue1, which is the metadata service used in the gLite middleware.
− Security and User Management: provides the user the functionality needed to work with VOMS database, in order to support user authentication, authorization, and credential management.
VOMS support the implementation of particular policies for data access and use.
− Scheduling: provide optimal resource allocation and sharing. Static or dynamic load balancing provide the best efficiency, costs, and use of available resources.
− Monitoring: support evaluation of the execution performance, and statistical analysis.
− SWAT Management and Execution: provide the functionality to control the execution over the Grid of the SWAT modules and related data.
− Workflow Management: supports the graph description of the processing, service composition, Grid mapping, workflow interpretation and execution, and fault recovering.
− Spatial Data Acquisition: support the working with sensors. They supervise the sensor status, data acquisition and transformation, store, and processing.
− Visualization and GIS Mapping: support data visualization in graphical user interfaces of the interactive applications, and maps generation and visualization.
2.2.3 EnviroGRIDS Applications
The enviroGRIDS Portal exposes to the user a set of tools and applications of which functionality is composed of the services provided by the below level. There are four types of interactive applications and tools available through the enviroGRIDS Portal:
1. Applications/ SWAT Scenarios Development Tools: the user may develop various scenarios for natural phenomena (e.g. through a graph based language) and use cases, and perform their execution over the Grid. The user may visualize the results and analyze statistical data.
2. Data Management Tools: data administrators and data provider may access, upload, update, and organize spatial data.
3. Decision Maker Tools: provide the user possibility to develop and execute various scenarios by different data series, and to analyze and make predictions on the phenomenon evolution. Graphical data visualization and mapping are available as well.
4. Citizen Tools: provide the citizen, as an Internet visitor, the execution of a given set of scenarios by limited set of data, and graphical visualization of the results.
1 AMGA Metadata Catalogue, http://amga.web.cern.ch/amga/index.html
3 EnviroGRIDS System Interoperability
3.1 Architectural Layers
System architecture (Figure 2) consists of the following architectural layers:
− BSC-OS Portal consists of a set of applications accessed by five types of users: administrator, data provider, Earth Science specialist, decision maker, and citizen. Scenarios describe the logics of applications and use cases of specific domains of interest such as hydrology, SWAT based computation, etc. A Web portal allows users to access specific resources through Web applications.
− Web infrastructure consists mainly on servers, geospatial services and data repositories that provide the basic functionality and data for the user’s applications.
− Grid infrastructure provides high power computation resources by Computing Elements (CE) and great capacity of storing spatial data by Storage Elements (SE).
3.2 Grid Oriented Architecture
The Grid based solution of the enviroGRIDS architecture offers a series of advantages by which resource management, resource sharing, data and processing distribution, security, are just the most important. The Grid is based on the concept of service. Grid architecture conforms generally the SOA architecture and particularly the OGSA framework.
Figure 2 EnviroGRIDS system conceptual architecture.
Web
Users
Geospatial Web Services
Grid
Computing Elements BSC-OS Portal
Storage Elements
3.3 Geospatial and Grid interoperability in enviroGRIDS
Interoperability is a great challenge in the enviroGRIDS system development2. Such a technology can significantly reduce problems associated with archiving, manipulating, analyzing, and utilizing large volumes of geospatial data at distributed locations. The OGC Web Services and the gLite middleware communicate and interact with each other in order to combine the complex specialized geospatial functionality with high power computation of the Grid.
3.3.1 Overview and Related Achievements
The integration of Geospatial infrastructure into the Grid environment has been a topic of interest in other research projects such as GDI-Grid3, GEO-Grid4, SEE/SAW-GEO5, and works but successful solutions have not been yet integrated into productive infrastructures, as Padberg and Kiehle highlighted in their paper6.
The G-OWS Working Group7 is dealing with the interoperability between Earth Science and Grid environment, and has as goals the specification of standards and the implementation of OGC Web services for the gLite middleware. Their integration takes into consideration the improvement of OGC Web Services with gLite security and VO management services. These activities have as final goal the design and development of gLite reference implementations of OWS for SDI community. Several important differences between OGC Web services - representing the SDI functionalities - and Grid services - representing the functionalities promoted by the Grids infrastructure - have been presented and analyzed by Padberg and Greve8, having the scope of solving the gap between the two platforms and achieving their interoperability. Most of the approaches in this research area consider the Globus based Grid environment that is also the case in the solution proposed by Chen et al. 9. In this paper the authors present a Grid- enabled catalog services model that combines the Grid Metadata Catalog Service (MCS) and the OGC CWS. They have developed an intelligent Grid Service Mediator (iGSM) to extend also the other OGC Web services to the Grid environment.
2 Rodila D., Gorgan D., Bacu V., The Interoperability between OGC Services and Grid Environment in enviroGRIDS project. MiDiS-2010, The First International Workshop on Middleware for Large Scale Distributed Systems, November 4-6, 2010, Fukuoka, Japan (under review)
3 GDI-Grid project, http://www.gdi-grid.de
4 GEO-Grid - The Global Earth Observation Grid, http://www.geogrid.org
5 SEE/SAW-GEO project, http://edina.ac.uk/projects/seesaw/
6 A. Padberg and C. Kiehle, Towards a Grid-Enabled SDI: Matching the Paradigms of OGC Web Services and Grid Computing, GSDI 11 World Conference, Rotterdam, Netherlands (2009)
7 G-OWS (gLite-OWS) Working Group, https://www.g-ows.org/pmwiki.php
8 A. Padberg and K. Greve, Gridification of OGC Web Services: Challenges and Potential. In: GIS.Science 14/2009, H. 3, S. 77-81 (2009)
9 A. Chen, L. Di, Y. Bai, Y. Wei, Grid-enabled Web Services for Geospatial Interoperability. Published by:
American Geophysical Union, Fall Meeting (2007)
3.3.2 OGC Web Services
The OGC specifications10 were developed to facilitate and standardize the sharing, interaction and visualization of geospatial data. The standards are mostly based upon the HTTP protocol and interact through messages over the Internet but new trends also consider including the usage of SOAP protocol and WSDL due to the large number of incompatibilities with standard Web services. The most representative geospatial services from OGC that are considered to be used also in the enviroGRIDS are detailed in the followings:
1. The Web Map Service (WMS) provides interface standards to retrieve maps of geospatial data.
The service gives access only to the graphical representation of the geospatial data and not to the actual data. A request to a WMS service should contain the geographic layers and the area upon which the layer are applied - the area to be processed. The response to such a request will contain one or more map images.
2. The Feature Service (WFS) provides a standardized way for accessing raw geographic data over the Web. It provides control over how to actually access the data: the data can be downloaded, analyzed, combined with other data from other Web services, etc and not just visualized as in the case of WMS. The WFS is normally specified to access vector datasets consisting of features of geospatial data, encoded in Geography Markup Language (GML).
3. The Web Coverage Service (WCS) provides standards for accessing raster datasets. (Raster data refers to an abstraction of the real world where spatial data is expressed as a matrix of cells or pixels, each such cell containing a value). This service only gives access to different type of data in Grid but does not provide transactional capabilities.
4. The Catalogue Service for the Web (CSW) provides interface standards to publish, discover search and query metadata about geospatial data, services or related resources.
5. The Web Processing Service (WPS) provides standards for processing and calculations of geospatial data. It can expose GIS functionalities in a standard way over the Internet, having support for SOAP, GET and POST communication.
To make the OGC Web services and the gLite middleware interoperable, they must be able to interact and communicate with each other, obtaining the benefits from both areas and providing specialized geospatial functionality with high computational power. The interoperability must be considered at different levels of interest:
• Data management level - services for providing the matching of metadata, data access, catalogs, etc. should be provided. It should also be considered the integration and compatibility of storage elements, computing elements, resource management services, etc
• Execution level - appropriate services have to handle task parallelization, jobs submission, process synchronization, fault recovery, result merging, etc.
• Security level - provide services for integrating the Grid based security and its capabilities into the OGC Web services which imply: certificate and credential management, single sign-on mechanism, user authentication, authorization, secure transport, etc.
3.3.3 Specific Case of Interoperability in EnviroGRIDS
The following examples of interoperability highlight the specific cases encountered in enviroGRIDS.
10 OGC WMS, Web Map Service, http://www.opengeospatial.org/standards/wms (wfs, wcs, wps)
3.3.3.1 Portal Level Interoperability
The functionalities of interest for the interoperability problem in the enviroGRIDS Portal are those associated with the specialist users. The specialist user is able to develop and execute scenarios using different data which will be visualized and analyzed by decision maker users to make predictions based on the results. The citizen user has restricted access in the Portal area and is able to visualize some sets of results in different formats. The application flows of users can also be seen in the Figure 10. At this level, the OGC Web services will be used mostly for visualization and thus we will focus more on the WMS service. The data backend for this type of service could be of different types such as a geospatial database, shape-files, results coming from other OGC Services (i.e. WCS, WFS), etc. The interoperability with the Grid of this type of service could be either for data retrieval or for layer processing over different regions of interest.
3.3.3.2 Execution Level Interoperability
At this level the OGC services will be mostly used for purpose of processing and retrieval of data in different formats. The most used OGC services will normally be WFC, WCS and WPS. The interoperability between OGC services and Grid environment will make available the property of parallel execution of complex and computational intensive operations. It will make use of the functionalities exposed mostly by WCS and WFS services in the Grid environment context. The complex operations will be split into simple requests and executed in parallel when possible.
3.3.3.3 Data Level Interoperability
The OGC Services along with Grid services used for data management would be the most appropriate and the data parallelism property will be the explored Grid functionality. One of the main concerns regarding the OGC Grid interoperability is related to the data distribution on the Grid:
• The Web existing databases do not support Grid integration (certificate authentication)
• There is a need for an additional layer above the Web databases to support Grid integration
• Using the Globus toolkit, this layer is realized by the OGSA-DAI11, which manages the transparent access to the Grid databases but in the gLite middleware, this layer does not exist and the necessary functionalities have to be developed (transparent management to the Grid databases, credential management for accessing the databases, etc.)
The functionalities offered by the Grid infrastructure can improve in most of the cases the execution time of a complex process by taking advantage of the parallelization techniques. Still, there are cases in which the overhead introduced by the Grid (job creation, submission and management, monitoring, result collection, etc.) is not compensated and the execution time will actually increase. These are also cases of simple processes that do not need high computational resources. In the interoperability context, these aspects have to be taken into consideration.
3.3.3.4 Web and Grid Service Classification
In our approach for solving the interoperability problem, a Mediator Component will play an important mediator role between OGC WS and the Grid infrastructure12, 13. This component will be responsible for
11 OGSA-DAI, The Open Grid Services Architecture Database Access and Integration, http://www.ogsadai.org.uk/
12 D. Gorgan, D. Rodila, V. Bacu, G. Giuliani, N. Ray, OGC and Grid Interoperability in enviroGRIDS Project. EGU2010 – European Geosciences Union General Assembly, 02 - 07 May 2010, Vienna, Austria
differentiating the cases in which the advantages brought by Grid technologies are visible from those in which the usage of Grid is not needed. This component will also be responsible for mapping the Grid security over the OGC Web services and for mediating their communication.
Functionalities for splitting the execution into sub-tasks and for merging the results must also be provided at the level of this component. Depending on the type of OGC Web service needed to be gridified, the Mediator Component will have different functionalities regarding the complexity analysis, the request division into sub-requests and the merge of the results. For example, for WMS these functionalities will be adapted to work for data layers, for WFS they will be designed to work with features and so on.
To gridify an OGC Web service, it has to be enhanced with this Mediator Component that will handle the integration with the Grid environment and will maintain the standard Web interface of the service. Now we will illustrate a specific use case and a specific data flow used at each level (Figure 3):
1. A user makes a request to a workflow that contains several OGC service calls to retrieve data stored both on the Grid and on the Web.
13 Y. Shu, J.F. Zhang, X. Zhou, A Grid-enabled Architecture for Geospatial Data Sharing, Proceedings of the APSCC 2006, IEEE Computer Society Press, pp. 369-375. (2006)
Figure 3 Gridification process based on Mediator Component.
2. The Mediator Component integrated in the gridified OGC service will parse the structure of the workflow and schedule the execution of each component OGC Service.
3. To execute a Grid-enhanced OGC Web service, the Mediator Component will analyze the complexity of the service request and will decide whether the request should be submitter to the Grid or not.
a. The requests that are not submitted to the Grid are executed as standard OGC requests, on the Web.
b. The requests that must be submitted to the Grid must be further analyzed to see which type of parallelism is applicable: task parallelism, data parallelism or both.
i. For task parallelism, the request functionality will be split in several sub-tasks and submitted to the Grid as jobs.
ii. For data parallelism, the requested data will be split to be retrieved and process by different worker nodes on the Grid.
c. The results of different jobs have to be collected and merge to obtain the final result of the initial request (functionality also accomplished by the Mediator Component).
Figure 4 Case1: Web Geospatial database, no Grid workers, Case2: Web Geospatial database, Grid workers.
Case 2 Case 1
Considering the options resulted after the Mediator Component has performed the complexity analysis, there are the following possible cases for an OGC Web service gridification14:
1. The service uses Web Geospatial database. The simple request (those for which the execution time does not exceed the overhead introduced by Grid) are executed directly on the Web server and the Grid environment is no longer used - see Case 1 in Figure 4.
2. The service uses Web Geospatial database but it is using the Grid environment for the execution.
The request is split into several sub-requests that are executed on individual workers. The workers connect to the Web Geospatial database and retrieve the necessary data - see Case 2 in Figure 4.
3. The service request is split into several jobs that will be executed on different workers (Computation Elements). The workers are each responsible for connecting to the SEs (Storage Elements) and obtaining the necessary data - see Case 3 in Figure 5.
4. The service connects directly to the Grid database, using Grid certificates and some special libraries. The connection is established using HTTPS and the data is copied using dedicated scripts - see Case 4 in Figure 5.
14 Rodila D., Gorgan D., Integration of Spatial Data Infrastructures with Grid Environment, SYNASC 2010 Symposium, September, 2010 (paper under review)
Figure 5 Case3: Grid Geospatial database, Grid workers, Case4: Grid Geospatial database, no Grid workers.
Case 3 Case 4
The general case that combines all of the four previous presented cases is illustrated in Figure 6. In this case the Grid is used both for data parallelism (the Grid Geospatial databases are accessible in parallel with the Web Geospatial databases) and computation parallelism (Grid workers are processing in parallel).
We have identified some general issues that arise for all cases:
• Making a Grid call from any application that runs outside the Grid, on the Web has to be enhanced with the necessary Grid security (map certificates credentials to the call). A possible solution for this case would be to intercept the call and to enhance it with appropriate certificates credentials and after that forward the call to the Grid.
• A call cannot always be identified with an authorized user so a call could not always have permission to run on the Grid. A possible solution would be to map a Grid certificate to the service itself but an OGC Web service can be used by several users so the request/call will no longer be identified.
• The gLite middleware is a file based system and does not have support for geospatial databases.
The solution would be to create a component for managing the Grid security and the parallel access of data, specific for gLite middleware.
Those are just a few of the most important issues that make the interoperability challenge so hard to achieve. Other problems will be taken as well into consideration and experimented during the enviroGRIDS project development in order to find out a practical and efficient solution.
Figure 6 General Case: Web and Grid Geospatial database, Grid workers.
4 EnviroGRIDS Portal Requirements
4.1 Portal Overview
The BSC-OS Portal exposes to the user a set of tools and applications supported by the functionality and components provided through the services. There are available personalized tools for different category of users: system administrator, data provider, Earth Science specialist, decision maker and citizen. Each category of user has particular accessing rights to data and system resources.
4.2 User Requirements
4.2.1 User Categories
Each user category may access different resources through particular accessing rights:
• Administrator – manages user accounts, creates new users, updates information on the Web Portal, manages data resources, sets up virtual organization, administrates virtual organization membership, etc.
• Data provider – enters datasets in repositories, fills in metadata, visualizes selected data, manages stored data, etc. The user belongs to enviroGRIDS VO and uses a certificate to get authentication into the system.
• Earth Science (ES) specialist – called in this document, simply by the term specialist, describes, develops and executes various scenarios for SWAT and other hydrological models. Develops use cases and generates reports for particular studies available to decision makers and citizens. The user belongs to enviroGRIDS VO and use a certificate to get authentication into the system.
• Decision maker – just visualizes and analyzes the reports of model execution for various scenarios. Visualizes statistics and reports that make possible decision and prediction activities.
The user does not need to authenticate into the Grid. The decision maker uses just the username and password to access the reports prepared for the private user community.
• Citizen – just visualizes data and reports already prepared by the ES specialists. The citizen accesses and visualizes just the public reports.
4.2.2 Functional User Requirements
This section analysis what each user category expects the portal has to do and provide. It describes use cases and scenarios through which the user accesses the system’s functionality.
4.2.2.1 Spatial Data Management
Management of distributed spatial data sources is one of the key requirements on current Geospatial portals15,16. Current principles of SDI systems are those which would facilitate spatial data management and publication. Main requirements are
15 Petr Horak, Sarka Horakova, Karel Charvat, Martin Vlk, Using Geohosting Principles for Publication of Users’ GMES Data within the EarthLookCZ Project, in INSPIRE, GMES and GEOSS Activities, Methods and Tools towards a Single Information Space in Europe for the Environment, ISBN 978-9934-8105-0-3, Riga 2009
• Ability of fast access to up-dated information
• Searching and classification of information
• Possibility of using information in other systems/subsystems and it publication as well
• Connect in situ measurement sensors
These functions related to searching and accessibility of spatial data are relatively common in different client applications, but the possibility of publication and sharing the spatial data by users is still problematic. User data could be used in numerous Web applications, but still are missing the tools supporting users to make geodata accessible in the form of Web services. For most users, it is problematic to make their data available to public access.
The system has to support solutions for spatial and non spatial data publication. Data Management has to include four main functionalities, usually grouped in four modules:
• Non spatial data and metadata publishing
• Spatial Data importing module
• Creating and publication of maps
• Creating projects for mobile editing of data in situ
The Data Management module has to make user data accessible on Internet, creating new map compositions by combining various data sources (users’ sources or external ones) and their publication on the Web, possibly also for other work with data saved on the server.
Data for creating a new map composition has to be integrated from the following sources:
• WMS server with known address
• WFS server with known address
• WMS or WFS server found by an integrated catalogue service
• File Repository of the portal
• Geodatabase of the portal
• Geodatabase connected through ODBC
Projects of new map compositions have to be defined by XML files. So far there is no standard form supporting such compositions storing. For instance Web Map context supports only composition from WMS. The data and their composition have to be accessible by several ways:
• Using different visualization clients for example Openlayers, Googlemaps, DHTML clients etc.
• By a desktop map browser
• Through OGC Web Map Service (WMS)
• Through OGC Web Feature Service (WFS)
16 Karel Charvat, Jan Jezek, Jachym Cepicky: Sensors and Analysis in Web Environment, in INSPIRE, GMES and GEOSS Activities, Methods and Tools towards a Single Information Space in Europe for the Environment, ISBN 978-9934-8105-0-3, Riga 2009
• Used by other analytical tools trough Web Processing Services (WPS)
For the integration of Sensor environment with other data it is necessary to support the next functionalities:
• Describe sensors in a standard way
• Standardize the access to observed data
• Standardize the process of what is commonly known as sensor planning
• Building a framework and encoding for measurements and observations
• Some sensors are already on the Web and they are able to return location information as well as observations and measurements
The integration has to be based on Open Geospatial Consortium standard framework Sensor Web Enablement (SWE)17,18
4.2.2.2 Hydrologic Model
The majority of the hydrological models used in the project should be well established tools and should normally have a conceptual structure. Those models have two main characteristics: their model structure is specified prior to their usage in modeling; and the model parameters do not have a physical meaning (they are not independently measurable) and have to be estimated through calibration against observed data. One of the main problems that arise when using the hydrological models is that different combinations of parameters or even different model structures yield similar results in terms of a defined performance measure, or objective function. This problem leads to difficulties in interpreting past behavior of the catchment system but also future predictions. The need for a model calibration becomes thus a major limitation, especially where no stream-flow measurements are available. To calibrate a selected model structure to a large number of catchments, statistical relationships between the parameters and the characteristics of the catchment such as size, land use or soil types must be established. These relationships can then be used to derive parameter values for a not-sized (not-calibrated) catchment. Manual calibration is time consuming and difficult in the presence of parameter dependence, especially for non-specialist users.
A user interface should be designed and developed to ease the calibration and the execution of a hydrological model, interface that will be accessible from the portal application. The main function of the user interface is therefore to provide a link between the input and the output of a calibration process and the hydrological model. As the interface will be integrated in the portal application, it should respect the same design structure and appearance as the rest of the portal.
4.2.2.2.1 Functional requirements for calibration and execution of hydrological models 1. Select the calibration program
The user should be able to choose the calibration program that he/she wants to apply to a hydrological model from a list of available programs. Examples of calibration programs are: SCUFI2, GLUE, ParaSol, MCMC, etc.
2. Select the hydrological model to be calibrated/executed
17 http://eo1.gsfc.nasa.gov/new/extended/sensorWeb/sensorWeb.html
18 http://www.opengeospatial.org/projects/groups/sensorweb
A list of available hydrological model should be available to the user. For each such model a short description should also be attached, serving as a hint for its representative functionalities.
3. Select the objective function
In the model calibration process, the parameters are adjusted until the system output and the model output show an acceptable level of agreement. This level of agreement is usually measured using an objective function, which the user should be able to select from a list of available functions.
4. Load the list of parameters for the chosen model
An agreed number of the hydrological model parameters should be available for configuration. Calibrated parameters are conditioned on the choice of objective function, the types and numbers of data points and the procedure used for calibration, among other factors. When the parameters are loaded in the list, they should contain a default value and the user should be able to change it, between feasible value ranges.
Those value ranges are predefined for each parameter. For better user interactivity and better performance, a comprehensive description of all the values and their significance should also be available for the user.
5. Specify the number of simulations
Optionally, the user should be able to choose the number of simulations to perform when calibrating the chosen model.
6. Calibrate intermediate and final variables
Traditional approach to calibration distributed-hydrologic models implies comparison of observed and simulated runoff, method which is not sufficient. Intermediate variables computed by the hydrologic model could be characterized by parameter values that do not replicate those hydrological processes in the physical system. The interface for calibration should provide facilities to calibrate intermediate and final variables produced by the selected model, giving the user higher confidence in the model output by assuring that the chosen intermediate and final states of the model are simulated consistently with
“observed” values.
7. Execute model calibration
One of the main functionality of the user interface provided in the portal is the actual calibration of the selected hydrological model. After the user configured correctly all the calibration inputs (selects the calibration program, selects the hydrological model, selects the objective function, assigns appropriate values to the model parameters), the user should be able to start the calibration process. A window with the calibration process progress should provide information regarding the status of the process, i.e. the monitoring functionality should also be provided.
8. Extract required outputs (corresponding to measured data) from the model output files.
The user interface should provide functionality for visualizing the output files obtained from the calibration process in different formats, necessary for different user scenarios. The user should be able to select the outputs he/she wants to visualize (corresponding to measured data). Functionalities for exporting the output files in different formats should also be provided.
9. Save iteration history
Functionalities for saving the history of the iterations are useful for failure cases. Using this functionality, the user is able to detect which iteration crashed and to restart only that iteration, not the entire process.
Considering the long periods of time the calibration process usually last, this functionality can improve considerably the execution time for the processes that failed in the first case.
10. Restart an unfinished iteration
This situation appears in the cases when iteration stopped before completing all the simulations. In order to accomplish this functionality, the following actions should be taken:
- Determine where the simulation runs stopped
- Check the last simulation to identify the place of the failed iteration - Restart the iteration
11. Split the iteration in several runs
In the case iteration can be split into several independent runs, the interface should provide the user the functionality to split and run in parallel the iteration.
12. Validate the calibration
Appropriate methods for validating the calibration results should be provided. In case the calibration process does not pass the validation phase, the user should have the possibility to rerun the calibration. This implies that the calibration configuration chosen by the user in the initialization phase should be saved as well in a history file. The calibration rerun is thus executed without any other configuration intervention from the user point of view,
13. Model execution
After the hydrological model is calibrated and the calibration has passed the validation phase, the user should be able to run the calibrated hydrological model.
14. Visualize execution status
The interface should provide execution monitoring. The user should be able to visualize the status of the model execution with short description of the current executing step.
15. Visualize execution output
The interface should provide functionalities (or links to external tools) for visualizing the output of the execution. The user should be able to select from different visualization formats such as maps, graphs, charts, etc. as well as to export the results in different formats for later use.
16. Compare execution results
The interface should provide functionalities for comparing the results from different execution processes.
The user should be able to choose previous saved executions results and compare them, or even compare the current execution results with previous ones. The differences of the results should be highlighted. Also a short description of the output parameters that are different in the comparison should be useful in the user interpretation of the results.
17. Display execution time
The interface should provide the display of the execution time at the end of the execution as well as during the execution (i.e. in the monitoring window).
Other functionalities that should be provided by the graphical user interface:
• Facilitate the coupling between external system analysis tools and the hydrological model
• Allows any required changes to be made with a minimum of effort
• Graphical displays should consist of those for user orientation and for data presentation (a multiple graphical display windows environment)
• Examples of data presentation are model inputs and simulation results presented in graphical plot displays
4.2.2.3 Satellite Image Processing
This section describes the functionalities the user expects from the system to perform on satellite data and images. It describes the operations the user accomplishes on satellite images, through the portal application.
The satellite images could reveal information on the earth surface, weather, geographic areas, pollution, and natural phenomena. The processing of satellite data requires high computation resources and massive data storage capacity. The main processing consists of imagery classification that is actually a search of information through various combinations of multispectral bands. The data exploration and interpretation are a multivariable process considering satellite image types (e.g. MODIS, Landsat, and QuickBird), geographic area particularities, soil composition, vegetation cover, and context (e.g. clouds, snow, and season). All these specific and variable conditions require flexible tools and friendly user interfaces to support an optimal research for the appropriate solutions.
4.2.2.3.1 User requirements
1. Satellite image visualization tool – view, transformations (zoom, rotate)
The user should have the possibility to search for a particular satellite image. He/she has to select on a map the area of interest, the type of satellite image and the acquisition time interval. The system will return from the database a list of images, which is matching with the selected requirements. The user may interact with the images by transformations (zoom in, zoom out, rotate, save, and scale), or he/she may visualize the metadata that is attached to the image. The metadata describes geographical location, satellite image type, data content type, etc.
2. Flexible description of complex processes (workflows) – workflow editor, workflow instantiation tool, workflow management tool
Complex algorithms used to process satellite images could be defined as workflows. The workflow is an acyclic graph describing a complex process. The graph nodes could be resources (input data), operators (basic functionality), services (complex functionality), and sub-graphs (already defined workflows). The workflow editor should allow the definition of complex workflows, and support different interaction techniques.
3. Execution of complex processes (workflows)
Satellite image processing algorithms described as workflows are instantiated to particular images. Only the instantiated workflows can be executed. The user should be able to search for a particular workflow, instantiate it to a satellite image and then monitor the workflow execution. The monitoring component will show information on the status of the execution, the overall workflow execution completion, intermediate output visualization, etc.
4. Satellite image processing basic algorithms
Some basic algorithms, such as vegetation index computation or some simple water detection algorithms will be available to be used. For algorithms that are more complex the user has the possibility to develop complex workflows.
5. Output visualization (pseudo-coloring tool)
The pseudo-coloring tool allows the user to define the classification of the processing output (that mean the selection of the interval that define a particular class).
6. Save the satellite images in different formats
An important feature of the application is the ability to use different satellite image types that come in different formats (e.g. GeoTIFF, JPEG, GIF, and PNG). The system should be able to convert any one format to another one.
7. Crop the satellite image to an area of interest
The user might be interested in processing only a subset of the satellite image data. Another important feature is the crop function that will allow extracting a sub-area of the satellite image.
8. Display information about the image
The user may visualize the metadata attached to the image. Using a “color picker” tool the user may visualize information on the pixels from the image (numerical value, color code of the pixel, other information).
4.2.2.3.2 Use Case Scenarios
Let us exemplify a few simple user scenarios in order to highlight the user functional requirements in the context of graphical user interface.
a. Satellite image visualization
1. The users select the area of interest by direct manipulation techniques 2. The system returns a list of images matching the user requirements
3. The user may visualize the satellite image spectral bands, statistical information, and metadata that is attached to the image
4. The user may combine the satellite image spectral bands to create pseudo-colored images b. Flexible description of complex processes (workflows)
1. The user visualizes the available operators, services, subgraphs that could be used to develop complex processes
2. Using direct manipulation techniques (e.g. selection, drag and drop) the user creates a hyper-graph as a composition of basic operators, services and subgraphs
3. The user instantiates the input nodes of the workflow to bandwidths of particular satellite images c. Execution of complex processes (workflows)
1. The user visualizes the processes already instantiated (i.e. iPDG) 2. The user selects the iPDG that will be executed
3. The system executes the selected iPDG and monitors the execution
4. The user may visualize information of execution progress and intermediate results 5. The user may visualize and download the output of the execution
d. Output visualization
1. The user selects the output to be pseudo-colored
2. The user may split the numerical interval contained in the output image, specifies a fill color for this interval and attaches a label to each interval
3. The user visualizes and downloads the resulted image
4.2.2.4 Data Visualization and Report Generation
The Earth Science specialists access the BSC-OS Portal in order to visualize data and develop reports for decision makers and citizens. The user should be able to understand the spatial data concepts and to be skilled on using the script language to describe the report content and format.
Figure 7 GUI prototype: (a) User describes the report content by editing the script code; (b) The information is shown on the portal using widgets like maps, charts or tables. They can be
organized using HTML and JavaScript.
(b)
(a)
The data visualization and report generation application should provide the following main features for the user:
• User authentication to allow the data access, visualization and inclusion into reports
• Search, discover and access the following raw data types: text files, vector data from geospatial database, images, and hydrological output data
Figure 8 GUI prototype: Decision makers and Citizens may visualize the reports by fixed and interactive manner.
