Data access, interoperability, query processing, trans- trans-actionstrans-actions

As explained earlier, data sources in Grids are distributed among several sites and heterogeneous in several aspects. This heterogeneity appears in:

• storage systems: Each site of a Grid would hold its own storage system:

The underlying storage systems can be any means to store data on permanent devices. The most encountered storage systems are raw file systems, most of the time based on some Unix-like file system (AFS, EFS). Another observed option in the High Performance Storage System [Teaff et al., 1995 ] which gives optimized access to files mainly in cluster when the file size is large. Databases are also seen in Grids, from open source implementation like MySQL or PostGres to commercial products like DB2 or Oracle-11G, the latter being advertised as a grid database, and mainly offers transparent access to different storage sites.

• data organization: Data access is also depending on the organization of the data, in terms of schemas, internal structure. Files structured or semi-structured will provide for instance means to extract more easily metadata facilitating their management.

• technologies: The tools to access the data depend on the storage system type and organization of the data. Local or Global API (Posix-like), specific protocols (like RFIO commands for HPSS), remote procedure

110 Fundamentals of Grid Computing

calls (RPC), HTTP, web/grid services are all complementary means to actually get data pieces.

To handle this heterogeneity and distribution, a key concept in Grid com-puting is transparency. The idea is to provide a unique way to access the data, regardless of the actual storage systems, data organization or under-lying access technologies. Data virtualization allows to access data virtually through a specific interface, mapped then to the actual data access patterns.

This requires obviously appropriate naming and indexing of the data pieces (see previous section).

Most of the implementations rely on a limited set of operations representing the standard/common operations for all kinds of storage, that can be mapped onto these.

Concerning file access, the idea is to develop Posix-like functions (open, close, read, write, seek, stat,. . . ) over any actual storage system to hide their heterogeneity. Grid wide access operations can be performed over distributed resources, with tools like object-based file system (XtreemFS [Hupfeld et al., 2008 ], VisageFS [Thizbolt et al., 2007 ], GridFS [Bhardwaj and Sinha, 2005 ] or even NFSv4), but due to consistency checks for ACID properties and latency delays, these solutions give low performances with scenarios when deployed at large scale on a grid.

Other problems include versioning and synchronization of replicas other than distributed environment. Replication will be detailed in Section 4.5.4.

In some cases, the data pieces can be split between different grid sites (like in XtreemFS), each of them holding a part of the data pieces. While being very efficient for increasing performances (for instance using parallel transfers, see Section 4.5.3), it creates more complexity for checksum computations, auditing, pinning and backups.

As we already mentioned in a previous section, some operations are held on a set of data pieces rather than on single data pieces. Bulk operations that are found (like in SRB) deal with registration, loading, deletion. The Hierarchical Data Format (HDFv5) gives a standard characterization of groups of files in a container.

The Data Access and Integration (OGSA-DAI) initiative [Antonioletti et al., 2005 ] in the OGSA architecture aims at providing an uniform access and integration to several kind of data sources, from relational databases, web databases, XML or even semantic enriched RDF files. The idea is to provide some mediator/wrapper between the actual data sources and the grid (thus the clients) that hides the heterogeneity of the access methods. Then the access itself is done via API calls, SQL queries or web browser (like OGSA-WebDB [Pahlevi and Kojima, 2004 ]). At the moment OGSA-DAI supports MySQL, IBM DB2, Microsoft SQL Server, Oracle, PostgreSQL and the eXist XML database.

Query processing is the process to actually access the pieces of data. As data are distributed, possibly replicated, there is the need to have some specific

Data management in grids 111 ways to perform the query processing. Intra-query parallelism in distributed query processing (DQP) deals with the access to the data when the data related to a single query are distributed among several sources. Inter-query parallelism appears when concurrent accesses to the same query service are provided. In order to provide an efficient query processing engine, both sides must be investigated and the query scheduler optimizes the query planning thanks to environment information. For instance, critical operations like table joins in databases must be done in the right order and on the right data sources or replicas. This optimization problem is not due to the distributed nature of the grid and appears also in traditional databases, but the problem is additionally stressed with the dynamic of the system and potential high latency of the network. The mainframe is to search for the potential data in the system, move the data to a processing engine and then perform an operation, with potential optimization in the order of the operation or the selected available operation service [Lynden et al., 2009 ]. Solutions based on mobile agents [Morvan and Hameurlain, 2009 ] aim at moving the query to the data rather than moving the data to the query engine.

Transactions are another important part in traditional databases. The idea is to group a set of operations on the data sources in an atomic operation:

If one of the operation fails, the whole transaction is cancelled. In Grids, mainly two problems cause these transactions operations to be difficult: per-formance and checkpointing. In order to cancel a transaction not being able to finish, there is the need to make some checkpointing of the data sources before individual operations so that they can be restored in case of failure.

Without a global coordinator of the transaction process, with network high latency and several thousands of clients, it becomes difficult to ensure the correctness or the freshness of the data, thus leading to inconsistency in the data. This problem is increased when data are replicated, since in that case all replicas must be in the same state at the end of the transaction. Works like GridTP [Qi et al., 2004 ], Turker [T¨urker et al., 2005 ], HVEM transac-tion [Jung and Yeom, 2008 ] show that some partial solutransac-tions exist, relaxing some constraints on consistency or using some existing hierarchical structure in the data set. See [Wang et al., 2008 ] for an historic survey of transaction management from flat to grid transactions.

Access is not enough: One should not forget how to transfer, how to con-trol (security), how to process: GridFS for instance combines the benefits of different approaches (Remote File Access, File Staging) and offers special functionalities for Grid Computing (estimated transfer cost and file copy to local system).

4.5.3 Transport

While data transport is not directly related to data management itself, it is nevertheless useful to spend some time to describe briefly its links with data management. Besides the classical data access through sockets, RPC or RMI

112 Fundamentals of Grid Computing

calls, web services invocation, FTP procedures, grids have seen the emerging of dedicated services to transport data between sites in the grid or to the final clients. The idea behind these services is to provide some additional possibilities in terms of performance or security thanks to the grid structure and middleware.

As an example, parallel streams can be used for transferring efficiently one piece of data. Indeed, as data are replicated in the system, it is obvious that opening different data streams between machines will decrease the trans-fer time. Experiences show that even when the data piece is on one single node, the time to deliver is shortened with parallel streams. Another exam-ple concerns the compression and the encryption of data during the transfer.

GridFTP [Allcock et al., 2005 ] (at operating systems low level) or the Globus Reliable File Transfer service and gLite File Transfer Service (at higher ab-straction level of service oriented grid architecture) [Globus, 2009b ], [EGEE, 2009 ] are typical existing efficient possibilities of transferring data in the Grids. In RLS, the reliability of the transfer is resistant to failures of the network or sites. If one site can not complete a transfer, a replica is selected and the transfer continues.