The Grid DataBlade processes queries on the database server, thereby minimizing the amount of network input/output and client-side CPU time required. In addition, we use an internal tiling scheme and affine transformations to provide an effective and very efficient environment for housing and accessing gridded data.
Why use a database to store gridded data?
Until recently, gridded data was usually stored in files and users ran applications on these files to generate required data products. However, there is now increasing recognition that it is often advantageous to store gridded data in a modern database and allow users to extract data products by running queries against the database. Three advantages of this approach are:
- Data access through SQL. By storing gridded data together with its metadata and other data, applications can use a simple SQL interface to perform complex queries based on any of these data items, including dynamically-derived properties of the gridded data.
- Client-server issues. In a traditional database or file-based environment, custom functionality is implemented purely on the client side, resulting in "fat clients". Modern DBMSs, in addition to allowing user-defined data types (UDTs), also allow user-defined routines (UDRs) to be defined in languages such as C or Java. This provides much greater control over where the grid processing functionality resides. With the Grid DataBlade, either an entire grid can be extracted and then manipulated on the client, or just that portion of the grid that is actually required can be retrieved.
- Synchronized concurrent access to data. Multiple users of the database can safely query the same data simultaneously. With a file-based approach, there is a danger that the update activity of one application might result in another application seeing inconsistent data.
Overall, the advantages of gridded databases (data integrity and consistency, as well as independent and effective access to data by diverse users of multiple applications) are very appealing, and substantial benefits can indeed be achieved through the use of modern object-relational DBMS technology (including BLOBs, UDRs, and UDTs).What features does the Grid DataBlade have?
The Grid DataBlade has numerous features to provide powerful and flexible processing and convenience of use. Our Grid DataBlade technology:
- is designed to handle 4D grids as a standard; grids of lesser dimensionality are handled by taking one or more of the grid dimensions to be 1
- stores grids in BLOBs using a tiling scheme, with user control over the tile size; this allows very efficient generation of data products that involve only a small portion of the data in the overall grid (e.g., extracting planar oblique slices from volumetric grids, or retrieving 1D probes ("sticks") from 3D or 4D grids)
- can store the data in, and convert it between, the many planar mapping projections and datum shifts available in the PROJ.4 library
- can handle non-uniformly spaced grids in any or all of the grid dimensions
- can handle the presence of multiple vector and/or scalar values at each grid point
- provides interpolation options using N-linear or nearest-neighbor schemes, as well as allowing for user-defined interpolation methods
- provides for convenient loading and extraction of grid files to/from the database, either as raw binary data or via a specific form of the commonly used NetCDF format, termed Grid Import-Export Format (GIEF)
- comes with client programs for loading gridded data in a variety of formats (such as GRIB), and for visualizing data extracted from the database on the fly
- provides application programming interfaces for C, Java, and SQL
- can provide more than 50-fold increases in speed of data product generation compared to more conventional approaches
- is supplied with full user/programmer documentation
The Grid DataBlade was originally developed for use with IBM Informix. Subsequently we have ported it for use with PostgreSQL (referred to as the BCS Grid Extension), and also with Oracle (referred to as the BCS Grid Cartridge). All three versions have identical functionality, and it can be ported to other databases as required.Who is currently using the Grid DataBlade?
The Grid DataBlade is currently in use by government, defense, and private sector customers.Can the Grid DataBlade be used with satellite data?
The Grid DataBlade has a special feature that allows the input and storage of unprojected satellite swath data as a grid, and provides for the data to then be extracted in a user-specified projection. This feature stores the original scan line data along with the satellite position and orientation metadata for each line, and so avoids the resampling that would be required (and the loss of accuracy that would result) if the data was stored in a projected format.
What standards will the Grid DataBlade be compliant with?
We are familiar with existing and emerging standards, such as those from the Open Geospatial Consortium (OGC). For example, the Web Coverage Service (WCS) is a standard web service for exchanging geospatial data. WCS provides available data together with its detailed descriptions, allows complex queries against this data, and returns data with its original semantics. We are currently working towards providing a web coverage server that complies with this OGC WCS standard.