Habermann, Ted (The HDF Group)
Jelenak, Aleksandar (The HDF Group)
The NetCDF SWG has laid an important foundation for understanding how data models for Earth Science data and related encodings should be standardized in the context of OGC. The CF-netCDF Core and Extensions Primer (OGC 10-091r3) provides an overview of the many possible components of the CF-netCDF suite and shows how those components fit together into a coherent whole.
The netCDF SWG has focused on netCDF data models, encodings, and conventions that are implemented in Version 3 of the netCDF software libraries and related tools (OGC 10-092r3). During 2008, Version 4 of the netCDF libraries introduced a new binary encoding using the HDF5 format. This encoding is included in OGC 10-091r3 Figure 1 as a binary encoding for the netCDF-3 Classic Data Model. OGC 11-165r2 identifies work related to netCDF-4 in Section viii Future Work: "This standard is written for netCDF version 3, but netCDF-4 is now being released. Currently the plan is to submit a separate extension standard for CF-netCDF version 4 as it becomes more heavily used in the community." Since the publication of that document, netCDF-4 has become the de facto version of netCDF for many new projects. This suggests that the time has come for standardization of netCDF-4.
The netCDF-4 Enhanced Data Model is an extension of the HDF5 data model and the binary encoding is HDF5. The HDF5 ecosystem includes many elements that go significantly beyond the binary encoding for netCDF shown in OGC 10-091r3. That ecosystem includes a general data container model (that netCDF specializes), libraries in multiple languages for writing and reading data in HDF5 files, tools for accessing, analyzing, and visualizing those data, and tools for serving those data using web services. Also, OGC members are starting to use HDF5 as a standard format for more complex data types. In particular, HDF5 has been proposed as at least two standards for point cloud data in the OGC Point Cloud DWG, has been demonstrated as a GDAL compliant format for images (KEA), and is also used as an open standard for other geographic data, e.g. the Bathymetric Attributed Grid (BAG) format that is the basis for IHO standards. Given this situation, it is appropriate to consider the standardization of other elements of the HDF ecosystem in addition to the binary encoding of the netCDF enhanced data model shown in Figure 1 of OGC 10-091r3.
The purpose of this Standards Working Group is to:
· Standardize the HDF5 data model, its binary encodings, related extensions, and other appropriate items, and
· Support other OGC working groups (SWG and DWG) in understanding the HDF5 data model and applying it in their domains of interest (e.g. point clouds, global grids, …).