LOFAR and HDF5: toward a new radio data standard

For decades now, scientific data volumes have experienced relentless, exponential growth. As a result, legacy astronomical data formats are straining under a burden not conceived of when these formats were first introduced. With future astronomical projects ensuring this trend, ASTRON and the LOFAR project are exploring the use of the Hierarchical Data Format version 5 (HDF5) for LOFAR radio data encapsulation. Most of LOFAR's standard data products will be stored natively using the HDF5 format. In addition, HDF5 analogues for traditional radio data structures such as visibility data and spectral image cubes are also being developed. The HDF5 libraries allow for the construction of potentially distributed, entirely unbound files. The nature of the HDF5 format further provides the ability to custom design a data encapsulation format. The LOFAR project has designed several data formats that will accommodate all LOFAR data products, examples of which are presented in this paper. With proper development and support, it is hoped that these data formats will be adopted by other astronomical projects as they, too, attempt to grapple with a future filled with mountains of data.