Updated: 27 Sep 2024
EVDC supports data in .hdf and .nc file formats and uses the GEOMS metadata standard.
How to create HDF data files:
The standard HDF libraries to be downloaded from NSCA contains binary programs suitable for creating HDF files according to the HDF4/5 standard. These binaries do, however, not automatically format the files according to the GEOMS standard used by EVDC.
To make sure your files will be formatted correctly for EVDC, we recommend you to use the specially developed GEOMS software tool developed by NASA. A description of this tool and access to download it is provided below. A detailed description of what data and metadata that need to be included in a GEOMS compatible file is given in the GEOMS document in the "browse metadata" section under the "Documentation" menu.
File formatting tool
The preferred way to create an hdf data file is by using the idlcr8hdf program , an IDL software that is delivered both as a .pro-file for users with an IDL liense, and as a .sav-file for those without a license. The program takes two ASCII files as input, one data file and one metadata file, and formats an GEOMS compatible file (HDF4/5 or netCDF).
A short "how-to" guide for getting started with file creation is found below.
The program can be downloaded here: idlcr8hdf.pro (For users with IDL license) and idlcr8hdf.sav (For users without IDL license) idlcr8hdf requires, as input, a Table Attribute Value (TAV) file. Download it from here: tableattrvalue_04R088.dat
The complete guide is found at idlcr8hdf-v4.0_Readme.pdf It is advisable to study the GEOMS manual and to look at example files to get an idea on how to use the program and how to format your files. Please also contact NILU to get assistance with any file formatting issues.
idlcr8hdf is the core program for other GEOMS supported programs, including geoms_harmon, geoms_qa, and cds_convert. These programs, respectively, convert HDF files, created using obsolete standards, to current GEOMS standards; perform QA on GEOMS compliant HDF files; and convert measurements written in formats supported by other datacenters (e.g. WOUDC, NASA/Ames (NDACC), GlobWave) to GEOMS compliant HDF.
1) Download the latest Table Atttribute File tableattrvalue_04R088.dat and the idlcr8hdf suite from above.
2) Start IDL
3) type .r idlcr8hdf, followed by idlcr8hdf. A terminal window will appear. Select output file type (HDF4 HDF5 or NetCDF). Once the file option is selected, the program will prompt for input file and directory selection by Dialog Boxes. Inputs are: The Metadata Template file, the Data file(s), the TAV file, and the Output Directory.
4) Metadata: The purpose of the metadata is to describe the users actual scientific measurements, label = value / PI_NAME = Fjaeraa; Ann Mari. Most of the attribute values do not change from measurement to measurement, which makes it possible to use a single metadata template as input with multiple data files.
5) Data: The data file is a plain ASCII file.
The templates are found at Data Formatting Templates
Quality Assurance: file format checker
Before you upload data to EVDC, you can check your file format online with the AVDC/EVDC GEOMS QA checker. The data files will checked against the newest GEOMS QA, vocabulary (TAV), and templates.
If your files successfully pass the QA checker, you can use the "Upload Data" page http://evdc.esa.int/upload/ to upload your data to the EVDC archive.
Link to tool: File format checker
Note: the file format checker is hosted and maintained at the AVDC (Aura Validation Data Centre) but shared with, and used by, EVDC in the same version.
How to create HDF data files with python3:
To work locally with the code:
Clone the git repository in a Linux/Unix environment
git clone https://git.nilu.no/geoms/geoms_qa
cd geoms_qa
git checkout dev
Running:
python3 ./pycr8hdf.py -h
will print a help message:
usage: pycr8hdf.py [-h] [-m META_FILE] [-d DATA_FILE] [-t TAV_FILE] [-o OUT_DIR] [-f FILE_FORMAT] [-l]
Create a GEOMS file in hdf5 format from .meta and .data files
optional arguments:
-h, --help show this help message and exit
-m META_FILE, --meta_file META_FILE
GEOMS meta data file (default: None)
-d DATA_FILE, --data_file DATA_FILE
GEOMS data file (default: None)
-t TAV_FILE, --tav_file TAV_FILE
GEOMS TAV file (default: ./test_data/tableattrvalue_04R072.dat)
-o OUT_DIR, --outdir OUT_DIR
Directory to write output. Same as input directory if not given. (default: None)
-f FILE_FORMAT, --format FILE_FORMAT
File output format ('h5', 'nc4') (default: h5)
-l, --log_file If true, write output to log file pycr8hdf.log in output directory (default: False)
Example of run:
./pycr8hdf.py -m test_data/balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.data -d test_data/balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.data -t test_data/tableattrvalue_04R072.dat -o test_data/ -f h5