EVDC supports data in .hdf and .nc file formats and uses the GEOMS metadata standard.

How to create HDF data files:

The standard HDF libraries to be downloaded from NSCA contains binary programs suitable for creating HDF files according to the HDF4/5 standard. These binaries do, however, not automatically format the files according to the GEOMS standard used by EVDC.

To make sure your files will be formatted correctly for EVDC, we recommend you to use the specially developed GEOMS software tool developed by NASA. A description of this tool and access to download it is provided below. A detailed description of what data and metadata that need to be included in a GEOMS compatible file is given in the GEOMS document in the "browse metadata" section under the "Documentation" menu.

File formatting tool

The preferred way to create an hdf data file is by using the idlcr8hdf program , an IDL software that is delivered both as a .pro-file for users with an IDL liense, and as a .sav-file for those without a license. The program takes two ASCII files as input, one data file and one metadata file, and formats an GEOMS compatible file (HDF4/5 or netCDF).

A short "how-to" guide for getting started with file creation is found below.

The program can be downloaded here: idlcr8hdf.pro (For users with IDL license) and idlcr8hdf.sav (For users without IDL license) idlcr8hdf requires, as input, a Table Attribute Value (TAV) file. Download it from here: tableattrvalue_04R092.dat

The complete guide is found at idlcr8hdf-v4.0_Readme.pdf It is advisable to study the GEOMS manual and to look at example files to get an idea on how to use the program and how to format your files. Please also contact NILU to get assistance with any file formatting issues.

idlcr8hdf is the core program for other GEOMS supported programs, including geoms_harmon, geoms_qa, and cds_convert. These programs, respectively, convert HDF files, created using obsolete standards, to current GEOMS standards; perform QA on GEOMS compliant HDF files; and convert measurements written in formats supported by other datacenters (e.g. WOUDC, NASA/Ames (NDACC), GlobWave) to GEOMS compliant HDF.

1) Download the latest Table Atttribute File tableattrvalue_04R092.dat and the idlcr8hdf suite from above.

2) Start IDL

3) type .r idlcr8hdf, followed by idlcr8hdf. A terminal window will appear. Select output file type (HDF4 HDF5 or NetCDF). Once the file option is selected, the program will prompt for input file and directory selection by Dialog Boxes. Inputs are: The Metadata Template file, the Data file(s), the TAV file, and the Output Directory.

4) Metadata: The purpose of the metadata is to describe the users actual scientific measurements, label = value / PI_NAME = Fjaeraa; Ann Mari. Most of the attribute values do not change from measurement to measurement, which makes it possible to use a single metadata template as input with multiple data files.

5) Data: The data file is a plain ASCII file.

The templates are found at Data Formatting Templates

Quality Assurance: file format checker

Before you upload data to EVDC, you can check your file format online with the AVDC/EVDC GEOMS QA checker. The data files will checked against the newest GEOMS QA, vocabulary (TAV), and templates.

If your files successfully pass the QA checker, you can use the "Upload Data" page http://evdc.esa.int/upload/ to upload your data to the EVDC archive.

Link to tool: File format checker

Note: the file format checker is hosted and maintained at the AVDC (Aura Validation Data Centre) but shared with, and used by, EVDC in the same version.

How to create HDF data files with python3:

To work locally with the code:

Clone the git repository in a Linux/Unix environment

git clone https://git.nilu.no/geoms/geoms_qa

cd geoms_qa

git checkout dev

Running:

python3 ./pycr8hdf.py -h

will print a help message:

usage: pycr8hdf.py [-h] [-m META_FILE] [-d DATA_FILE] [-t TAV_FILE] [-o OUT_DIR] [-f FILE_FORMAT] [-l]

Create a GEOMS file in hdf5 format from .meta and .data files

optional arguments:

-h, --help show this help message and exit

-m META_FILE, --meta_file META_FILE

GEOMS meta data file (default: None)

-d DATA_FILE, --data_file DATA_FILE

GEOMS data file (default: None)

-t TAV_FILE, --tav_file TAV_FILE

GEOMS TAV file (default: ./test_data/tableattrvalue_04R072.dat)

-o OUT_DIR, --outdir OUT_DIR

Directory to write output. Same as input directory if not given. (default: None)

-f FILE_FORMAT, --format FILE_FORMAT

File output format ('h5', 'nc4') (default: h5)

-l, --log_file If true, write output to log file pycr8hdf.log in output directory (default: False)

Example of run:

./pycr8hdf.py -m test_data/balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.data -d test_data/balloon_sonde.o3_noaa.esrl000_rd_south.pole_20231207t211354z_20231207t230247z_004.data -t test_data/tableattrvalue_04R072.dat -o test_data/ -f h5