EM Data Format Standards SIG

Submitted by schultz on Fri, 2009-09-25 00:00

Working to promote data interchange, archive and distribution standards...

Questions about data format standards

Submitted by schultz on Mon, 2009-12-21 16:09.

 

Some questions to comment on: For instance, for time series how would you like to see metadata defined? We could post a copy of the LabView interface we developed for the NIMS MT system, to show you the metadata entry page to see if the information in that is adequate, or if additional metadata is needed - for instance we don't include any permitting information - and that might be interesting to add if it was scrubbed adequately - say for instance - just to include something about the landowner & contact details? Also how do we include instrument response information (say poles and zeros for instance)? What if some parameter (for instance, gain) changes in mid-run? What if you have some sort of mid-run calibration pulse or electrode resistivity measurement - shold that be included in the data stream?

For response functions - is EDI format adequate? What are its shortcomings? For instance can we have a better description of robustly estimated complex Fourier (cross)powers? Is there an adequate description of the full covariance matrix? Suggestions welcome for drafting a proposed EEDI (extended EDI) format...

 

Data Format Standard

Submitted by scottu on Tue, 2009-12-22 18:34.

With the apparent funding of the "National Geoelectromagnetic Facility", it seems to me that a discussion of a format for the numerous required data structures is useful.  We have clearly "outgrown" EDI format, and these instruments will be required to do things other than MT.  It has been strongly recommended that we not head down the XML route, given the overhead.

 

 There are several formats that we could use.  NETCDF, which has numerous readers, is one.  The SEG has declared that SEG-D is the format for archive of EM timeseries data and has added support to allow frequency domain data to be stored in the format.  There are certianly many others.  As a beginning, it might be useful to look at the applications that support NETCDF:  http://www.unidata.ucar.edu/software/netcdf/software.html

 

We will need to support both natural source and controlled source data.  It will need to support both frequency domain and time domain surveys.  It will need to support large numbers of instruments.  It should be easy to write, easy to read. 

 

I think that for an open source community to be successful, file standards are necessary precondition.  So I hope that we can have a meaningful dialog here that will allow us to put together a useful format that these new instruments can use as a native format.