TOYODA Eizi's blog: WCMP

Showing posts with label WCMP. Show all posts

2014-06-28

"jmd" metadata tools updated for WCMP1.3

A few years ago I made an experimental generator tool of WIS Discovery Metadata, called "jmd". It is found at http://www.gisc.kishou.go.jp/xsd/jmd0.1/.

During the work of writing guideline for WCMP 1.3, I need to create many metadata records as example. It was painful to write raw XML and correct it by modify-validate-fix cycle, so I remembered the "jmd" tool. So I've updated the tool to generate modern style of ISO19139 according to the new WMO profile.

2014-06-25

default CRS for meteorological data

There is an argument (again) that all coordinates in CF should have CRS specification.
http://mailman.cgd.ucar.edu/pipermail/cf-metadata/2014/057478.html

My thought:

- I agree Jim says default CRS is needed for backward compatibility.

- The default should be WGS84 http://epsg.io/4326

- For NWP model data, even if the modeller says the coordinates are spherical, that means only the equations are spherical. I've never seen surface elevation and observations are injected *after* coordinate conversion.

- For observational data, I hope the data source is migrated well to WGS84. Archived data is an issue, but that can't be an excuse to continue to create the issue.

Similar argument should exist for vertical CRS. I think it is convenient to use "MSL height" http://epsg.io/5714 which has no clear definition as an default.

2014-03-12

Review of changes to ISO 19115-1:2014 and 19115-3

This is a paper submitted to IPET-MDRD-1 meeting: http://wis.wmo.int/file=907

2014-02-28

Fwd: [IPET-MDRD] Updates to Manuals on GTS and WIS

There was an announcement from WMO Secretariat:
-----------------------------------------------------------

2014-02-13

reviewing TT-ApMD2 conclusion

I's only one month before the IPET-MDRD-1 meeting. Oh no.

I need to create input to the meeting by pushing actions identified in the TT-ApMD2 meeting last year. As as first step, I'm reviewing the final report of the meeting

Intermediate result: redmine tickets as my personal memoranda.

2014-01-29

New mandatory regulations of WCMP 1.3 compared to version 1.2

The WMO Core Metadata Profile (WCMP) version 1.3 has more mandatory requirements compared to the previous version 1.2. The Manual on WIS (update 2013, see my previous post for text) has following mention on this matter:

Note: metadata created using profile version 1.2 is compatible with that created under version 1.3 other than that the records may have been completed inconsistently and therefore may fail the version 1.3 conformance checking rules.

So it is natural to ask "What are the checking rules newly introduced?" Please check following list.

6.1.2 Each WIS Discovery Metadata record shall validate without error against the rule-based constraints listed in ISO/TS 19139:2007 Annex A (Table A.1). [These constraints cannot be tested by XSD (W3C XML Schema), so it is highly likely to fail to meet this requirement even if your records pass XSD validation]
6.2.1 Each WIS Discovery Metadata record shall name explicitly all namespaces used within the record; use of default namespaces is prohibited. [the latter part is a kind of difficult to detect, as the XPath language cannot find where the namespace declaration is present.]
6.3.1 Each WIS Discovery Metadata record shall declare the following XML namespace for GML: http://www.opengis.net/gml/3.2. [WCMP1.2 preferred this namespace, but it was not in so strong language]
Intended scope of distribution
- [§9.1 ¶1] The scope of distribution for data within WIS shall be expressed using the following controlled vocabulary: "GlobalExchange", “RegionalExchange” and “OriginatingCentre” (if the scope of distribution is documented).
- Requirement 9.1.1: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the scope of distribution using the keyword “GlobalExchange” of type “dataCenterdataCentre” from thesaurus WMO_DistributionScopeCode.
Data policy
- Requirement 9.3.1: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the WMO Data License as Legal Constraint (type: “otherConstraints”) using one and only one term from the WMO_DataLicenseCode code list.
- [§9.3 ¶5] The presence of more than one WMO Data Policy statement in a single metadata record yields an ambiguous state; a WIS Discovery Metadata record describing data for global exchange shall declare only a single WMO Data Policy.
Requirement 9.3.2: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the GTS Priority as Legal Constraint (type: “otherConstraints”) using one and only one term from the WMO_GTSProductCategoryCode code list.
[§8.1 ¶9] WMO Core Metadata Profile mandates dateStamp in format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS. [WCMP 1.2 has similar words, but it didn't say "shall"].

For entire test set, my working draft of the Schematron rule set (pretty-printed by a stylesheet on PC browser) is available here.

2014-01-22

WIS Metadata todo's found during CAS/ET-WDC

todos new and gone:

people wanted to use the gmd:geographicelement tag with only xlink:href attribute with URL of Google Maps. I had to say so far there is no WCMP regulation against that. I hope this is not controversial for people demanding strong-typed XLink.
I heared GEO AQ CoP metadata discussion held in Dublin (Sep 2012) recommended the gmd:thesaurusName tag with only xlink:href attribute. That is different from WCMP's present use of gmd:thesaurusName/*/gmd:title string to identify the thesaurus.
some people wanted to use a bounding box for a country even though the data is for a single station. They say we don't know the boundary of a station in WIGOS sense. It's totally user unfriendly in the data discovery context. Metadata records with unnecessarily wide bounding box matches the search for other part of the country, and the user will be extremely frustrated when they cannot narrow down to remove the noise. I __strongly recommended__ to forget the word "bounding" and use point coords, i.e. using the same west == east and north == south. It was totally unexpected response of users and I felt it might be necessary to add a notice in WCMP.
don't know good Vertical CRS identifier for "metre above sea level" without clear datum definition. Meteorologists don't want precision for centimeter, but clear identification of "m asl" is necessary and to be __never__ confused with ellipsoid height. Some tried to use "WGS84" but I'm not sure whether it's geoid or ellipsoid height. If that's about EPSG:4326, that's only the 2d horizontal CRS and it's totally undefined. IPET-MDRD may take time to conclude, but the users need the answer today!!! I might have to propose something.

resolved todo:

The WMO category code "radiation" was proposed for measurements of infrared and visible electromagnetic waves. The ET incuding WRDC St Petersburg agreed to withdraw the proposal and use existing entry "actinometry". The term radiation has unclear meaning and may include nuclear radiation.

2013-12-20

XSLT (XPath) cannot find where the namespace declaration is

Yesterday I got stack a tricky features of XML.

Well, somehow I had to remove redundant the default namespace declarations (something looking like attribute "xmlns") from a large number of huge size of XML documents. That job is incomplete, and this is intermediate memorandum.

Before really doing the job, I wanted to see how many instances are found where. It was surprising that XSLT cannot do the job. The "namespace::" syntax of XPath does not find the literal text in XML seriarization, but rather matches conceptual namespace nodes which are copied to all child elements [XPath]. So it cannot detect redundant NS declaration.

I think it is necessary to program with XML parser. I wrote a ruby script to work with libxml2 Reader interface.

2013-12-18

XSLT to extract metadata from OAI-PMH GetRecord response

Between the GISCs of WMO Information System, a metadata record is exchanged using OAI-PMH. OAI-PMH is an HTTP-based protocol, in which the server's response is an XML document that encapsulates metadata record(s).

It sounds so easy. It's just extracting /OAI-PMH/GetRecord/record/metadata/gmd:MD_Metadata. Following command would suffice:

$ xmllint --xpath '//*[local-name()="MD_Metadata"][1]' input.xml > output.xml

Even with older version of libxml2, a few lines of equivalent XSLT would do the same job. Until today I have thought so. But it was no good.

WMO Core Metadata Profile version 1.3 somehow prohibits the use of default namespace declaration. But above command produces undesired default namespace declaration if the OAI-PMH uses it. Oh no....!

It is a bit tricky to remove namespace declaration. The exclude-result-prefixes parameter
works only in the literal result elements. That means you have to write <gmd:MD_Metadata> instead of xsl:copy-of or xsl:copy or xsl:element.

<xsl:stylesheet version="1.0"
 xmlns:gmd="http://www.isotc211.org/2005/gmd"
 xmlns:oai="http://www.openarchives.org/OAI/2.0/"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 exclude-result-prefixes="oai"
 >
<xsl:output method="xml" omit-xml-declaration="no" />
<xsl:template match="/">
 <xsl:apply-templates select=".//oai:metadata/*[1]"/>
</xsl:template>
<xsl:template match="gmd:MD_Metadata">
 <!-- this is the literal result element -->
 <gmd:MD_Metadata>
  <!-- you might wish to override xsi:schemaLocation by ISO standard -->
  <xsl:apply-templates select="*|@*|text()"/>
  </gmd:MD_Metadata>
</xsl:template>
<xsl:template match="*">
 <!-- xsl:copy-of brings undesirable xmlns= even under MD_Metadata -->
 <xsl:copy>
 <xsl:apply-templates select="*|@*|text()"/>
 </xsl:copy>
</xsl:template>
<xsl:template match="@*">
 <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>

2013-12-17

Note on NASA DIF (Directory Interchange Format) and GCMD Keywords

For long time I knew only the name of DIF (Directory Interchange Format) used in GCMD (Global Change Master Directory) which is a catalogue operated by NASA. Recent days I'm getting interacting with more people who are interested in using GCMD keywords in the WMO/WIS Discovery Metadata which is extension of ISO 19139.

Resources I found in a quick research:

DIF Writer's Guide

An entry point of most of official resources.
Contains XML Schema or template that illustrates the structure.

Some elements refer to keywords i.e. controlled vocabulary

given as 10 tables in (at least) CSV format

NASA gives a mapping table to ISO 19115

some elements seem to be given in old names and structure

AADC (Australian Antarctic Data Centre) provides a converter into various profiles of ISO 19115/19139

probably well-done, but apparently using some extension to XSLT

In the WIS community there was a question about gmd:keywordType and the uniqueness of gmd:thesaurusName. Our profile WCMP 1.3 requires a single thesaurusName appears only once. The GCMD keyword tables contain different types. If a metadata creator wish to use gmd:keywordType to clarify the category of the keywords, he/she has to divide the MD_Keyword block for different keywordType.

In the original mapping by GCMD, there is no such issue. The ISO element "keyword" is mapped from only "Keyword" in DIF which is free text. But the most complex DIF element "Parameters" is mapped to old ISO element "category" which is probably superseded by topicCategory which is unfortunately enumeration and hence no longer extendable. So the mapping does not have contemporary meaning, really unfortunately.

So I move to more realistic mapping implementation by AADC. It creates MD_Keywords from following DIF elements:

Parameters - Controlled
Keyword - Free text
Sensor_Name (Instruments) - Controlled
Source_Name (Platform) - Controlled
Paleo_Temporal_Coverage - Free numeric date range
Paleo_Temporal_Coverage/Chronostratigraphic_Unit - Controlled but the list not online, probably something like shown in Wikipedia
Project - Controlled
IDN_Node - some identifier I don't know
Location - Controlled

Apparently there should be a need to care about a need for using GCMD thesaurusName for multiple keyword types.

TT-ApMD-2 (see para 28) was aware about that situation, and recommended slightly changing the title of thesaurusName/*/title like following:

"NASA/Global Change Master Directory (GCMD) Earth Science Keywords. Version 8.0.0.0.0. (for theme)"

I know this is ugly and there are still some opinions, and really hope we get some agreement....

WMO Common Code Table C-15 (Physical quantities) and QUDT unit of measurement

New common code table C-15 (Physical quantities) is under development in the WMO Manual on Codes. This is to be served as online registory http://codes.wmo.int/common/c-15. In my understanding the primary motivation at the moment is to provide semantic description of quantities used in the Aviation XML.

The table is of course a list of entries, each describes a quantity, for example "airTemperature" http://codes.wmo.int/common/c-15/me/airTemperature. Looking at the table, there is a field "generalization" with value "ThermodynamicTemperature" that links to http://qudt.org/vocab/quantity#ThermodynamicTemperature.

This is a link to QUDT. The top page describes only SI and CGS systems, but there seems to be care for other conventional units.

2013-12-04

"iso" URN namespace - machine-readable reference to ISO standards

I found IETF RFC5141 http://tools.ietf.org/html/rfc5141 defines the "iso" namespace of the URN. That makes it possible to cite ISO standards in a computer-readable manner. The metadata standards used in the WMO Core Metadata Profile can be called like following:

urn:iso:std:iso:19115:2003:en
urn:iso:std:iso:19115:cor-1:2006:en
urn:iso:std:iso:ts:19139:2007:en

Interestingly enough, the author (from ISO) says the ISO requests this namespace because the URN scheme using OID (such as urn:oid:1:0:19115 for ISO 19115) is not human-readable.

I'm not trying to change WCMP (for example gmd:metadataStandardName) since (for now) I don't know request that the field has to be computer-readable. But I think it is worth sharing.

[article also posted to WMO/IPET-MDRD]

2013-11-26

what is XML namespace - ISO 19139 in mind

Today I had an opportunity to explain what is XML namespace. This might happen again, so I save the text for future reference.

GML versions - why WCMP 1.3 specifies GML 3.2 only?

Question:

Why the WMO Core Metadata Profile (WCMP) v1.3 tells only GML 3.2 must be used? Isn't it good to make it open-ended? GML 3.3 is already published as OGC standard, and further development might be coming.

Answer:

We need clear guidance on validation using XSD (W3C XML Schema) from an official source. That was (still is) only official version published by ISO.

utilizing XLink in WIS Discovery Metadata to compactify common information

When looking at multiple metadata records, we can easily find a huge number of repetition of the same information with significant size, for example point of contact, citation, or reference system. It is a natural idea to utilize XLink attributes in ISO 19139 XML Schema, and that has been discussed so many times in WIS community for years.

But I have to say there is no tangible outcome so far. Everybody like to talk about the vision, but apparently don't know how to get there. I feel I'm obliged to explain already-known difficulties.

Note: following is only my personal view, and is not to be considered official position of any organization.

Quick review - DataCite Metadata Schema v3.0

Data Publication is a recent movement in the data-intensive science domains, which tries to define the service of data as a part of scholarly work to be acknowledged, thus tries to make data referable from literature. ICSU WDS has established a working group for the data publication, and DataCite is a non-profit organization (seemingly) involved in it.

They published the DataCite Metadata Schema v3.0, which is interesting.

[WCMP] conceptual "extension" encoded as restrictions in terms of XML

The WMO core metadata profile, version 1.3, says it is a restricted profile of ISO 19115:

A category-1 profile places additional restrictions on the use of an International Standard to meet the more specific requirements of a given community

But I found some people described it's an extended profile because it has additional codelists. Which is right?

In my view, both.

Apparently opposite terminologies reflect different levels of viewpoint: conceptual and concrete, or more frankly, UML and XML.

[WCMP] Questions on Schema and Schematrons re v1.2 and 1.3

There are many questions regarding WMO Core Metadata Profile (WCMP). I think some are really worth sharing.

[WCMP] how to cite International Meteorological Vocabulary (WMO 182)?