2014-01-30

when trouble receiving WMO Google Groups message

There is an member in having trouble receiving emails from TT-ApMD's Google Group.  I think it is good to share the instruction for future re-use and reference for other members.



You can visit online archive to check (1) whether you are member and subscribing emails, and (2) any threads posted so far.
 
 
Maybe it is better to check the Atom Feed to verify every messages posted.
 
 
If there's trouble in membership, I will ask the WMO Secretariat to address the issue.

2014-01-29

New mandatory regulations of WCMP 1.3 compared to version 1.2

The WMO Core Metadata Profile (WCMP) version 1.3 has more mandatory requirements compared to the previous version 1.2.  The Manual on WIS (update 2013, see my previous post for text) has following mention on this matter:

Note: metadata created using profile version 1.2 is compatible with that created under version 1.3 other than that the records may have been completed inconsistently and therefore may fail the version 1.3 conformance checking rules.

So it is natural to ask "What are the checking rules newly introduced?"  Please check following list.


  • 6.1.2 Each WIS Discovery Metadata record shall validate without error against the rule-based constraints listed in ISO/TS 19139:2007 Annex A (Table A.1). [These constraints cannot be tested by XSD (W3C XML Schema), so it is highly likely to fail to meet this requirement even if your records pass XSD validation]
  • 6.2.1 Each WIS Discovery Metadata record shall name explicitly all namespaces used within the record; use of default namespaces is prohibited.  [the latter part is a kind of difficult to detect, as the XPath language cannot find where the namespace declaration is present.]
  • 6.3.1 Each WIS Discovery Metadata record shall declare the following XML namespace for GML: http://www.opengis.net/gml/3.2. [WCMP1.2 preferred this namespace, but it was not in so strong language]
  • Intended scope of distribution
    • [§9.1 ¶1] The scope of distribution for data within WIS shall be expressed using the following controlled vocabulary: "GlobalExchange", “RegionalExchange” and “OriginatingCentre” (if the scope of distribution is documented).
    • Requirement 9.1.1: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the scope of distribution using the keyword “GlobalExchange” of type “dataCenterdataCentre” from thesaurus WMO_DistributionScopeCode.
  • Data policy
    • Requirement 9.3.1: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the WMO Data License as Legal Constraint (type: “otherConstraints”) using one and only one term from the WMO_DataLicenseCode code list.
    • [§9.3 ¶5] The presence of more than one WMO Data Policy statement in a single metadata record yields an ambiguous state; a WIS Discovery Metadata record describing data for global exchange shall declare only a single WMO Data Policy.
  • Requirement 9.3.2: A WIS Discovery Metadata record describing data for global exchange via the WIS shall indicate the GTS Priority as Legal Constraint (type: “otherConstraints”) using one and only one term from the WMO_GTSProductCategoryCode code list.
  • [§8.1 ¶9] WMO Core Metadata Profile mandates dateStamp in format YYYY-MM-DD or YYYY-MM-DDTHH:MM:SS. [WCMP 1.2 has similar words, but it didn't say "shall"].

For entire test set, my working draft of the Schematron rule set (pretty-printed by a stylesheet on PC browser) is available here.

2014-01-22

WIS Metadata todo's found during CAS/ET-WDC

todos new and gone:

  • people wanted to use the gmd:geographicelement tag with only xlink:href attribute with URL of Google Maps. I had to say so far there is no WCMP regulation against that.  I hope this is not controversial for people demanding strong-typed XLink.
  • I heared GEO AQ CoP metadata discussion held in Dublin (Sep 2012) recommended the gmd:thesaurusName tag with only xlink:href attribute.  That is different from WCMP's present use of gmd:thesaurusName/*/gmd:title string to identify the thesaurus.
  • some people wanted to use a bounding box for a country even though the data is for a single station.  They say we don't know the boundary of a station in WIGOS sense.  It's totally user unfriendly in the data discovery context.  Metadata records with unnecessarily wide bounding box matches the search for other part of the country, and the user will be extremely frustrated when they cannot narrow down to remove the noise.  I __strongly recommended__ to forget the word "bounding" and use point coords, i.e. using the same west == east and north == south.  It was totally unexpected response of users and I felt it might be necessary to add a notice in WCMP.
  • don't know good Vertical CRS identifier for "metre above sea level" without clear datum definition.  Meteorologists don't want precision for centimeter, but clear identification of "m asl" is necessary and to be __never__ confused with ellipsoid height.  Some tried to use "WGS84" but I'm not sure whether it's geoid or ellipsoid height.  If that's about EPSG:4326, that's only the 2d horizontal CRS and it's totally undefined.  IPET-MDRD may take time to conclude, but the users need the answer today!!!  I might have to propose something.
resolved todo:
  • The WMO category code "radiation" was proposed for measurements of infrared and visible electromagnetic waves.  The ET incuding WRDC St Petersburg agreed to withdraw the proposal and use existing entry "actinometry".  The term radiation has unclear meaning and may include nuclear radiation.

2014-01-12

[Ruby] use self.class to override the constants in the subclasses

In Ruby names starting with uppercase letters are constants.  When you use a reference to a constant in a method, that refers to the constant defined in the class.  If you make a subclass, the method was inherited to the subclass, but the constant is still resolved against the superclass where the method was originally defined in.

If that is undesirable, we can use "self.class::NAME" synatax to refer to find the constant defined in the class in which the method is really used.

Tested in Ruby 1.8.5:

$ cat test.rb
class A
  Config = "constant in class A"
  def initialize
    p self.class.name
    p Config
    p self.class::Config
  end
end

class B < A
  Config = "constant in class B"
end

A.new
B.new
$ ruby test.rb
"A"
"constant in class A"
"constant in class A"
"B"
"constant in class A"
"constant in class B"

2014-01-09

GTS trouble affected GISC cache (loss of data 12Z to 18Z 8 Wed)

There was a trouble in international bulletin exchange over GTS, which caused significant loss of data in the GISC cache around 12UTC to 18UTC on 8 Wed January.   That includes all GTS messages including coarse-resolution GRIB1 bulletins from JMA GSM.

RTH Tokyo was receiving a huge number of a single bulletin repeatedly, and went beyond its capacity.  Delivery to GISC cache system had to be turned off during the recovery operation.
I'm afraid there is no plan to re-deliver the lost data to GISC cache, as it is classified as WIS Part B, i.e. the service in best effort basis.  Operation-critical users are advised to subscribe to the WIS Part A service, for example dedicated-bandwidth GTS for neighbouring NMHSs, or JMBSC for commercial weather enterprises.

High-resolution GSM data service was not affected since it is coming through different channel.
I apologise all users for inconveniences.  All delivery is now working normally.

2013-12-20

XSLT (XPath) cannot find where the namespace declaration is

Yesterday I got stack a tricky features of XML.

Well, somehow I had to remove redundant the default namespace declarations (something looking like attribute "xmlns") from a large number of huge size of XML documents.  That job is incomplete, and this is intermediate memorandum.

Before really doing the job, I wanted to see how many instances are found where.  It was surprising that XSLT cannot do the job.  The "namespace::" syntax of XPath does not find the literal text in XML seriarization, but rather matches conceptual namespace nodes which are copied to all child elements [XPath].  So it cannot detect redundant NS declaration.

I think it is necessary to program with XML parser.  I wrote a ruby script to work with libxml2 Reader interface.



2013-12-18

XSLT to extract metadata from OAI-PMH GetRecord response

Between the GISCs of WMO Information System, a metadata record is exchanged using OAI-PMH.  OAI-PMH is an HTTP-based protocol, in which the server's response is an XML document that encapsulates metadata record(s).

It sounds so easy.  It's just extracting /OAI-PMH/GetRecord/record/metadata/gmd:MD_Metadata. Following command would suffice:

$ xmllint --xpath '//*[local-name()="MD_Metadata"][1]' input.xml > output.xml

Even with older version of libxml2, a few lines of equivalent XSLT would do the same job. Until today I have thought so.  But it was no good.

WMO Core Metadata Profile version 1.3 somehow prohibits the use of default namespace declaration.  But above command produces undesired default namespace declaration if the OAI-PMH uses it.  Oh no....!

It is a bit tricky to remove namespace declaration.  The exclude-result-prefixes parameter
works only in the literal result elements.  That means you have to write <gmd:MD_Metadata> instead of xsl:copy-of or xsl:copy or xsl:element.

<xsl:stylesheet version="1.0"
 xmlns:gmd="http://www.isotc211.org/2005/gmd"
 xmlns:oai="http://www.openarchives.org/OAI/2.0/"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
 exclude-result-prefixes="oai"
 >
<xsl:output method="xml" omit-xml-declaration="no" />
<xsl:template match="/">
 <xsl:apply-templates select=".//oai:metadata/*[1]"/>
</xsl:template>
<xsl:template match="gmd:MD_Metadata">
 <!-- this is the literal result element -->
 <gmd:MD_Metadata>
  <!-- you might wish to override xsi:schemaLocation by ISO standard -->
  <xsl:apply-templates select="*|@*|text()"/>
  </gmd:MD_Metadata>
</xsl:template>
<xsl:template match="*">
 <!-- xsl:copy-of brings undesirable xmlns= even under MD_Metadata -->
 <xsl:copy>
 <xsl:apply-templates select="*|@*|text()"/>
 </xsl:copy>
</xsl:template>
<xsl:template match="@*">
 <xsl:copy-of select="." />
</xsl:template>
</xsl:stylesheet>