Daniel Veillard's libxml2 is one of the most commonly-available XML parser library which includes an implementation of XSD validator. Version 2.7.8 and before had a bug that breaks xmllint(1) command with XSD schema of GML and hence ISO 19139.
I wrote a patch in 2011, and I found it was incorporated in the Version 2.8.0. Great!
2013-08-19
Update and maintenance of WIS OAI Monitor: id case insensitivity and reliability of incremental harvesting
These weeks I did extensive (but incomplete unfortunately) maintenance work of the monitoring site of OAI-PMH synchronization of WIS Discovery metadata. I believe I ought to share what's happening.
= 1. Reliability of Incremental Harvesting
Apparently my monitor had indicated unnatural large number of diffs (differences between the same metadata sets between centers). After performing full harvesting, the number reduced significantly.
That meanst that the changes (mostly additions) were not effectively harvested by incremental harvesting i.e. the OAI-PMH "ListIdentifiers" request with "from=" parameter having a little (24h in my case) before the present time. I think that is due to some problem in the implementation or the operation, but I still need more information to give effective guidance.
Anyway the monitoring software must be changed in its design strategy.
Right now it retrieves full OAI sets every 3 hours from relatively fast servers (>300 record/sec), and only "increments" (changes within past 24 hours) are retrieved from relatively slow servers (around 3 record/sec). It was deemed necessary to achieve high temporal resolution, but really unfortunately, the loss of incremental harvesting happens on slow servers. As a result, a reliable monitoring should use full harvesting for all kinds of servers, under current situation.
Right now the entire WIS data catalogue consists of 1.4e5 records. It's a simple math that it takes at least 13 hours to take full harvest from 3 record/sec server. In reality it is not nice thing to maintain (even) daily monitoring since monitoring should not take up majority of server resources.
Something has to be done.
= 2. Case-insensitivity of Metadata id's
Another source of increase of #diffs is the case-insensitivity of metadata identifier. The first meeting of IPET-MDI agreed that the identifier should be treated as case-insensitive, and so was written in the WMO Core Metadata Profile, which is now an appendix to the Manual on WIS.
Before 2013-08-16T12Z, the monitoring did comparison of the OAI sets in case-sensitive manner. I didn't notice problem when it was written. But recently there were some records whose id's were partly lowercased. Some other centres followed the change, and others not. It is highly suspected that the synchronization does not work for the latter, but it is out of the scope of id-only monitoring.
Anyway the rule is rule, so I changed the software to make the case-insensitive diffs.
= 1. Reliability of Incremental Harvesting
Apparently my monitor had indicated unnatural large number of diffs (differences between the same metadata sets between centers). After performing full harvesting, the number reduced significantly.
That meanst that the changes (mostly additions) were not effectively harvested by incremental harvesting i.e. the OAI-PMH "ListIdentifiers" request with "from=" parameter having a little (24h in my case) before the present time. I think that is due to some problem in the implementation or the operation, but I still need more information to give effective guidance.
Anyway the monitoring software must be changed in its design strategy.
Right now it retrieves full OAI sets every 3 hours from relatively fast servers (>300 record/sec), and only "increments" (changes within past 24 hours) are retrieved from relatively slow servers (around 3 record/sec). It was deemed necessary to achieve high temporal resolution, but really unfortunately, the loss of incremental harvesting happens on slow servers. As a result, a reliable monitoring should use full harvesting for all kinds of servers, under current situation.
Right now the entire WIS data catalogue consists of 1.4e5 records. It's a simple math that it takes at least 13 hours to take full harvest from 3 record/sec server. In reality it is not nice thing to maintain (even) daily monitoring since monitoring should not take up majority of server resources.
Something has to be done.
= 2. Case-insensitivity of Metadata id's
Another source of increase of #diffs is the case-insensitivity of metadata identifier. The first meeting of IPET-MDI agreed that the identifier should be treated as case-insensitive, and so was written in the WMO Core Metadata Profile, which is now an appendix to the Manual on WIS.
Before 2013-08-16T12Z, the monitoring did comparison of the OAI sets in case-sensitive manner. I didn't notice problem when it was written. But recently there were some records whose id's were partly lowercased. Some other centres followed the change, and others not. It is highly suspected that the synchronization does not work for the latter, but it is out of the scope of id-only monitoring.
Anyway the rule is rule, so I changed the software to make the case-insensitive diffs.
2013-07-04
bookmarks for IPET-DRMM
= general
1st meeting of IPET-DRMM - docplan
http://www.wmo.int/pages/prog/www/ISS/Meetings/IPET-DRMM_Tokyo2013/IPET-DRMM_DocPlan.html
TDCF tables
http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/LatestVERSION/LatestVERSION.html
= Web resources
Experimental tracker
http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/LatestVERSION/LatestVERSION.html
Status flow
https://docs.google.com/drawings/d/1xNt5pbjsqiTTpvlmZKwFf4CkXxQ7Sp5mVh3dtvV5iao/edit?usp=sharing
= AHL
T1T2 usage (Manual, VolC1, and Tokyo Cache)
https://docs.google.com/spreadsheet/ccc?key=0AtzHt7XJp45TdE0wVVFVVnU2amdobVl5bHFtUzlXdmc&usp=sharing
Routeing Catalogue
http://www.wmo.int/pages/prog/www/ois/Operational_Information/RtngCat_en.html
= Not actually related
TT-ApMD-2
http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=TT-ApMD-2
AvXML RC2
http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=AvXML-1.0RC2-Release
1st meeting of IPET-DRMM - docplan
http://www.wmo.int/pages/prog/www/ISS/Meetings/IPET-DRMM_Tokyo2013/IPET-DRMM_DocPlan.html
TDCF tables
http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/LatestVERSION/LatestVERSION.html
= Web resources
Experimental tracker
http://www.wmo.int/pages/prog/www/WMOCodes/WMO306_vI2/LatestVERSION/LatestVERSION.html
Status flow
https://docs.google.com/drawings/d/1xNt5pbjsqiTTpvlmZKwFf4CkXxQ7Sp5mVh3dtvV5iao/edit?usp=sharing
= AHL
T1T2 usage (Manual, VolC1, and Tokyo Cache)
https://docs.google.com/spreadsheet/ccc?key=0AtzHt7XJp45TdE0wVVFVVnU2amdobVl5bHFtUzlXdmc&usp=sharing
Routeing Catalogue
http://www.wmo.int/pages/prog/www/ois/Operational_Information/RtngCat_en.html
= Not actually related
TT-ApMD-2
http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=TT-ApMD-2
AvXML RC2
http://www.wmo.int/pages/prog/www/WIS/wiswiki/tiki-index.php?page=AvXML-1.0RC2-Release
2013-06-12
Outcome of WIGOS Metadata Task Team (TT-WDM-1)
Note: below is just my personal analysis of publicly-available documents.
There was a meeting TT-WMD-1 in March.
* Final Report: http://www.wmo.int/pages/prog/www/WIGOS-WIS/reports/TT-WMD-1_Final-Report_Geneva_2013.doc (pdf-converted: http://goo.gl/6jcpX)
FR§5.2 says WIGOS will develop a web portal called WIR. Existing OSCAR will be a component of WIR in following structure:
WIR
|
+- SORT: standards of observing methods
| (I guess WIS metadata don't link to that in near future)
|
|
+- OSCAR: DB of requirements and capabilities of observatios
|
+- OSCAR/Surface: evolution of Volume A
+- OSCAR/Distributed: DBs made for each regional/progamme community
FR§5.2.6 says the radar DB http://wwr.dmi.gov.tr/ is an example of OSCAR/Distributed. I wonder where is existing satellite DB in OSCAR is located, perhaps as an instance of OSCAR/Distributed.
FR Appendix II (p.18) gives a draft spec of "Core WIGOS Metadata". It's a list of elements with Mandatory/Conditional/Optional flags. The elements are in conceptual level, so the encoding or instantiation has to be developed later before actual catalogue-building can be started.
Regarding the future of Volume A, FR§6.3 says it's primarily the role of CBS, but in my understanding CIMO will augment a number of elements (such as shown in Doc.7 to that meeting http://www.wmo.int/pages/prog/www/WIGOS-WIS/meetings/TT-WMD-1/Doc7_Metadata.doc ).
FR§5.2.4 shows an idea of linkage from WIGOS (WIR) to WIS. That must raise requirement of some interface - i.e. rules of linkage (URL) to data searched by some criteria (ex station, variable, equipment etc) and sustainable operation with organizational arrangement (I mean which GISC does the service).
Regarding the future of Volume A, FR§6.3 says it's primarily the role of CBS, but in my understanding CIMO will augment a number of elements (such as shown in Doc.7 to that meeting http://www.wmo.int/pages/prog/www/WIGOS-WIS/meetings/TT-WMD-1/Doc7_Metadata.doc ).
FR§5.2.4 shows an idea of linkage from WIGOS (WIR) to WIS. That must raise requirement of some interface - i.e. rules of linkage (URL) to data searched by some criteria (ex station, variable, equipment etc) and sustainable operation with organizational arrangement (I mean which GISC does the service).
Similarly it is conceivable to link from WIS to WIGOS.
2013-05-13
[DRMM] 2013 inter-session amendment of Manual on Codes approved
WMO Secretariat updated its operational newsletter
http://www.wmo.int/pages/prog/www/ois/Operational_Information/Newsletters/current_news_en.html
They say:
"Adoption of amendments to the Manual on Codes (WMO No. 306) between CBS sessions
http://www.wmo.int/pages/prog/www/ois/Operational_Information/Newsletters/current_news_en.html
They say:
"Adoption of amendments to the Manual on Codes (WMO No. 306) between CBS sessions
In accordance with the Procedure for the Adoption of Amendments to the Manual on Codes between CBS sessions, and as concurred by the president of CBS, the WMO Circular letter No. OBS/WIS/DRMM/DRC (PR-6688) (English http://goo.gl/CaA3y, French http://goo.gl/tOXFP, Russian http://goo.gl/rx7ow, Spanish http://goo.gl/ZSOXx, Arabic http://goo.gl/l3xPA) dated 25 February 2013, including draft amendments to the Manual on Codes, was dispatched to all WMO Members for comments.
The WMO Secretariat received replies approving the draft amendments in the Circular letter by 24 April 2013. The amendments are therefore approved in accordance with the Procedure
The president of CBS agreed to notify all WMO Members and members of EC of the approval of the amendments by the WWW Operational Newsletter. The implementation date of amendments is on 14 November 2013."
2013-05-08
Fast track amendment (2013 round #1) of WMO Manual on Codes
WMO announced the first round in 2013 of the fast track amendment (*) of WMO Manual on Codes was approved on 28 April, to take effect on 8 May 2013, i.e. today.
The result is now posted on WMO website. The latest versions of GRIB & BUFR tables are 11 and 20 respectively.
IPET-MDRD/TT-MDI reviews WIS discovery metadata records by WMO programmes
Recently our team had opportunity to review a set of WIS discovery metadata (i.e. geographical metadata in ISO 19139 format) made by GAWSIS (GAW Station Info Service by MeteoSwiss).
That was really good opportunity to refresh my memory, implement missing links, and practice interaction between WMO programmes. Our role is to establish conventions (notations) to express various metadata which is special to WMO, so it is essential to have input from WMO programmes.
Subscribe to:
Posts
(
Atom
)