Showing posts with label WMOCodes. Show all posts
Showing posts with label WMOCodes. Show all posts

2015-01-12

TDCF migration of upper-air sounding data

The World Meteorological Organization (WMO) has been working for TDCF migration, which is transition from observation reports and forecast messages in TAC (traditional alphanumeric codes) into TDCF (table-driven code forms) namely BUFR (binary universal representation) and CREX.  For most of message types, the parallel distributions were scheduled up to November 2014.  

Some countries actually stopped their TAC distributions, and
it turned out that the meteorology community needs more work to get fully ready to the transition, especially for upper-air sounding.  The situation was complicated since WMO tried to change not only code but reporting practice, as discussed in following forum thread:

Traditionally, the alphanumeric messages for a single ascent of radiosonde are supposed to be disseminated in four parts:

  • Part A: standard isobaric levels up to 100 hPa
  • Part B: significant levels up to 100 hPa
  • Part C: standard isobaric levels above 100 hPa
  • Part D: significant levels above 100 hPa

The standard isobaric levels are 1000, 925, 850, 700, 500, 400, 300, 250, 150, and 100 for Part A; and 70, 50, 30 and 10 hPa for Part C.  Weather charts are supposed to be drawn at those levels (JMA's charts are available at http://www.jma.go.jp/jp/metcht/kosou.html).

The significant levels (in Parts B and D) supplements standard levels so that linear interpolation of all reported data (both standard and significant levels) gives enough accuracy.  Roughly they are chosen at kinks of plots of temperature, humidity, and winds  (a good illustration of temperature siglevs is given at ECMWF website).  Please see the Manual on Codes Volume I.1 for the complex rules for selecting significant levels.

Then, why do we split messages at 100 hPa?  The answer is time of delivery.  The 100 hPa level lies 16 km above sea level.  Typical radiosonde has 6 m/s speed.  It takes 45 minutes to reach that level.  The time may double if we wait for the end of ascent: that's why WMO member states agreed to split the report.

In the new paradigm of BUFR, there is a rule called B/C 20 and 25 to determine how to migrate from TAC to BUFR:
https://www.wmo.int/pages/prog/www/WMOCodes/BC_Regulations/BC20-PILOT.pdf
https://www.wmo.int/pages/prog/www/WMOCodes/BC_Regulations/BC25-TEMP.pdf
It describes two messages should be sent for a single ascent:

  • Standard and significant levels up to 100 hPa
  • Standard and significant levels above 100 hPa
I guess the designers of the rules thought "the reason for splitting Parts A-B and C-D is simply for size of bulletins;  now the size limitation of GTS bulletins are being eliminated;  why don't we merge unnecessary split?"

That was correct, but the meaning or value of the reporting style was not stressed.  So implementers simply used the template only and ignored the text regulations.  Most implementations simply converts existing TAC to BUFR in parts.

It was supposed that BUFR would get the mainstream, i.e. observation systems directly generated BUFR and TAC were made as a compatibility service.  By doing so we could eliminate various problems of TAC reports.  But that mainstreaming did not take place in many WMO members.  I suspect the change of the rule is even not known for many people working for observation systems.  So the part-for-part converter had to be chosen.

There are many benefits of BUFR-mainstreamed data processing, which is basically straightforward handling of numbers.  For example 1000's digit of geopotential height in lower troposphere is omitted in FM 35 TEMP.   If 700 hPa height is 3451 metres, only digits "451" is shown in the report, and the recipient has to guess the lost digit using "common sense".  It is usually 3451 m in most of the world, but it can be 2451 m in very cold places like Antarctica, so it is a headache for automated system.  A simple-minded converter always generates BUFR with 3451 m value for TAC input "451" which may be incorrect.  But the benefit has not so stressed for these decades, and I don't think many understand it.

Right now there are many proposals to change the part-by-part converted BUFRs to distinguish Parts A-D.  But I'd really remind the situation filled by a whole bunch of unimplemented regulations and ununderstood good intentions.  I think we must be really careful for changing rules, since current one takes many years to be understood partly.

2014-11-10

Changes of present weather symbols for synoptic charts - long messy history

I've got involved in a question:

"There are many websites describing plot symbols for weather maps.  Why some are different?"

Is there really difference?  Yes, unfortunately.

Currently valid definition of the symbols are given in the Attachment II-4 of the Manual on GDPFS Volume I (WMO Pub No.485).  The symbols are given in 10x10 table indexed by two-digit number ww, and the meaning of the numbers are defined in the code table 4677 of the Manual on Codes, Volume I.1 (WMO Pub. 306).  Those are all online in PDF --- thanks to the Internet.  But you can find several websites showing slightly different tables, for example http://www.australianweathernews.com/learn_about_surface_charts.htm or http://www.asahi-net.or.jp/~ns8m-hgc/w-wld1.htm .  The differences are (as far as I found):

-       00-03 have circle-like symbols, which are currently absent,

-       07 does not have alternative symbol like cursive lowercase L for ocean spray,

-       11 is *two* horizontal broken bars, unlike current *three* broken bars ( or http://www.fileformat.info/info/unicode/char/2637/index.htm), and

-       12 is *one* broken bar over another bar, unlike current *two* broken bars over another bar ( or http://www.fileformat.info/info/unicode/char/2633/index.htm).

 That's the table used before 1962.  I found the same table in the WMO Technical Regulations published in 1959.

-       1962: Rec.67 CSM-III decided the symbols for 11 and 12 should be the present style. The text suggests there were confusion before that.

-       1970: 6.4.2.2 CSM-V introduced the symbol for spray (code 07).  It also decided removal of many code points including 00-03, but those are not carried out at that time.

-       1980: Rec.7 CBS-Ext.(80) superseded the symbol table with new one in which code points 00-03 remain blank.a

The final reports of CBS (Commission for Basic Systems) and CSM (Commission for Synoptic Meteorology) are found here: http://goos.kishou.go.jp/ws/ETMC/code_task/doc_cmm_cbs.html

 

2014-09-29

ECMWF workshop on closing GRIB-netCDF gap

I've participated the ECMWF workshop on closing GRIB-netCDF gap http://www.ecmwf.int/en/workshop-closing-grib/netcdf-gap. It was really exciting experience and it's my greatest honour that I was only participant from Asia-Pacific regions to the invitation-only meeting.  So far only presentations are available, but I expect and hope something like summary will be posted also.
Followings are notes before forgetting.  This is just my note and the thought is not necessarily the official position of whatever else, such as WMO nor JMA.

GRIB 1-to-2 Compatibility of Parameter Tables

If my brain was not too damaged by English ale, there was a voice in cocktail conversation that the parameter table of GRIB2 is (and every other tables are) designed to be upper compatible to the counterpart(s) of GRIB Edition 1 (if any).  Unfortunately another voice seems also right that this important "principle" was not written in any formal document such as the Manual or relevant meeting reports.  But apparently the principle explains well the presence of redundant parameters in GRIB2.

As was presented by myself, the GRIB2 parameter tables contain some redundancy such as 0-0-4 Maximum temperature [K] or 0-1-8 Total precipitation [kg.m-2].  I don't care whether they were introduced by the "principle".  Anyway the principle was forgotten (or did not exist from the beginning) before the 2008 meeting when the expert team tried to deprecate the parameters.  If GRIB2 had reference implementation including GRIB1 converter or concrete validation procedure including handling of units, people should have realized the breaking impact at that time.

Much more care is needed to remove some components than to add.

Dead idea of netCDF as GRIB2

Mr Chris Little said WMO "outvoted" the idea of making GRIB2 as netCDF + compression + WMO tables.  I didn't know that.  Formal records (such as CBS/EC reports) do not retain any trace of such idea, but the story is consistent with early FWIS task team reports that indicates much favour on Unidata technologies.  Anyway the idea "netCDF as GRIB Next" has been raised repeatedly, and governance only does not accounts well for technical choice.  Many argued separation of data model and representation; that directly leads to the question why WMO (actually operational meteorology) is not happy with WMO-controlled convention on netCDF.

Most people talk about the data size and compression.  As John Caron reports there will need a little more development for netCDF to compete in size efficiency with GRIB of 16-bit pack with JPEG2K compression.  Furthermore GRIB2 does have run-length encoding which lacks proper documentation unfortunately.  But this is accidental (rather than fundamental) difference in my eyes, and it's the matter of time basically.

I would rather stress on the "sequential" nature of GRIB.
  • Two GRIB messages concatenated in a file works. [as Baudouin point out]
  • At the beginning of output the programmer does not have to fix any the list of ensemble member, forecast time, vertical level, nor parameter.
  • The size of file is kept minimum if the data is sparse matrix in terms of level and parameter or other keys.
  • The reader has to scan the entire file even if the only one plane is interested in
NetCDF3 has almost opposite nature so I would call "direct access".  There can be the way in middle that could be called "indexed sequential" as in JMA's internal format NuSDaS.  I've presented that argument in Japan Geoscience Union meeting in April, but the community has little contact with Europe, so it is real good opportunity to revisit it.

Another point might sound ridiculous for some eyes, but there are some message switching systems on the GTS that requires the four octets of binary messages end with "7777".  If it is university website I'd easily change that in hours, but mission-critical operational systems is managed in totally different way.

Anyway the discussion made me less pessimistic on the future of GRIB 3, but I'd really like to be clear about the reasoning if GRIB Edition 3 is not to remain the name only.

Three-week Rule is not Short enough?

GRIB is defined in the WMO Manual on Codes, which is properly amended by the Congress held once in four years.  That is way too long for data format maintenance, so various delegation of authority has been made so that some no-harm amendment can be done twice a year.  But it's still long to make proposal from national focal point and wait for one year in the worst case.  CF allows any amendment in email based discussion which is considered approved with tree weeks without objection, which makes agile introduction of contemporary requirements.  Did you ever heard such an argument?

It was really shocking for me to see Heiko Klein reports he has 1500 standard name candidates which he hesitates to send to the CF list.  That is a sign for me that something is failing in the CF framework.  Lack of local namespace makes it impossible to share the data outside the institute.

There was many voices in the workshop (mainly from software developer) that the local table of GRIB has to be published and consolidated in computer-readable manner.  The discussion went to direction that "don't worry, WMO will suppress much harder the use of local numbers" but that seems to going to reproduce the issue reported by Heiko.

Data creators are often busy.  They hate to wait for procedures even for three weeks.  And there is subtle fear that the worldwide coordination process finds more logical formulation of data.  So I often hear many people trying to escape from the standardization claiming "it's just locally used data".  Unfortunately the promise keeps short in many cases and we get in greater trouble.

Semantic Web and Abstract Reference Software

Sorry I'll write on the topic later.


2014-08-26

METNO C monitoring - change TAF/METAR from Moscow, new CLIMAT BUFR from Minsk Belarus

CLIMAT BUFR from Belarus is quite interesting as it is going to use new template for timezone east of Greenwich.
-------- Forwarded Message --------

NOXX02 LSSW 260400
METNO C3514
284 DM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 2/5/2010 A FCKZ31
RUMS FM 51-X EXT. 02,05,08,11,14,17,20,23 UAOO UARR UATE UATG UAUU
285 AM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 8/25/2014 A FCKZ31
RUMS FM 51-X EXT. 02,05,08,11,14,17,20,23 UAAH UACK UACP UADD UAOO
UARR UASK UASP UASS UAUU
286 DM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 2/5/2010 A FTKZ31
RUMS FM 51-X EXT. 05,11,17,23 UAAA UACC UAKK UATT
287 AM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 8/25/2014 A FTKZ31
RUMS FM 51-X EXT. 05,11,17,23 UAAA UACC UAII UAKK UATE UATG UATT
288 DM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 6/17/2004 A SAKZ31
RUMS FM 15-X EXT. HALF-HLY & HLY UAAA UACC UAKK UAOO UARR UATE UATG
UATT UAUU
289 AM 8/25/2014 6 MOSCOW RUSSIAN FEDERATION MOSCOW 8/25/2014 A SAKZ31
RUMS FM 15-X EXT. HALF-HLY & HLY UAAA UAAH UACC UACK UACP UADD UAII
UAKK UAOO UARR UASK UASP UASS UATE UATG UATT UAUU
290 AA 8/25/2014 6 MOSCOW BELARUS MINSK 8/25/2014 E ISCD01 UMMN FM
94-XIII MONTHLY 26554 26666 26825 26850 26863 26941 26951 33008 33019
33036 33038 33041 CLIMAT

2014-06-12

Asperatus and WMO International Cloud Atlas

Around 2011 and 2012, there appeared a movement to create a new cloud classification called Undulatus Asperatus and work with WMO to give it official recognition.
http://cloudappreciationsociety.org/asperatus-update/comment-page-2/

http://online.wsj.com/news/articles/SB10001424127887324251504578579204045089848

It was pretty impressive that the Royal Met Soc was supporting the idea:
http://www.rmets.org/asperatus-new-cloud-variety

Recently I learned that CIMO (Commission for Instrument and Methods of Observation) took responsibility of revising the International Cloud Atlas (ICA), and established a task team. The team had a meeting in 2013 and the report is available:
http://www.wmo.int/pages/prog/www/IMOP/reports/2013/CIMO-TT-ICA_FinalRep.pdf#page=20

Their recommendation is that the Asperatus cloud would be added not as a new Genera to existing ten (such as Cumulus or Altostratus), but as a new supplementary feature.

That should be a feasible compromise, since it has little impact of realtime synoptic meteorology, while skywatchers will have autholized terminology to describe the cloud with distinct appearance.

Recent document submitted to coming CIMO session says the task team is expected to finalize the revision of ICA by the end of 2015, and then it has to be authorized by some upper bodies, at least by the Executive Council of WMO which was held June every year.  So the new ICA will be available in the latter half of 2016, if everything goes perfectly.

2014-04-11

note on relative humidity in real nwp

1. ECMWF proposal to introduce new parameters for clarification.

https://groups.google.com/a/wmo.int/forum/#!topic/cbs-ipet-drmm/mjqgaFE3gSE

That proposes new parameters:

* 0-1-93 Relative humidity with respect to water
* 0-1-94 Relative humidity with respect to ice

There is already "0-1-1 Relative humidity" but ECMWF propses adding
*both* water and ice because 0-1-1 was used for RH computed using the
"virtual" saturation (not commonly-understood term) defined by water-ice
mixture.

2. JMA uses such RH, at least in the global model. Saturated vapor
pressure is linear combination of water and ice, and the mixing ratio
changes linearly from -15 C to 0 C.

Sources:

* Para 3.2.5 of NWP Outline 2013
http://www.jma.go.jp/jma/jma-eng/jma-center/nwp/outline2013-nwp/index.htm

* Para 3.5.2(4) of NWP text #43 (in Japanese)
http://www.jma.go.jp/jma/kishou/books/nwptext/43/chapter3.pdf#page=28

3. Large-scale condensation scheme

The reason to have such virtual saturation should be large-scale
condensation.
For example following article describes a formulation that leads to JMA
schme.

W.K. Tao, J. Simpson, and M. McCumber, 1989: An Ice-Water Saturation
Adjustment. MWR 117 231-235.
http://journals.ametsoc.org/doi/abs/10.1175/1520-0493%281989%29117%3C0231:AIWSA%3E2.0.CO%3B2

The minimum temperature of presence of water droplets was undefined in
Tao et al (1989). Smith (1990), cited by JMA Outline, reports -15 C was
a choice by tuning.

R.N.B. Smith, 1990: A scheme for predicting layer clouds and their water
content in a general circulation model. QJRMS 116 435-460.
http://onlinelibrary.wiley.com/doi/10.1002/qj.49711649210/abstract

2013-12-17

WMO Common Code Table C-15 (Physical quantities) and QUDT unit of measurement

New common code table C-15 (Physical quantities) is under development in the WMO Manual on Codes.  This is to be served as online registory http://codes.wmo.int/common/c-15.  In my understanding the primary motivation at the moment is to provide semantic description of quantities used in the Aviation XML.

The table is of course a list of entries, each describes a quantity, for example "airTemperature" http://codes.wmo.int/common/c-15/me/airTemperature.  Looking at the table, there is a field "generalization" with value "ThermodynamicTemperature" that links to http://qudt.org/vocab/quantity#ThermodynamicTemperature.

This is a link to QUDT.  The top page describes only SI and CGS systems, but there seems to be care for other conventional units. 

2013-12-06

ambiguity in pressure level heights of TAC TEMP which really casued trouble


It comes to attention recently that unnatural values in geopotential height is sometimes reported in BUFR TEMP message for 89532 SYOWA in Antarctica. That message is converted by RTH Tokyo from the traditional alphanumeric code (TAC) FM 35. The issues is partly a problem in the conversion software (handling of negative values), but also partly stemming from inherent ambiguity in the TAC TEMP format; location-independent algorithm fails to estimate of "upper digits" especially on 700 hPa.
The TAC/BUFR conversion is commonly seen worldwide, and other converter might have that problem, though the situation is not surveyed yet.