Unicode support should be a default feature in any parser generator, but many of them are using their custom non-standard syntax that requires care to support Unicode. Some are allowing at most 16 bit Unicode code points in the grammar syntax, that forces the developer to write per byte Unicode character recognition.
Tunnel Grammar Studio is using the Augmented BNF (ABNF) Internet Standard 68 syntax, defined in RFC 5234 and updated by RFC 7405 for case-sensitive string support. It allows the developer directly to write grammars with a full Unicode support.
There are many different character decoders available to be included into the generated parser: UTF-8, UTF-16/32 (little and big endian) as well as a Universal decoder that will first check the input stream for UTF-8/16/32 and then fall back automatically to ISO 8859-1 (Latin-1) or to ASCII depending from the compiled decoders.
All of this is available out of the box, without the need, the user to write any additional code, but the grammar and to choose the required input decoders. This flexibility ensures that the generated parser can be deployed universally and internationally, no matter what input format the end-user will use.

For more information: Tunnel Grammar Studio

At 2019 Nov 17, 16:44 GMT in news channel Versions