Unicode support should be a default feature in any parser generator, but many of them are using their custom non-standard syntax that requires care to support Unicode. Some are allowing at most 16 bit Unicode code points in the grammar syntax, that forces the developer to write per byte Unicode character recognition.
Tunnel Grammar Studio is using the Augmented BNF (ABNF) Internet Standard 68 syntax, defined in RFC 5234 and updated by RFC 7405 for case-sensitive string support. It allows the developer direct… Read morely to write grammars with a full Unicode support.
There are many different character decoders available to be included into the generated parser: UTF-8, UTF-16/32 (little and big endian) as well as a Universal decoder that will first check the input stream for UTF-8/16/32 and then fall back automatically to ISO 8859-1 (Latin-1) or to ASCII depending from the compiled decoders.
All of this is available out of the box, without the need, the user to write any additional code, but the grammar and to choose the required input decoders. This flexibility ensures that the generated parser can be deployed universally and internationally, no matter what input format the end-user will use.

For more information: Tunnel Grammar Studio

At 2019 Nov 17, 16:44 GMT in news channel Versions

More and more devices in current times are multi-core and it requires the software developers to integrate up-to-date solutions for the programs running on these devices to satisfy the user expectations for a high performance.
Tunnel Grammar Studio generates parsers that can be run in a single thread (executed in steps or till completion: error/success) or in up to three threads for multi-threaded parsing, where each thread operates on specific part of the parsing pipeline.
This technology provides more speed… Read more for parsing large grammars, that are available for advanced domain specific languages and serialized data structures. The tasks in the generated parsing machines are simultaneous and parallelized, so the usage of the available CPU cores is better. There is 5 different architectures of the Parsing Machine, based on its 3 main modules: Lexer - the lexical analysis that groups characters into tokens, Parser - that is recognizing the tokens as a valid sequence of the language and the Builder - that constructs the abstract or concrete syntax tree.

For more information: Tunnel Grammar Studio

At 2019 Nov 17, 16:32 GMT in news channel Versions

Tunnel Grammar Studio can generate a parser from an ABNF grammar. Later on, the generated parser, as a part of the client software, will perform the parsing on the supplied user input. Often the structure of parsers, whatever mechanism they use, prevent them from identifying the error type and location precisely. Tunnel Grammar Studio has build in options to generate parsers with different error detection mechanisms. Before the compilation there are multiple options to be chosen: error location tracking, th… Read moree found erroneous token and, the possible valid tokens that were expected at this location. There are different options how the error location will be reported: line/char offset - the location structure contains line and Unicode character offset in the line, code point and byte offsets from the input stream start. Each automatically generated syntax tree (abstract or concrete) node may have its own start/final location information at the input stream. For the visitor syntax tree architect model, on each event (start/final for rules, groups and tokens) the current input location is dynamically available, which makes it easy to generate custom syntax trees.

For more information: Tunnel Grammar Studio

At 2019 Nov 17, 16:28 GMT in news channel Versions

While using the parser generator, proper error reporting is important for its usability. In case the error is not reported with details or its location is not clear, it may slow down the working process and make the developer spend unnecessary time searching for the error / guessing what the issue may be.

Tunnel Grammar Studio checks first the syntax errors at compilation time, then an analysis is made to detect references to missing rules, invalid repetition ranges and rules recognition collisions. At compi… Read morele time, after the syntax is checked, an analysis is initiated to detect LL(k>1) collisions. All found errors and warnings plus additional relevant messages are logged into the GUI log list. The error messages are clearly explained and its automatic localization allows their fast correction.

For more information: Tunnel Grammar Studio

At 2019 Nov 17, 16:28 GMT in news channel Versions

Scalable Vector Graphic (SVG) export is now available for the grammar layout visualization and debugger syntax trees views.

At 2018 Sep 10, 13:00 GMT in news channel Versions