Update grammars

Each programming language needs to be parsed in order to extract its syntax and semantic: the so-called grammar of a language. In rust-code-analysis, we use tree-sitter as parsing library since it provides a set of distinct grammars for each of our supported programming languages. But a grammar is not a static monolith, it changes over time, and it can also be affected by bugs, hence it is necessary to update it every now and then.

As now, since we have used bash scripts to automate the operations, grammars can be updated natively only on Linux and MacOS systems, but these scripts can also run on Windows using WSL.

In rust-code-analysis we use both third-party and internal grammars. The first ones are published on crates.io and maintained by external developers, while the second ones have been thought and defined inside the project to manage variant of some languages used in Firefox. We are going to explain how to update both of them in the following sections.

Third-party grammars

Update the grammar version in Cargo.toml and enums/Cargo.toml. Below an example for the tree-sitter-java grammar

tree-sitter-java = "x.xx.x"

where x represents a digit.

Run ./recreate-grammars.sh to recreate and refresh all grammars structures and data

./recreate-grammars.sh

Once the script above has finished its execution, you need to fix, if there are any, all failed tests and problems introduced by changes in the grammars.

Commit your changes and create a new pull request

Internal grammars

Update the version of tree-sitter-cli in the package.json file of the internal grammar and then install the updated version.

Update dependency version field in Cargo.toml and enums/Cargo.toml. Below an example for the tree-sitter-ccomment grammar

tree-sitter-ccomment = { path = "./tree-sitter-ccomment", version = "=x.xx.x" }

where x represents a digit.

Open the Cargo.toml file of the chosen grammar and:

  • Set its version to the same value present in the main Cargo.toml file
  • Increase the tree-sitter version to the most recent one

Run the appropriate script to update the grammar by recreating and refreshing every file and script.

For tree-sitter-ccomment and tree-sitter-preproc run ./generate-grammars/generate-grammar.sh followed by the name of the grammar. Below an example always using the tree-sitter-ccomment grammar

./generate-grammars/generate-grammar.sh tree-sitter-ccomment

Instead, for tree-sitter-mozcpp and tree-sitter-mozjs, use their specific scripts.

For tree-sitter-mozcpp, run

./generate-grammars/generate-mozcpp.sh

For tree-sitter-mozjs, run

./generate-grammars/generate-mozjs.sh

Once the script above has finished its execution, you need to fix, if there are any, all failed tests and problems introduced by changes in the grammars.

Commit your changes and create a new pull request