LSP Error Testing
Introduction
Conjure Oxide includes a server that uses a Language Server Protocol (LSP) to check Essence files for errors before the file is parsed. The LSP server communicates with the Diagnostics API to check for errors, and it will return the error message along with the range of where the error occurred in the given Essence file. The server will then use the diagnosis to perform error underlining, and syntax and semantic highlighting (more details in LSP Documentation and Diagnostics API documentation).
There are two types of error when trying to diagnose a given Essence file, which are syntactic and semantic errors:
- Syntactic errors are errors that stem from the tokens in the Essence file not being in the correct syntax when given into the parser.
- Semantic errors are errors that may pass the syntactic error checking (i.e. the file lines have the correct syntax), but ultimately is unable to be parsed due to errors relating to the entire context of the file.
To be able to accurately diagnose the Essence files for errors, tests cases have been written for situations that Diagnostics API and Native parser might encounter during diagnosing and parsing, referencing the Essence Error Classification).
Testing
Parser
Test cases that test the coverage of the Native parser use Roundtrip testing. The syntax-error and semantic-error test directories exist in the test-integration crate as child directories of roundtrip\native-errors. Furthermore, config.toml sets only the Native parser to be used when testing test cases.
These test cases will run alongside all the integration tests in the tests-integration crate, which can be done by using:
cargo test
Alternatively, to run only the Roundtrip tests, use the command:
cargo test tests_roundtrip
Diagnostics API
For testing the Diagnostics API, test cases are written in the tests directory of the conjure-cp-essence-parser crate, which uses normal Rust unit testing. To run these tests, use the command:
cargo test -p conjure-cp-essence-parser
Or, to run the test files individually, use the command:
cargo test -p conjure-cp-essence-parser --test {file_name}
The sections below will categorise all test cases into one of the categories based on the error classification, with an explanation of the purpose of each test case.
Native parser error tests
| Test name | Error category | Input essence | Purpose |
|---|---|---|---|
missing-token-01 | Syntax error: Missing token | find x: | To detect missing domains |
missing-token-02 | Syntax error: Missing token | find: bool | To detect missing identifiers |
missing-token-03 | Syntax error: Missing token | letting x be | To detect missing values |
missing-token-04 | Syntax error: Missing token | find x,y,z: int(1..3) such that x = y = such that z = 3 | To detect missing operators in comparison statements |
missing-token-05 | Syntax error: Missing token | find x: int(1..3 | To make sure that a domain is properly closed |
missing-token-06 | Syntax error: Missing token | find x bool | To detect missing colons |
unexpected-token-01 | Syntax error: Unexpected token | find x: int(1..3x) | To detect unexpected variables in a domain’s range |
unexpected-token-02 | Syntax error: Unexpected token | find x: int(1..3)) | To detect unexpected closing parentheses |
malformed-top-level-01 | Syntax error: Malformed top level statement | find a,b,c: int(1..3) print a | To make sure unknown commands will not be parsed |
invalid-domain-01 | Semantic error: Invalid domain | find x: int(10..1) | To make sure that the start of a domain cannot be greater than the end of the domain |
invalid-index | Semantic error: Invalid index | letting s be (0,1,1,0) letting t be (0,0,0,1) find a : bool such that a = (s[5] = t[1]) | To make sure that indexes that are out of range will raise an error |
keyword-as-identifiers-01 | Keyword as identifier | find bool: bool | To make sure that the keyword ‘bool’ cannot be used as an identifier |
keyword-as-identifiers-02 | Semantic error: Keyword as identifier | find letting,b,c: int(1..3) | To make sure that the keyword ‘letting’ cannot be used as an identifier |
type-mismatch | Semantic error: Type Mismatch | letting y be false find x: int(5..10) such that 5 + y = 6 | To make sure that if the type of a variable is incorrect for an expression, it will raise an error |
unsafe-division | Semantic error: Unsafe division | find x: int(5..10) such that x/0 = 3 | To make sure that expressions with division over zero will raise an error |
Note: tests that have not been implemented will have an input file with a .disabled extension, these will be removed once the features are implemented.
Diagnostics API error tests
| Test name | Error category | Input essence | Purpose |
|---|---|---|---|
detects_keyword_as_identifier_find | Syntax error: Keyword as identifiers | find find,b,c: int(1..3) | To make sure that the keyword ‘find’ cannot be used as an identifier |
detects_keyword_as_identifier_letting | Syntax error: Keyword as identifiers | find letting,b,c: int(1..3) | To make sure that the keyword ‘letting’ cannot be used as an identifier |
detects_keyword_as_identifier_bool | Syntax error: Keyword as identifiers | find bool: bool | To make sure that the keyword ‘bool’ cannot be used as an identifier |
invalid_top_level_statement_expression | Syntax error: Malformed top level statements | a,a,b: int(1..3) | To make sure that a missing ‘find’ keyword will cause a top level error |
malformed_find_2 | Syntax error: Malformed top level statements | find >=lex,b,c: int(1..3) | To make sure that complex operators cannot be used as an identifier |
malformed_find_3 | Syntax error: Malformed top level statements | find +,b,c: int(1..3) | To make sure that operators cannot be used as an identifier |
unexpected_colon_used_as_identifier | Syntax error: Malformed top level statements | find :,b,c: int(1..3) | To make sure that colons cannot be used as an identifier |
missing_colon_domain_in_find_statement_1st_line | Syntax error: Malformed top level statements | find x | To make sure that missing the colon and domain in a ‘find’ statement will cause a top level error |
missing_colon_domain_in_find_statement_2nd_line | Syntax error: Malformed top level statements | find x: int(1..3) find y | To make sure that malformed top level checks work not only for the first line |
unexpected_print_2nd_line | Syntax error: Malformed statements Syntax error: Unexpected token | find a,b,c: int(1..3) print a | To make sure that unknown commands will not be parsed |
missing_identifier | Syntax error: Missing token | find: bool | To detect missing identifiers |
missing_colon | Syntax error: Missing token | find x bool | To detect missing colons |
missing_domain | Syntax error: Missing token | find x: bool find y: | To detect missing domains |
missing_constraint | Syntax error: Missing token | find x: bool such that | To detect missing constraints |
multiple_missing_tokens | Syntax error: Missing token | find x: int(1..3 letting x be | To detect multiple missing tokens |
missing_domain_in_tuple_domain | Syntax error: Missing token | find x: tuple() | To detect missing domains in tuple domains |
missing_operator_in_comparison | Syntax error: Missing token | find x: int such that 5 = | To detect missing operators in comparison statements |
missing_right_operand_in_and_expr | Syntax error: Missing token | find x: int such that x /\ | To detect missing right operand in an ‘and’ expression |
missing_period_in_domain | Syntax error: Missing token | find a: int(1.3) | To detect if a domain is missing a period |
unexpected_closing_paren | Syntax error: Unexpected token | find x: int(1..3)) | To detect unexpected closing parentheses |
unexpected_identifier_in_range | Syntax error: Unexpected token | find x: int(1..3x) | To detect unexpected variables in a domain’s range |
unexpected_semicolon | Syntax error: Unexpected token | find x: int(1..3) such that x = 6; | To detect unexpected semicolons at the end of an expression |
unexpected_extra_comma_in_find | Syntax error: Unexpected token | find x,, y: int(1..3) | To detect extra commas in the variable list of a ‘find’ statement |
unexpected_token_in_implication | Syntax error: Unexpected token | find x: int(1..3) such that x -> %9 | To detect unexpected token in an implication expression |
unexpected_token_in_matrix_domain | Syntax error: Unexpected token | find x: matrix indexed by [int, &] of int | To detect unexpected token in a matrix domain |
unexpected_token_in_set_literal | Syntax error: Unexpected token | find x: set of int such that x = {1, 2, @} | To detect unexpected token in a set literal |
multiple_unexpected_tokens | Syntax error: Unexpected token | find x: set of int; such that x = {1, 2, @} | To make sure that the syntactic checker can detect multiple unexpected tokens |
unexpected_x_in_all_diff | Syntax error: Unexpected token | find a : bool such that a = allDiff([1,2,4,1]x) | To detect unexpected tokens in all diffs |
unexpected_int_at_the_end | Syntax error: Unexpected token | find a : bool such that a = allDiff([1,2,4,1])8 | To detect unexpected tokens at the end of a line |
unexpected_token_in_identifier | Syntax error: Unexpected token | find v@lue: int(1..3) | To detect unexpected tokens in a variable name |
detects_undefined_variable | Semantic error: Undefined variable | find x: int(1..10) such that x = y | To make sure that undefined variables will raise an error |
no_errors_for_valid_code | None | find x, y: int(1..10) such that x + y = 10 | To make sure that the semantic checker will not give an error when there is none |
range_points_to_error_location | None | find x: int(1..10) `such that x = undefined_var | To make sure that the range given by the semantic checker is the range of the error |
domain_start_greater_than_end | Semantic error: Invalid domain | find x: int(10..1) | To make sure that the start of a domain cannot be greater than the end of the domain |
incorrect_type_for_equation | Semantic error: Type mismatch | letting y be false find x: int(5..10) such that 5 + y = 6 | To make sure that if the type of a variable is incorrect for an expression, it will raise an error |
dividing_over_zero | Semantic error: Unsafe division | find x: int(5..10) such that x/0 = 3 | To make sure that expressions with division over zero will raise an error |
invalid_index | Semantic error: Invalid index | letting s be (0,1,1,0) letting t be (0,0,0,1) find a : bool such that a = (s[5] = t[1]) | To make sure that indexes that are out of range will raise an error |
duplicate_declaration_of_variable | Semantic error: Redefined variable | find x: int(1..10) find x: int(2..3) | To make sure that redefining a variable will raise an error |
extra_comma_in_variable_list | Semantic error: Unexpected token | find x,: int(1..10) | To make sure that having an extra comma in the variable list is not allowed |
Note: currently all semantic tests are set to be ignored, as they have not been implemented yet. Once they have been implemented, the #[ignore] will be removed.