YQL fix complete logic

## Why these changes are proposed?

I figured out that all this time I did not understand how `antlr4-c3` really works.

It is configured with

```ts
ignoredTokens: Set<number>
preferredRules: Set<number>
```

On completion, it returns

```ts
class CandidatesCollection {
    tokens: Map<number, TokenList>
    rules: Map<number, ICandidateRule>
}

class ICandidateRule {
    startTokenIndex: number
    ruleList: RuleList
}

type TokenList = number[]
type RuleList = number[]
```

I thought that `rules` is a mapping from a `TokenId` to a `ParserCallStack`, but it totally is not true, as `rules` key is in fact a `RuleId`.

The documentation says

> Whenever the c3 engine hits a __lexer token__ ... it will check the call stack for it and, if that contains any of the **preferred rules**, will select that **instead** of the __lexer token__.

> [Preferred] Rules which replace any candidate token they contain.

So when we add the rule `X` to `preferredRules`, then when `C3` hits a lexer token with a `Parser Call Stack` containing the rule `X`, it will not add it to `CandidatesCollection::tokens`, but instead it will add an entry to `CandidatesCollection::rules` with a `Parser Call Stack`. It used when we have `ID_PLAIN` in a grammar, but this `ID_PLAIN` has different meaning depending on the context (`Parser Call Stack`), e.g. it can be a Table Name, Column Name and so on... So we can ask C3 not to include `ID_PLAIN` in `CandidatesCollection::tokens`, but instead emit `rules` like `table_id`, `column_id` and so on into the `CandidatesCollection::rules`.

So we have a `Parser Call Stack` for `rules`, but not for `tokens`.

## How it works correctly now then?

Described in the comments section.

## How to live on?

- [BAD ] Make a rule for each token to capture keywords at `CandidatesCollection::rules`.
- [GOOD] Extend `antlr4-c3` to include `Parser Call Stack` for `CandidatesCollection::tokens`.

---

Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1131
commit_hash:1a954f55098b9c090ab87e88f8bee61d9ff319ed

0 files changed, 0 insertions, 0 deletions