| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
| |
commit_hash:67576d4a9e4a1f4bbbb204bcb0b98c255009b1da
|
| |
|
|
|
|
|
|
| |
When table `folder/table` exists. On prefix
``` SELECT * FROM `folder/ ``` users want to
accept ``` table` ``` rather than just `table`.
This patch is about it.
commit_hash:12d36cbf037db91f49136ab8e013b160a28b5b1b
|
| |
|
|
| |
commit_hash:56c7deefce71a95f10a4738b0a51e6ba384dd983
|
| |
|
|
| |
commit_hash:6d67ec1fa5023083debd89aaa99950019ca37c90
|
| |
|
|
|
| |
Added support for token filtration. It is needed because YQL grammar supports multiple systems with various capabilities. For example, some of those do not support `INTERSECT ALL` and therefore `ALL` after `INTERSECT` should be *disabled*. Another example is a support of an only `EVALUATE FOR`, so `FOR` is *forced* to be precendenced by `EVALUATE`.
commit_hash:a926c8d3911f8dd025ab58eb7f81e2370fcd9376
|
| |
|
|
|
| |
Added support for a columns completion at a simple select. For example, `` SELECT # FROM hahn.`home/yql/tutorial/users` `` и `` USE hahn; SELECT $ FROM `home/yql/tutorial/users` ``.
commit_hash:2254449e91255c19792a1dc521825e44dda7d63b
|
| |
|
|
|
| |
Instead of relying on keyword sequences provided by C3 and using hacks like pattern matching on keyword sequences (`format.cpp`), emit composite types manually.
commit_hash:999d8d4b514433fbdf7885aa0122b2010e985efb
|
| |
|
|
|
|
|
|
|
|
|
|
| |
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/62
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1323
commit_hash:ebab6c3290ba984174c85bba35eeb066b53af5aa
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Clang Tidy is cool, but too slow ;(. Btw, should we respect `misc-include-cleaner` check?
```bash
ya dump compile-commands > /tmp/cc/compile_commands.json
find . -type f \( -name "*.cpp" -o -name "*.h" \) -exec clang-tidy {} -p /tmp/cc {} -checks="misc-*" \;
```
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1305
commit_hash:c2ecf638062dcdd0730fed3381da4a230f5352c3
|
| |
|
|
| |
commit_hash:6476b453be16b7db003e75789e3b630c1168f14f
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Introduced scoped name collection at global analysis
- Added `BingingsNameService` constructed after global analysis and united with the base service.
- Added dollar detection at local analysis.
After this point design should be reexamined and module should be refactored.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/57
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1304
commit_hash:a7bf41703f0ee846e359407b53a5d62fc05928ea
|
| |
|
|
| |
commit_hash:ab24b3db82789b77ad465efa64de93c5eab43629
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Completion candidates comes unclosed from right and with zeroed `CursorShift`. In future there will be an atomic patch that will close candidates and set `CursorShift` correctly. Not it is zeroed to integrate with clients (e.g. YDB CLI).
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/53
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1299
Co-authored-by: vvvv <[email protected]>
commit_hash:38bcd21f0d6959275af89a981fd9b296374aea51
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There was a mistake thinking that provider is a part of a cluster name. Actually `yt:saurus` and `saurus` are same clusters and provider (`yt`) serves only as a hint for translation expressions such as `yt:$cluster_name`.
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/54
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1301
commit_hash:19a45cf0bb6c7dda26747b2e3371646272f79bce
|
| |
|
|
|
|
|
|
|
|
|
| |
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/50
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1292
commit_hash:a6d543bc18b2f9239bf0b33afd8497a7919fe4b3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
I leaved the YDB configuration empty as need your feedback.
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/39
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1289
commit_hash:408c7e0f101027fc6ca3d86d251b5639fcae07bb
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The simplest solution actually is an SQL injection.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/ydb-platform/ydb/pull/18146
- Related to https://github.com/vityaman/ydb/issues/46
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1283
commit_hash:7043e7cdd1e0d378926783d31cf5f8e055393aaa
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Introduce global analysis which will be extended with more ParseTree analysis later.
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/40
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1281
commit_hash:848fb3fdd6bc9612769c47d66198fca018ff465f
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
On `SELECT * FROM pn` it completed to `SELECT * FROM ppnv1`, but expected ``` SELECT * FROM `pnv1` ```.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/44
- Related to https://github.com/ydb-platform/ydb/pull/18146
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1280
commit_hash:e937239fae7bf609bde6358b9b27c2b504c8ba32
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fixes superfluous candidates (`OPTIONAL`, `CALLABLE`, ...) at `SELECT * FROM #`.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1273
commit_hash:d34d81ea86b8b6b5d339174be20d14d0371c3de6
|
| |
|
|
|
|
|
|
|
|
|
|
| |
---
- Related to `YQL-19747`
- Related to https://github.com/vityaman/ydb/issues/8
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1268
commit_hash:76e4e2c8b225994cc842c1ee00a0ec8e6eed5c05
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Outcomes:
1. Need to think more deeply about library interfaces. If you think that an interface is redundant, think again, because author knows a usecase that you do not imagine.
2. If you are lazy now to implement something robustly, think twice because the day will come and this code
will fire.
The problem was that on input ``` ... cluster.`/yql/t#` ``` C3 receives only a prefix ``` ... cluster.`/yql/t ``` and a token stream produced was incorrect and therefore completions too.
---
- Related to `YQL-19747`
- Related to https://github.com/ytsaurus/ytsaurus/pull/1257
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1271
commit_hash:8bf5f2683c9e41963003da923f255f63b898aac5
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
---
- Related to `YQL-19747`
- On top of https://github.com/ytsaurus/ytsaurus/pull/1253
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/14
- Related to https://github.com/vityaman/ydb/issues/35
- Related to https://github.com/vityaman/ydb/issues/40
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1257
commit_hash:0b842abb27184c88b8177beeea29fb1ea86b7a04
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We faced with a failing assertion at `NYql::NormalizeName`. This patch expects it and fallbacks to `ToLowerUTF8`.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/38
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1255
commit_hash:946162904ce3b74f490debb7f9e0587f136b4075
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Essentially we need the `UnionNameService` to integrate dynamic name services into the `sql/v1/complete`. This decorator queries multiple name services and merges their responses. We will use it to union static `NameServices` and the `SchemaNameService` (and the `ClusterNameService`, and so on).
It does not deal with exceptions and fails the whole request on some subtask failed. Client should prepare robust children (later `SwallowingNameService` will be added to return an empty `NameResponse` on errors to provide best-effort dynamic object loading).
`StaticNameService` was split into micro-`NameService`s :)
`NameConstraints` are extracted to provide name qualification via `Qualified` and `Unqualified` methods. This is needed because depending on a context `NameService` can return unqualified names (for example, on `PRAGMA yt.#`). As internal indexes for scanning and ranking are built on a sorted list of, then `Ranking` actually needs an ability to get fully-qualified names, so now it via the `NameConstraints`.
Also this design potentially let us to improve internal indexes by using partitioning by a namespace. Other option was to make `PragmaName` and `FunctionName` more structured via adding a separate field for a namespace, but it seems to me that it will force to do more parsing from indexes results to a `Name`. Anyway this is an internal component so it can be changed if needed. I still doubt this decision because structured `PragmaName { Namespace, Identifier }` seems to be cleaner and there should be no noticeable overhead because of COW strings.
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1252
commit_hash:e8a1fbb17e2ca3ffe72aafbea943a38624d74491
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Clients might want to use custom `NameSet` and `FrequencyData` in their environment, for example, to get their private UDFs and to have more relevant ranking that includes their private UDF and respect their usage pattern.
To achieve this goal I decided to load pure `NameSet` and `FrequencyData` and provide functions for pruning. Also I checked defaults and decided that it is more common for a client to create a `StaticNameService` from pure `NameSet` and `FrequencyData` to keep their pruning consistent.
I also extracted a separate module `ranking`. It will be needed when I will implement `UnionNameService` to union `StaticNameService` and `SchemaNameService`. `UnionNameService` will load `Limit` names from each child and then crop them to sorted prefix of length `Limit` using `IRanking`.
---
- Related to `YQL-19747`
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/14
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1246
Co-authored-by: vvvv <[email protected]>
Co-authored-by: vvvv <[email protected]>
commit_hash:cdca301a58a34e56d537a447b4ff779cd70faea6
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- [x] Chained futures, add `CompleteAsync` method (then will migrate the YDB CLI on it).
- [x] Removed deadlined and fallback NameService as unused
- [x] Annotate thread-safe methods with `const` and use `AtomicSharedPtr` for them.
- [x] Move `name` to `name/service` with backward compatibility with the YDB CLI.
---
`CompletionEngine` is left thread-unsafe because of the dependency chain `CompletionEngine -> LocalSyntaxAnalysis -> C3Engine` which is thread-unsafe, but readonly indexed data structures such as `Ranking` and `NameService` are annotated with const and distributed via shared pointers.
I removed deadlined and fallback name services because the first is stupid the second is ahead of its time and is better to be added later to keep interfaces as minimal as possible.
---
The migration on async complete plan:
1. Introduce CompleteAsync
2. Migrate clients on CompleteAsync
3. Make Complete to return Future
4. Migrate clients on Complete
5. Remove CompleteAsync
---
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/33
- Related to https://github.com/vityaman/ydb/issues/31
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1241
Co-authored-by: vvvv <[email protected]>
Co-authored-by: vvvv <[email protected]>
commit_hash:497cc081ab78bebf7354e0acfaa418d936cc8240
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When I tried to implement a folder and object names completion at `ID_QUOTED` I faced with a problem, that I actually can't detect, that cursor is at `ID_QUOTED` token because `TCompletionInput::Text` it was cut until the `TCompletionInput::CursorPosition`, therefore at input ``` SELECT * FROM `#` ``` prefix was ``` SELECT * FROM `# ``` and then lexer failed.
While we actually want tokenize the whole current statement, `C3` still needs to receive a prefix as input. I tried to tokenize the whole statement and then on input `SELECT Optional<#>` got nothing because `<>` is solid token in the `SQL`. The only way to fix it I found is to cut a query to prefix until the cursor position.
BTW, current implementation is not so efficient as we tokenize the input multiple times. Especially `SplitQueryToStatemnts` seems heavy. In future we anyway will parse the whole input so will need to design APIs to receive ready token streams to do statements splitting, for example, just not to do the work twice.

So I introduce you the following changes
- [x] Select the whole current statement, not just prefix.
- [x] Find the token at caret and output no candidates when caret is at `STRING_VALUE`, `DIGIGTS` and so on.
- [x] Change `C3` wrapper interface to take `TCompletionInput` to hide an implementation detail that it runs on cut prefix.
- [x] `#` annotated queries in unit tests.
- [x] Detect `CaretTokenPosition` -- if is it enclosed with a token or between two.
- [x] Ensure that `maxErrors` in `ILexer::Tokenize` is positive. Just a tiny bugfix.
---
- Related to https://github.com/ytsaurus/ytsaurus/pull/1209
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/14
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1225
commit_hash:a434b9888ec8a7356247d63d9f1420e256ae4fca
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
I was lazy to search for a most frequent used name among equivalent by the relation `(a ~ b) iff (NormalizeName(a) = NormalizeName(b))`. Because it seems that names we receive from JSONs are canonized and therefore in a preferable style by the opinion of the YQL language designers. But because of duplicates at `statements_opensource.json` we have, for example, both `IGNORETYPEV3` and `IGNORE_TYPE_V3` in candidates list. I think that we should just remove `IGNORETYPEV3` from the JSON.
---
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/21
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1229
commit_hash:fe73374ae27df1fcacb0adccda930ec98ed1d7a6
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- [x] Rank keywords just by plain usages
- [x] `LocalSyntaxAnalysis` now returns a mapping `:: Keyword -> [Following Keywords]`.
- [x] Extracted keyword sequence formatting from `syntax/local` to `syntax/format`.
- [x] Extracted token display logic from `syntax/local` to `antlr4/vocabulary` as it is ANTLR dependent.
---
Example
```python
$ ./yql_complete <<< "select "
[Keyword] CAST(
[Keyword] NULL
[Keyword] NOT
[FunctionName] If(
[FunctionName] Yson::ConvertToString(
[FunctionName] Count(
[FunctionName] Sum(
[FunctionName] Unwrap(
[FunctionName] Coalesce(
[Keyword] DISTINCT
[Keyword] ALL
[Keyword] CASE
[FunctionName] Max(
[Keyword] FALSE
[FunctionName] Some(
```
---
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/17
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1197
commit_hash:f42cb4aaffe6de7c9137069c4d9c635ee110a805
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Forgot to support it.
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/24
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1195
commit_hash:a2f5937dbca5712f3ecbfccdf66662ce99e70619
|
| |
|
|
|
|
|
|
|
|
| |
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/vityaman/ydb/issues/19
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1189
commit_hash:7f1cb1dcf0617aa2c94c3f2188fc9bd481380252
|
| |
|
|
|
|
|
|
|
|
|
|
| |
- [x] Complete after PRAGMA
- [x] Complete multi-token names correctly, for example, `yt.` returns only `DisableStrict`, not `yt.DisableStrict` and `DateTime::` returns `Split`, not `DateTime::Split`.
I tried to implement it using `CompletedToken` edition, but not all completion environments support candidates with various `contextLen` (`Replxx` does not). So I decided that completions should rewrite only the current token, not sequences. For example, on `DateTime::Spl` rewrite only `Spl`. It makes sense as multi-token names have some namespace separated by a punctuation, so used types only namespace and gets names inside of it.
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1181
commit_hash:9d8967ac43b9348f6dbb53837d92a9dcc9b51f48
|
| |
|
|
|
|
|
|
|
| |
No description
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1185
commit_hash:1def5874ff6a9a5b3dcdd0ad285d2e64b16c9306
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
As I understand, type name should not be completed at `SELECT |`, so I added a check that we are at `invoke_expr` context.
Currently composite type keywords are suggested at `SELECT |` and also are uppercased. I will fix it separately when this merged during
- https://github.com/users/vityaman/projects/5?pane=issue&itemId=105056723&issue=vityaman%7Cydb%7C8
---
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Related to https://github.com/users/vityaman/projects/5/views/1?pane=issue&itemId=105056423&issue=vityaman%7Cydb%7C7
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1182
commit_hash:e87565867cf9fa82d9ac49a88d59b293d6686fe7
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Token sequences plan
- [x] [Easy] Support `GROUP BY`, `ORDER BY`.
- [x] [Easy] Support `Optional<`, `List<`, `Dict<`.
- [x] [Easy] Support `Avg(`, `Sum(`.
---
Co-authored-by: Victor Smirnov [[email protected]]
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1173
commit_hash:a443dec666c486fef7f891be04d68a786be83049
|
| |
|
|
|
|
|
|
|
|
|
|
| |
- [x] Fix bug with incorrect no-case sorting.
- [x] Get names from `sql_functions.json` and `types.json`.
- [x] Add types and functions ranking according to `rules_corr_basic.json` data via a `PartialSort`.
- [x] Add benchmark workspace.
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1167
commit_hash:84d93265fb69bf5651f905d6af038056657e9a16
|
| |
|
|
|
|
|
|
|
|
|
| |
- Function names are suggested now
- Changed the module structure
- Checking ruleIndex independence on mode (ansi | default) via unit tests
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1163
commit_hash:1b1a27d2cff8db663c5c7e8efb57896476823315
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- [x] Parse YQL grammar to extract lexer grammar into `TLexerGrammar`.
- [x] Translate `TLexerGrammar` into regexes.
- [x] Implement a lexer via regexes `TRegexLexer` to test generated regexes validity.
- [x] Test on `Default` syntax mode.
- [x] Test on `ANSI` syntax mode.
---
- Related to https://github.com/ydb-platform/ydb/issues/15129
- Requirement for https://github.com/ytsaurus/ytsaurus/pull/1112
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1127
commit_hash:03ffffe81cdafe7f93a4d3fd9a3212fe67f1c72d
|
| |
|
|
|
|
|
|
|
| |
- Related to https://github.com/ydb-platform/ydb/issues/9056
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1151
commit_hash:6e1e429a2ea805016bf00a1e60b501b7fc8dc8de
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When we run completion engine on multi-statement query, where preceding statements are syntactically incorrect, `antlr4-c3` does not return candidates. Running engine only on a current statement provides a best-effort attempt to provide candidates.
- Related to https://github.com/ydb-platform/ydb/issues/9056
- Depends on https://github.com/ytsaurus/ytsaurus/pull/1127 (`ELexerFlavor`)
---
Pull Request resolved: https://github.com/ytsaurus/ytsaurus/pull/1144
commit_hash:0ced9443a9712191f5420246531f781ca4bc5f42
|
|
|
commit_hash:5fc851d2c72810067fe0d407b66535b17de63129
|