| Commit message (Collapse) | Author | Age | Files | Lines | |
|---|---|---|---|---|---|
| * | [YQL-17573] Fix handling groping sets + window functions/top level dstinct ↵ | Andrey Neporada | 2024-04-03 | 19 | -180/+376 |
| | | | | | (#3415) | ||||
| * | YQL-17766 Fix decimal incompatibility between yqlrun and yt (#2631) | kinash-varvara | 2024-04-03 | 13 | -58/+192 |
| | | |||||
| * | Fixed TPCH schema in CLI (#3438) | Pavel Velikhov | 2024-04-03 | 1 | -10/+10 |
| | | |||||
| * | failed status code metrics and logs have been added YQ-2535 (#2805) | Oleg Doronin | 2024-04-03 | 22 | -68/+415 |
| | | |||||
| * | Temporary restore old config examples due to #3434 (#3448) | Ivan Blinkov | 2024-04-03 | 6 | -80/+451 |
| | | |||||
| * | fix race condition (#3440) | ivanmorozov333 | 2024-04-03 | 8 | -51/+87 |
| | | |||||
| * | Decrease default value of SharedCacheSizeMb to 1024 (#3420) | Nikolay Shestakov | 2024-04-03 | 1 | -2/+1 |
| | | |||||
| * | Enable quoting disk space usage by storage pool kind (#2678) | Daniil Demin | 2024-04-03 | 24 | -125/+1382 |
| | | | | | | | | | | | | | | | | | | | KIKIMR-18302 Users pay differently for HDD and SSD storage. They create storage pools differentiated by the underlying storage kind for their databases. Moreover, they can specify the preferred storage kind for each column in a table (see [column groups](https://ydb.tech/docs/en/yql/reference/syntax/create_table#column-family) in the docs for the CREATE TABLE statement). However, up until this PR they didn't know, how much storage was used on each of the storage pool kinds. (And we didn't have storage pool kinds quotas to disable writes to the database, which exceeded the limit on one of its storage pools.) We would like to provide users with an aggregate of the disk space usage of the database so they can order additional disks before the space is physically depleted. This is done by aggregating the [by channel disk space usage statistics](https://github.com/ydb-platform/ydb/blob/7a673cf01feefbe95bf5e7396d9179a5f283aeba/ydb/core/protos/table_stats.proto#L57) that the SchemeShard receives from the data shards (see [TEvPeriodicTableStats](https://github.com/ydb-platform/ydb/blob/7a673cf01feefbe95bf5e7396d9179a5f283aeba/ydb/core/protos/tx_datashard.proto#L789)). Channels are mapped to the corresponding storage pool kinds via the information that the SchemeShard has about the database (in code databases are subdomains) and the storage pools it was created with. Aggregation is done on two levels: by tables and by database. Aggregate by the table path can be seen in the UI in the path description of the table under the Describe -> PathDescription -> TableStats -> StoragePools field. Aggregate by the database can be seen in the UI in the Describe -> PathDescription -> DomainDescription -> DiskSpaceUsage -> StoragePoolsUsage field. In addition, we implemented "storage_pools_quotas" that the user can specify in the "DatabaseQuotas" section of the config of the database that the user would like to create. There are 3 parameters in each [storage pool quota](https://github.com/jepett0/ydb/blob/a19c3b4dcc28fb1da6d04ecfb139ffdfe90c72fb/ydb/public/api/protos/ydb_cms.proto#L98): - pool kind, - hard quota (if any storage pool exceeds its hard quota, writes to the **whole** database (not just the storage pool that has exceeded the quota!) are restricted), - soft quota (if all storage pools use less storage than the corresponding soft quota, then the database opens for writing again). "storage_pools_quotas" can be used together with the existing ["data_size_hard_quota"](https://github.com/jepett0/ydb/blob/a19c3b4dcc28fb1da6d04ecfb139ffdfe90c72fb/ydb/public/api/protos/ydb_cms.proto#L82) and ["data_size_soft_quota"](https://github.com/jepett0/ydb/blob/a19c3b4dcc28fb1da6d04ecfb139ffdfe90c72fb/ydb/public/api/protos/ydb_cms.proto#L88) that do not differentiate between storage pools. Exceedance of __any__ hard quota (either the storage pool one, or the entire "data_size_hard_quota") disables writes to the database. To reenable writes, __all__ disk space usage (either the [storage pool one](https://github.com/jepett0/ydb/blob/a19c3b4dcc28fb1da6d04ecfb139ffdfe90c72fb/ydb/core/tx/schemeshard/schemeshard_info_types.h#L1460), or the aggregated [TotalSize](https://github.com/jepett0/ydb/blob/a19c3b4dcc28fb1da6d04ecfb139ffdfe90c72fb/ydb/core/tx/schemeshard/schemeshard_info_types.h#L1452)) must be below the corresponding soft quota. One important thing to note about the storage pools usage statistics is that it is delivered to the SchemeShard with a considerable delay (about 1 minute). This means that the storage pools usage will be checked against the storage pools quotas with a delay and some data can be written above the hard limit. (And the other way around too: deleting some data to open the database for writes will be noticed by the SchemeShard with a considerable delay (about 420 seconds in my tests with a default compaction policy, I don't know where this number comes from). This is due to the fact that the new data is stored in the LSM tree (I guess) and is written to the appropriate storage pool later, after compaction. | ||||
| * | KIKIMR-20945: pin old configs in manual instructions (#3434) | Ivan Blinkov | 2024-04-03 | 2 | -6/+6 |
| | | |||||
| * | sequential read for huge portions (limits for raw bytes in mem) (#3418) | ivanmorozov333 | 2024-04-03 | 35 | -395/+975 |
| | | |||||
| * | YQ-2068 common TAsyncInputTransformHelper for both sync and async Compute ↵ | zverevgeny | 2024-04-03 | 4 | -52/+46 |
| | | | | | Actors (#3410) | ||||
| * | Fix SqlParsingOnly.AlterTableAddIndexWithIsNotSupported (#3431) | Mikhail Surin | 2024-04-03 | 1 | -1/+1 |
| | | |||||
| * | Use designated initializers to pass arguments in to KQP planer (#3385) | Daniil Cherednik | 2024-04-03 | 5 | -67/+92 |
| | | | | Co-authored-by: Vlad Kuznetsov <va.kuznecov@physics.msu.ru> | ||||
| * | log converters errors (#3402) | Олег | 2024-04-03 | 5 | -36/+60 |
| | | |||||
| * | Changed default behaviour of external_throttling section (#3408) | Oleg Shatov | 2024-04-03 | 2 | -18/+2 |
| | | |||||
| * | Fix block minmax aggregation (#3407) | amikish | 2024-04-03 | 2 | -0/+8 |
| | | |||||
| * | Changed YQL TPCH schema to accomodate long ints (#3417) | Pavel Velikhov | 2024-04-03 | 1 | -10/+10 |
| | | |||||
| * | Add tracing to load actor (#3395) | Sergey Belyakov | 2024-04-03 | 3 | -4/+32 |
| | | |||||
| * | early finish has bee fixed (for another in-flight TEvOutputChannelDat… (#3384) | Oleg Doronin | 2024-04-03 | 1 | -2/+1 |
| | | |||||
| * | Return .mapping.json to keep libraries automergeable (#3413) | AlexSm | 2024-04-03 | 1 | -0/+13 |
| | | |||||
| * | limit retries in starting state (#3394) | Vitalii Gridnev | 2024-04-03 | 1 | -0/+8 |
| | | |||||
| * | add helpers for huge portions scan processing (#3396) | ivanmorozov333 | 2024-04-03 | 41 | -283/+1092 |
| | | |||||
| * | Bigdate native YT types support (#3324) | amikish | 2024-04-03 | 3 | -10/+28 |
| | | |||||
| * | YDB-633: continue reorganizing docs (#3353) | Ivan Blinkov | 2024-04-03 | 257 | -2770/+1073 |
| | | |||||
| * | [docs] update docs_release.yaml (#3406) | Ivan Blinkov | 2024-04-03 | 1 | -1/+0 |
| | | |||||
| * | Fix types for blind deletes (#3404) | Mikhail Surin | 2024-04-03 | 2 | -2/+45 |
| | | |||||
| * | YDB FQ: basic integration tests for YDB as an external data source (#3399) | Vitaly Isaev | 2024-04-02 | 49 | -166/+616 |
| | | |||||
| * | fix csv reading (#3403) | Олег | 2024-04-02 | 1 | -1/+1 |
| | | |||||
| * | Update CODEOWNERS for /ydb/library/yql/yt (#3380) | Aleksandr Gaev | 2024-04-02 | 1 | -1/+2 |
| | | |||||
| * | Add dstool different token types support (#3397) | Robert Drynkin | 2024-04-02 | 1 | -16/+25 |
| | | |||||
| * | [YQL-17573] Support more callables in FlatMap over Extend optimizer (#3383) | Andrey Neporada | 2024-04-02 | 1 | -2/+2 |
| | | |||||
| * | Fix viewer in Safari (#2041) | Semyon Danilov | 2024-04-02 | 2 | -8/+20 |
| | | |||||
| * | YDB-2822 Fix UUID column usage in CDC (#3229) | Semyon Danilov | 2024-04-02 | 9 | -43/+176 |
| | | |||||
| * | Like to find/grep YQ-3015 (#3390) | Oleg Doronin | 2024-04-02 | 3 | -3/+3 |
| | | |||||
| * | Fixed double finishing of spans in grpc request proxy (#3208) | Oleg Shatov | 2024-04-02 | 3 | -37/+51 |
| | | |||||
| * | Support returning list in yql v1 (#2784) | Mikhail Surin | 2024-04-02 | 15 | -39/+305 |
| | | |||||
| * | TFederatedTopicWriteSession: use SelfContext instead of shared_ptr (#3372) | qyryq | 2024-04-02 | 3 | -101/+210 |
| | | |||||
| * | Make clang-14 happy (#3389) | Daniil Cherednik | 2024-04-02 | 1 | -1/+1 |
| | | | | Not for changelog (changelog entry is not required) | ||||
| * | sync merge (#3387) | Олег | 2024-04-02 | 0 | -0/+0 |
| | | |||||
| * | Merge (#3386) | Олег | 2024-04-02 | 0 | -0/+0 |
| | | |||||
| * | return errors in case wrong schema in bulk update requests (#3327) | Олег | 2024-04-02 | 18 | -91/+211 |
| | | |||||
| * | YQ-2918: propagate stats aggregation from plan for V2 (#3367) | Egor Zudin | 2024-04-02 | 10 | -24/+188 |
| | | |||||
| * | File downloader: add logs (#3376) | Alexey Ozeritskiy | 2024-04-02 | 2 | -2/+3 |
| | | |||||
| * | Fix a race between mediator state and pipeline restoring active transactions ↵ | Aleksei Borzenkov | 2024-04-02 | 4 | -22/+288 |
| | | | | | (#3358) | ||||
| * | Add balancing monitoring (#3265) | Robert Drynkin | 2024-04-02 | 4 | -0/+13 |
| | | |||||
| * | Support of Interval type in ToPg (#3368) | Vitaly Stoyan | 2024-04-02 | 14 | -0/+207 |
| | | |||||
| * | Changed interconnect disconnected handler in tenantinfo handler (#2813) | Andrei Rykov | 2024-04-02 | 1 | -105/+130 |
| | | |||||
| * | Disable style tests in ydb/library/yql/providers/generic/connector/tests (#3373) | Vitaly Isaev | 2024-04-02 | 9 | -15/+1 |
| | | |||||
| * | Fix memleak: cyclic smart pointers (+ logs) (#3370) | Alexey Ozeritskiy | 2024-04-02 | 1 | -5/+9 |
| | | |||||
| * | Missed peerdir (#3364) | Hor911 | 2024-04-02 | 1 | -0/+1 |
| | | |||||
 |
index : ydb | |
| Mirror of YDB github repos |
| aboutsummaryrefslogtreecommitdiffstats |