diff options
528 files changed, 10437 insertions, 2867 deletions
diff --git a/ydb/docs/en/core/best_practices/_includes/batch_upload.md b/ydb/docs/en/core/best_practices/_includes/batch_upload.md index 2791ab4d1c..2455c5a33a 100644 --- a/ydb/docs/en/core/best_practices/_includes/batch_upload.md +++ b/ydb/docs/en/core/best_practices/_includes/batch_upload.md @@ -1,32 +1,29 @@ -# Uploading large data volumes +# Uploading data to {{ ydb-short-name }} -This section provides recommendations on how to efficiently upload large amounts of data to {{ ydb-short-name }}. - -{% if oss == true %} - -An example of uploading data in [C++](https://a.yandex-team.ru/arc/trunk/arcadia/kikimr/public/sdk/cpp/examples/batch_upload). - -{% endif %} +This section provides recommendations on how to efficiently upload data to {{ ydb-short-name }}. There are anti-patterns and non-optimal settings for uploading data. They don't guarantee acceptable data uploading performance. To accelerate data uploads, consider the following recommendations: * Shard a table when creating it. This lets you effectively use the system bandwidth as soon as you start uploading data. - By default, a new table consists of a single shard. {{ ydb-short-name }} supports automatic table sharding by data volume. This means that a table shard is divided into two shards when it reaches a certain size. - The acceptable size for splitting a table shard is 2 GB. As the number of shards grows, the data upload bandwidth increases, but it remains low for some time at first. - Therefore, when uploading a large amount of data for the first time, we recommend initially creating a table with the desired number of shards. You can calculate the number of shards based on 1 GB of data per shard in a resulting set. +By default, a new table consists of a single shard. {{ ydb-short-name }} supports automatic table sharding by data volume. This means that a table shard is divided into two shards when it reaches a certain size. +The acceptable size for splitting a table shard is 2 GB. As the number of shards grows, the data upload bandwidth increases, but it remains low for some time at first. +Therefore, when uploading a large amount of data for the first time, we recommend initially creating a table with the desired number of shards. You can calculate the number of shards based on 1 GB of data per shard in a resulting set. * Insert multiple rows in each transaction to reduce the overhead of the transactions themselves. - Each transaction in {{ ydb-short-name }} has some overhead. To reduce the total overhead, you should make transactions that insert multiple rows. Good performance indicators terminate a transaction when it reaches 1 MB of data or 100,000 rows. - When uploading data, avoid transactions that insert a single row. +Each transaction in {{ ydb-short-name }} has some overhead. To reduce the total overhead, you should make transactions that insert multiple rows. Good performance indicators terminate a transaction when it reaches 1 MB of data or 100,000 rows. +When uploading data, avoid transactions that insert a single row. * Within each transaction, insert rows from the primary key-sorted set to minimize the number of shards that the transaction affects. - In {{ ydb-short-name }}, transactions that span multiple shards have higher overhead compared to transactions that involve exactly one shard. Moreover, this overhead increases with the growing number of table shards involved in the transaction. - We recommend selecting rows to be inserted in a particular transaction so that they're located in a small number of shards, ideally, in one. +In {{ ydb-short-name }}, transactions that span multiple shards have higher overhead compared to transactions that involve exactly one shard. Moreover, this overhead increases with the growing number of table shards involved in the transaction. +We recommend selecting rows to be inserted in a particular transaction so that they're located in a small number of shards, ideally, in one. +* If you need to push data to multiple tables, we recommend pushing data to a single table within a single query. +* If you need to push data to a table with a synchronous secondary index, we recommend that you first push data to a table and, when done, build a secondary index. * You should avoid writing data sequentially in ascending or descending order of the primary key. - Writing data to a table with a monotonically increasing key causes all new data to be written to the end of the table, since all tables in YDB are sorted by ascending primary key. As YDB splits table data into shards based on key ranges, inserts are always processed by the same server that is responsible for the "last" shard. Concentrating the load on a single server results in slow data uploading and inefficient use of a distributed system. +Writing data to a table with a monotonically increasing key causes all new data to be written to the end of the table, since all tables in YDB are sorted by ascending primary key. As YDB splits table data into shards based on key ranges, inserts are always processed by the same server that is responsible for the "last" shard. Concentrating the load on a single server results in slow data uploading and inefficient use of a distributed system. +* Some use cases require writing the initial data (often large amounts) to a table before enabling OLTP workloads. In this case, transactionality at the level of individual queries is not required and you can use ```BulkUpsert``` calls in the API and SDK. Since no transactionality is used, this approach has much lower overhead as compared to YQL queries. In case of a successful response to the query, the ```BulkUpsert``` method guarantees that all data added within this query is committed. {% note warning %} -When you create a table in the current version, you can only configure the sharding model (or Partitioning Policy) from the SDK for [Java](https://github.com/yandex-cloud/ydb-java-sdk) and [Python](https://github.com/yandex-cloud/ydb-python-sdk). +The ```BulkUpsert``` method isn't supported on tables with secondary indexes. {% endnote %} @@ -37,3 +34,4 @@ We recommend the following algorithm for efficiently uploading data to {{ ydb-sh 3. Partition the resulting data set by the number of shards in the table. Each part will contain a set of consecutive rows. 4. Upload the resulting parts to the table shards concurrently. 5. Make a ```COMMIT``` after every 100,000 rows or 1 MB of data. + diff --git a/ydb/docs/en/core/best_practices/_includes/paging.md b/ydb/docs/en/core/best_practices/_includes/paging.md index 661f281555..130ae7dba6 100644 --- a/ydb/docs/en/core/best_practices/_includes/paging.md +++ b/ydb/docs/en/core/best_practices/_includes/paging.md @@ -52,15 +52,17 @@ ORDER BY city, number LIMIT $limit; {% note warning "NULL value in key column" %} -In YDB, all columns, including key ones, may have a NULL value. Despite this, using NULL as key column values is highly discouraged, since the SQL standard doesn't allow NULL to be compared. As a result, concise SQL statements with simple comparison operators won't work correctly. Instead, you'll have to use cumbersome statements with IS NULL/IS NOT NULL expressions. +In {{ ydb-short-name }}, all columns, including key ones, may have a NULL value. Despite this, using NULL as key column values is highly discouraged, since the SQL standard doesn't allow NULL to be compared. As a result, concise SQL statements with simple comparison operators won't work correctly. Instead, you'll have to use cumbersome statements with IS NULL/IS NOT NULL expressions. {% endnote %} ## Examples of paginated output implementation {% if oss %} -* [C++](https://a.yandex-team.ru/arc/trunk/arcadia/kikimr/public/sdk/cpp/examples/pagination) + +* [C++](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp/examples/pagination) {% endif %} * [Java](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/src/main/java/com/yandex/ydb/examples/pagination) -* [Python](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/pagination) +* [Python](https://github.com/ydb-platform/ydb-python-sdk/tree/main/examples/pagination) +* [Go](https://github.com/ydb-platform/ydb-go-examples/tree/master/pagination) diff --git a/ydb/docs/en/core/best_practices/_includes/schema_design.md b/ydb/docs/en/core/best_practices/_includes/schema_design.md index d0c3aae9a2..45379bb56b 100644 --- a/ydb/docs/en/core/best_practices/_includes/schema_design.md +++ b/ydb/docs/en/core/best_practices/_includes/schema_design.md @@ -13,12 +13,12 @@ General recommendations for choosing a primary key: You should carefully choose a primary key and avoid situations where a small part of the database is under much heavier load than the rest of the database. -For example, since all tables in {{ ydb-short-name }} are sorted by ascending primary key, writes in a data table with a monotonically increasing primary key cause new data to be written to the end of the table. As YDB splits table data into partitions based on key ranges, inserts are always processed by the same server that is responsible for the "last" partition. Concentrating the load on a single server results in slow data uploading and inefficient use of a distributed system. +For example, since all tables in {{ ydb-short-name }} are sorted by ascending primary key, writes in a data table with a monotonically increasing primary key cause new data to be written to the end of the table. As {{ ydb-short-name }} splits table data into partitions based on key ranges, inserts are always processed by the same server that is responsible for the "last" partition. Concentrating the load on a single server results in slow data uploading and inefficient use of a distributed system. As an example, let's take logging of user events to a table with the ```( timestamp, userid, userevent, PRIMARY KEY (timestamp, userid) )``` schema. ```timestamp``` monotonically increases, and as a result, all records are written to the end of the table and the "last" partition, which is responsible for this range of keys, serves all records in the table. This leads to lower writing performance. -YDB supports automatic partition splitting when the threshold size is reached (usually about 2 GB). However, in our case, after splitting, a new partition starts taking all the write load and the situation repeats. +{{ ydb-short-name }} supports automatic partition splitting when the threshold size is reached (usually about 2 GB). However, in our case, after splitting, a new partition starts taking all the write load and the situation repeats. ## Techniques that let you evenly distribute load across table partitions {#balance-shard-load} @@ -31,10 +31,10 @@ Writing data to a table with the ```( timestamp, userid, userevent, PRIMARY KEY Let's take a table with the ```( timestamp, userid, userevent, PRIMARY KEY (userid, timestamp) )``` schema. As the entire primary key or its first component, you can use a hash of the source key. For example: ``` -( HASH(timestamp, userid), timestamp, userid, userevent, PRIMARY KEY (HASH(timestamp, userid), userid, timestamp) ) +( HASH(timestamp, userid), timestamp, userid, userevent, PRIMARY KEY (HASH(userid), userid, timestamp) ) ``` -If you select the hashing function correctly, the rows are distributed evenly enough throughout the key space, which results in an even load on the system. In this case, if the ```userid, timestamp``` fields are present in the key after ```HASH(timestamp, userid)```, this preserves data locality and sorting by time for a specific user. +If you select the hashing function correctly, the rows are distributed evenly enough throughout the key space, which results in an even load on the system. In this case, if the ```userid, timestamp``` fields are present in the key after ```HASH(userid)```, this preserves data locality and sorting by time for a specific user. ### Reducing the number of partitions affected by a single query {#decrease-shards} @@ -47,3 +47,4 @@ In {{ ydb-short-name }}, all columns, including key ones, may contain a NULL val ## Row size limit {#limit-string} To achieve high performance, we don't recommend writing rows larger than 8 MB and key columns larger than 2 KB to the DB. + diff --git a/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md b/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md index cb928162b2..89c016ee95 100644 --- a/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md +++ b/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md @@ -127,3 +127,4 @@ SELECT series_id, FROM series view views_index WHERE views == 0; ``` + diff --git a/ydb/docs/en/core/best_practices/_includes/table_sharding.md b/ydb/docs/en/core/best_practices/_includes/table_sharding.md index 0d48336513..2c53361206 100644 --- a/ydb/docs/en/core/best_practices/_includes/table_sharding.md +++ b/ydb/docs/en/core/best_practices/_includes/table_sharding.md @@ -1,14 +1,16 @@ # Table partitioning recommendations -YDB tables are sorted by primary key in ascending order. Tables are partitioned by splitting the range of key values into consecutive non-overlapping ranges called partitions. +{{ ydb-short-name }} tables are sorted by primary key in ascending order. Tables are partitioned by splitting the range of key values into consecutive non-overlapping ranges called partitions. -Thanks to partitioning, table data can be distributed across multiple storage devices and the load from operations on a table can use more processor cores and network bandwidth. However, too many partitions in a table may add overhead on RAM and CPU time usage. So optimal partitioning directly affects the efficiency of query execution. To achieve optimal partitioning, YDB provides tools for the initial partitioning of a table when creating it as well as methods for subsequent automatic partitioning. +Thanks to partitioning, table data can be distributed across multiple storage devices and the load from operations on a table can use more processor cores and network bandwidth. However, too many partitions in a table may add overhead on RAM and CPU time usage. So optimal partitioning directly affects the efficiency of query execution. To achieve optimal partitioning, {{ ydb-short-name }} provides tools for the initial partitioning of a table when creating it and methods for subsequent automatic partitioning. When creating a table, you can set the initial partitioning in one of two ways: + * Set up even partitioning by the first column of the primary key if this column is UInt32 or UInt64. In this case, you need to explicitly specify the number of partitions. * Set the exact partitioning keys that determine the number and boundaries of the original partitions. -YDB supports two methods for automatic table partitioning: +{{ ydb-short-name }} supports two methods for automatic table partitioning: + * By data size. * By load on a datashard (a system component that serves a table partition). @@ -17,3 +19,4 @@ Automatic partitioning modes can be enabled separately or together. Both modes c Automatic partitioning by size is parameterized by the value of the partition size setting. When that value is reach, the partition is split. The default value is 2 GB. A key for splitting is selected based on the histogram of data size distribution across partitions by key sub-ranges. If the total data size in adjacent partitions becomes less than half of the setting size, these partitions are merged. Automatic partitioning by load is triggered based on CPU usage by the datashard serving the partition. All datashards monitor their CPU usage. If a high (>50%) level of usage is detected at some point in time, the partition is split. A key is selected using statistics on accessing the keys of a datashard's own partition. + diff --git a/ydb/docs/en/core/best_practices/_includes/timeouts.md b/ydb/docs/en/core/best_practices/_includes/timeouts.md index 54c583612f..28754f481b 100644 --- a/ydb/docs/en/core/best_practices/_includes/timeouts.md +++ b/ydb/docs/en/core/best_practices/_includes/timeouts.md @@ -28,7 +28,7 @@ Timeout usage example: ```python from kikimr.public.sdk.python import client as ydb - + def execute_in_tx(session, query): settings = ydb.BaseRequestSettings() settings = settings.with_timeout(0.5) # transport timeout @@ -46,13 +46,13 @@ Timeout usage example: - C++ ```cpp - #include <kikimr/public/sdk/cpp/client/ydb.h> - #include <kikimr/public/sdk/cpp/client/ydb_table.h> - #include <kikimr/public/sdk/cpp/client/ydb_value.h> - + #include <ydb/public/sdk/cpp/client/ydb.h> + #include <ydb/public/sdk/cpp/client/ydb_table.h> + #include <ydb/public/sdk/cpp/client/ydb_value.h> + using namespace NYdb; using namespace NYdb::NTable; - + TAsyncStatus ExecuteInTx(TSession& session, TString query, TParams params) { return session.ExecuteDataQuery( query @@ -74,7 +74,7 @@ Timeout usage example: "a.yandex-team.ru/kikimr/public/sdk/go/ydb" "a.yandex-team.ru/kikimr/public/sdk/go/ydb/table" ) - + func executeInTx(ctx context.Context, s *table.Session, query string) { newCtx, close := context.WithTimeout(ctx, time.Millisecond*300) // client and by default operation timeout newCtx2 := ydb.WithOperationTimeout(newCtx, time.Millisecond*400) // operation timeout override @@ -86,3 +86,4 @@ Timeout usage example: ``` {% endlist %} + diff --git a/ydb/docs/en/core/best_practices/index.md b/ydb/docs/en/core/best_practices/index.md index a4077dc318..91ae3a01bb 100644 --- a/ydb/docs/en/core/best_practices/index.md +++ b/ydb/docs/en/core/best_practices/index.md @@ -1 +1,2 @@ This section provides recommendations on how to use {{ ydb-short-name }} features. + diff --git a/ydb/docs/en/core/concepts/_assets/BS_overview.svg b/ydb/docs/en/core/concepts/_assets/BS_overview.svg new file mode 100644 index 0000000000..eb08e1cb2e --- /dev/null +++ b/ydb/docs/en/core/concepts/_assets/BS_overview.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="771px" height="814px" viewBox="-0.5 -0.5 771 814" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T14:50:26.556Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="ZU8BhaNDal54XeL1MvRJ" version="12.7.0" type="device"><diagram id="0HZmCEQLlPINtnbGPtoa" name="Page-1">7Zxdl6I4EIZ/jZc9BxK+vBx1puecOd3HGXd3Zq72RIjCNoobsdX99RuUIKTo9psgTt80KTTIW4+VqiTSwt3J6pGRmf8UeTRsIc1btXCvhZCOnDb/l1jWW4utWVvDmAVe+qKdYRD8R1OjlloXgUfnhRfGURTGwaxodKPplLpxwUYYi5bFl42isHjVGRlTYBi4JITWH4EX+1urg+yd/QsNxr64sm6lNzwh4sXpncx94kXLnAl/auEui6J4ezRZdWmYiCd02b7v8xtnsw/G6DQ+5A3+n/Tx6fnpBaOnUdd9oN/+cXsPzraXVxIu0hsef+93k35ZtFrzzr5+66cfP14LTVi0mHo06VZr4c7SD2I6mBE3ObvkFHCbH09C3tL54SgIw24URmzzXjxyXOq63D6PWfRCc2eGjmmYSYdjRryA31T+XZs/fg7edKrDK2UxXeVMqQiPNJrQmPFb0dKzRuqPtUA0bS937s1sfs61VmojKVHjrOed6Pwg1f0IH7SBD3okJgOfMO8d6fXjpR+NULn0njW0TKty6ZGmWnoRZe5Qe0e59jrQ/kvwSi8qu2dSxzPKZHfQEFvVy44t5bKj+wv5WaCpTczX8d0EHln8GkR9A4q/npJJ4LaQFfKrd4aMH42To2ee0fKX6sAp/O7jovJFhafRlEruSE0kDMZT3nS5ppTbO4mWAU85P6YnJoHnJZcpdXXxm3gB/zhF95SMC0aJc9DVnGMe7RzUXOfI44dy71j3N3pkY3Z9Rg9YtjV19JDFr8HoAeu1BuatQHfl9YK42BEDA27uwIDbuFYDA4LVXOMHBhPVbWBAsLjrDLoNC02y7OqHBASruUFM4vusJ2T3HDpwXC80wXKv8aHJcmoXmmBd130aNCw0ybLXIDTBgm1PaGpwNS27R31osoF3Ei90EnVZ078b6isKWEYDzec+mSWH7oKF6w4j7guN94tf9BSLkm9clPDevhDJul0U88GGYpaS7FxNTFgW/0GnhN9ho7MdHZsFR2Doh3aVEUVc/waZNnTtQ1FMW7T3UK2L5OLycsKyNqO6wQMltmtGNUzhb4Vqy7Alqh9sdCDXyLka1zAbF2lhU5m2dNkRJVQ7lVINc/NOGA0HccSSXWmyJ3iPwWye6JWhHkYL7/hUkFBnVFqdWq5Dh1eapSnb+VAaRHT9anqXZ9s/CPPoFMh92xMAsvyG8mwbw2y7serLtY569Q2YyJyn+RUYNZVvjjIQUKnfC+Yv6qXCklQl9V7FUsEMQrlK8teuBkDBIb4eQFnt2gEFR2flKskRyjKUqwRH0b9qAZQcoSxTuVRwRkq5SnKEUg+UcFPtgJIjlHqgzBvIoWzlC30mzKF6g1a22KpaLylM2SWTABXrBTcPKFdJDlM1oKpkz3Z9qJJjVQ2oOiA9p1PvY/ITyVY2ZeeRuZ9NhuYk4sqw9c9Ezg9t0fy1aZqi2Vulam9b63yrT1nAbyuZBHx/Ni8mbEzjd25LLNlRr/CzTeiYnPBmie7CxmhI4mT3Zv5TlDkjvUI/CjZz/4IKeUHZkPw5jxbMpem7di4FHUn8OPIc2FYY0M8Gjeyuz6AFlik3QEtNIMDyHn0xi3AsBKCjiiE4oAD7DcEbvjO1C0EAOqoYggP2RPyG4A3fWeIrdC4EoKNqIbAOWG7NQeCGZD4P3Dddr9lF3xun+J6ugvin6Jsf5zDirV1PSWO9j5etH95RQD80IRFreDUhUBpB2nIecWo+Iuep1wbwuOx1H4A6dooEYq19Awxqt8mgvDfN0E6E0DS5p4pdiZ/uVsXhcXnxPg6NIoX6OQzuuPuVI7KcwTmXKYYfcmP+HIR7C+G9qApZbg3VpoTL4zL3fZgis8gpOilZKw+W7XdBTRoHJ337A2j7Nqk0JZyQfSKWllh+FHtwHOmzXJvL44qJfVxaAMyqAugZkdG+TQYNLG/GNU8NjoYjd2VcLz6un7tjtP57+X3of6SDl+6/X/v9B7huU+edR2DauYS8N2eipRGt0o1Hpdqj+9FeftKNevHh0lJjxZcfWKBe/AMmUU5Yp7pkdKh0f1SpRiUP+7noqt5ZapmSWlVukSpV64BqtGqN5JinHilYC9UIqey5trVh6oAUvWqR5FCunqmSR7TWhylsV8cUb+6eWLxNz3fPfcaf/gc=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f8cecc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f8cecc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-fff2cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#fff2cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-d5e8d4-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#d5e8d4"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient></defs><g><rect x="0" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="60" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="0" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="60" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="0" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="60" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="0" y="330" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 360px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Hive</div></div></div></foreignObject><text x="60" y="364" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Hive</text></switch></g><rect x="160" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="220" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="160" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="220" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="40" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 41px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 1</div></div></div></foreignObject><text x="60" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="200" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 201px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 2</div></div></div></foreignObject><text x="220" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="320" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="380" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="320" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="380" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="320" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Hive</div></div></div></foreignObject><text x="380" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Hive</text></switch></g><rect x="353" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 354px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 3</div></div></div></foreignObject><text x="373" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="480" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="540" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="480" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BSC</div></div></div></foreignObject><text x="540" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BSC</text></switch></g><rect x="480" y="50" width="120" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 60px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static<br />Node 1</div></div></div></foreignObject><text x="540" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static...</text></switch></g><rect x="640" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="700" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="640" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">CMS</div></div></div></foreignObject><text x="700" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">CMS</text></switch></g><rect x="640" y="50" width="120" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 60px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static<br />Node 2</div></div></div></foreignObject><text x="700" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static...</text></switch></g><rect x="640" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeBroker</div></div></div></foreignObject><text x="700" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeBroker</text></switch></g><path d="M 150 -100 L 145 -100 Q 140 -100 140 -90 L 140 30 Q 140 40 135 40 L 132.5 40 Q 130 40 135 40 L 137.5 40 Q 140 40 140 50 L 140 170 Q 140 180 145 180 L 150 180" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,140,40)" pointer-events="all"/><rect x="95" y="0" width="90" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 88px; height: 1px; padding-top: 10px; margin-left: 96px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Tenant 1</div></div></div></foreignObject><text x="140" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Tenant 1</text></switch></g><path d="M 390.5 -22.5 L 385.5 -22.5 Q 380.5 -22.5 380.5 -12.5 L 380.5 30 Q 380.5 40 375.5 40 L 373 40 Q 370.5 40 375.5 40 L 378 40 Q 380.5 40 380.5 50 L 380.5 92.5 Q 380.5 102.5 385.5 102.5 L 390.5 102.5" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,380.5,40)" pointer-events="all"/><rect x="335" y="0" width="90" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 88px; height: 1px; padding-top: 10px; margin-left: 336px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Tenant 2</div></div></div></foreignObject><text x="380" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Tenant 2</text></switch></g><path d="M 627.5 -102.5 L 622.5 -102.5 Q 617.5 -102.5 617.5 -92.5 L 617.5 30 Q 617.5 40 612.5 40 L 610 40 Q 607.5 40 612.5 40 L 615 40 Q 617.5 40 617.5 50 L 617.5 172.5 Q 617.5 182.5 622.5 182.5 L 627.5 182.5" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,617.5,40)" pointer-events="all"/><rect x="577.5" y="0" width="80" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 10px; margin-left: 579px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static</div></div></div></foreignObject><text x="618" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static</text></switch></g><path d="M 552.5 357.5 C 494.5 357.5 480 385 526.4 390.5 C 480 402.6 532.2 429 569.9 418 C 596 440 683 440 712 418 C 770 418 770 396 733.75 385 C 770 363 712 341 661.25 352 C 625 335.5 567 335.5 552.5 357.5 Z" fill="#dae8fc" stroke="#6c8ebf" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 288px; height: 1px; padding-top: 385px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobStorage</div></div></div></foreignObject><text x="625" y="389" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobStorage</text></switch></g><rect x="480" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="540" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="640" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="700" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="480" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk</div></div></div></foreignObject><text x="550" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk</text></switch></g><rect x="640" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk</div></div></div></foreignObject><text x="710" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk</text></switch></g><rect x="480" y="610" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="620" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 650px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">VDisk</div></div></div></foreignObject><text x="550" y="654" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">VDisk</text></switch></g><rect x="640" y="610" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="620" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 650px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">VDisk</div></div></div></foreignObject><text x="710" y="654" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">VDisk</text></switch></g><rect x="480" y="690" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="700" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 730px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="550" y="734" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="640" y="690" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="700" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 730px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="710" y="734" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><path d="M 140 810 L 140 48" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 300 812 L 300 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 460 812 L 460 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 617 812 L 617 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 120 360 L 493.94 373.77" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 499.18 373.96 L 492.06 377.2 L 493.94 373.77 L 492.32 370.2 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 120 280 L 513.79 362.68" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 518.93 363.76 L 511.36 365.75 L 513.79 362.68 L 512.79 358.9 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 542.87 235.69 L 593.13 335.31" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 540.5 231 L 546.78 235.67 L 542.87 235.69 L 540.53 238.82 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 595.5 340 L 589.22 335.33 L 593.13 335.31 L 595.47 332.18 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 440 304 L 546.75 354.77" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 551.49 357.02 L 543.67 357.17 L 546.75 354.77 L 546.67 350.85 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 700 310 L 665.57 347.32" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 662.01 351.18 L 664.18 343.66 L 665.57 347.32 L 669.33 348.41 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="0" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="60" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="160" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="220" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="320" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="380" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="0" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="10" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 11px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="70" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="160" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="170" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 171px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="230" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="320" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="330" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="390" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_assets/Slide3_group_layout.svg b/ydb/docs/en/core/concepts/_assets/Slide3_group_layout.svg new file mode 100644 index 0000000000..3d6bf588a8 --- /dev/null +++ b/ydb/docs/en/core/concepts/_assets/Slide3_group_layout.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="731px" height="468px" viewBox="-0.5 -0.5 731 468" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T15:31:24.461Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="c6UdJlCHfOx2ZCPmeLO2" version="12.7.0" type="device"><diagram id="5oa2Jgz5xl6jGbGaUmvA" name="Page-1">5V1dc6JIFP01Pu4W0Hz5GGOSqZrZbKqcqszuy1YLjbKDtIvtROfXL0ijQpOEZLzeJpqH2Bdo4Zzb3fcciBmQ68XmLqPL+R88ZMnAMsLNgIwHlmUZrpX/KiLbMmKaQ7uMzLI4lLFDYBL/ZDJoyOg6DtmqtqPgPBHxsh4MeJqyQNRiNMv4U323iCf1T13SGVMCk4AmavQxDsW8jPqWd4h/YvFsXn2y6Q7LLQta7SyvZDWnIX86CpGbAbnOOBflu8XmmiUFehUu5XG3z2zdn1jGUtHlgG+bz2L995rc2/b9n9Gnydf/JuQ32csPmqzlBT+M49X3PGQOyJVpGNXJi22FSMbXaciKTo0BGT3NY8EmSxoUW5/yJMhjc7FI8lbewyiKk+SaJzzbHUsip/jJ4yuR8e/saIu7exVH8FQcxctXHp9lNIxZbVu0e+XbVCiq62KZYJujkITmjvEFE9k236XaWtEkE9WXzacD6VYFxvyIcFvGqMyz2b7nAxX5G8nGG5ixFGZOykRImR8FrUwEPptG50fcVhG3zwk4AQbcYX5otwHuW1OyS31owF29ALdhAY8i5gatGR56w6lhwAO+nzA0AdyBBjyyngHcnbrOGTJ8D50mgLvAgPsBawd86ju2c44M9/UC3HutnDEvpJwhrmbljA87FJiZr69eGxND1yP0DHNPE3HsoTCEBdw16NBsBdwae+5udW2k/h5sYCJszVbdSslCMWEYPt3hrTBhGI6HyoRmy7Gpqt2TMmFOqcmsdibcm6tbRCY0W6dNYHWbZ77BonYmzPHoGo8JRzPVa6qytyqZrMt2gPb1EVrNZAIrZP08oL1FgTYcgDWyfi4QPuTQKlk7HwgfclUnnxhy3ZwgfMiBBbB+XhA+5KoEbpQ2l+oG4Zc21bp/OX4Q+nCwgNVvfxwhfCrA5W9fLCF8KoBv+/bHE8KnAlju9scUwqdClcFV6UQu2xUiFnrpBKyX9XOF2u5ennc4AOtl/VwhfMih9bJ2rhA+5MC3jPVzhdAhJ8AyWD9XCB/y5x95JpftCuGXNgRYF+vnCuEPB2D92x9XCJ8KcP3bF1cInwrg28D9cYXwqQCWu/1xhdCp8NQJqiqd7Mt2hfYTGFrp5EE/uKKdK2S33MQ873CANuK0c4XwIQd/cEU3VwgfcnAjTjdXCB9yaCNOO1cIHfLqfJ4vbS7VFcIvbXzgB1f0c4Xwh4MFC3l/XCF8KoANuv64QvhUABt0/XGF8KkAlrv9cYXwqVBl8D0P2cBqKZnyaxR13GkSz9L8fZAjwnK4RgUScUCTK7lhEYdhcfgoY6v4J53uuioIXfI4FbuLcUYDZ1z0tRZ8VX6DlHmihbmOtadC7bRAbYFBrcpfCbWa/T2H2hxiY63qXok1+WhYExMba1XwSqztj4a1Tc6H9T+Pj3eLh2QxSefOreV9vbv68rnTXziycMYmsskzMeczntLk5hAd1dfWwz5fOF9K7P5lQmzlF/oVsDbW23z1lBtN6yWkV3ydBeylCyr3K875RT4yllAR/2C13tvglYc+FMlx4NFtCAO/QZCg2YwJeVCDo/1Z/AJtCmvjSQFYxjfbX6x8Gkwow+kEg8BtlPKOOgj29v7xKHChRkEHUcXS8Kr4Asm8lfKUvTN9X03LV+aBKnbi7CXN9C2HmZK+r3bUnKeAh0EHBXbEW5DQ1SoO+k2d03w44p3MNftpCgJg5jrcRngrczlh2fZbMcv97lTNv+Skt2uMN7XWVraeZbyE4IVCpfoTFU1Tw2xy2jk3nPb5+Uy50eF+h/65UT2oomluKGtp19zwm6lx5sKnwyPS+idH5VdpmhzvLgewk6PDraEPVcedqhhALuO6fP/Xh+bt3St1kzilI2jmLk05nWwdVYZcsyNo5t6mnT4ecyfTvEpH72Yubx7+AUS5++H/aJCb/wE=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f5f5f5-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f5f5f5"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-dae8fc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#dae8fc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-d5e8d4-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#d5e8d4"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-ffe6cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#ffe6cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-fff2cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#fff2cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f8cecc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f8cecc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-e1d5e7-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#e1d5e7"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-60a917-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#60a917"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-008a00-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#008a00"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-1ba1e2-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#1ba1e2"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-0050ef-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#0050ef"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient></defs><g><rect x="90" y="67" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 87px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 1:1000</div></div></div></foreignObject><text x="190" y="91" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 1:1000</text></switch></g><rect x="90" y="27" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="27" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="27" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="27" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="27" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="67" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 87px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 1:1001</div></div></div></foreignObject><text x="430" y="91" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 1:1001</text></switch></g><rect x="330" y="27" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="27" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="27" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="27" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="27" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="187" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 207px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 2:1000</div></div></div></foreignObject><text x="190" y="211" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 2:1000</text></switch></g><rect x="90" y="147" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="147" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="147" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="147" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="147" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="187" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 207px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 2:1001</div></div></div></foreignObject><text x="430" y="211" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 2:1001</text></switch></g><rect x="330" y="147" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="147" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="147" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="147" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="147" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="307" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 327px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 3:1000</div></div></div></foreignObject><text x="190" y="331" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 3:1000</text></switch></g><rect x="90" y="267" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="267" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="267" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="267" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="267" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="307" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 327px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 3:1001</div></div></div></foreignObject><text x="430" y="331" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 3:1001</text></switch></g><rect x="330" y="267" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="267" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="267" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="267" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="267" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="427" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 447px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 4:1000</div></div></div></foreignObject><text x="190" y="451" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 4:1000</text></switch></g><rect x="90" y="387" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="387" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="387" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="387" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="387" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="427" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 447px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 4:1001</div></div></div></foreignObject><text x="430" y="451" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 4:1001</text></switch></g><rect x="330" y="387" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="387" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="387" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="387" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="387" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="0" y="57" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 67px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 1</div></div></div></foreignObject><text x="25" y="71" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 1</text></switch></g><rect x="0" y="177" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 187px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 2</div></div></div></foreignObject><text x="25" y="191" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 2</text></switch></g><rect x="0" y="297" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 307px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 3</div></div></div></foreignObject><text x="25" y="311" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 3</text></switch></g><rect x="0" y="417" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 427px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 4</div></div></div></foreignObject><text x="25" y="431" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 4</text></switch></g><path d="M 610 67 L 576.37 67" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 571.12 67 L 578.12 63.5 L 576.37 67 L 578.12 70.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="610" y="37" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 67px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="670" y="71" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><path d="M 570 367 L 570 7" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 7 L 510 20.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 25.88 L 506.5 18.88 L 510 20.63 L 513.5 18.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 127 L 510 140.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 145.88 L 506.5 138.88 L 510 140.63 L 513.5 138.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 247 L 510 260.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 265.88 L 506.5 258.88 L 510 260.63 L 513.5 258.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 367 L 510 380.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 385.88 L 506.5 378.88 L 510 380.63 L 513.5 378.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 7 L 570 7" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 127 L 570 127" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 247 L 570 247" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 367 L 570 367" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_assets/Slide_blob.svg b/ydb/docs/en/core/concepts/_assets/Slide_blob.svg new file mode 100644 index 0000000000..502ed357b3 --- /dev/null +++ b/ydb/docs/en/core/concepts/_assets/Slide_blob.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="691px" height="551px" viewBox="-0.5 -0.5 691 551" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T16:05:40.464Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="HF0g-8VF_2eGPBkH-fRg" version="12.7.0" type="device"><diagram id="LqqcKPgxwdzCRuipa-4Q" name="Page-1">3ZtRc5s4EMc/DTO9h3SQBBg/GidNc3PXyYw7vV7fFKMYWox8shzb/fQnjADDUocktpH74qBFYPhp/9LuKrbIeL65FXQR/c1DlljYDjcWubYwxraH1Z/Mss0tCA2d3DITcahtlWES/2TaaGvrKg7ZstZRcp7IeFE3Tnmasqms2agQfF3v9siT+rcu6IwBw2RKE2j9Jw5llFt9PKjsH1k8i4pvRt4wPzOnRWf9JsuIhny9ZyI3FhkLzmV+NN+MWZLRK7jk1334xdnywQRLZZcLxt6Hb1+Cu/8E+7ZBn/5EH++/Ply5bn6bJ5qs9Bvrp5XbAoHgqzRk2V1siwTrKJZssqDT7OxajbqyRXKeqBZSh488lXoUka/aNIlnqWok7FE9ZQAfWr/HExOSbfZM+iVuGZ8zKbaqiz5LNE/tUaTwlHU1PG7BPNobGmJrI9UuMStvXVFTBxrcSyAOnofIQuVWusmFjPiMpzS5qaxBHXPV5y/OFxrudyblVtOlK8kPo1eIxfZrdr/3btH8V99+17je1Fpb3frlGC35SkzZARCFQKmYMdnB6zIoB4dcsITK+Kmuxbbx05fe81g9c+kq2K77Cm66QP6k+qqGF5SP8XrHQMAvPtOHhMm78G0iO4KKkFdH40MV+S0ick6lIYcYoaHzawZ31Awankkz/us0MxKCbve6LbIOy+7fgwaN5avRH9nuof7qIH+CowoYA6ccR1QFGknv+i2VaIp+OwQSv6V+SUf94jOtecQ7j37B95DD+iUN/Tb6n0a/cFG5ZSkTii9Pe5cwwYZJGMCaSLboHZNjm4UJTnRjzn/ErH9QvlmgPAAqSPjDburuG5VrWPQLE8h7KkxIEzzDwgwfgAKIlhFdZIfTlUi2gaDTH9m6/ByrCmzekvkaQa6vhgfjgxfIs1G4uEIOhNlWuCid9eg0h60KbXE79Yqyzquo60wVECaUIQMRT2ky0ifmcRjmMR1bxj+ztFc7rQ4s1H3dwHKvs3upMG6pCxdHYo0ayYLbUiRqQY1PRbooUu2h9pyLx4z8OmYPUiZnpQwLLv7FQ8YD+znIbbPG6SDDpJjAWfjSKBPHMFeGqctvQNnBhlGGOQ++/GnZtQ2jDMPbbF6232Hvj8uHPXgW9nkjDRgiX75He6atgTBy/pxv7UwkF3TG7tJHDqi/KHlr8pdZBfQoc3DdX8sq1B5L1ArTPxVNbMb+aVU7rlWO3+Oyknz8zR+N1JjqMSYN7xg0Bv3EO6YY5knljktRB+yzbtSg05JWIrtFOyerh2CY8VS8kHG8hr3zgslLxQvOOj3zKktG/fGCaUjF65NpvNz+9Qh3B3RgoLEZHBg0IwOna2SAThYZEDN2ld+wwA87LvDEOfYC/zY/hgGubd1gyx9aI2fGUgt7dJ75XZLlEaj/lRlMnb1PBQTWFm4VlUW2MFflpj73C4lxyMxIBF4vd9I1nieeUXInMOxGh/SO+/desPbbvXsv3PAuBO86BPcfjTcFbwAyuMxcmOC7/ssz8c0SPMwbM1HXFG+awPtfnmBp1ewVvX9kbss/bCgYo91nYO8+0e4z2H26lmLh52dHuk+T6lmL180fVRxjlJpVkZY8a9gySOTlg6Sa1Q+M8hpd9TstcvM/</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="0" y="250" width="520" height="300" fill="#ffffff" stroke="#000000" pointer-events="all"/><path d="M 170 60 L 170 155 L 260 155 L 260 243.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 260 248.88 L 256.5 241.88 L 260 243.63 L 263.5 241.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="130" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 131px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TabletId</div></div></div></foreignObject><text x="170" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TabletId</text></switch></g><path d="M 250 60 L 250 110 L 75 110 L 75 253.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 75 258.88 L 71.5 251.88 L 75 253.63 L 78.5 251.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="210" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 211px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel</div></div></div></foreignObject><text x="250" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel</text></switch></g><path d="M 330 60 L 330 170 L 275 170 L 275 353.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 275 358.88 L 271.5 351.88 L 275 353.63 L 278.5 351.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="290" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 291px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Generation</div></div></div></foreignObject><text x="330" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Generation</text></switch></g><rect x="370" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 371px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Step</div></div></div></foreignObject><text x="410" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Step</text></switch></g><rect x="450" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 451px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Cookie</div></div></div></foreignObject><text x="490" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Cookie</text></switch></g><rect x="530" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 531px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobSize</div></div></div></foreignObject><text x="570" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobSize</text></switch></g><rect x="610" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PartId</div></div></div></foreignObject><text x="650" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PartId</text></switch></g><path d="M 420 -200 L 415 -200 Q 410 -200 410 -190 L 410 70 Q 410 80 405 80 L 402.5 80 Q 400 80 405 80 L 407.5 80 Q 410 80 410 90 L 410 350 Q 410 360 415 360 L 420 360" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(-90,410,80)" pointer-events="all"/><rect x="385" y="90" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 100px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">BlobId</div></div></div></foreignObject><text x="410" y="104" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobId</text></switch></g><rect x="155" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 170px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">64</div></div></div></foreignObject><text x="170" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">64</text></switch></g><rect x="240" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">8</div></div></div></foreignObject><text x="250" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="315" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 330px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="330" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="395" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="410" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="475" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 490px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">24</div></div></div></foreignObject><text x="490" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">24</text></switch></g><rect x="545" y="0" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 570px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">28 (26)</div></div></div></foreignObject><text x="570" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">28 (26)</text></switch></g><rect x="640" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 650px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">4</div></div></div></foreignObject><text x="650" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="15" y="260" width="120" height="280" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 267px; margin-left: 16px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletStorageInfo</div></div></div></foreignObject><text x="75" y="279" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletStorageInfo</text></switch></g><path d="M 125 310 L 170 310 L 170 405 L 208.63 405" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 213.88 405 L 206.88 408.5 L 208.63 405 L 206.88 401.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="25" y="290" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 310px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 0</div></div></div></foreignObject><text x="75" y="314" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 0</text></switch></g><rect x="25" y="330" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 350px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 1</div></div></div></foreignObject><text x="75" y="354" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 1</text></switch></g><rect x="25" y="370" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 390px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 2</div></div></div></foreignObject><text x="75" y="394" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 2</text></switch></g><rect x="25" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel N</div></div></div></foreignObject><text x="75" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel N</text></switch></g><rect x="215" y="360" width="120" height="180" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 367px; margin-left: 216px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletChannelInfo</div></div></div></foreignObject><text x="275" y="379" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletChannelInfo</text></switch></g><path d="M 325 410 L 398.63 410" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 410 L 396.88 413.5 L 398.63 410 L 396.88 406.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0≤gen<10</div></div></div></foreignObject><text x="275" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0≤gen<10</text></switch></g><rect x="405" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><path d="M 325 460 L 398.63 460" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 460 L 396.88 463.5 L 398.63 460 L 396.88 456.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">10≤gen<125</div></div></div></foreignObject><text x="275" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">10≤gen<125</text></switch></g><rect x="405" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 54321</div></div></div></foreignObject><text x="455" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 54321</text></switch></g><path d="M 325 510 L 398.63 510" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 510 L 396.88 513.5 L 398.63 510 L 396.88 506.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">125≤gen</div></div></div></foreignObject><text x="275" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">125≤gen</text></switch></g><rect x="405" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><rect x="425" y="260" width="90" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 275px; margin-left: 470px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 18px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Tablet</div></div></div></foreignObject><text x="470" y="280" fill="#000000" font-family="Helvetica" font-size="18px" text-anchor="middle">Tablet</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_assets/Slide_group_content.svg b/ydb/docs/en/core/concepts/_assets/Slide_group_content.svg new file mode 100644 index 0000000000..5e5796fae0 --- /dev/null +++ b/ydb/docs/en/core/concepts/_assets/Slide_group_content.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="616px" height="412px" viewBox="-0.5 -0.5 616 412" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T15:37:07.887Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="3zDAlwPQQ3q6GuUUS0rv" version="12.7.0" type="device"><diagram id="wSBYkmeAYjroDNpxkVYP" name="Page-1">3Ztbc6IwFMc/jY87I4EgPmpt7V66++DM7nMKKWYHiBPj9dNv0KDi0VaLmLBP4knA5PdPwrmMLfchXQ4FmYxfeESTFmpHy5Y7aCHkIMdVH7lltbX4Pt4aYsEi3WlvGLE11ca2ts5YRKeljpLzRLJJ2RjyLKOhLNmIEHxR7vbGk/KvTkhMgWEUkgRa/7BIjrfWAHX29mfK4nHxy47f3bakpOisZzIdk4gvDkzuY8t9EJzL7VW6fKBJDq/gsr3v6UzrbmCCZvKSG14S5q9fv/0apDEJw/iFDkbrL1jLMyfJTM9Yj1auCgSCz7KI5k9pt9z+YswkHU1ImLculOjKNpZpor456vKNZ/KJpCzJ9X6myZxKFhLVAIerZzCnQtLlgUkPf0h5SqVYqS66FWFNXa8lt1gki70yTkfbxoeq+NpI9GqId8/eA1MXmtkV/IoR3IzfDTA5HbeEyTmByQtOYKqNEgKQ2uYpBUeU0AlKJyB5dUGCO9ExDgl5lkHyACTz+81tWwYJA0iueUi+ZZB8AMkzDslDlkHqAEjYPCTbDu4AQPKNQ8K2HdxdAKljHJJv28HtQEfpjbD8NkFJkqpP6DepucoyGZKwOFPXoSJDhTLkRJQTnvR0Q8qiKL+9L+iUrcnr5lE58glnmdxMCvdbeJA/ayb5dBuU3Qh6t11m7kLmp1x4VBtz6HdtmftJzvVVqKs4v4p4SlgG7c2XxOkEJU18KAk+IQmuTRLo5V0nCXScmyYJci2TBPqU10kC3fTGSdK1TBLowV4nCQwKmiaJiy2TBPrL10kCQ5CmSeI5lkkCvfPrJIEBT+Mkse31DmOB6ySB4VXTJMGWvd4L7/rTksBgrnGSWPZ6RzAWBIxpFNOR/sqFHPOYZyR53Fv75ZB63+cH5xPN7i+VcqULbznWC8PsKZ+JkL43gaJcR0RM5TsdNfd8Mu8KJWhCJJuXk+O35+4A7r8HX0ff1dIbKpqTr+rm3pBmVKjBcLVse4psz9984oHZbVAqvj0ogZj6AdT+SRc3epMcZZV2+YiDTXIyY1Jb9O7BULFiaanWAqZ/VMA0nXDyYFhXsehUJ74dLmvwwRAMHtL24Otgy/DBcKlioequm7djGh8MbSqWsO66eY3jg2FIxeLWXTevaXwYhgwVy1733Lye6bMPQzezYkHsnpvXPD4YHQUW4zvavObxnauw2FJorFMNJzh6kZsuQeJzxZWdGoYLWndVIzCtxrm6yk4N6OP/t2p4xvdG1ZJK88vzyEclTYynJnHVmkrz6/Ougy3TpGpRxfChdhMvq22XJj70sgDkO6TwP5cV/jC/X5xKH+b3i0DbkgR/Me7LE/wKYg9dnuCvPQsflNf5Lro4/BuMf2Khu9cvdPV1/w+lTdvB/7zcx38=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="167" y="250" width="170" height="160" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="83" y="50" width="480" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="93" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 94px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0</div></div></div></foreignObject><text x="113" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0</text></switch></g><rect x="153" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 154px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">1</div></div></div></foreignObject><text x="173" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">1</text></switch></g><rect x="213" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 214px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">2</div></div></div></foreignObject><text x="233" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">2</text></switch></g><rect x="273" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 274px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">3</div></div></div></foreignObject><text x="293" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">3</text></switch></g><rect x="333" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 334px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">4</div></div></div></foreignObject><text x="353" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="393" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 394px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">5</div></div></div></foreignObject><text x="413" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">5</text></switch></g><rect x="453" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 454px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">6</div></div></div></foreignObject><text x="473" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">6</text></switch></g><rect x="513" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 514px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">7</div></div></div></foreignObject><text x="533" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">7</text></switch></g><rect x="0" y="70" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 80px; margin-left: 35px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 0</div></div></div></foreignObject><text x="35" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 0</text></switch></g><rect x="88" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 113px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />0</div></div></div></foreignObject><text x="113" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="148" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 173px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />1</div></div></div></foreignObject><text x="173" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="208" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 233px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />2</div></div></div></foreignObject><text x="233" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="268" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 293px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />3</div></div></div></foreignObject><text x="293" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="328" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 353px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />4</div></div></div></foreignObject><text x="353" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="388" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 413px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />5</div></div></div></foreignObject><text x="413" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="448" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 473px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />6</div></div></div></foreignObject><text x="473" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="508" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 533px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />7</div></div></div></foreignObject><text x="533" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><path d="M 473 140 L 473 106.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 473 101.12 L 476.5 108.12 L 473 106.37 L 469.5 108.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="353" y="140" width="240" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 150px; margin-left: 473px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Courier New; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">VDISK[GroupId:Generation:0:6:0]</div></div></div></foreignObject><text x="473" y="154" fill="#000000" font-family="Courier New" font-size="12px" text-anchor="middle">VDISK[GroupId:Generation:0:6:0]</text></switch></g><rect x="177" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0</div></div></div></foreignObject><text x="197" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0</text></switch></g><rect x="230" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">1</div></div></div></foreignObject><text x="250" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">1</text></switch></g><rect x="285" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">2</div></div></div></foreignObject><text x="305" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">2</text></switch></g><rect x="177" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">3</div></div></div></foreignObject><text x="197" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">3</text></switch></g><rect x="230" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">4</div></div></div></foreignObject><text x="250" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="285" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">5</div></div></div></foreignObject><text x="305" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">5</text></switch></g><rect x="177" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">6</div></div></div></foreignObject><text x="197" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">6</text></switch></g><rect x="230" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">7</div></div></div></foreignObject><text x="250" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">7</text></switch></g><rect x="285" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">8</div></div></div></foreignObject><text x="305" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="97" y="270" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 280px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 0</div></div></div></foreignObject><text x="132" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 0</text></switch></g><rect x="97" y="320" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 330px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 1</div></div></div></foreignObject><text x="132" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 1</text></switch></g><rect x="97" y="370" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 380px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 2</div></div></div></foreignObject><text x="132" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 2</text></switch></g><rect x="172" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 197px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />0</div></div></div></foreignObject><text x="197" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="225" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />1</div></div></div></foreignObject><text x="250" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="280" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 305px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />2</div></div></div></foreignObject><text x="305" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><path d="M 390 330 L 331.37 330" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 326.12 330 L 333.12 326.5 L 331.37 330 L 333.12 333.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="390" y="315" width="160" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe flex-start; width: 1px; height: 1px; padding-top: 322px; margin-left: 392px;"><div style="box-sizing: border-box; font-size: 0; text-align: left; "><div style="display: inline-block; font-size: 12px; font-family: Courier New; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">VDISK[GroupId:Generation:1:2:0]</div></div></div></foreignObject><text x="392" y="334" fill="#000000" font-family="Courier New" font-size="12px">VDISK[GroupId:Generation:1...</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_includes/connect.md b/ydb/docs/en/core/concepts/_includes/connect.md new file mode 100644 index 0000000000..f0470801bd --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect.md @@ -0,0 +1,106 @@ +# Connecting to a database + +{% include [overlay/ui.md](connect_overlay/ui.md) %} + +To connect to the {{ ydb-short-name }} database from the {{ ydb-short-name }} CLI or an application using the {{ ydb-short-name }} SDK, you'll need to provide the following data: + +1. Endpoint +1. Authentication information +1. Database location + +## Endpoint {#endpoint} + +The `endpoint` is a string in `protocol://host:port` format provided by the {{ ydb-short-name }} cluster owner to correctly route client requests to its databases via the network infrastructure and to establish a network connection. For cloud databases, the `endpoint` is shown in the management console on the page of the desired database. In addition, you can usually obtain it using the CLI of the cloud provider. In corporate environments, the names of {{ ydb-short-name }} endpoints are provided by administrators or can also be obtained in the internal cloud management console. + +{% include [overlay/endpoint.md](connect_overlay/endpoint.md) %} + +Examples: + +- `grpc://localhost:7135` indicates a data exchange protocol without encryption (gRPC), the server is run on the same host as the client and accepts connections on port 7135. +- `grpcs://ydb.somecorp-int.ru` indicates a data exchange protocol with encryption (gRPCs), the server is run on the host ydb.somecorp-int.ru in the isolated SomeCorp corporate intranet and accepts connections on the YDB default port 2135. +- `grpcs://ydb.serverless.yandexcloud.net:2135` indicates a data exchange protocol with encryption (gRPCs), the public server Serverless YDB Yandex.Cloud ydb.serverless.yandexcloud.net, and port 2135. + +{% include [overlay/endpoint_example.md](connect_overlay/endpoint_example.md) %} + +## Authentication information {#auth} + +Once a network connection is established, the server starts to accept client requests with authentication information for processing. Based on this information, the server identifies the client's account and verifies permission to execute the request. + +The basic operation model assumes that there is a separate system for managing credentials and permissions: [IAM (Identity and Access Management)](https://en.wikipedia.org/wiki/Identity_management). IAM issues a token linked to the account, which the {{ ydb-short-name }} client passes to the server along with a request. The {{ ydb-short-name }} server accesses the IAM system to verify the token and permission to perform the requested operation and caches the result. + +{{ ydb-short-name }} also supports username and password based authentication without interacting with IAM, but using this model in practice is limited to simple configurations, primarily local ones. + +### Authentication modes {#auth-modes} + +{{ ydb-short-name }} clients (CLI or SDK) support the following token selection modes to pass tokens in requests: + +- **Anonymous**: An empty token is passed in a request. +- **Access Token**: A fixed token is set as a parameter for an SDK or CLI and passed in a request. +- **Refresh Token**: An [OAuth token](https://auth0.com/blog/refresh-tokens-what-are-they-and-when-to-use-them/) of a user's personal account is set as a parameter for an SDK or CLI, using which the client regularly accesses the IAM API in the background for rotating (obtaining a new) token that is passed in a request. +- **Service Account Key**: The service account attributes and the signing key are set as a parameter for an SDK or CLI, using which the client regularly accesses the IAM API in the background for rotating (obtaining a new) token that is passed in a request. +- **Metadata**: An SDK or CLI regularly accesses the local service for rotating (obtaining a new) token that is passed in a request. + +Any owner of a valid token can get permission to perform operations, so the main task of the security system is to ensure the token's privacy and not to compromise it. + +Authentication modes with token rotation, such as **Refresh Token** and **Service Account Key**, provide a higher level of security compared to the **Access Token** mode that uses a fixed token, since only secrets with a short validity period are transmitted to the {{ ydb-short-name }} server over the network. + +The highest level of security and performance is provided when using the **Metadata** mode, since it eliminates the need to work with secrets when deploying an application and allows accessing the IAM system and caching a token in advance, before running the application. + +When choosing the authentication mode among those supported by the server and environment, follow the recommendations below: + +- **Anonymous** is usually used on self-deployed local {{ ydb-short-name }} clusters that are inaccessible over the network. +- **Access Token** is used when other modes are not supported on the server side or for setup/debugging purposes. It does not require client interactions with the IAM system. However, if the IAM system supports an API for token rotation, fixed tokens issued by this IAM usually have a short validity period, which makes it necessary to update them manually in the IAM system on a regular basis. +- **Refresh Token** can be used when performing one-time manual operations under a personal account, for example, related to DB data maintenance, performing ad-hoc operations in the CLI, or running applications from a workstation. You can manually obtain this token once in the IAM system for a long period and store it in an environment variable on a personal workstation to be used automatically and without additional authentication parameters when starting the CLI. +- **Service Account Key** is mainly used for applications designed to run in environments where the **Metadata** mode is supported, when testing them outside these environments (for example, on a workstation). It can also be used for applications outside these environments, working as an analog of **Refresh Token** for service accounts. Unlike a personal account, access objects and service account roles can be restricted. +- **Metadata** is used when deploying applications in clouds. Currently, this mode is supported on VMs and in Yandex Cloud Functions. + +The token to specify in request parameters can be obtained in the IAM system that the specific {{ ydb-short-name }} deployment is associated with. In particular, for the {{ ydb-short-name }} service, Yandex.Cloud uses Yandex.Passport OAuth and Yandex.Cloud service accounts. When using {{ ydb-short-name }} in a corporate context, centralized authentication systems that are standard for the company can be used. + +{% include [overlay/auth_choose.md](connect_overlay/auth_choose.md) %} + +When using modes in which the {{ ydb-short-name }} client accesses the IAM system, the IAM URL that provides an API for issuing tokens can be set additionally. By default, the existing SDK and CLI make an attempt to access the Yandex.Cloud IAM API hosted at `iam.api.cloud.yandex.net:443`. + +## Database location {#database} + +Database location (`database`) is a string that defines where the queried database is located in the {{ ydb-short-name }} cluster. Represented as a [file path](https://en.wikipedia.org/wiki/Path_(computing)) with `/` used as a separator. It always starts with a `/`. + +A {{ ydb-short-name }} cluster may have multiple databases deployed, and their location is defined by the cluster configuration. Like the endpoint, `database` for cloud databases is displayed in the management console on the desired database page, and can also be obtained via the CLI of the cloud provider. + +For cloud solutions, databases are created and hosted on the {{ ydb-short-name }} cluster in self-service mode, with no need in attendance of the cluster owner or administrators. + +{% include [overlay/database.md](connect_overlay/database.md) %} + +{% note warning %} + +Applications should not in any way interpret the number and value of `database` directories, since they are set in the {{ ydb-short-name }} cluster configuration. For example, when working with {{ ydb-short-name }} in Yandex.Cloud, the `database` current structure is `region_name/cloud_id/database_id`. However, this format may be changed in the future for new databases. + +{% endnote %} + +Examples: + +- `/ru-central1/b1g8skpblkos03malf3s/etn01q5ko6sh271beftr` is the Yandex.Cloud database with the `etn01q3ko8sh271beftr` ID in the `b1g8skpbljhs03malf3s` cloud, deployed in the `ru-central1` region. +- `/local` is the default database for custom deployment [using Docker](../../getting_started/ydb_docker.md) + +{% include [overlay/database_example.md](connect_overlay/database_example.md) %} + +## Configuring connection parameters on the client {#client-config} + +For information about how to define connection parameters on the client, see the following articles: + +* [Connecting to and authenticating with a database in the {{ ydb-short-name }} CLI](../../reference/ydb-cli/connect.md) +* [Authentication in the {{ ydb-short-name }} SDK](../../reference/ydb-sdk/auth.md) + +## Additional information {#addition} + +### A root certificate for TLS {#tls-cert} + +When using an encrypted protocol ([gRPC over TLS](https://grpc.io/docs/guides/auth/), or gRPCS), a network connection can only be continued if the client is sure that it receives a response from the genuine server that it is trying to connect to, rather than someone in-between intercepting its request on the network. This is ensured by verifications through a [chain of trust](https://en.wikipedia.org/wiki/Chain_of_trust), to enable which, the client needs to have a root certificate installed. + +The OS that the client runs on already include a set of root certificates from the world's major certification authorities. However, the {{ ydb-short-name }} cluster owner can use its own CA that is not associated with any of the global ones, which is often the case in corporate environments, and is almost always used for self-deployment of clusters with connection encryption support. In this case, the cluster owner must somehow transfer its root certificate for use on the client side. This certificate can be installed in the certificate store of the OS where the client is run (manually by a user or a corporate team of OS admins), or embedded in the client itself (as in the {{ ydb-short-name }} CLI and SDK Yandex.Cloud). + +### API for getting IAM tokens {#token-refresh-api} + +To rotate tokens, the {{ ydb-short-name }} SDK and CLI use a gRPC request to the Yandex.Cloud IAM API: [IamToken - create](https://cloud.yandex.com/en/docs/iam/api-ref/grpc/iam_token_service#Create). In **Refresh Token** mode, the token specified in the OAuth parameter is passed in the `yandex_passport_oauth_token` attribute. In **Service Account Key** mode, based on the specified attributes of the service account and encryption key, a JSON Web Token (JWT) with a short validity period is generated and passed in the `jwt` attribute. The source code for generating the JWT is available as part of the SDK (for example, the `get_jwt()` method in the [Python code](https://github.com/ydb-platform/ydb-python-sdk/blob/main/ydb/iam/auth.py)). + +In the {{ ydb-short-name }} SDK and CLI, you can substitute the host used for accessing the API for obtaining tokens. This makes it possible to implement a similar API in corporate contexts. + diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/auth_choose.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/auth_choose.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/auth_choose.md diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/database.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/database.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/database.md diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/database_example.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/database_example.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/database_example.md diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint.md diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint_example.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint_example.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/endpoint_example.md diff --git a/ydb/docs/en/core/concepts/_includes/connect_overlay/ui.md b/ydb/docs/en/core/concepts/_includes/connect_overlay/ui.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/connect_overlay/ui.md diff --git a/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md b/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md index 7b4c7f8645..97ef4ef07e 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md +++ b/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md @@ -2,10 +2,10 @@ To host databases in {{ ydb-short-name }}, use the following directory structure: ```/domain/account/directory/database```, where: -* *A domain* is the first-level element where all cluster resources are located. There can only be one domain per cluster. -* *An account* is a second-level element that defines the owner of a DB group. -* *A directory* is a third-level element set by the account owner for grouping their databases. -* *A database* is a fourth-level element that corresponds to the DB. +- *A domain* is the first-level element where all cluster resources are located. There can only be one domain per cluster. +- *An account* is a second-level element that defines the owner of a DB group. +- *A directory* is a third-level element set by the account owner for grouping their databases. +- *A database* is a fourth-level element that corresponds to the DB. At the top hierarchy level, the cluster contains a single domain that hosts one or more accounts, while accounts host one or more directories that host databases. diff --git a/ydb/docs/en/core/concepts/_includes/databases/cluster.md b/ydb/docs/en/core/concepts/_includes/databases/cluster.md index fc4fc5a78b..816b7757b6 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/cluster.md +++ b/ydb/docs/en/core/concepts/_includes/databases/cluster.md @@ -3,3 +3,4 @@ _A {{ ydb-short-name }} cluster_ is a set of {{ ydb-short-name }} nodes that the load is distributed across and that communicate with each other through a network. A single cluster may host multiple databases, and specific cluster nodes are allocated for each database. One database is fully hosted in one cluster. There are {{ ydb-short-name }} clusters with a single data center and geo-distributed clusters. The smallest single-datacenter cluster consists of a single node. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/compute.md b/ydb/docs/en/core/concepts/_includes/databases/compute.md index 7284642097..cf4dc20369 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/compute.md +++ b/ydb/docs/en/core/concepts/_includes/databases/compute.md @@ -1,3 +1,4 @@ ## Computing resources {#compute-resources} -{{ ydb-short-name }} clusters can be deployed on bare metal, VMs, or in a container virtualization environment.
\ No newline at end of file +{{ ydb-short-name }} clusters can be deployed on bare metal, VMs, or in a container virtualization environment. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/database.md b/ydb/docs/en/core/concepts/_includes/databases/database.md index 12aef03453..d057ef34f7 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/database.md +++ b/ydb/docs/en/core/concepts/_includes/databases/database.md @@ -1,8 +1,8 @@ ## Database {#database} -_A {{ ydb-short-name }} database ({{ ydb-short-name }} DB)_ is an isolated consistent set of data that is accessed via {{ ydb-full-name }}. +_A {{ ydb-short-name }} database ({{ ydb-short-name }} DB)_ is an isolated consistent set of data that is accessed via {{ ydb-short-name }}. -{{ ydb-full-name }}: +{{ ydb-short-name }}: * Accepts network connections from clients and processes their queries to update and read data. * Authenticates and authorizes clients. @@ -13,3 +13,4 @@ _A {{ ydb-short-name }} database ({{ ydb-short-name }} DB)_ is an isolated consi * Generates metrics that can be used to monitor the DB performance. Resources for the {{ ydb-short-name }} database (CPU, RAM, nodes, and disk space) are allocated within a {{ ydb-short-name }} cluster. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/intro.md b/ydb/docs/en/core/concepts/_includes/databases/intro.md index ddf87b8818..f833273d05 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/intro.md +++ b/ydb/docs/en/core/concepts/_includes/databases/intro.md @@ -2,5 +2,5 @@ title: "Terms and definitions {{ ydb-full-name }}" description: "A database (DB) {{ ydb-full-name }}: is an isolated, consistent set of data that is accessed through the {{ ydb-full-name }}, service, which provides scalability, fault tolerance, and automatic data replication." --- - # Terms and definitions + diff --git a/ydb/docs/en/core/concepts/_includes/databases/regions.md b/ydb/docs/en/core/concepts/_includes/databases/regions.md index 1c84c96c6a..a21125b9df 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/regions.md +++ b/ydb/docs/en/core/concepts/_includes/databases/regions.md @@ -6,3 +6,4 @@ _A wide geographic region_ is a territory where the distance between availabilit A geo-distributed {{ ydb-short-name }} cluster contains nodes located in different availability zones within a wide geographic region. {{ ydb-short-name }} performs synchronous data writes to each of the availability zones, ensuring uninterrupted performance if an availability zone fails. In geographically distributed clusters, you can choose a policy for distributing computing resources across data centers. This lets you find the right balance between minimum execution time and minimum downtime if a data center goes offline. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/slots.md b/ydb/docs/en/core/concepts/_includes/databases/slots.md index 6469a59c63..1139d05acc 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/slots.md +++ b/ydb/docs/en/core/concepts/_includes/databases/slots.md @@ -3,3 +3,4 @@ _A slot_ is a part of the server resources allocated for running a single {{ ydb-short-name }} cluster node, it has a fixed size of 10CPU/50GB RAM. Slots are used if a {{ ydb-short-name }} cluster is deployed on bare-metal instances whose resources are sufficient to host multiple slots. If you use VMs for cluster deployment, their capacity is selected so that the use of slots is not required: one node serves one database and one database can use multiple nodes allocated to it. You can see a list of DB slots in the output of the `discovery list` command of the {{ ydb-short-name }} console client. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md b/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md index 62f4e22d42..b460eb8da9 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md +++ b/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md @@ -5,3 +5,4 @@ A storage group is a redundant array of independent disks that are networked in Each storage group corresponds to a specific storage schema that affects the number of disks used, the failure model, and the redundancy factor. The ``block4-2`` schema is commonly used for single data center clusters, where the storage group is located on 8 disks in 8 racks, can withstand the failure of any two disks, and ensures a redundancy factor of 1.5. In multiple data center clusters, we use the ``mirror3dc`` schema, where storage groups are made up of 9 disks, 3 in each of the three data centers, which can survive the failure of a data center or disk, and ensures a redundancy factor of 3. {{ ydb-short-name }} lets you allocate additional storage groups for available DBs as your data grows. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md b/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md index 5141bc5226..7501678e61 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md @@ -1,3 +1,4 @@ ## Network Block Store Volume -YDB can be used as a platform for creating a wide range of data storage and processing systems, for example, by implementing a [network block device](https://en.wikipedia.org/wiki/Network_block_device) on YDB. Network block devices implement an interface for a local block device, as well as ensure fault-tolerance (through redundancy) and good scalability in terms of volume size and the number of input/output operations per unit of time. The downside of a network block device is that any input/output operation on such device requires network interaction, which might increase the latency of the network device compared to the local device. You can deploy a common file system on a network block device and/or run an application directly on the block device, such as a database management system. +{{ ydb-short-name }} can be used as a platform for creating a wide range of data storage and processing systems, for example, by implementing a [network block device](https://en.wikipedia.org/wiki/Network_block_device) on {{ ydb-short-name }}. Network block devices implement an interface for a local block device, as well as ensure fault-tolerance (through redundancy) and good scalability in terms of volume size and the number of input/output operations per unit of time. The downside of a network block device is that any input/output operation on such device requires network interaction, which might increase the latency of the network device compared to the local device. You can deploy a common file system on a network block device and/or run an application directly on the block device, such as a database management system. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/dir.md b/ydb/docs/en/core/concepts/_includes/datamodel/dir.md new file mode 100644 index 0000000000..05138506ef --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/datamodel/dir.md @@ -0,0 +1,4 @@ +## Directories {#dir} + +For convenience, the service supports creating directories like in a file system, meaning the entire database consists of a directory tree, while tables and other entities are in the leaves of this tree (similar to files in the file system). A directory can host multiple subdirectories and tables. The names of the entities they contain are unique. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/intro.md b/ydb/docs/en/core/concepts/_includes/datamodel/intro.md index 5d7c35d9aa..57acaa8bc0 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/intro.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/intro.md @@ -1,5 +1,5 @@ # Data model and schema -This section describes the entities that YDB uses within DBs. -The YDB core lets you flexibly implement various storage primitives, so new entities may appear in the future. +This section describes the entities that {{ ydb-short-name }} uses within DBs. +The {{ ydb-short-name }} core lets you flexibly implement various storage primitives, so new entities may appear in the future. diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/pq.md b/ydb/docs/en/core/concepts/_includes/datamodel/pq.md index 52f6071407..34c72a32fb 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/pq.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/pq.md @@ -1,3 +1,4 @@ ## Persistent queues -A persistent queue consists of one or more partitions, where each partition is a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) [message queue](https://en.wikipedia.org/wiki/Message_queue) ensuring reliable delivery between two or more components. Data messages are untyped and constitute a data blob. Partitioning is a parallel processing tool that helps ensure high queue bandwidth. Mechanisms are provided to implement at least once and exactly once persistent queue delivery guarantees. A persistent queue in YDB is similar to a topic in [Apache Kafka](https://en.wikipedia.org/wiki/Apache_Kafka). +A persistent queue consists of one or more partitions, where each partition is a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) [message queue](https://en.wikipedia.org/wiki/Message_queue) ensuring reliable delivery between two or more components. Data messages are untyped and constitute a data blob. Partitioning is a parallel processing tool that helps ensure high queue bandwidth. Mechanisms are provided to implement at least once and exactly once persistent queue delivery guarantees. A persistent queue in {{ ydb-short-name }} is similar to a topic in [Apache Kafka](https://en.wikipedia.org/wiki/Apache_Kafka). + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/table.md b/ydb/docs/en/core/concepts/_includes/datamodel/table.md index 04285c76ae..0e0fdf619c 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/table.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/table.md @@ -1,6 +1,6 @@ ## Table {#table} -A table in YDB is a [relational table](https://en.wikipedia.org/wiki/Table_(database)) that contains a set of related data consisting of rows and columns. Each row is a set of cells designed to store values of certain types according to the data schema. The data schema defines the names and types of table columns. An example of a data schema is shown below. +A table in {{ ydb-short-name }} is a [relational table](https://en.wikipedia.org/wiki/Table_(database)) that contains a set of related data consisting of rows and columns. Each row is a set of cells designed to store values of certain types according to the data schema. The data schema defines the names and types of table columns. An example of a data schema is shown below.  @@ -8,38 +8,90 @@ A table in YDB is a [relational table](https://en.wikipedia.org/wiki/Table_(data Figure 1 shows the schema of a ```Series``` table with four columns, named ```SeriesId```, ```ReleaseDate```, ```SeriesInfo```, and ```Title```, with the ```Uint64?``` data type for the first two columns and ```String?``` for the others. The ```SeriesId``` column is declared the primary key. -YDB uses [YQL]{% if audience != "external" %}(https://yql.yandex-team.ru/docs/ydb/types/){% else %}(../../datatypes.md){% endif %} data types. {% if audience != "external" %} [Simple YQL data types](https://yql.yandex-team.ru/docs/ydb/types/primitive/) {% else %} [Simple YQL data types](../../../yql/reference/types/primitive.md) {% endif %} can be used as column types. All columns may contain a special NULL value to indicate a missing value. +{{ ydb-short-name }} uses [YQL]{% if audience != "external" %}(https://yql.yandex-team.ru/docs/ydb/types/){% else %}(../../datatypes.md){% endif %} data types. {% if audience != "external" %} [Simple YQL data types](https://yql.yandex-team.ru/docs/ydb/types/primitive/) {% else %} [Simple YQL data types](../../../yql/reference/types/primitive.md) {% endif %} can be used as column types. All columns may contain a special NULL value to indicate a missing value. -YDB tables always have one or more columns that make up the key ([primary key](https://en.wikipedia.org/wiki/Unique_key)). Each table row has a unique key value, so there can be no more than one row per key value. A YDB table is always ordered by key. This means that you can efficiently make point reads by key and range-based queries by key or key prefix (actually using an index). In the example above, the key columns are highlighted in gray and marked with a special sign. Tables consisting only of key columns are supported. However, you can't create tables without a primary key. +{{ ydb-short-name }} tables always have one or more columns that make up the key ([primary key](https://en.wikipedia.org/wiki/Unique_key)). Each table row has a unique key value, so there can be no more than one row per key value. {{ ydb-short-name }} tables are always ordered by key. This means that you can efficiently make point reads by key and range-based queries by key or key prefix (actually using an index). In the example above, the key columns are highlighted in gray and marked with a special sign. Tables consisting only of key columns are supported. However, you can't create tables without a primary key. Often, when you design a table schema, you already have a set of fields, which can naturally serve as the primary key. Be careful when selecting the key to avoid hotspots. -For example, if you insert data into a table with a monotonically increasing key, you write the data to the end of the table. However, since YDB splits table data by key range, your inserts are always processed by the same server, so you lose the main benefits of a distributed database. +For example, if you insert data into a table with a monotonically increasing key, you write the data to the end of the table. But since {{ ydb-short-name }} splits table data by key range, your inserts are always processed by the same server, so you lose the main benefits of a distributed database. To distribute the load evenly across different servers and avoid hotspots when working with large tables, we recommend hashing the natural key and using the hash as the first component of the primary key, as well as changing the order of the primary key components. ### Partitioning tables {#partitioning} A database table can be sharded by primary key value ranges. Each shard of the table is responsible for a specific range of primary keys. Key ranges served by different shards do not overlap. Different table shards can be served by different distributed database servers (including ones in different locations). They can also move independently between servers to enable rebalancing or ensure shard operability if servers or network equipment goes offline. -If there is not a lot of data, the table may consist of a single shard. As the amount of data served by the shard grows, YDB automatically splits the shard into two shards. The data is split by the median value of the primary key, depending on the amount of data: as soon as the shard exceeds the set data volume, it is split in two. +If there is not a lot of data or load, the table may consist of a single shard. As the amount of data served by the shard or the load on the shard grows, {{ ydb-short-name }} automatically splits this shard into two shards. The data is split by the median value of the primary key if the shard size exceeds the threshold. If partitioning by load is used, the shard first collects a sample of the requested keys (that can be read, written, and deleted) and, based on this sample, selects a key for partitioning to evenly distribute the load across new shards. So in the case of load-based partitioning, the size of new shards may significantly vary. -The shard split threshold and automatic splitting can be configured (enabled/disabled) individually for each database table. +The size-based shard split threshold and automatic splitting can be configured (enabled/disabled) individually for each database table. -In addition to automatically splitting shards, you can create an empty table with a predefined number of shards. You can manually set the exact shard key split range or evenly split into a predefined number of shards. In this case, ranges are created based on the first component of the primary key. You can set even splitting for tables that have an integer as the first component of the primary key. +In addition to automatically splitting shards, you can create an empty table with a predefined number of shards. You can manually set the exact shard key split range or evenly split into a predefined number of shards. In this case, ranges are created based on the first component of the primary key. You can set even splitting for tables that have a Uint64 or Uint32 integer as the first component of the primary key. + +Partitioning parameters refer to the table itself and not to secondary indexes built on its data. Each index is served by its own set of shards and decisions to split or merge its partitions are made independently based on the default settings. These settings may become available to users in the future like the settings of the main table. + +A split or merge operation usually takes about 500 milliseconds. During this time, the data involved in the operation becomes temporarily unavailable for reads and writes. Special wrapper methods in the {{ ydb-short-name }} SDK make automatic retries when receiving information saying that the shard is being split or merged, without raising it to the application level. Please note that if the system is overloaded for some reason (for example, due to a general shortage of CPU or insufficient DB disk throughput), split and merge operations may take longer. The following table partitioning parameters are defined in the data schema: -| Parameter name | Description | Type | Acceptable values | Update possibility | Reset capability | -| ------------- | --------- | --- | ------------------- | --------------------- | ------------------ | -| ```AUTO_PARTITIONING_BY_SIZE``` | Automatic partitioning by partition size | Enum | ```ENABLED```, ```DISABLED``` | Yes | No | -| ```AUTO_PARTITIONING_PARTITION_SIZE_MB``` | Preferred size of each partition, in MB | Uint64 | Natural numbers | Yes | No | -| ```AUTO_PARTITIONING_MIN_PARTITIONS_COUNT``` | The minimum number of partitions before automatic partition merging stops | Uint64 | Natural numbers | Yes | No | -| ```AUTO_PARTITIONING_MAX_PARTITIONS_COUNT``` | The maximum number of partitions before automatic partitioning stops | Uint64 | Natural numbers | Yes | No | -| ```UNIFORM_PARTITIONS``` | The number of partitions for uniform initial table partitioning. The type of the primary key's first column must be Uint64 or Uint32 | Uint64 | Natural numbers | No | No | -| ```PARTITION_AT_KEYS``` | Boundary values of keys for initial table partitioning. It's a list of boundary values separated by commas and surrounded with brackets. Each boundary value can be either a set of values of key columns (also separated by commas and surrounded with brackets) or a single value if only the values of the first key column are specified. Examples: ```(100, 1000)```, ```((100, "abc"), (1000, "cde"))``` | Expression | | No | No | +#### AUTO_PARTITIONING_BY_SIZE + +* Type: Enum (`ENABLED`, `DISABLED`) +* Default value: `ENABLED` + +Automatic partitioning by partition size. If the size of a partition exceeds the value specified by the [`AUTO_PARTITIONING_PARTITION_SIZE_MB`](#auto_partitioning_partition_size_mb) parameter, it is enqueued for splitting. If the total size of two or more adjacent partitions is less than 50% of the [`AUTO_PARTITIONING_PARTITION_SIZE_MB`](#auto_partitioning_partition_size_mb) parameter value, they are enqueued for merging. + +#### AUTO_PARTITIONING_BY_LOAD + +* Type: Enum (`ENABLED`, `DISABLED`) +* Default value: `DISABLED` + +Automatic partitioning by load. If a shard consumes more than 50% of the CPU for a few dozens of seconds, it is enqueued for splitting. If the total load on two or more adjacent shards uses less than 35% of a single CPU core within an hour, they are enqueued for merging. + +Performing split or merge operations uses the CPU and takes time. Therefore, when dealing with varying load, in addition to enabling this mode, we recommend that you set the [`AUTO_PARTITIONING_MIN_PARTITIONS_COUNT`](#auto_partitioning_min_partitions_count) parameter to a value other than 1 so that a decrease in the load does not lead to a decrease in the number of partitions below the required one and there is no need to split them again when the load increases. + +When choosing the minimum number of partitions, it makes sense to consider that one table partition can only be hosted on one server and use no more than 1 CPU core for data update operations. Hence, you can set the minimum number of partitions for a table on which a high load is expected to at least the number of nodes (servers) or, preferably, to the number of CPU cores allocated to the database. + +#### AUTO_PARTITIONING_PARTITION_SIZE_MB + +* Type: Uint64 +* Default value: 2000 MB ( 2GB ) + +The partition size threshold in MB. If exceeded, a shard is split. Valid when the [`AUTO_PARTITIONING_BY_SIZE`](#auto_partitioning_by_size) mode is enabled. + +#### AUTO_PARTITIONING_MIN_PARTITIONS_COUNT + +* Type: Uint64 +* Default value: 1 + +Partitions are only merged if their actual number exceeds the value specified by this parameter. When using automatic partitioning by load, we recommend that you set this parameter to a value other than 1, so that periodic load drops don't lead to a decrease in the number of partitions below the required one. + +#### AUTO_PARTITIONING_MAX_PARTITIONS_COUNT + +* Type: Uint64 +* Default value: 50 + +Partitions are only split if their number doesn't exceed the value specified by this parameter. With any automatic partitioning mode enabled, we recommend that you set a meaningful value for this parameter and monitor when the actual number of partitions approaches this value, otherwise splitting of partitions will sooner or later stop under an increase in data or load, which will lead to a failure. + +#### UNIFORM_PARTITIONS + +* Type: Uint64 +* Default value: Not applicable + +The number of partitions for uniform initial table partitioning. The type of the primary key's first column must be Uint64 or Uint32. A created table is immediately divided into the specified number of partitions. + +When automatic partitioning is enabled, make sure to set the correct [`AUTO_PARTITIONING_MIN_PARTITIONS_COUNT`](#auto_partitioning_min_partitions_count) parameter value so that all partitions are not merged into one immediately after creating the table. + +#### PARTITION_AT_KEYS + +* Type: Expression +* Default value: Not applicable + +Boundary values of keys for initial table partitioning. It's a list of boundary values separated by commas and surrounded with brackets. Each boundary value can be either a set of values of key columns (also separated by commas and surrounded with brackets) or a single value if only the values of the first key column are specified. Examples: ```(100, 1000)```, ```((100, "abc"), (1000, "cde"))```. + +When automatic partitioning is enabled, make sure to set the correct [`AUTO_PARTITIONING_MIN_PARTITIONS_COUNT`](#auto_partitioning_min_partitions_count) parameter value so that all partitions are not merged into one immediately after creating the table. ### Reading data from replicas {#read_only_replicas} -When making queries in YDB, the actual execution of a query to each shard is performed at a single point serving the distributed transaction protocol. By storing data in shared storage, you can run one or more shard followers without allocating additional space in the storage: the data is already stored in replicated format and you can serve more than one reader (but the writer is still strictly one at every moment). +When making queries in {{ ydb-short-name }}, the actual execution of a query to each shard is performed at a single point serving the distributed transaction protocol. By storing data in shared storage, you can run one or more shard followers without allocating additional space in the storage: the data is already stored in replicated format and you can serve more than one reader (but the writer is still strictly one at every moment). Reading data from followers allows you: @@ -56,13 +108,13 @@ You can enable running read replicas for each shard of the table in the table da The internal state of each of the followers is restored exactly and fully consistently from the leader state. -Besides the data status in storage, followers also receive a stream of updates from the leader. Updates are sent in real time, immediately after the commit to the log. However, they are sent asynchronously, resulting in some delay (usually no more than dozens of milliseconds, but sometimes longer in the event of cluster connectivity issues) in applying updates to followers relative to their commit on the leader. Therefore, reading data from followers is only supported in the [transaction mode](../transactions.md#modes) `StaleReadOnly()`. +Besides the data status in storage, followers also receive a stream of updates from the leader. Updates are sent in real time, immediately after the commit to the log. However, they are sent asynchronously, resulting in some delay (usually no more than dozens of milliseconds, but sometimes longer in the event of cluster connectivity issues) in applying updates to followers relative to their commit on the leader. Therefore, reading data from followers is only supported in the [transaction mode](../transactions#modes) `StaleReadOnly()`. If there are multiple followers, their delay from the leader may vary: although each follower of each of the shards retains internal consistency, artifacts may be observed between different shards. Please allow for this in your application code. For that same reason, it's currently impossible to perform cross-shard transactions from followers. ### Deleting expired data (TTL) {#ttl} -YDB enables an automatic background delete of expired data. The table data schema may have a column with a Datetime or Timestamp value defined. Its value for all rows will be compared with the current time in the background. Rows for which the current time becomes greater than the column value, factoring in the specified delay, will be deleted. +{{ ydb-short-name }} enables an automatic background delete of expired data. The table data schema may have a column with a Datetime or Timestamp value defined. Its value for all rows will be compared with the current time in the background. Rows for which the current time becomes greater than the column value, factoring in the specified delay, will be deleted. | Parameter name | Type | Acceptable values | Update possibility | Reset capability | | ------------- | --- | ------------------- | --------------------- | ------------------ | @@ -70,6 +122,17 @@ YDB enables an automatic background delete of expired data. The table data schem For more information about deleting expired data, see [Time to Live (TTL)](../../../concepts/ttl.md) +### Renaming a table {#rename} + +{{ ydb-short-name }} lets you rename an existing table, move it to another directory of the same database, or replace one table with another, deleting the data in the replaced table. Only the metadata of the table is changed by operations (for example, its path and name). The table data is neither moved nor overwritten. + +Operations are performed in isolation, the external process sees only two states of the table: before and after the operation. This is critical, for example, for table replacement: the data of the replaced table is deleted by the same transaction that renames the replacing table. During the replacement, there might be errors in queries to the replaced table that have [retryable statuses](../../../reference/ydb-sdk/error_handling.md#termination-statuses). + +The speed of renaming is determined by the type of data transactions currently running against the table and doesn't depend on the table size. + +- [Renaming a table in YQL](../../../yql/reference/syntax/alter_table.md#rename) +- [Renaming a table via the CLI](../../../reference/ydb-cli/commands/tools/rename.md) + ### Bloom filter {#bloom-filter} Using a [Bloom filter](https://en.wikipedia.org/wiki/Bloom_filter) lets you more efficiently determine if some keys are missing in a table when making multiple point queries by primary key. This reduces the number of required disk I/O operations but increases the amount of memory consumed. diff --git a/ydb/docs/en/core/concepts/_includes/distributed_storage/common_scheme_ydb.md b/ydb/docs/en/core/concepts/_includes/distributed_storage/common_scheme_ydb.md new file mode 100644 index 0000000000..c19ff72844 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/distributed_storage/common_scheme_ydb.md @@ -0,0 +1,129 @@ +## General YDB schema + +An approximate general YDB schema is shown below. + + + +### Nodes + +A single YDB installation consists of a *cluster* that is divided into *nodes*. A node is a single process in the system, usually kikimr. This node is part of a cluster and can exchange data with its other nodes via *Interconnect*. Each *node* has its own ID which is usually named NodeId. NodeID is an integer from 1, consisting of 20 bits. NodeID 0 is reserved for internal needs and usually indicates the current node or no node. + +A number of services are run on each node and implemented via *actors*. + +Nodes can be static and dynamic. + +A configuration of static nodes, that is, their complete list with the address for connecting via Interconnect specified, is stored in a configuration file and is read once when the process is started. A set of static nodes changes very rarely. This usually happens when expanding clusters or moving nodes from one physical machine to another. To change the set of static nodes, apply the updated configuration to **every** node and then perform a rolling restart of the entire cluster. + +Dynamic nodes are not known in advance and are added to the system as new processes are started. This may be due, for example, to the creation of new tenants in YDB installations as a database. When registering a dynamic node, its process first connects to one of the static nodes via gRPC, transmits information about itself through a special service called Node Broker, and receives the NodeID to log in with in response. The mechanism for assigning nodes is pretty much similar to the DHCP in the context of distributing IP addresses. + +### Tablets + +Special microservices called *tablets* are run on each node. Each tablet has a specific type and ID and is a singleton, meaning that only one tablet with a specific ID can be running in the entire cluster at any given time. A tablet can be started on any node that is suitable for it. An important characteristic of a tablet is its *generation* that increases with each subsequent run. Please note that due to the distributed nature of the system and in case of various kinds of issues, such as with network partitioning, a situation may arise when the same tablet will actually be running on two different nodes at the same time. However, BlobStorage guarantees that only one of them will be able to successfully complete operations that change its state and the generation that each successful operation is performed in will not decrease over time. + +You can find out on what node the tablet in the current generation is running through the *StateStorage* service. To send messages to tablets, use a special set of libraries named *tablet pipe*. With it, knowing the ID of the target tablet, you can easily send the desired message to it. + +A tablet can be divided into two parts: the basic tablet and the user logic. + +The basic tablet is a set of tables, each of which may consist of one or more key columns of an arbitrary type and a set of data columns. Each table may have its own schema. In addition, tables can be created and deleted while the tablet is running. The interface of the basic tablet lets you perform read and update operations on these tables. + +The user logic is located between the basic tablet and the user and lets you process specific requests for this type of tablet, reliably saving changes in BlobStorage. A commonly used template for tablet operation is storing all data in memory, reading it only at the start, and synchronously changing the data in memory and in storage after a successful commit. + +#### How does a tablet store data and what data is it? + +The basic tablet is an LSM tree that contains all of its table data. One level below the basic tablet is BlobStorage that, roughly speaking, is KeyValue storage that stores binary large objects (blobs). *BLOB* is a binary fragment from 1 byte to 10 MB in size, which has a fixed ID (that is usually called *BlobId* and is of the TLogoBlobID type) and contains related data. Storage is immutable, meaning that only one value corresponds to each ID and it cannot change over time. You can write and read a blob and then delete it when you no longer need it. + +For BlobStorage, blobs are an opaque entity. A tablet can store several types of blobs. The most frequently written blob is a log blob (meaning a recovery log). A tablet's log is arranged in a list of blobs, each of which contains information about the change being made to the tables. When run, the tablet finds the last blob in the log and then recursively reads all related blobs following the links. The log may also mention snapshot blobs, which is a type of blobs that contain data from multiple log blobs after merging them (the merge operation in the LSM tree). + +The tablet writes blobs of different types to different *channels*. A channel specifies the branch of storage to store blobs in and performs various functions, such as: + +1. Choosing the storage type (different channels can be linked to different types of storage devices: SSD, HDD, or NVME). +1. Load balancing, because each channel has a limit on IOPS, available space and bandwidth. +1. Specifying the data type. When restoring the log, only the blobs from the null channel are read, which lets you distinguish them from other blobs. + +#### Blob ID format + +Each blob has a 192-bit ID consisting of the following fields (in the order used for sorting): + +1. TabletId (64 bits): ID of the blob owner tablet. +1. Channel (8 bits): Channel sequence number. +1. Generation (32 bits): Number of the generation in which the tablet that wrote this blob was run. +1. Step (32 bits): Internal number of the blob group within the Generation. +1. Cookie (24 bits): ID that can be used if the Step is insufficient. +1. CrcMode (2 bits): Selects mode for redundant blob integrity control at the BlobStorage level. +1. BlobSize (26 bits): Blob data size. +1. PartId (4 bits): Part number when blob erasure coding is used. At the "BlobStorage <-> tablet" interaction level, this parameter is always 0, meaning the entire blob. + +Two blobs are considered different if at least one of the first five parameters (TabletId, Channel, Generation, Step, or Cookie) differs in their IDs. So it is impossible to write two blobs that only differ in BlobSize and/or CrcMode. + +For debugging purposes, there is string blob ID formatting that has interactions `[TabletId:Generation:Step:Channel:Cookie:BlobSize:PartId]`, for example, `[12345:1:1:0:0:1000:0]`. + +When writing a blob, the tablet selects the Channel, Step, and Cookie parameters. The TabletId is fixed and must indicate the tablet performing the write operation, while the Generation parameter must indicate the generation that the tablet performing the operation is running in. + +When performing reads, the blob ID is specified, which can be arbitrary, but preferably preset. + +#### Groups + +Blobs are written in a logical entity called *group*. On each node, a special actor called DS proxy is created for each group that the blob is written to. This actor is responsible for performing all operations related to the group. The actor is created automatically through the NodeWarden service that will be described below. + +Physically, a group is a set of multiple physical devices (OS block devices) that are located on different nodes so that the failure of one device correlates as little as possible with the failure of another device. These devices are usually located in different racks or different datacenters. On each of these devices, some space is allocated for the group, which is managed by a special service called *VDisk*. Each VDisk runs on top of a block device from which it is separated by another service called *PDisk*. Blobs are broken into fragments based on *erasure coding* with these fragments written to VDisks. Before splitting into fragments, optional encryption of the data in the group can be performed. + +This scheme is shown in the figure below. + + + +VDisks from different groups are shown as multicolored squares; one color stands for one group. + +A group can be treated as a set of VDisks: + + + +Each VDisk within the group has a sequence number, the disks are numbered from 0 to N-1, where N is the number of disks in the group. + +In addition, the group disks are combined into fail domains, while the fail domains are combined into fail realms. As a rule, each fail domain has exactly one disk inside (although, in theory, it may have more, but this has found no application in practice), and multiple fail realms are only used for groups that host their data in three datacenters at once. In addition to the sequence number in the group, each VDisk is assigned an ID that consists of a fail realm index, fail domain index inside the fail realm, and the VDisk index inside the fail domain. In string form, this ID is written as `VDISK[GroupId:GroupGeneration:FailRealm:FailDomain:VDisk]`. + +All the fail realms have the same number of fail domains and all the fail domains have the same number of disks inside. The number of the fail realms, the number of the fail domains inside the fail realm, and the number of the disks inside the fail domain make up the geometry of the group. The geometry depends on the way the data is encoded in the group. For example, for block-4-2 numFailRealms = 1, numFailDomainsInFailRealm >= 8 (only 8 fail realms are used in practice), numVDisksInFailDomain >= 1 (strictly 1 fail domain is used in practice). For mirror-3-dc numFailRealms >= 3, numFailDomainsInFailRealm >= 3, and numVDisksInFailDomain >= 1 (3x3x1 are used). + +Each PDisk has an ID that consists of the number of the node that it is running on and the internal number of the PDisk inside this node. This ID is usually written as NodeId:PDiskId. For example, 1:1000. If you know the PDisk ID, you can calculate the service ActorId of this disk and send it a message. + +Each VDisk runs on top of a specific PDisk and has a *slot ID* consisting of three fields (NodeID:PDiskId:VSlotId) and the above-mentioned VDisk ID. Strictly speaking, there are different concepts: a slot is the space reserved on the PDisk and occupied by the VDisk, and the VDisk is a group component that occupies a certain slot and performs operations on it. Similarly to PDisks, if you know the slot ID, you can calculate the service ActorId of the running VDisk and send it a message. To send messages from the DS proxy to the VDisk, an intermediate actor called *BS_QUEUE* is used. + +The composition of each group is not fixed: it may change while using the system. For this purpose, the concept of "group generation" is introduced. Each "GroupId:GroupGeneration" pair corresponds to a fixed set of slots (a vector that consists of N slot IDs, where N is the size of the group), storing the data of the entire group. *Please note that a group generation and a tablet generation are not related in any way*. + +As a rule, groups of two adjacent generations differ by no more than one slot. + +#### Subgroups + +For each blob, a special concept of a *subgroup* is introduced, which is an ordered subset of group disks with a strictly fixed number of elements, depending on the type of encoding (the number of elements in the group must be at least the same), where the data of this blob will be stored. For single-datacenter groups with conventional encoding, this subset is selected as the first N elements of a cyclic disk permutation in the group, where the permutation depends on the BlobId hash. + +Each disk in the subgroup corresponds to a disk in the group, but is limited by the allowed number of stored blobs. For example, to encode block-4-2 with four data parts and two parity parts, the functional purpose of disks in the subgroup is as follows: + +| Number in the subgroup | Possible PartIds | +| ------------------- | ------------------- | +| 0 | 1 | +| 1 | 2 | +| 2 | 3 | +| 3 | 4 | +| 4 | 5 | +| 5 | 6 | +| 6 | 1,2,3,4,5,6 | +| 7 | 1,2,3,4,5,6 | + +In this case, PartID=1..4 corresponds to the data parts (which are obtained by splitting the original blob into 4 equal parts), and PartID=5..6 are parity parts. Disks numbered 6 and 7 in the subgroup are called *handoff disks*. Any part, either one or more, can be written to them. Disks 0..5 can only store the corresponding blob parts. + +In practice, when performing writes, the system tries to write 6 parts to the first 6 disks of the subgroup and, in the vast majority of cases, these attempts are successful. However, if any of the disks is not available, the write operation fails and the system uses handoff disks where parts of the disks that did not respond in time are sent. This may result in a situation when multiple parts of the same blob are written to one handoff disk. This is acceptable, although it makes no sense in terms of storage: each part must have its own unique disk. + +#### Tablet channel history + +As mentioned above, each group has a fixed amount of data that can fit into it and also divides the bandwidth by throughput and the number of operations per second between all consumers. The load on tablets may vary. As a result a group may become overloaded. In this regard, the concept of history is introduced, with which, for each tablet, knowing a blob's Channel and Generation, you can determine the group that this blob is written to. + +This mechanism works as follows: + + + +For each channel, the TTabletStorageInfo structure contains the TTabletChannelInfo substructure with generation ranges and the group number corresponding to each range. The ranges are strictly adjacent to each other, the last range is open. Group numbers may overlap in different ranges and even across different channels: this is not prohibited and is quite common. + +When writing a blob, a tablet selects the most recent range for the corresponding channel, since the write operation is always performed on behalf of the current generation of the tablet. When reading a blob, the group number is fetched based on the BlobId.Generation of the blob being read. + +#### Garbage collection + +#### Tablet suspension diff --git a/ydb/docs/en/core/concepts/_includes/distributed_storage/detailed_distributed_storage.md b/ydb/docs/en/core/concepts/_includes/distributed_storage/detailed_distributed_storage.md new file mode 100644 index 0000000000..ce160da18e --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/distributed_storage/detailed_distributed_storage.md @@ -0,0 +1,357 @@ +## BlobStorage implementation details + +### Managing BlobStorage resources on nodes via NodeWarden + +### How PDisk works + +#### Data format + +### How VDisk works + +#### Large and small blobs + +#### Fresh segment + +#### SSTable + +#### Syncing metadata + +#### Replicating data + +#### Data format + +##### Donor disks + +### How DS proxy works + +## BS_CONTROLLER + +### Communicating with NodeWarden + +### Box and Storage Pool + +### Self Heal + +### Entities and tables + +All information stored in BS_CONTROLLER can be grouped into: + +1. BS_CONTROLLER settings. +1. Low-level configuration (PDisks, VDisks, and groups). +1. Top-level configuration (Boxes and StoragePools). +1. Runtime information (persistent disk and group metrics, the status of scrubbing of individual VDisks, information about node drive serial numbers). +1. OperationLogs. + +The operation logic is built so that all table data at the start of the BS_CONTROLLER tablet is loaded into memory and stored there for the entire time of its operation (with the exception of OperationLog). When executing transactions, the data is updated in tables and in memory. + +The tables are discussed in more detail below. + +#### BS_CONTROLLER settings + +The settings are stored in the State table. + +#### Low-level configuration + +Low-level configuration includes information about the available cluster resources: PDisks, VDisks, and groups. These tables are interconnected via PK <-> FK links. + +#### Top-level configuration + +Top-level configuration stores data necessary for linking Boxes and nodes where they are located... + +#### Database schema + +Box + +| Column | ID | PK | Type | Default | Destination | +| ------------ | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| Name | 2 | | Utf8 | | | +| Generation | 3 | | Uint64 | | | + +BoxHost + +| Column | ID | PK | Type | Default | Destination | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| FQDN | 3 | * | Utf8 | | | +| IcPort | 4 | * | Int32 | | | +| HostConfigId | 5 | | Uint64 | | | + +BoxHostV2 + +| Column | ID | PK | Type | Default | Destination | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| FQDN | 2 | * | Utf8 | | | +| IcPort | 3 | * | Int32 | | | +| HostConfigId | 4 | | Uint64 | | | + +BoxStoragePool + +| Column | ID | PK | Type | Default | Destination | +| ---------------------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| Name | 20 | | Utf8 | | | +| ErasureSpecies | 3 | | Int32 | | | +| RealmLevelBegin | 4 | | Int32 | | | +| RealmLevelEnd | 5 | | Int32 | | | +| DomainLevelBegin | 6 | | Int32 | | | +| DomainLevelEnd | 7 | | Int32 | | | +| NumFailRealms | 8 | | Int32 | | | +| NumFailDomainsPerFailRealm | 9 | | Int32 | | | +| NumVDisksPerFailDomain | 10 | | Int32 | | | +| VDiskKind | 11 | | Int32 | | | +| SpaceBytes | 12 | | Uint64 | | | +| WriteIOPS | 13 | | Uint64 | | | +| WriteBytesPerSecond | 14 | | Uint64 | | | +| ReadIOPS | 15 | | Uint64 | | | +| ReadBytesPerSecond | 16 | | Uint64 | | | +| InMemCacheBytes | 17 | | Uint64 | | | +| Kind | 18 | | Utf8 | | | +| NumGroups | 19 | | Uint32 | | | +| Generation | 21 | | Uint64 | | | +| EncryptionMode | 22 | | Uint32 | 0 | | +| SchemeshardId | 23 | | Uint64 | | | +| PathItemId | 24 | | Uint64 | | | +| RandomizeGroupMapping | 25 | | Bool | false | | + +BoxStoragePoolPDiskFilter + +| Column | ID | PK | Type | Default | Destination | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| Type | 3 | * | Int32 | | | +| SharedWithOs | 4 | * | Bool | | | +| ReadCentric | 5 | * | Bool | | | +| Kind | 6 | * | Uint64 | | | + +BoxStoragePoolUser + +| Column | ID | PK | Type | Default | Destination | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| UserId | 3 | * | String | | | + +BoxUser + +| Column | ID | PK | Type | Default | Destination | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| UserId | 2 | * | String | | | + +DriveSerial + +| Column | ID | PK | Type | Default | Destination | +| ------------- | ---- | ---- | -------- | -------------- | ------------ | +| Serial | 1 | * | String | | | +| BoxId | 2 | | Uint64 | | | +| NodeId | 3 | | Uint32 | | | +| PDiskId | 4 | | Uint32 | | | +| Guid | 5 | | Uint64 | | | +| LifeStage | 6 | | Uint32 | | | +| Kind | 7 | | Uint64 | | | +| PDiskType | 8 | | Int32 | | | +| PDiskConfig | 9 | | String | | | + +DriveStatus + +| Column | ID | PK | Type | Default | Destination | +| ----------- | ---- | ---- | -------- | -------------- | ------------ | +| FQDN | 1 | * | Utf8 | | | +| IcPort | 2 | * | Int32 | | | +| Path | 3 | * | Utf8 | | | +| Status | 4 | | Uint32 | | | +| Timestamp | 5 | | Uint64 | | | + +Group + +| Column | ID | PK | Type | Default | Destination | +| ---------------------- | ---- | ---- | -------- | -------------- | ------------ | +| ID | 1 | * | Uint32 | | | +| Generation | 2 | | Uint32 | | | +| ErasureSpecies | 3 | | Uint32 | | | +| Owner | 4 | | Uint64 | | | +| DesiredPDiskCategory | 5 | | Uint64 | | | +| DesiredVDiskCategory | 6 | | Uint64 | | | +| EncryptionMode | 7 | | Uint32 | 0 | | +| LifeCyclePhase | 8 | | Uint32 | 0 | | +| MainKeyId | 9 | | String | | | +| EncryptedGroupKey | 10 | | String | | | +| GroupKeyNonce | 11 | | Uint64 | 0 | | +| MainKeyVersion | 12 | | Uint64 | 0 | | +| Down | 13 | | Bool | false | | +| SeenOperational | 14 | | Bool | false | | + +GroupLatencies + +| Column | ID | PK | Type | Default | Destination | +| ----------------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupId | 1 | * | Uint32 | | | +| PutTabletLogLatencyUs | 4 | | Uint32 | | | +| PutUserDataLatencyUs | 5 | | Uint32 | | | +| GetFastLatencyUs | 6 | | Uint32 | | | + +GroupStoragePool + +| Column | ID | PK | Type | Default | Destination | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupId | 1 | * | Uint32 | | | +| BoxId | 2 | | Uint64 | | | +| StoragePoolId | 3 | | Uint64 | | | + +HostConfig + +| Column | ID | PK | Type | Default | Destination | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| HostConfigId | 1 | * | Uint64 | | | +| Name | 2 | | Utf8 | | | +| Generation | 3 | | Uint64 | | | + +HostConfigDrive + +| Column | ID | PK | Type | Default | Destination | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| HostConfigId | 1 | * | Uint64 | | | +| Path | 2 | * | Utf8 | | | +| Type | 3 | | Int32 | | | +| SharedWithOs | 4 | | Bool | | | +| ReadCentric | 5 | | Bool | | | +| Kind | 6 | | Uint64 | | | +| PDiskConfig | 7 | | String | | | + +MigrationEntry + +| Column | ID | PK | Type | Default | Destination | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| PlanName | 1 | * | String | | | +| EntryIndex | 2 | * | Uint64 | | | +| GroupId | 3 | | Uint32 | | | +| OriginNodeId | 4 | | Uint32 | | | +| OriginPDiskId | 5 | | Uint32 | | | +| OriginVSlotId | 6 | | Uint32 | | | +| TargetNodeId | 7 | | Uint32 | | | +| TargetPDiskId | 8 | | Uint32 | | | +| Done | 9 | | Bool | | | + +MigrationPlan + +| Column | ID | PK | Type | Default | Destination | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| Name | 1 | * | String | | | +| State | 2 | | Uint32 | | | +| Size | 3 | | Uint64 | | | +| Done | 4 | | Uint64 | | | + +Node + +| Column | ID | PK | Type | Default | Destination | +| ------------------- | ---- | ---- | -------- | -------------- | ------------ | +| ID | 1 | * | Uint32 | | | +| NextPDiskID | 2 | | Uint32 | | | +| NextGroupKeyNonce | 9 | | Uint64 | 0 | | + +OperationLog + +| Column | ID | PK | Type | Default | Destination | +| --------------- | ---- | ---- | -------- | ------------------- | ------------ | +| Index | 1 | * | Uint64 | | | +| Timestamp | 2 | | Uint64 | | | +| Request | 3 | | String | | | +| Response | 4 | | String | | | +| ExecutionTime | 5 | | Uint64 | TDuration::Zero() | | + +PDisk + +| Column | ID | PK | Type | Default | Destination | +| ---------------- | ---- | ---- | -------- | -------------- | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| Path | 3 | | Utf8 | | | +| Category | 4 | | Uint64 | | | +| Guid | 7 | | Uint64 | | | +| SharedWithOs | 8 | | Bool | | | +| ReadCentric | 9 | | Bool | | | +| NextVSlotId | 10 | | Uint32 | 1000 | | +| PDiskConfig | 11 | | String | | | +| Status | 12 | | Uint32 | | | +| Timestamp | 13 | | Uint64 | | | +| ExpectedSerial | 14 | | String | | | +| LastSeenSerial | 15 | | String | | | +| LastSeenPath | 16 | | String | | | + +PDiskMetrics + +| Column | ID | PK | Type | Default | Destination | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| Metrics | 3 | | String | | | + +ScrubState + +| Column | ID | PK | Type | Default | Destination | +| ---------------------- | ---- | ---- | -------- | ------------------ | ------------ | +| NodeId | 1 | * | Uint32 | | | +| PDiskId | 2 | * | Uint32 | | | +| VSlotId | 3 | * | Uint32 | | | +| State | 5 | | String | | | +| ScrubCycleStartTime | 6 | | Uint64 | TInstant::Zero() | | +| ScrubCycleFinishTime | 8 | | Uint64 | TInstant::Zero() | | +| Success | 7 | | Bool | false | | + +State + +| Column | ID | PK | Type | Default | Destination | +| -------------------------- | ---- | ---- | -------- | --------------------------------------------- | ------------ | +| FixedKey | 1 | * | Bool | | | +| NextGroupID | 2 | | Uint32 | | | +| SchemaVersion | 4 | | Uint32 | 0 | | +| NextOperationLogIndex | 5 | | Uint64 | | | +| DefaultMaxSlots | 6 | | Uint32 | 16 | | +| InstanceId | 7 | | String | | | +| SelfHealEnable | 8 | | Bool | false | | +| DonorModeEnable | 9 | | Bool | true | | +| ScrubPeriodicity | 10 | | Uint32 | 86400*30 | | +| SerialManagementStage | 11 | | Uint32 | | | +| NextStoragePoolId | 12 | | Uint64 | 0 | | +| PDiskSpaceMarginPromille | 13 | | Uint32 | 150 | | +| GroupReserveMin | 14 | | Uint32 | 0 | | +| GroupReservePart | 15 | | Uint32 | 0 | | +| MaxScrubbedDisksAtOnce | 16 | | Uint32 | Max<ui32>() | | +| PDiskSpaceColorBorder | 17 | | Uint32 | NKikimrBlobStorage::TPDiskSpaceColor::GREEN | | + +VDiskMetrics + +| Column | ID | PK | Type | Default | Destination | +| ----------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupID | 1 | * | Uint32 | | | +| GroupGeneration | 2 | * | Uint32 | | | +| Ring | 3 | * | Uint32 | | | +| FailDomain | 4 | * | Uint32 | | | +| VDisk | 5 | * | Uint32 | | | +| Metrics | 11 | | String | | | + +VSlot + +| Column | ID | PK | Type | Default | Destination | +| --------------------- | ---- | ---- | -------- | ------------------ | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| VSlotID | 3 | * | Uint32 | | | +| Category | 4 | | Uint64 | | | +| GroupID | 5 | | Uint32 | | | +| GroupGeneration | 6 | | Uint32 | | | +| RingIdx | 7 | | Uint32 | | | +| FailDomainIdx | 8 | | Uint32 | | | +| VDiskIdx | 9 | | Uint32 | | | +| GroupPrevGeneration | 10 | | Uint32 | 0 | | +| Mood | 11 | | Uint32 | 0 | | +| LastSeenReady | 12 | | Uint64 | TInstant::Zero() | | + +### Messages via pipe + +### Transaction types + diff --git a/ydb/docs/en/core/concepts/_includes/distributed_storage/intro.md b/ydb/docs/en/core/concepts/_includes/distributed_storage/intro.md new file mode 100644 index 0000000000..9828485727 --- /dev/null +++ b/ydb/docs/en/core/concepts/_includes/distributed_storage/intro.md @@ -0,0 +1,3 @@ +# Disk subsystem of a cluster aka YDB BlobStorage + +YDB BlobStorage is a YDB subsystem that is responsible for reliable data storage. diff --git a/ydb/docs/en/core/concepts/_includes/index/intro.md b/ydb/docs/en/core/concepts/_includes/index/intro.md index 3ad1dc709c..bca0c2eb47 100644 --- a/ydb/docs/en/core/concepts/_includes/index/intro.md +++ b/ydb/docs/en/core/concepts/_includes/index/intro.md @@ -2,10 +2,9 @@ title: Yandex Database (YDB). DBMS overview description: "Yandex Database (YDB): is a horizontally scalable distributed fault-tolerant DBMS. YDB is designed to meet high performance requirements. For example, a typical server can handle dozens of thousands of queries per second. The system is designed to handle hundreds of petabytes of data." --- +# {{ ydb-short-name }} overview -# Yandex Database (YDB) Overview - -*{{ ydb-full-name }}* ({{ ydb-short-name }}) is a horizontally scalable distributed fault-tolerant DBMS. {{ ydb-short-name }} is designed to meet high performance requirements. For example, a typical server can handle dozens of thousands of queries per second. The system is designed to handle hundreds of petabytes of data. {{ ydb-short-name }} can operate in single data center and geo-distributed (cross data center) modes on a cluster of thousands of servers. +*{{ ydb-short-name }}* ({{ ydb-short-name }}) is a horizontally scalable distributed fault-tolerant DBMS. {{ ydb-short-name }} is designed to meet high performance requirements. For example, a typical server can handle dozens of thousands of queries per second. The system is designed to handle hundreds of petabytes of data. {{ ydb-short-name }} can operate in single data center and geo-distributed (cross data center) modes on a cluster of thousands of servers. {{ ydb-short-name }} provides: @@ -15,7 +14,7 @@ description: "Yandex Database (YDB): is a horizontally scalable distributed faul * High availability with automatic failover in case a server, rack, or availability zone goes offline. * Automatic data partitioning as data or load grows. -To interact with {{ ydb-short-name }}, you can use the [YDB CLI](../../../reference/ydb-cli/index.md) or [SDK](../../../reference/ydb-sdk/index.md) for {% if oss %}C++,{% endif %} Java, Python, Node.js, PHP, and Go. +To interact with {{ ydb-short-name }}, you can use the [{{ ydb-short-name }} CLI](../../../reference/ydb-cli/index.md) or [SDK](../../../reference/ydb-sdk/index.md) for {% if oss %}C++,{% endif %} Java, Python, Node.js, PHP, and Go. {{ ydb-short-name }} supports a relational [data model](../../../concepts/datamodel.md) and manages tables with a predefined schema. To make it easier to organize tables, directories can be created like in the file system. @@ -26,3 +25,4 @@ Database commands are mainly written in YQL, an SQL dialect. This gives the user {{ ydb-short-name }} natively supports different processing options, such as [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) and [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing). The current version offers limited analytical query support. This is why we can say that {{ ydb-short-name }} is currently an OLTP database. {{ ydb-short-name }} is used in Yandex services as a high-performance [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) DBMS. {{ yandex-cloud }} services such as Yandex Object Storage and Yandex Block Storage use {{ ydb-short-name }} for storing data and are based on its components. + diff --git a/ydb/docs/en/core/concepts/_includes/index/when_use.md b/ydb/docs/en/core/concepts/_includes/index/when_use.md index d5873d7a43..915fdf4880 100644 --- a/ydb/docs/en/core/concepts/_includes/index/when_use.md +++ b/ydb/docs/en/core/concepts/_includes/index/when_use.md @@ -1,11 +1,12 @@ ## Use cases -YDB can be used as an alternative solution in the following cases: +{{ ydb-short-name }} can be used as an alternative solution in the following cases: * When using NoSQL systems if strong data consistency is required. * When using NoSQL systems if you need to make transactional changes to data stored in different rows of one or more tables. * In systems that need to process and store large amounts of data and allow for almost infinite horizontal scalability (using industrial clusters of 5000+ nodes, processing millions of RPS and storing petabytes of data). -* In systems with low load, when support for a separate DB instance will be a waste of money (consider using YDB in serverless mode instead). +* In systems with low load, when support for a separate DB instance will be a waste of money (consider using {{ ydb-short-name }} in serverless mode instead). * In systems with unpredictable or seasonally fluctuating load (you can add/reduce computing resources on request and/or in serverless mode). * In high-load systems that shard load across relational DB instances. -* When developing a new product that has no reliable forecast of expected load or is natively designed for an expected high load beyond the capabilities of traditional relational databases.
\ No newline at end of file +* When developing a new product that has no reliable forecast of expected load or is natively designed for an expected high load beyond the capabilities of traditional relational databases. + diff --git a/ydb/docs/en/core/concepts/_includes/limits-ydb.md b/ydb/docs/en/core/concepts/_includes/limits-ydb.md index 2431f3f445..591a65d3b6 100644 --- a/ydb/docs/en/core/concepts/_includes/limits-ydb.md +++ b/ydb/docs/en/core/concepts/_includes/limits-ydb.md @@ -1,13 +1,13 @@ # Database limits -This section describes the parameters of limits set in YDB. +This section describes the parameters of limits set in {{ ydb-short-name }}. ## Schema object limits The table below shows the limits that apply to schema objects: tables, databases, and columns. The_Object_ column specifies the type of schema object that the limit applies to. The _Error type_ column shows the status that the query ends with if an error occurs. For more information about statuses, see [Error handling in the API](../../reference/ydb-sdk/error_handling.md). -| Objects | Limit | Value | Explanation | Internal<br>name | Error<br>type | +| Object | Limit | Value | Explanation | Internal<br>name | Error<br>type | | :--- | :--- | :--- | :--- | :---: | :---: | | Database | Maximum path depth | 32 | Maximum number of nested path elements (directories, tables) | MaxDepth | SCHEME_ERROR | | Database | Maximum number of paths (schema objects) | 200,000 | Maximum number of path elements (directories, tables) in the database | MaxPaths | GENERIC_ERROR | @@ -38,6 +38,7 @@ The table below lists the limits that apply to query execution. The _Call_ colum | :--- | :--- | :--- | :--- | :---: | | Maximum number of rows in query results | 1000 | ExecuteDataQuery | Complete results of some queries executed using the `ExecuteDataQuery` method may contain more rows than allowed. In this case, the maximum allowed number of rows will be returned in response to the query and the result will have the `truncated` flag | SUCCESS | | Maximum query result size | 50 MB | ExecuteDataQuery | Complete results of some queries may exceed the set limit. In this case, the query fails and no data is returned | PRECONDITION_FAILED | -| Maximum number of sessions per cluster node | 1000 | CreateSession | Using the library for working with YDB, an application can create sessions within a connection. Sessions are linked to a node. You can create a limited number of sessions with a single node | OVERLOADED | +| Maximum number of sessions per cluster node | 1000 | CreateSession | Using the library for working with {{ ydb-short-name }}, an application can create sessions within a connection. Sessions are linked to a node. You can create a limited number of sessions with a single node | OVERLOADED | | Maximum query text length | 10 KB | ExecuteDataQuery | Limit on the length of YQL query text | BAD_REQUEST | | Maximum size of parameter values | 50 MB | ExecuteDataQuery | Limit on the total size of parameters passed when executing a previously prepared query | BAD_REQUEST | + diff --git a/ydb/docs/en/core/concepts/_includes/scan_query.md b/ydb/docs/en/core/concepts/_includes/scan_query.md index 1060320a81..0114586d1c 100644 --- a/ydb/docs/en/core/concepts/_includes/scan_query.md +++ b/ydb/docs/en/core/concepts/_includes/scan_query.md @@ -1,4 +1,4 @@ -# Scan Queries in Yandex Database +# Scan queries in {{ ydb-short-name }} *Scan Queries* is a separate data access interface designed primarily for performing analytical ad hoc queries on a DB. @@ -52,4 +52,5 @@ class TTableClient { }; ``` -{% endif %}
\ No newline at end of file +{% endif %} + diff --git a/ydb/docs/en/core/concepts/_includes/secondary_indexes.md b/ydb/docs/en/core/concepts/_includes/secondary_indexes.md index 9e41aec395..9674b11680 100644 --- a/ydb/docs/en/core/concepts/_includes/secondary_indexes.md +++ b/ydb/docs/en/core/concepts/_includes/secondary_indexes.md @@ -67,7 +67,6 @@ Creating an index is an asynchronous operation. If the client-server connection --columns title \ series ``` - * `--endpoint`: DB endpoint. * `--database`: Full DB path. * `-sync|-async`: Index type. @@ -230,3 +229,4 @@ If you don't specify the index type, a synchronous index is created by default. #### What's next For other examples of working with secondary indexes, see the [recommendations](../../best_practices/secondary_indexes.md). + diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md index d1ccf18914..540c543c9e 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md @@ -7,11 +7,11 @@ keywords: - dedicated editable: false --- - -# Serverless and Dedicated modes {{ ydb-full-name }} +# Serverless and Dedicated modes {{ ydb-short-name }} You can create and use multiple {{ ydb-short-name }} databases. When creating a database, one of two operating modes is selected for each database: Serverless or Dedicated. The mode can't be changed later. * _Serverless_: A DB that doesn't require you to configure, administer, or monitor load or manage resources. To create a database, you only need to enter a name, and you'll get the URL for the connection. Payment is charged for the execution of queries and the actual amount of stored data. -* _Dedicated_: You determine the computing resources that will be reserved for the database: CPU and RAM on the nodes, the number of nodes, and the storage size. You need to make sure there are sufficient resources to handle the load and add more when necessary. Payment is charged for dedicated resources per hour, regardless of their actual use.
\ No newline at end of file +* _Dedicated_: You determine the computing resources that will be reserved for the database: CPU and RAM on the nodes, the number of nodes, and the storage size. You need to make sure there are sufficient resources to handle the load and add more when necessary. Payment is charged for dedicated resources per hour, regardless of their actual use. + diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md index 60be12a250..c85c91f0dd 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md @@ -6,7 +6,7 @@ When executing a query to the serverless database, an indicator is calculated th Because serverless database resources are indefinitely large, the maximum consumption of RUs over a period of time can also be any value, leading to surprisingly high charges. For example, this can happen as a result of an error in the code causing an infinite loop of queries. -With the cloud deployment of YDB, there is a limit on the total number of RUs per second at the level of cloud quotas. But this limit is technical and large enough that the potential invoice amount can still be significantly higher than expected. You can only increase this parameter by contacting technical support. +With the cloud deployment of {{ ydb-short-name }}, there is a limit on the total number of RUs per second at the level of cloud quotas. But this limit is technical and large enough that the potential invoice amount can still be significantly higher than expected. You can only increase this parameter by contacting technical support. The **Throttling limit** on a serverless database allows you to set the maximum consumption rate of RUs per second. Considering a 5-minute accumulation of unused RUs, even with small limits (10 RU/s by default when creating a database), you can successfully perform various development and testing tasks and run applications with a small load. At the same time, the amount of possible charges will be limited to an upper limit of 2572000 seconds per month (30 days) multiplied by the price per 1 million RUs. Based on the pricing policy as of the date of this documentation (₽13.36), the maximum possible amount of charges per month is about ₽340. @@ -39,3 +39,4 @@ The **Maximum amount of data** limit for a serverless database limits the amount You can change the **Maximum amount of data** limit interactively at any time, both via the graphical console and the CLI and raise or reduce it without limitations. This allows you to quickly adjust it as needed. We don't recommend setting the **Maximum amount of data** limit below the current actual amount because in this state, all data modification operations, including DELETE, become unavailable. You will only be able to reduce the amount of data with the DROP TABLE or DROP INDEX commands. If the limit is accidentally set below the actual volume, we recommend returning it to the operating value exceeding the actual volume with some redundancy. + diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md index d0f6aa8e92..44803b1f0c 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md @@ -2,18 +2,19 @@ ### Separate compute and storage layers {#separate-layers} -Please note that YDB has two separate explicit layers: storage and compute. The storage layer is responsible for securely storing data on storage devices and replicating data between nodes to ensure fault tolerance. +Please note that {{ ydb-short-name }} has two separate explicit layers: storage and compute. The storage layer is responsible for securely storing data on storage devices and replicating data between nodes to ensure fault tolerance. -In YDB, user data is stored in tables that are partitioned. A shard is an entity that is responsible for storing a table partition (typically one). An entity called a tablet is responsible for changing data in a shard. It's a component that implements consistent changes in the shard data and solves the issue of distributed consensus. A tablet instance can be viewed as an object that is generated in the process address space and consumes CPU resources and RAM. Tablets store all their statuses in the storage layer. This means, among other things, that a tablet instance can be raised in an arbitrary process that the storage layer is available from. The YDB compute layer essentially consists of tablets and the YQL query execution layer. +In {{ ydb-short-name }}, user data is stored in tables that are partitioned. A shard is an entity that is responsible for storing a table partition (typically one). An entity called a tablet is responsible for changing data in a shard. It's a component that implements consistent changes in the shard data and solves the issue of distributed consensus. A tablet instance can be viewed as an object that is generated in the process address space and consumes CPU resources and RAM. Tablets store all their statuses in the storage layer. This means, among other things, that a tablet instance can be raised in an arbitrary process that the storage layer is available from. The {{ ydb-short-name }} compute layer essentially consists of tablets and the YQL query execution layer. It should be noted that the concept of a database comprises user tables and, accordingly, tablet instances serving these tables as well as certain system entities. For example, there is a tablet called SchemeShard. It serves the data schema of all tables. There is a coordination system for distributed transactions whose items are also tablets. -### YDB Dedicated mode {#dedicated} +### {{ ydb-short-name }} Dedicated mode {#dedicated} -Dedicated mode assumes that resources for tablet instances and for executing YQL queries are selected from the compute resources explicitly allocated to the database. Compute resources are VMs that have a certain amount of vCPUs and memory. The task of selecting the optimal amount of resources for the DB is currently the user's responsibility. If there aren't enough resources to serve the load, the latency of requests increases, which may eventually lead to the denial of service for requests, such as that with the ```OVERLOADED``` return code. The user can add compute resources (VMs) to the database in the UI or CLI to ensure it has the necessary amount of computing resources. Adding compute resources to the DB is relatively fast and comparable to the time it takes to start a VM. After that, YDB automatically balances tablet instances across the cluster based on the resources added. +Dedicated mode assumes that resources for tablet instances and for executing YQL queries are selected from the compute resources explicitly allocated to the database. Compute resources are VMs that have a certain amount of vCPUs and memory. The task of selecting the optimal amount of resources for the DB is currently the user's responsibility. If there aren't enough resources to serve the load, the latency of requests increases, which may eventually lead to the denial of service for requests, such as that with the ```OVERLOADED``` return code. The user can add compute resources (VMs) to the database in the UI or CLI to ensure it has the necessary amount of computing resources. Adding compute resources to the DB is relatively fast and comparable to the time it takes to start a VM. After that, {{ ydb-short-name }} automatically balances tablet instances across the cluster based on the resources added. -### YDB Serverless mode {#serverless} +### {{ ydb-short-name }} Serverless mode {#serverless} -In Serverless mode, the YDB infrastructure determines the amount of computing resources to be allocated to serve the user database. The amount of allocated resources can be either very large (an arbitrary number of cores) or very small (significantly less than one core). If a user created a DB that contains a single table with a single entry and makes DB queries very rarely, YDB actually uses a small amount of RAM on tablet instances that are part of the user DB. This is possible due to the fact that the user database components are objects rather than processes. If the load increases, the DB components start using more CPU time and memory. If the load grows to the point where there are not enough VM resources, the YDB infrastructure can balance the system granularly, generating tablet instances on other VMs. +In Serverless mode, the {{ ydb-short-name }} infrastructure determines the amount of computing resources to be allocated to serve the user database. The amount of allocated resources can be either very large (an arbitrary number of cores) or very small (significantly less than one core). If a user created a DB that contains a single table with a single entry and makes DB queries very rarely, {{ ydb-short-name }} actually uses a small amount of RAM on tablet instances that are part of the user DB. This is possible due to the fact that the user database components are objects rather than processes. If the load increases, the DB components start using more CPU time and memory. If the load grows to the point where there are not enough VM resources, the {{ ydb-short-name }} infrastructure can balance the system granularly, generating tablet instances on other VMs. + +This technology lets you package virtual entities (tablet instances) very tightly together into physical resources based on actual consumption. This makes it possible to invoice the user for the operations performed rather than the allocated resources. -This technology lets you package virtual entities (tablet instances) very tightly together into physical resources based on actual consumption. This makes it possible to invoice the user for the operations performed rather than the allocated resources.
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_includes/transactions.md b/ydb/docs/en/core/concepts/_includes/transactions.md index c898223d0c..1c662a047d 100644 --- a/ydb/docs/en/core/concepts/_includes/transactions.md +++ b/ydb/docs/en/core/concepts/_includes/transactions.md @@ -1,14 +1,14 @@ -# YDB transactions and queries +# {{ ydb-short-name }} transactions and queries -This section describes the specifics of YQL implementation for YDB transactions. +This section describes the specifics of YQL implementation for {{ ydb-short-name }} transactions. ## Query language -The main tool for creating, modifying, and managing data in YDB is a declarative query language called YQL. YQL is an SQL dialect that can be considered a database interaction standard. YDB also supports a set of special RPCs that can be used, for example, to manage a tree schema or cluster. +The main tool for creating, modifying, and managing data in {{ ydb-short-name }} is a declarative query language called YQL. YQL is an SQL dialect that can be considered a database interaction standard. {{ ydb-short-name }} also supports a set of special RPCs that can be used, for example, to manage a tree schema or cluster. ## Transaction modes {#modes} -By default, YDB transactions are performed in *Serializable* mode. It provides the strictest [isolation level](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) for custom transactions. This mode guarantees that the result of successful parallel transactions is equal to their specific execution sequence, while there are no [read anomalies](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Read_phenomena) for successful transactions. +By default, {{ ydb-short-name }} transactions are performed in *Serializable* mode. It provides the strictest [isolation level](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) for custom transactions. This mode guarantees that the result of successful parallel transactions is equal to their specific execution sequence, while there are no [read anomalies](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Read_phenomena) for successful transactions. If the requirements for the consistency or freshness of data read by a transaction can be lowered, the user can use execution modes with lower guarantees: @@ -25,11 +25,11 @@ The constructs implemented in YQL can be divided into two classes: [data definit For more information about supported YQL constructs, see the [YQL documentation](../../yql/reference/index.md). -Listed below are the features and limitations of YQL support in YDB, which might not be obvious at first glance and are worth noting. +Listed below are the features and limitations of YQL support in {{ ydb-short-name }}, which might not be obvious at first glance and are worth noting. * Multi-statement transactions (transactions made up of a sequence of YQL statements) are supported. Transactions may interact with client software, or in other words, client interactions with the database might look as follows: `BEGIN; make a SELECT; analyze the SELECT results on the client side; ...; make an UPDATE; COMMIT`. We should note that if the transaction body is fully formed before accessing the database, it will be processed more efficiently. -* YDB does not support transactions that combine DDL and DML queries. The traditional [ACID](https://en.wikipedia.org/wiki/ACID) transaction paradigm only applies to DML queries, meaning that change data. DDL queries must be idempotent, meaning repeatable if an error occurs. If you need to apply an action to a schema, remember that each action is transactional, while a set of actions is not. -* The version of YQL in YDB uses the [Optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) mechanism. Optimistic locking is applied to entities affected during a transaction. When the transaction is complete, the mechanism verifies that the locks have not been invalidated. For the user, locking optimism means that when transactions are competing with one another, the one that finishes first wins. Competing transactions fail with the ```Transaction locks invalidated``` error. +* {{ ydb-short-name }} does not support transactions that combine DDL and DML queries. The traditional [ACID](https://en.wikipedia.org/wiki/ACID) transaction paradigm only applies to DML queries, meaning that change data. DDL queries must be idempotent, meaning repeatable if an error occurs. If you need to apply an action to a schema, remember that each action is transactional, while a set of actions is not. +* The version of YQL in {{ ydb-short-name }} uses the [Optimistic Concurrency Control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) mechanism. Optimistic locking is applied to entities affected during a transaction. When the transaction is complete, the mechanism verifies that the locks have not been invalidated. For the user, locking optimism means that when transactions are competing with one another, the one that finishes first wins. Competing transactions fail with the ```Transaction locks invalidated``` error. * All changes made during the transaction accumulate in the database server memory and are committed when the transaction completes. If the locks are not invalidated, all the changes accumulated are committed atomically, but if at least one lock is invalidated, none of the changes are committed. The above model involves certain restrictions: changes made by a single transaction must fit inside the available memory, and the transaction "doesn't see" its changes. A transaction should be formed in such a way so that the first part of the transaction only reads data, while the second part of the transaction only changes data. The query structure then looks as follows: @@ -42,10 +42,11 @@ A transaction should be formed in such a way so that the first part of the trans COMMIT; ``` -For more information about YQL support in YDB, see the [YQL documentation](../../yql/reference/index.md). +For more information about YQL support in {{ ydb-short-name }}, see the [YQL documentation](../../yql/reference/index.md). ## Distributed transactions {#distributed-tx} -A database table in YDB can be sharded by the range of the primary key values. Different table shards can be served by different distributed database servers (including ones in different locations). They can also move independently between servers to enable rebalancing or ensure shard operability if servers or network equipment goes offline. +A database table in {{ ydb-short-name }} can be sharded by the range of the primary key values. Different table shards can be served by different distributed database servers (including ones in different locations). They can also move independently between servers to enable rebalancing or ensure shard operability if servers or network equipment goes offline. + +{{ ydb-short-name }} supports distributed transactions. Distributed transactions are transactions that affect more than one shard of one or more tables. They require more resources and take more time. While point reads and writes may take up to 10 ms in 99 percentile, distributed transactions typically take from 20 to 500 ms. -YDB supports distributed transactions. Distributed transactions are transactions that affect more than one shard of one or more tables. They require more resources and take more time. While point reads and writes may take up to 10 ms in 99 percentile, distributed transactions typically take from 20 to 500 ms. diff --git a/ydb/docs/en/core/concepts/_includes/ttl.md b/ydb/docs/en/core/concepts/_includes/ttl.md index 68f34b9ae8..301a8ec884 100644 --- a/ydb/docs/en/core/concepts/_includes/ttl.md +++ b/ydb/docs/en/core/concepts/_includes/ttl.md @@ -63,7 +63,7 @@ Currently, you can manage TTL settings using: * [YQL](../../yql/reference/index.md) * [Console client {{ ydb-short-name }}]{% if audience != "external" %}(https://cloud.yandex.com/en-ru/docs/ydb/quickstart/yql-api/ydb-cli){% else %}(../../quickstart/yql-api/ydb-cli.md){% endif %}. -* {{ ydb-short-name }} {% if oss %}C++ and{% endif %} Python [SDK](../../reference/ydb-sdk/index.md) +* {{ ydb-short-name }} {% if oss %}C++ and{% endif %} Python [SDK](../../reference/ydb-sdk/index.md) {% note info %} @@ -241,3 +241,4 @@ The current TTL settings can be obtained from the table description: ``` {% endlist %} + diff --git a/ydb/docs/en/core/concepts/cluster/_assets/BS_overview.svg b/ydb/docs/en/core/concepts/cluster/_assets/BS_overview.svg new file mode 100644 index 0000000000..eb08e1cb2e --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_assets/BS_overview.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="771px" height="814px" viewBox="-0.5 -0.5 771 814" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T14:50:26.556Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="ZU8BhaNDal54XeL1MvRJ" version="12.7.0" type="device"><diagram id="0HZmCEQLlPINtnbGPtoa" name="Page-1">7Zxdl6I4EIZ/jZc9BxK+vBx1puecOd3HGXd3Zq72RIjCNoobsdX99RuUIKTo9psgTt80KTTIW4+VqiTSwt3J6pGRmf8UeTRsIc1btXCvhZCOnDb/l1jWW4utWVvDmAVe+qKdYRD8R1OjlloXgUfnhRfGURTGwaxodKPplLpxwUYYi5bFl42isHjVGRlTYBi4JITWH4EX+1urg+yd/QsNxr64sm6lNzwh4sXpncx94kXLnAl/auEui6J4ezRZdWmYiCd02b7v8xtnsw/G6DQ+5A3+n/Tx6fnpBaOnUdd9oN/+cXsPzraXVxIu0hsef+93k35ZtFrzzr5+66cfP14LTVi0mHo06VZr4c7SD2I6mBE3ObvkFHCbH09C3tL54SgIw24URmzzXjxyXOq63D6PWfRCc2eGjmmYSYdjRryA31T+XZs/fg7edKrDK2UxXeVMqQiPNJrQmPFb0dKzRuqPtUA0bS937s1sfs61VmojKVHjrOed6Pwg1f0IH7SBD3okJgOfMO8d6fXjpR+NULn0njW0TKty6ZGmWnoRZe5Qe0e59jrQ/kvwSi8qu2dSxzPKZHfQEFvVy44t5bKj+wv5WaCpTczX8d0EHln8GkR9A4q/npJJ4LaQFfKrd4aMH42To2ee0fKX6sAp/O7jovJFhafRlEruSE0kDMZT3nS5ppTbO4mWAU85P6YnJoHnJZcpdXXxm3gB/zhF95SMC0aJc9DVnGMe7RzUXOfI44dy71j3N3pkY3Z9Rg9YtjV19JDFr8HoAeu1BuatQHfl9YK42BEDA27uwIDbuFYDA4LVXOMHBhPVbWBAsLjrDLoNC02y7OqHBASruUFM4vusJ2T3HDpwXC80wXKv8aHJcmoXmmBd130aNCw0ybLXIDTBgm1PaGpwNS27R31osoF3Ei90EnVZ078b6isKWEYDzec+mSWH7oKF6w4j7guN94tf9BSLkm9clPDevhDJul0U88GGYpaS7FxNTFgW/0GnhN9ho7MdHZsFR2Doh3aVEUVc/waZNnTtQ1FMW7T3UK2L5OLycsKyNqO6wQMltmtGNUzhb4Vqy7Alqh9sdCDXyLka1zAbF2lhU5m2dNkRJVQ7lVINc/NOGA0HccSSXWmyJ3iPwWye6JWhHkYL7/hUkFBnVFqdWq5Dh1eapSnb+VAaRHT9anqXZ9s/CPPoFMh92xMAsvyG8mwbw2y7serLtY569Q2YyJyn+RUYNZVvjjIQUKnfC+Yv6qXCklQl9V7FUsEMQrlK8teuBkDBIb4eQFnt2gEFR2flKskRyjKUqwRH0b9qAZQcoSxTuVRwRkq5SnKEUg+UcFPtgJIjlHqgzBvIoWzlC30mzKF6g1a22KpaLylM2SWTABXrBTcPKFdJDlM1oKpkz3Z9qJJjVQ2oOiA9p1PvY/ITyVY2ZeeRuZ9NhuYk4sqw9c9Ezg9t0fy1aZqi2Vulam9b63yrT1nAbyuZBHx/Ni8mbEzjd25LLNlRr/CzTeiYnPBmie7CxmhI4mT3Zv5TlDkjvUI/CjZz/4IKeUHZkPw5jxbMpem7di4FHUn8OPIc2FYY0M8Gjeyuz6AFlik3QEtNIMDyHn0xi3AsBKCjiiE4oAD7DcEbvjO1C0EAOqoYggP2RPyG4A3fWeIrdC4EoKNqIbAOWG7NQeCGZD4P3Dddr9lF3xun+J6ugvin6Jsf5zDirV1PSWO9j5etH95RQD80IRFreDUhUBpB2nIecWo+Iuep1wbwuOx1H4A6dooEYq19Awxqt8mgvDfN0E6E0DS5p4pdiZ/uVsXhcXnxPg6NIoX6OQzuuPuVI7KcwTmXKYYfcmP+HIR7C+G9qApZbg3VpoTL4zL3fZgis8gpOilZKw+W7XdBTRoHJ337A2j7Nqk0JZyQfSKWllh+FHtwHOmzXJvL44qJfVxaAMyqAugZkdG+TQYNLG/GNU8NjoYjd2VcLz6un7tjtP57+X3of6SDl+6/X/v9B7huU+edR2DauYS8N2eipRGt0o1Hpdqj+9FeftKNevHh0lJjxZcfWKBe/AMmUU5Yp7pkdKh0f1SpRiUP+7noqt5ZapmSWlVukSpV64BqtGqN5JinHilYC9UIqey5trVh6oAUvWqR5FCunqmSR7TWhylsV8cUb+6eWLxNz3fPfcaf/gc=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f8cecc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f8cecc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-fff2cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#fff2cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-d5e8d4-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#d5e8d4"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient></defs><g><rect x="0" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="60" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="0" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="60" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="0" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="60" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="0" y="330" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 360px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Hive</div></div></div></foreignObject><text x="60" y="364" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Hive</text></switch></g><rect x="160" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="220" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="160" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="220" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="40" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 41px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 1</div></div></div></foreignObject><text x="60" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="200" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 201px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 2</div></div></div></foreignObject><text x="220" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="320" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="380" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="320" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DataShard</div></div></div></foreignObject><text x="380" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DataShard</text></switch></g><rect x="320" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Hive</div></div></div></foreignObject><text x="380" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Hive</text></switch></g><rect x="353" y="50" width="40" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 60px; margin-left: 354px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Dynamic<br />Node 3</div></div></div></foreignObject><text x="373" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Dynami...</text></switch></g><rect x="480" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="540" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="480" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BSC</div></div></div></foreignObject><text x="540" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BSC</text></switch></g><rect x="480" y="50" width="120" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 60px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static<br />Node 1</div></div></div></foreignObject><text x="540" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static...</text></switch></g><rect x="640" y="90" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 120px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">gRPC proxy/KQP</div></div></div></foreignObject><text x="700" y="124" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">gRPC proxy/KQP</text></switch></g><rect x="640" y="170" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 200px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">CMS</div></div></div></foreignObject><text x="700" y="204" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">CMS</text></switch></g><rect x="640" y="50" width="120" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 60px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static<br />Node 2</div></div></div></foreignObject><text x="700" y="64" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static...</text></switch></g><rect x="640" y="250" width="120" height="60" rx="9" ry="9" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 280px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeBroker</div></div></div></foreignObject><text x="700" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeBroker</text></switch></g><path d="M 150 -100 L 145 -100 Q 140 -100 140 -90 L 140 30 Q 140 40 135 40 L 132.5 40 Q 130 40 135 40 L 137.5 40 Q 140 40 140 50 L 140 170 Q 140 180 145 180 L 150 180" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,140,40)" pointer-events="all"/><rect x="95" y="0" width="90" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 88px; height: 1px; padding-top: 10px; margin-left: 96px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Tenant 1</div></div></div></foreignObject><text x="140" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Tenant 1</text></switch></g><path d="M 390.5 -22.5 L 385.5 -22.5 Q 380.5 -22.5 380.5 -12.5 L 380.5 30 Q 380.5 40 375.5 40 L 373 40 Q 370.5 40 375.5 40 L 378 40 Q 380.5 40 380.5 50 L 380.5 92.5 Q 380.5 102.5 385.5 102.5 L 390.5 102.5" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,380.5,40)" pointer-events="all"/><rect x="335" y="0" width="90" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 88px; height: 1px; padding-top: 10px; margin-left: 336px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Tenant 2</div></div></div></foreignObject><text x="380" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Tenant 2</text></switch></g><path d="M 627.5 -102.5 L 622.5 -102.5 Q 617.5 -102.5 617.5 -92.5 L 617.5 30 Q 617.5 40 612.5 40 L 610 40 Q 607.5 40 612.5 40 L 615 40 Q 617.5 40 617.5 50 L 617.5 172.5 Q 617.5 182.5 622.5 182.5 L 627.5 182.5" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(90,617.5,40)" pointer-events="all"/><rect x="577.5" y="0" width="80" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 10px; margin-left: 579px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Static</div></div></div></foreignObject><text x="618" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Static</text></switch></g><path d="M 552.5 357.5 C 494.5 357.5 480 385 526.4 390.5 C 480 402.6 532.2 429 569.9 418 C 596 440 683 440 712 418 C 770 418 770 396 733.75 385 C 770 363 712 341 661.25 352 C 625 335.5 567 335.5 552.5 357.5 Z" fill="#dae8fc" stroke="#6c8ebf" stroke-miterlimit="10" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 288px; height: 1px; padding-top: 385px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobStorage</div></div></div></foreignObject><text x="625" y="389" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobStorage</text></switch></g><rect x="480" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 481px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="540" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="640" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 641px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="700" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="480" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk</div></div></div></foreignObject><text x="550" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk</text></switch></g><rect x="640" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk</div></div></div></foreignObject><text x="710" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk</text></switch></g><rect x="480" y="610" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="620" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 650px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">VDisk</div></div></div></foreignObject><text x="550" y="654" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">VDisk</text></switch></g><rect x="640" y="610" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="620" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 650px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">VDisk</div></div></div></foreignObject><text x="710" y="654" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">VDisk</text></switch></g><rect x="480" y="690" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="490" y="700" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 730px; margin-left: 491px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="550" y="734" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="640" y="690" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="650" y="700" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 730px; margin-left: 651px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="710" y="734" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><path d="M 140 810 L 140 48" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 300 812 L 300 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 460 812 L 460 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 617 812 L 617 50" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="3 3" pointer-events="stroke"/><path d="M 120 360 L 493.94 373.77" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 499.18 373.96 L 492.06 377.2 L 493.94 373.77 L 492.32 370.2 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 120 280 L 513.79 362.68" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 518.93 363.76 L 511.36 365.75 L 513.79 362.68 L 512.79 358.9 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 542.87 235.69 L 593.13 335.31" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 540.5 231 L 546.78 235.67 L 542.87 235.69 L 540.53 238.82 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 595.5 340 L 589.22 335.33 L 593.13 335.31 L 595.47 332.18 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 440 304 L 546.75 354.77" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 551.49 357.02 L 543.67 357.17 L 546.75 354.77 L 546.67 350.85 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 700 310 L 665.57 347.32" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 662.01 351.18 L 664.18 343.66 L 665.57 347.32 L 669.33 348.41 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="0" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 1px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="60" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="160" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 161px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="220" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="320" y="450" width="120" height="60" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 480px; margin-left: 321px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">NodeWarden</div></div></div></foreignObject><text x="380" y="484" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">NodeWarden</text></switch></g><rect x="0" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="10" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 11px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="70" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="160" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="170" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 171px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="230" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><rect x="320" y="530" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="330" y="540" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 570px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="390" y="574" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/cluster/_assets/Slide3_group_layout.svg b/ydb/docs/en/core/concepts/cluster/_assets/Slide3_group_layout.svg new file mode 100644 index 0000000000..3d6bf588a8 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_assets/Slide3_group_layout.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="731px" height="468px" viewBox="-0.5 -0.5 731 468" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T15:31:24.461Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="c6UdJlCHfOx2ZCPmeLO2" version="12.7.0" type="device"><diagram id="5oa2Jgz5xl6jGbGaUmvA" name="Page-1">5V1dc6JIFP01Pu4W0Hz5GGOSqZrZbKqcqszuy1YLjbKDtIvtROfXL0ijQpOEZLzeJpqH2Bdo4Zzb3fcciBmQ68XmLqPL+R88ZMnAMsLNgIwHlmUZrpX/KiLbMmKaQ7uMzLI4lLFDYBL/ZDJoyOg6DtmqtqPgPBHxsh4MeJqyQNRiNMv4U323iCf1T13SGVMCk4AmavQxDsW8jPqWd4h/YvFsXn2y6Q7LLQta7SyvZDWnIX86CpGbAbnOOBflu8XmmiUFehUu5XG3z2zdn1jGUtHlgG+bz2L995rc2/b9n9Gnydf/JuQ32csPmqzlBT+M49X3PGQOyJVpGNXJi22FSMbXaciKTo0BGT3NY8EmSxoUW5/yJMhjc7FI8lbewyiKk+SaJzzbHUsip/jJ4yuR8e/saIu7exVH8FQcxctXHp9lNIxZbVu0e+XbVCiq62KZYJujkITmjvEFE9k236XaWtEkE9WXzacD6VYFxvyIcFvGqMyz2b7nAxX5G8nGG5ixFGZOykRImR8FrUwEPptG50fcVhG3zwk4AQbcYX5otwHuW1OyS31owF29ALdhAY8i5gatGR56w6lhwAO+nzA0AdyBBjyyngHcnbrOGTJ8D50mgLvAgPsBawd86ju2c44M9/UC3HutnDEvpJwhrmbljA87FJiZr69eGxND1yP0DHNPE3HsoTCEBdw16NBsBdwae+5udW2k/h5sYCJszVbdSslCMWEYPt3hrTBhGI6HyoRmy7Gpqt2TMmFOqcmsdibcm6tbRCY0W6dNYHWbZ77BonYmzPHoGo8JRzPVa6qytyqZrMt2gPb1EVrNZAIrZP08oL1FgTYcgDWyfi4QPuTQKlk7HwgfclUnnxhy3ZwgfMiBBbB+XhA+5KoEbpQ2l+oG4Zc21bp/OX4Q+nCwgNVvfxwhfCrA5W9fLCF8KoBv+/bHE8KnAlju9scUwqdClcFV6UQu2xUiFnrpBKyX9XOF2u5ennc4AOtl/VwhfMih9bJ2rhA+5MC3jPVzhdAhJ8AyWD9XCB/y5x95JpftCuGXNgRYF+vnCuEPB2D92x9XCJ8KcP3bF1cInwrg28D9cYXwqQCWu/1xhdCp8NQJqiqd7Mt2hfYTGFrp5EE/uKKdK2S33MQ873CANuK0c4XwIQd/cEU3VwgfcnAjTjdXCB9yaCNOO1cIHfLqfJ4vbS7VFcIvbXzgB1f0c4Xwh4MFC3l/XCF8KoANuv64QvhUABt0/XGF8KkAlrv9cYXwqVBl8D0P2cBqKZnyaxR13GkSz9L8fZAjwnK4RgUScUCTK7lhEYdhcfgoY6v4J53uuioIXfI4FbuLcUYDZ1z0tRZ8VX6DlHmihbmOtadC7bRAbYFBrcpfCbWa/T2H2hxiY63qXok1+WhYExMba1XwSqztj4a1Tc6H9T+Pj3eLh2QxSefOreV9vbv68rnTXziycMYmsskzMeczntLk5hAd1dfWwz5fOF9K7P5lQmzlF/oVsDbW23z1lBtN6yWkV3ydBeylCyr3K875RT4yllAR/2C13tvglYc+FMlx4NFtCAO/QZCg2YwJeVCDo/1Z/AJtCmvjSQFYxjfbX6x8Gkwow+kEg8BtlPKOOgj29v7xKHChRkEHUcXS8Kr4Asm8lfKUvTN9X03LV+aBKnbi7CXN9C2HmZK+r3bUnKeAh0EHBXbEW5DQ1SoO+k2d03w44p3MNftpCgJg5jrcRngrczlh2fZbMcv97lTNv+Skt2uMN7XWVraeZbyE4IVCpfoTFU1Tw2xy2jk3nPb5+Uy50eF+h/65UT2oomluKGtp19zwm6lx5sKnwyPS+idH5VdpmhzvLgewk6PDraEPVcedqhhALuO6fP/Xh+bt3St1kzilI2jmLk05nWwdVYZcsyNo5t6mnT4ecyfTvEpH72Yubx7+AUS5++H/aJCb/wE=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f5f5f5-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f5f5f5"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-dae8fc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#dae8fc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-d5e8d4-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#d5e8d4"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-ffe6cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#ffe6cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-fff2cc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#fff2cc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-f8cecc-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#f8cecc"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-e1d5e7-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#e1d5e7"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-60a917-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#60a917"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-008a00-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#008a00"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-1ba1e2-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#1ba1e2"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient><linearGradient x1="0%" y1="0%" x2="0%" y2="100%" id="mx-gradient-0050ef-1-ffffff-1-s-0"><stop offset="0%" style="stop-color:#0050ef"/><stop offset="100%" style="stop-color:#ffffff"/></linearGradient></defs><g><rect x="90" y="67" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 87px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 1:1000</div></div></div></foreignObject><text x="190" y="91" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 1:1000</text></switch></g><rect x="90" y="27" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="27" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="27" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="27" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="27" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="67" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 87px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 1:1001</div></div></div></foreignObject><text x="430" y="91" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 1:1001</text></switch></g><rect x="330" y="27" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="27" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="27" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="27" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="27" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="187" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 207px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 2:1000</div></div></div></foreignObject><text x="190" y="211" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 2:1000</text></switch></g><rect x="90" y="147" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="147" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="147" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="147" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="147" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="187" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 207px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 2:1001</div></div></div></foreignObject><text x="430" y="211" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 2:1001</text></switch></g><rect x="330" y="147" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="147" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="147" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="147" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="147" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="307" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 327px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 3:1000</div></div></div></foreignObject><text x="190" y="331" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 3:1000</text></switch></g><rect x="90" y="267" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="267" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="267" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="267" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="267" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="307" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 327px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 3:1001</div></div></div></foreignObject><text x="430" y="331" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 3:1001</text></switch></g><rect x="330" y="267" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="267" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="267" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="267" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="267" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="90" y="427" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 447px; margin-left: 91px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 4:1000</div></div></div></foreignObject><text x="190" y="451" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 4:1000</text></switch></g><rect x="90" y="387" width="40" height="40" fill="url(#mx-gradient-dae8fc-1-ffffff-1-s-0)" stroke="#6c8ebf" pointer-events="all"/><rect x="130" y="387" width="40" height="40" fill="url(#mx-gradient-d5e8d4-1-ffffff-1-s-0)" stroke="#82b366" pointer-events="all"/><rect x="170" y="387" width="40" height="40" fill="url(#mx-gradient-ffe6cc-1-ffffff-1-s-0)" stroke="#d79b00" pointer-events="all"/><rect x="210" y="387" width="40" height="40" fill="url(#mx-gradient-fff2cc-1-ffffff-1-s-0)" stroke="#d6b656" pointer-events="all"/><rect x="250" y="387" width="40" height="40" fill="url(#mx-gradient-f8cecc-1-ffffff-1-s-0)" stroke="#b85450" pointer-events="all"/><rect x="330" y="427" width="200" height="40" fill="url(#mx-gradient-f5f5f5-1-ffffff-1-s-0)" stroke="#666666" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 198px; height: 1px; padding-top: 447px; margin-left: 331px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #333333; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PDisk 4:1001</div></div></div></foreignObject><text x="430" y="451" fill="#333333" font-family="Helvetica" font-size="12px" text-anchor="middle">PDisk 4:1001</text></switch></g><rect x="330" y="387" width="40" height="40" fill="url(#mx-gradient-e1d5e7-1-ffffff-1-s-0)" stroke="#9673a6" pointer-events="all"/><rect x="370" y="387" width="40" height="40" fill="url(#mx-gradient-60a917-1-ffffff-1-s-0)" stroke="#2d7600" pointer-events="all"/><rect x="410" y="387" width="40" height="40" fill="url(#mx-gradient-008a00-1-ffffff-1-s-0)" stroke="#005700" pointer-events="all"/><rect x="450" y="387" width="40" height="40" fill="url(#mx-gradient-1ba1e2-1-ffffff-1-s-0)" stroke="#006eaf" pointer-events="all"/><rect x="490" y="387" width="40" height="40" fill="url(#mx-gradient-0050ef-1-ffffff-1-s-0)" stroke="#001dbc" pointer-events="all"/><rect x="0" y="57" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 67px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 1</div></div></div></foreignObject><text x="25" y="71" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 1</text></switch></g><rect x="0" y="177" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 187px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 2</div></div></div></foreignObject><text x="25" y="191" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 2</text></switch></g><rect x="0" y="297" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 307px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 3</div></div></div></foreignObject><text x="25" y="311" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 3</text></switch></g><rect x="0" y="417" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 427px; margin-left: 25px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Node 4</div></div></div></foreignObject><text x="25" y="431" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Node 4</text></switch></g><path d="M 610 67 L 576.37 67" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 571.12 67 L 578.12 63.5 L 576.37 67 L 578.12 70.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="610" y="37" width="120" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 67px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">DS proxy</div></div></div></foreignObject><text x="670" y="71" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">DS proxy</text></switch></g><path d="M 570 367 L 570 7" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 7 L 510 20.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 25.88 L 506.5 18.88 L 510 20.63 L 513.5 18.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 127 L 510 140.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 145.88 L 506.5 138.88 L 510 140.63 L 513.5 138.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 247 L 510 260.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 265.88 L 506.5 258.88 L 510 260.63 L 513.5 258.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 367 L 510 380.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 385.88 L 506.5 378.88 L 510 380.63 L 513.5 378.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><path d="M 510 7 L 570 7" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 127 L 570 127" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 247 L 570 247" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 510 367 L 570 367" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg b/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg new file mode 100644 index 0000000000..502ed357b3 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="691px" height="551px" viewBox="-0.5 -0.5 691 551" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T16:05:40.464Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="HF0g-8VF_2eGPBkH-fRg" version="12.7.0" type="device"><diagram id="LqqcKPgxwdzCRuipa-4Q" name="Page-1">3ZtRc5s4EMc/DTO9h3SQBBg/GidNc3PXyYw7vV7fFKMYWox8shzb/fQnjADDUocktpH74qBFYPhp/9LuKrbIeL65FXQR/c1DlljYDjcWubYwxraH1Z/Mss0tCA2d3DITcahtlWES/2TaaGvrKg7ZstZRcp7IeFE3Tnmasqms2agQfF3v9siT+rcu6IwBw2RKE2j9Jw5llFt9PKjsH1k8i4pvRt4wPzOnRWf9JsuIhny9ZyI3FhkLzmV+NN+MWZLRK7jk1334xdnywQRLZZcLxt6Hb1+Cu/8E+7ZBn/5EH++/Ply5bn6bJ5qs9Bvrp5XbAoHgqzRk2V1siwTrKJZssqDT7OxajbqyRXKeqBZSh488lXoUka/aNIlnqWok7FE9ZQAfWr/HExOSbfZM+iVuGZ8zKbaqiz5LNE/tUaTwlHU1PG7BPNobGmJrI9UuMStvXVFTBxrcSyAOnofIQuVWusmFjPiMpzS5qaxBHXPV5y/OFxrudyblVtOlK8kPo1eIxfZrdr/3btH8V99+17je1Fpb3frlGC35SkzZARCFQKmYMdnB6zIoB4dcsITK+Kmuxbbx05fe81g9c+kq2K77Cm66QP6k+qqGF5SP8XrHQMAvPtOHhMm78G0iO4KKkFdH40MV+S0ick6lIYcYoaHzawZ31Awankkz/us0MxKCbve6LbIOy+7fgwaN5avRH9nuof7qIH+CowoYA6ccR1QFGknv+i2VaIp+OwQSv6V+SUf94jOtecQ7j37B95DD+iUN/Tb6n0a/cFG5ZSkTii9Pe5cwwYZJGMCaSLboHZNjm4UJTnRjzn/ErH9QvlmgPAAqSPjDburuG5VrWPQLE8h7KkxIEzzDwgwfgAKIlhFdZIfTlUi2gaDTH9m6/ByrCmzekvkaQa6vhgfjgxfIs1G4uEIOhNlWuCid9eg0h60KbXE79Yqyzquo60wVECaUIQMRT2ky0ifmcRjmMR1bxj+ztFc7rQ4s1H3dwHKvs3upMG6pCxdHYo0ayYLbUiRqQY1PRbooUu2h9pyLx4z8OmYPUiZnpQwLLv7FQ8YD+znIbbPG6SDDpJjAWfjSKBPHMFeGqctvQNnBhlGGOQ++/GnZtQ2jDMPbbF6232Hvj8uHPXgW9nkjDRgiX75He6atgTBy/pxv7UwkF3TG7tJHDqi/KHlr8pdZBfQoc3DdX8sq1B5L1ArTPxVNbMb+aVU7rlWO3+Oyknz8zR+N1JjqMSYN7xg0Bv3EO6YY5knljktRB+yzbtSg05JWIrtFOyerh2CY8VS8kHG8hr3zgslLxQvOOj3zKktG/fGCaUjF65NpvNz+9Qh3B3RgoLEZHBg0IwOna2SAThYZEDN2ld+wwA87LvDEOfYC/zY/hgGubd1gyx9aI2fGUgt7dJ75XZLlEaj/lRlMnb1PBQTWFm4VlUW2MFflpj73C4lxyMxIBF4vd9I1nieeUXInMOxGh/SO+/desPbbvXsv3PAuBO86BPcfjTcFbwAyuMxcmOC7/ssz8c0SPMwbM1HXFG+awPtfnmBp1ewVvX9kbss/bCgYo91nYO8+0e4z2H26lmLh52dHuk+T6lmL180fVRxjlJpVkZY8a9gySOTlg6Sa1Q+M8hpd9TstcvM/</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="0" y="250" width="520" height="300" fill="#ffffff" stroke="#000000" pointer-events="all"/><path d="M 170 60 L 170 155 L 260 155 L 260 243.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 260 248.88 L 256.5 241.88 L 260 243.63 L 263.5 241.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="130" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 131px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TabletId</div></div></div></foreignObject><text x="170" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TabletId</text></switch></g><path d="M 250 60 L 250 110 L 75 110 L 75 253.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 75 258.88 L 71.5 251.88 L 75 253.63 L 78.5 251.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="210" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 211px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel</div></div></div></foreignObject><text x="250" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel</text></switch></g><path d="M 330 60 L 330 170 L 275 170 L 275 353.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 275 358.88 L 271.5 351.88 L 275 353.63 L 278.5 351.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="290" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 291px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Generation</div></div></div></foreignObject><text x="330" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Generation</text></switch></g><rect x="370" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 371px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Step</div></div></div></foreignObject><text x="410" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Step</text></switch></g><rect x="450" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 451px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Cookie</div></div></div></foreignObject><text x="490" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Cookie</text></switch></g><rect x="530" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 531px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobSize</div></div></div></foreignObject><text x="570" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobSize</text></switch></g><rect x="610" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PartId</div></div></div></foreignObject><text x="650" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PartId</text></switch></g><path d="M 420 -200 L 415 -200 Q 410 -200 410 -190 L 410 70 Q 410 80 405 80 L 402.5 80 Q 400 80 405 80 L 407.5 80 Q 410 80 410 90 L 410 350 Q 410 360 415 360 L 420 360" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(-90,410,80)" pointer-events="all"/><rect x="385" y="90" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 100px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">BlobId</div></div></div></foreignObject><text x="410" y="104" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobId</text></switch></g><rect x="155" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 170px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">64</div></div></div></foreignObject><text x="170" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">64</text></switch></g><rect x="240" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">8</div></div></div></foreignObject><text x="250" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="315" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 330px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="330" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="395" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="410" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="475" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 490px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">24</div></div></div></foreignObject><text x="490" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">24</text></switch></g><rect x="545" y="0" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 570px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">28 (26)</div></div></div></foreignObject><text x="570" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">28 (26)</text></switch></g><rect x="640" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 650px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">4</div></div></div></foreignObject><text x="650" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="15" y="260" width="120" height="280" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 267px; margin-left: 16px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletStorageInfo</div></div></div></foreignObject><text x="75" y="279" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletStorageInfo</text></switch></g><path d="M 125 310 L 170 310 L 170 405 L 208.63 405" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 213.88 405 L 206.88 408.5 L 208.63 405 L 206.88 401.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="25" y="290" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 310px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 0</div></div></div></foreignObject><text x="75" y="314" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 0</text></switch></g><rect x="25" y="330" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 350px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 1</div></div></div></foreignObject><text x="75" y="354" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 1</text></switch></g><rect x="25" y="370" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 390px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 2</div></div></div></foreignObject><text x="75" y="394" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 2</text></switch></g><rect x="25" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel N</div></div></div></foreignObject><text x="75" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel N</text></switch></g><rect x="215" y="360" width="120" height="180" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 367px; margin-left: 216px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletChannelInfo</div></div></div></foreignObject><text x="275" y="379" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletChannelInfo</text></switch></g><path d="M 325 410 L 398.63 410" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 410 L 396.88 413.5 L 398.63 410 L 396.88 406.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0≤gen<10</div></div></div></foreignObject><text x="275" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0≤gen<10</text></switch></g><rect x="405" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><path d="M 325 460 L 398.63 460" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 460 L 396.88 463.5 L 398.63 460 L 396.88 456.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">10≤gen<125</div></div></div></foreignObject><text x="275" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">10≤gen<125</text></switch></g><rect x="405" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 54321</div></div></div></foreignObject><text x="455" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 54321</text></switch></g><path d="M 325 510 L 398.63 510" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 510 L 396.88 513.5 L 398.63 510 L 396.88 506.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">125≤gen</div></div></div></foreignObject><text x="275" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">125≤gen</text></switch></g><rect x="405" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><rect x="425" y="260" width="90" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 275px; margin-left: 470px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 18px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Tablet</div></div></div></foreignObject><text x="470" y="280" fill="#000000" font-family="Helvetica" font-size="18px" text-anchor="middle">Tablet</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/cluster/_assets/Slide_group_content.svg b/ydb/docs/en/core/concepts/cluster/_assets/Slide_group_content.svg new file mode 100644 index 0000000000..5e5796fae0 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_assets/Slide_group_content.svg @@ -0,0 +1,3 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="616px" height="412px" viewBox="-0.5 -0.5 616 412" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T15:37:07.887Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="3zDAlwPQQ3q6GuUUS0rv" version="12.7.0" type="device"><diagram id="wSBYkmeAYjroDNpxkVYP" name="Page-1">3Ztbc6IwFMc/jY87I4EgPmpt7V66++DM7nMKKWYHiBPj9dNv0KDi0VaLmLBP4knA5PdPwrmMLfchXQ4FmYxfeESTFmpHy5Y7aCHkIMdVH7lltbX4Pt4aYsEi3WlvGLE11ca2ts5YRKeljpLzRLJJ2RjyLKOhLNmIEHxR7vbGk/KvTkhMgWEUkgRa/7BIjrfWAHX29mfK4nHxy47f3bakpOisZzIdk4gvDkzuY8t9EJzL7VW6fKBJDq/gsr3v6UzrbmCCZvKSG14S5q9fv/0apDEJw/iFDkbrL1jLMyfJTM9Yj1auCgSCz7KI5k9pt9z+YswkHU1ImLculOjKNpZpor456vKNZ/KJpCzJ9X6myZxKFhLVAIerZzCnQtLlgUkPf0h5SqVYqS66FWFNXa8lt1gki70yTkfbxoeq+NpI9GqId8/eA1MXmtkV/IoR3IzfDTA5HbeEyTmByQtOYKqNEgKQ2uYpBUeU0AlKJyB5dUGCO9ExDgl5lkHyACTz+81tWwYJA0iueUi+ZZB8AMkzDslDlkHqAEjYPCTbDu4AQPKNQ8K2HdxdAKljHJJv28HtQEfpjbD8NkFJkqpP6DepucoyGZKwOFPXoSJDhTLkRJQTnvR0Q8qiKL+9L+iUrcnr5lE58glnmdxMCvdbeJA/ayb5dBuU3Qh6t11m7kLmp1x4VBtz6HdtmftJzvVVqKs4v4p4SlgG7c2XxOkEJU18KAk+IQmuTRLo5V0nCXScmyYJci2TBPqU10kC3fTGSdK1TBLowV4nCQwKmiaJiy2TBPrL10kCQ5CmSeI5lkkCvfPrJIEBT+Mkse31DmOB6ySB4VXTJMGWvd4L7/rTksBgrnGSWPZ6RzAWBIxpFNOR/sqFHPOYZyR53Fv75ZB63+cH5xPN7i+VcqULbznWC8PsKZ+JkL43gaJcR0RM5TsdNfd8Mu8KJWhCJJuXk+O35+4A7r8HX0ff1dIbKpqTr+rm3pBmVKjBcLVse4psz9984oHZbVAqvj0ogZj6AdT+SRc3epMcZZV2+YiDTXIyY1Jb9O7BULFiaanWAqZ/VMA0nXDyYFhXsehUJ74dLmvwwRAMHtL24Otgy/DBcKlioequm7djGh8MbSqWsO66eY3jg2FIxeLWXTevaXwYhgwVy1733Lye6bMPQzezYkHsnpvXPD4YHQUW4zvavObxnauw2FJorFMNJzh6kZsuQeJzxZWdGoYLWndVIzCtxrm6yk4N6OP/t2p4xvdG1ZJK88vzyEclTYynJnHVmkrz6/Ougy3TpGpRxfChdhMvq22XJj70sgDkO6TwP5cV/jC/X5xKH+b3i0DbkgR/Me7LE/wKYg9dnuCvPQsflNf5Lro4/BuMf2Khu9cvdPV1/w+lTdvB/7zcx38=</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="167" y="250" width="170" height="160" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="83" y="50" width="480" height="60" fill="#ffffff" stroke="#000000" pointer-events="all"/><rect x="93" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 94px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0</div></div></div></foreignObject><text x="113" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0</text></switch></g><rect x="153" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 154px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">1</div></div></div></foreignObject><text x="173" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">1</text></switch></g><rect x="213" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 214px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">2</div></div></div></foreignObject><text x="233" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">2</text></switch></g><rect x="273" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 274px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">3</div></div></div></foreignObject><text x="293" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">3</text></switch></g><rect x="333" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 334px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">4</div></div></div></foreignObject><text x="353" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="393" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 394px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">5</div></div></div></foreignObject><text x="413" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">5</text></switch></g><rect x="453" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 454px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">6</div></div></div></foreignObject><text x="473" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">6</text></switch></g><rect x="513" y="60" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 80px; margin-left: 514px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">7</div></div></div></foreignObject><text x="533" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">7</text></switch></g><rect x="0" y="70" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 80px; margin-left: 35px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 0</div></div></div></foreignObject><text x="35" y="84" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 0</text></switch></g><rect x="88" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 113px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />0</div></div></div></foreignObject><text x="113" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="148" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 173px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />1</div></div></div></foreignObject><text x="173" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="208" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 233px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />2</div></div></div></foreignObject><text x="233" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="268" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 293px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />3</div></div></div></foreignObject><text x="293" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="328" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 353px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />4</div></div></div></foreignObject><text x="353" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="388" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 413px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />5</div></div></div></foreignObject><text x="413" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="448" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 473px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />6</div></div></div></foreignObject><text x="473" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="508" y="0" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 25px; margin-left: 533px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />7</div></div></div></foreignObject><text x="533" y="29" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><path d="M 473 140 L 473 106.37" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 473 101.12 L 476.5 108.12 L 473 106.37 L 469.5 108.12 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="353" y="140" width="240" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 150px; margin-left: 473px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Courier New; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">VDISK[GroupId:Generation:0:6:0]</div></div></div></foreignObject><text x="473" y="154" fill="#000000" font-family="Courier New" font-size="12px" text-anchor="middle">VDISK[GroupId:Generation:0:6:0]</text></switch></g><rect x="177" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0</div></div></div></foreignObject><text x="197" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0</text></switch></g><rect x="230" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">1</div></div></div></foreignObject><text x="250" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">1</text></switch></g><rect x="285" y="260" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 280px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">2</div></div></div></foreignObject><text x="305" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">2</text></switch></g><rect x="177" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">3</div></div></div></foreignObject><text x="197" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">3</text></switch></g><rect x="230" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">4</div></div></div></foreignObject><text x="250" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="285" y="310" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 330px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">5</div></div></div></foreignObject><text x="305" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">5</text></switch></g><rect x="177" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 178px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">6</div></div></div></foreignObject><text x="197" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">6</text></switch></g><rect x="230" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 231px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">7</div></div></div></foreignObject><text x="250" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">7</text></switch></g><rect x="285" y="360" width="40" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 38px; height: 1px; padding-top: 380px; margin-left: 286px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">8</div></div></div></foreignObject><text x="305" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="97" y="270" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 280px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 0</div></div></div></foreignObject><text x="132" y="284" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 0</text></switch></g><rect x="97" y="320" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 330px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 1</div></div></div></foreignObject><text x="132" y="334" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 1</text></switch></g><rect x="97" y="370" width="70" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 380px; margin-left: 132px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail realm 2</div></div></div></foreignObject><text x="132" y="384" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail realm 2</text></switch></g><rect x="172" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 197px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />0</div></div></div></foreignObject><text x="197" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="225" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />1</div></div></div></foreignObject><text x="250" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><rect x="280" y="200" width="50" height="50" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 225px; margin-left: 305px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">fail<br />domain<br />2</div></div></div></foreignObject><text x="305" y="229" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">fail...</text></switch></g><path d="M 390 330 L 331.37 330" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 326.12 330 L 333.12 326.5 L 331.37 330 L 333.12 333.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="390" y="315" width="160" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe flex-start; width: 1px; height: 1px; padding-top: 322px; margin-left: 392px;"><div style="box-sizing: border-box; font-size: 0; text-align: left; "><div style="display: inline-block; font-size: 12px; font-family: Courier New; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">VDISK[GroupId:Generation:1:2:0]</div></div></div></foreignObject><text x="392" y="334" fill="#000000" font-family="Courier New" font-size="12px">VDISK[GroupId:Generation:1...</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md new file mode 100644 index 0000000000..a00da63b70 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md @@ -0,0 +1,6 @@ +# General schema {{ ydb-short-name }} + +An approximate general {{ ydb-short-name }} schema is shown below. + + + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md new file mode 100644 index 0000000000..684a8f7b36 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md @@ -0,0 +1,12 @@ +## Nodes + +One {{ ydb-short-name }} installation consists of a *cluster* that is divided into *nodes*. A node is a single process in the system, usually kikimr. This node is part of a cluster and can exchange data with its other nodes via *Interconnect*. Each *node* has its own ID which is usually named NodeId. NodeID is an integer from 1, consisting of 20 bits. NodeID 0 is reserved for internal needs and usually indicates the current node or no node. + +A number of services are run on each node and implemented via *actors*. + +Nodes can be static and dynamic. + +A configuration of static nodes, that is, their complete list with the address for connecting via Interconnect specified, is stored in a configuration file and is read once when the process is started. A set of static nodes changes very rarely. This usually happens when expanding clusters or moving nodes from one physical machine to another. To change the set of static nodes, apply the updated configuration to **every** node and then perform a rolling restart of the entire cluster. + +Dynamic nodes are not known in advance and are added to the system as new processes are started. This may be due, for example, to the creation of new tenants in {{ ydb-short-name }} installations as a database. When registering a dynamic node, its process first connects to one of the static nodes via gRPC, transmits information about itself through a special service called Node Broker, and receives the NodeID to log in with in response. The mechanism for assigning nodes is pretty much similar to the DHCP in the context of distributing IP addresses. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md new file mode 100644 index 0000000000..9842fa66e5 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md @@ -0,0 +1,38 @@ +## Tablets + +Special microservices called *tablets* are run on each node. Each tablet has a specific type and ID and is a singleton, meaning that only one tablet with a specific ID can be running in the entire cluster at any given time. A tablet can be started on any node that is suitable for it. An important characteristic of a tablet is its *generation* that increases with each subsequent run. Please note that due to the distributed nature of the system and in case of various kinds of issues, such as with network partitioning, a situation may arise when the same tablet will actually be running on two different nodes at the same time. However, BlobStorage guarantees that only one of them will be able to successfully complete operations that change its state and the generation that each successful operation is performed in will not decrease over time. + +You can find out on what node the tablet in the current generation is running through the *StateStorage* service. To send messages to tablets, use a special set of libraries named *tablet pipe*. With it, knowing the ID of the target tablet, you can easily send the desired message to it. + +A tablet can be divided into two parts: the basic tablet and the user logic. + +The basic tablet is a set of tables, each of which may consist of one or more key columns of an arbitrary type and a set of data columns. Each table may have its own schema. In addition, tables can be created and deleted while the tablet is running. The interface of the basic tablet lets you perform read and update operations on these tables. + +The user logic is located between the basic tablet and the user and lets you process specific requests for this type of tablet, reliably saving changes in BlobStorage. A commonly used template for tablet operation is storing all data in memory, reading it only at the start, and synchronously changing the data in memory and in storage after a successful commit. + +### How does a tablet store data and what data is it? + +The basic tablet is an LSM tree that contains all of its table data. One level below the basic tablet is BlobStorage that, roughly speaking, is KeyValue storage that stores binary large objects (blobs). *BLOB* is a binary fragment from 1 byte to 10 MB in size, which has a fixed ID (that is usually called *BlobId* and is of the TLogoBlobID type) and contains related data. Storage is immutable, meaning that only one value corresponds to each ID and it cannot change over time. You can write and read a blob and then delete it when you no longer need it. + +To learn more about blobs and distributed storages, see [here](../../distributed_storage.md). + +For BlobStorage, blobs are an opaque entity. A tablet can store several types of blobs. The most frequently written blob is a log blob (meaning a recovery log). A tablet's log is arranged in a list of blobs, each of which contains information about the change being made to the tables. When run, the tablet finds the last blob in the log and then recursively reads all related blobs following the links. The log may also mention snapshot blobs, which is a type of blobs that contain data from multiple log blobs after merging them (the merge operation in the LSM tree). + +The tablet writes blobs of different types to different *channels*. A channel specifies the branch of storage to store blobs in and performs various functions, such as: + +1. Choosing the storage type (different channels can be linked to different types of storage devices: SSD, HDD, or NVME). +2. Load balancing, because each channel has a limit on IOPS, available space and bandwidth. +3. Specifying the data type. When restoring the log, only the blobs from the null channel are read, which lets you distinguish them from other blobs. + +### Tablet channel history + +As mentioned above, each group has a fixed amount of data that can fit into it and also divides the bandwidth by throughput and the number of operations per second between all consumers. The load on tablets may vary. As a result a group may become overloaded. In this regard, the concept of history is introduced, with which, for each tablet, knowing a blob's Channel and Generation, you can determine the group that this blob is written to. + +This mechanism works as follows: + + + +For each channel, the TTabletStorageInfo structure contains the TTabletChannelInfo substructure with generation ranges and the group number corresponding to each range. The ranges are strictly adjacent to each other, the last range is open. Group numbers may overlap in different ranges and even across different channels: this is not prohibited and is quite common. + +When writing a blob, a tablet selects the most recent range for the corresponding channel, since the write operation is always performed on behalf of the current generation of the tablet. When reading a blob, the group number is fetched based on the BlobId.Generation of the blob being read. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md new file mode 100644 index 0000000000..38d25100d2 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md @@ -0,0 +1,357 @@ +## BlobStorage implementation details + +### Managing BlobStorage resources on nodes via NodeWarden + +### How PDisk works + +#### Data format + +### How VDisk works + +#### Large and small blobs + +#### Fresh segment + +#### SSTable + +#### Syncing metadata + +#### Replicating data + +#### Data format + +##### Donor disks + +### How DS proxy works + +## BS_CONTROLLER + +### Communicating with NodeWarden + +### Box and Storage Pool + +### Self Heal + +### Entities and tables + +All information stored in BS_CONTROLLER can be grouped into: + +1. BS_CONTROLLER settings. +2. Low-level configuration (PDisks, VDisks, and groups). +3. Top-level configuration (Boxes and StoragePools). +4. Runtime information (persistent disk and group metrics, the status of scrubbing of individual VDisks, information about node drive serial numbers). +5. OperationLogs. + +The operation logic is built so that all table data at the start of the BS_CONTROLLER tablet is loaded into memory and stored there for the entire time of its operation (with the exception of OperationLog). When executing transactions, the data is updated in tables and in memory. + +The tables are discussed in more detail below. + +#### BS_CONTROLLER settings + +The settings are stored in the State table. + +#### Low-level configuration + +Low-level configuration includes information about the available cluster resources: PDisks, VDisks, and groups. These tables are interconnected via PK <-> FK links. + +#### Top-level configuration + +Top-level configuration stores data necessary for linking Boxes and nodes where they are located... + +#### Database schema + +Box + +| Column | ID | PK | Type | Default | Purpose | +| ------------ | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| Name | 2 | | Utf8 | | | +| Generation | 3 | | Uint64 | | | + +BoxHost + +| Column | ID | PK | Type | Default | Purpose | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| FQDN | 3 | * | Utf8 | | | +| IcPort | 4 | * | Int32 | | | +| HostConfigId | 5 | | Uint64 | | | + +BoxHostV2 + +| Column | ID | PK | Type | Default | Purpose | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| FQDN | 2 | * | Utf8 | | | +| IcPort | 3 | * | Int32 | | | +| HostConfigId | 4 | | Uint64 | | | + +BoxStoragePool + +| Column | ID | PK | Type | Default | Purpose | +| ---------------------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| Name | 20 | | Utf8 | | | +| ErasureSpecies | 3 | | Int32 | | | +| RealmLevelBegin | 4 | | Int32 | | | +| RealmLevelEnd | 5 | | Int32 | | | +| DomainLevelBegin | 6 | | Int32 | | | +| DomainLevelEnd | 7 | | Int32 | | | +| NumFailRealms | 8 | | Int32 | | | +| NumFailDomainsPerFailRealm | 9 | | Int32 | | | +| NumVDisksPerFailDomain | 10 | | Int32 | | | +| VDiskKind | 11 | | Int32 | | | +| SpaceBytes | 12 | | Uint64 | | | +| WriteIOPS | 13 | | Uint64 | | | +| WriteBytesPerSecond | 14 | | Uint64 | | | +| ReadIOPS | 15 | | Uint64 | | | +| ReadBytesPerSecond | 16 | | Uint64 | | | +| InMemCacheBytes | 17 | | Uint64 | | | +| Kind | 18 | | Utf8 | | | +| NumGroups | 19 | | Uint32 | | | +| Generation | 21 | | Uint64 | | | +| EncryptionMode | 22 | | Uint32 | 0 | | +| SchemeshardId | 23 | | Uint64 | | | +| PathItemId | 24 | | Uint64 | | | +| RandomizeGroupMapping | 25 | | Bool | false | | + +BoxStoragePoolPDiskFilter + +| Column | ID | PK | Type | Default | Purpose | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| Type | 3 | * | Int32 | | | +| SharedWithOs | 4 | * | Bool | | | +| ReadCentric | 5 | * | Bool | | | +| Kind | 6 | * | Uint64 | | | + +BoxStoragePoolUser + +| Column | ID | PK | Type | Default | Purpose | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| StoragePoolId | 2 | * | Uint64 | | | +| UserId | 3 | * | String | | | + +BoxUser + +| Column | ID | PK | Type | Default | Purpose | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| BoxId | 1 | * | Uint64 | | | +| UserId | 2 | * | String | | | + +DriveSerial + +| Column | ID | PK | Type | Default | Purpose | +| ------------- | ---- | ---- | -------- | -------------- | ------------ | +| Serial | 1 | * | String | | | +| BoxId | 2 | | Uint64 | | | +| NodeId | 3 | | Uint32 | | | +| PDiskId | 4 | | Uint32 | | | +| Guid | 5 | | Uint64 | | | +| LifeStage | 6 | | Uint32 | | | +| Kind | 7 | | Uint64 | | | +| PDiskType | 8 | | Int32 | | | +| PDiskConfig | 9 | | String | | | + +DriveStatus + +| Column | ID | PK | Type | Default | Purpose | +| ----------- | ---- | ---- | -------- | -------------- | ------------ | +| FQDN | 1 | * | Utf8 | | | +| IcPort | 2 | * | Int32 | | | +| Path | 3 | * | Utf8 | | | +| Status | 4 | | Uint32 | | | +| Timestamp | 5 | | Uint64 | | | + +Group + +| Column | ID | PK | Type | Default | Purpose | +| ---------------------- | ---- | ---- | -------- | -------------- | ------------ | +| ID | 1 | * | Uint32 | | | +| Generation | 2 | | Uint32 | | | +| ErasureSpecies | 3 | | Uint32 | | | +| Owner | 4 | | Uint64 | | | +| DesiredPDiskCategory | 5 | | Uint64 | | | +| DesiredVDiskCategory | 6 | | Uint64 | | | +| EncryptionMode | 7 | | Uint32 | 0 | | +| LifeCyclePhase | 8 | | Uint32 | 0 | | +| MainKeyId | 9 | | String | | | +| EncryptedGroupKey | 10 | | String | | | +| GroupKeyNonce | 11 | | Uint64 | 0 | | +| MainKeyVersion | 12 | | Uint64 | 0 | | +| Down | 13 | | Bool | false | | +| SeenOperational | 14 | | Bool | false | | + +GroupLatencies + +| Column | ID | PK | Type | Default | Purpose | +| ----------------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupId | 1 | * | Uint32 | | | +| PutTabletLogLatencyUs | 4 | | Uint32 | | | +| PutUserDataLatencyUs | 5 | | Uint32 | | | +| GetFastLatencyUs | 6 | | Uint32 | | | + +GroupStoragePool + +| Column | ID | PK | Type | Default | Purpose | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupId | 1 | * | Uint32 | | | +| BoxId | 2 | | Uint64 | | | +| StoragePoolId | 3 | | Uint64 | | | + +HostConfig + +| Column | ID | PK | Type | Default | Purpose | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| HostConfigId | 1 | * | Uint64 | | | +| Name | 2 | | Utf8 | | | +| Generation | 3 | | Uint64 | | | + +HostConfigDrive + +| Column | ID | PK | Type | Default | Purpose | +| -------------- | ---- | ---- | -------- | -------------- | ------------ | +| HostConfigId | 1 | * | Uint64 | | | +| Path | 2 | * | Utf8 | | | +| Type | 3 | | Int32 | | | +| SharedWithOs | 4 | | Bool | | | +| ReadCentric | 5 | | Bool | | | +| Kind | 6 | | Uint64 | | | +| PDiskConfig | 7 | | String | | | + +MigrationEntry + +| Column | ID | PK | Type | Default | Purpose | +| --------------- | ---- | ---- | -------- | -------------- | ------------ | +| PlanName | 1 | * | String | | | +| EntryIndex | 2 | * | Uint64 | | | +| GroupId | 3 | | Uint32 | | | +| OriginNodeId | 4 | | Uint32 | | | +| OriginPDiskId | 5 | | Uint32 | | | +| OriginVSlotId | 6 | | Uint32 | | | +| TargetNodeId | 7 | | Uint32 | | | +| TargetPDiskId | 8 | | Uint32 | | | +| Done | 9 | | Bool | | | + +MigrationPlan + +| Column | ID | PK | Type | Default | Purpose | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| Name | 1 | * | String | | | +| State | 2 | | Uint32 | | | +| Size | 3 | | Uint64 | | | +| Done | 4 | | Uint64 | | | + +Node + +| Column | ID | PK | Type | Default | Purpose | +| ------------------- | ---- | ---- | -------- | -------------- | ------------ | +| ID | 1 | * | Uint32 | | | +| NextPDiskID | 2 | | Uint32 | | | +| NextGroupKeyNonce | 9 | | Uint64 | 0 | | + +OperationLog + +| Column | ID | PK | Type | Default | Purpose | +| --------------- | ---- | ---- | -------- | ------------------- | ------------ | +| Index | 1 | * | Uint64 | | | +| Timestamp | 2 | | Uint64 | | | +| Request | 3 | | String | | | +| Response | 4 | | String | | | +| ExecutionTime | 5 | | Uint64 | TDuration::Zero() | | + +PDisk + +| Column | ID | PK | Type | Default | Purpose | +| ---------------- | ---- | ---- | -------- | -------------- | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| Path | 3 | | Utf8 | | | +| Category | 4 | | Uint64 | | | +| Guid | 7 | | Uint64 | | | +| SharedWithOs | 8 | | Bool | | | +| ReadCentric | 9 | | Bool | | | +| NextVSlotId | 10 | | Uint32 | 1000 | | +| PDiskConfig | 11 | | String | | | +| Status | 12 | | Uint32 | | | +| Timestamp | 13 | | Uint64 | | | +| ExpectedSerial | 14 | | String | | | +| LastSeenSerial | 15 | | String | | | +| LastSeenPath | 16 | | String | | | + +PDiskMetrics + +| Column | ID | PK | Type | Default | Purpose | +| --------- | ---- | ---- | -------- | -------------- | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| Metrics | 3 | | String | | | + +ScrubState + +| Column | ID | PK | Type | Default | Purpose | +| ---------------------- | ---- | ---- | -------- | ------------------ | ------------ | +| NodeId | 1 | * | Uint32 | | | +| PDiskId | 2 | * | Uint32 | | | +| VSlotId | 3 | * | Uint32 | | | +| State | 5 | | String | | | +| ScrubCycleStartTime | 6 | | Uint64 | TInstant::Zero() | | +| ScrubCycleFinishTime | 8 | | Uint64 | TInstant::Zero() | | +| Success | 7 | | Bool | false | | + +State + +| Column | ID | PK | Type | Default | Purpose | +| -------------------------- | ---- | ---- | -------- | --------------------------------------------- | ------------ | +| FixedKey | 1 | * | Bool | | | +| NextGroupID | 2 | | Uint32 | | | +| SchemaVersion | 4 | | Uint32 | 0 | | +| NextOperationLogIndex | 5 | | Uint64 | | | +| DefaultMaxSlots | 6 | | Uint32 | 16 | | +| InstanceId | 7 | | String | | | +| SelfHealEnable | 8 | | Bool | false | | +| DonorModeEnable | 9 | | Bool | true | | +| ScrubPeriodicity | 10 | | Uint32 | 86400*30 | | +| SerialManagementStage | 11 | | Uint32 | | | +| NextStoragePoolId | 12 | | Uint64 | 0 | | +| PDiskSpaceMarginPromille | 13 | | Uint32 | 150 | | +| GroupReserveMin | 14 | | Uint32 | 0 | | +| GroupReservePart | 15 | | Uint32 | 0 | | +| MaxScrubbedDisksAtOnce | 16 | | Uint32 | Max<ui32>() | | +| PDiskSpaceColorBorder | 17 | | Uint32 | NKikimrBlobStorage::TPDiskSpaceColor::GREEN | | + +VDiskMetrics + +| Column | ID | PK | Type | Default | Purpose | +| ----------------- | ---- | ---- | -------- | -------------- | ------------ | +| GroupID | 1 | * | Uint32 | | | +| GroupGeneration | 2 | * | Uint32 | | | +| Ring | 3 | * | Uint32 | | | +| FailDomain | 4 | * | Uint32 | | | +| VDisk | 5 | * | Uint32 | | | +| Metrics | 11 | | String | | | + +VSlot + +| Column | ID | PK | Type | Default | Purpose | +| --------------------- | ---- | ---- | -------- | ------------------ | ------------ | +| NodeID | 1 | * | Uint32 | | | +| PDiskID | 2 | * | Uint32 | | | +| VSlotID | 3 | * | Uint32 | | | +| Category | 4 | | Uint64 | | | +| GroupID | 5 | | Uint32 | | | +| GroupGeneration | 6 | | Uint32 | | | +| RingIdx | 7 | | Uint32 | | | +| FailDomainIdx | 8 | | Uint32 | | | +| VDiskIdx | 9 | | Uint32 | | | +| GroupPrevGeneration | 10 | | Uint32 | 0 | | +| Mood | 11 | | Uint32 | 0 | | +| LastSeenReady | 12 | | Uint64 | TInstant::Zero() | | + +### Messages via pipe + +### Transaction types + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md new file mode 100644 index 0000000000..3dde12875d --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md @@ -0,0 +1,74 @@ +## Description of the BlobStorage interface + +### Blob ID format + +Each blob has a 192-bit ID consisting of the following fields (in the order used for sorting): + +1. TabletId (64 bits): ID of the blob owner tablet. +2. Channel (8 bits): Channel sequence number. +3. Generation (32 bits): Number of the generation in which the tablet that wrote this blob was run. +4. Step (32 bits): Internal number of the blob group within the Generation. +5. Cookie (24 bits): ID that can be used if the Step is insufficient. +6. CrcMode (2 bits): Selects mode for redundant blob integrity control at the BlobStorage level. +7. BlobSize (26 bits): Blob data size. +8. PartId (4 bits): Part number when blob erasure coding is used. At the "BlobStorage <-> tablet" interaction level, this parameter is always 0, meaning the entire blob. + +Two blobs are considered different if at least one of the first five parameters (TabletId, Channel, Generation, Step, or Cookie) differs in their IDs. So it is impossible to write two blobs that only differ in BlobSize and/or CrcMode. + +For debugging purposes, there is string blob ID formatting that has interactions `[TabletId:Generation:Step:Channel:Cookie:BlobSize:PartId]`, for example, `[12345:1:1:0:0:1000:0]`. + +When writing a blob, the tablet selects the Channel, Step, and Cookie parameters. The TabletId is fixed and must indicate the tablet performing the write operation, while the Generation parameter must indicate the generation that the tablet performing the operation is running in. + +When performing reads, the blob ID is specified, which can be arbitrary, but preferably preset. + +### Groups + +Blobs are written in a logical entity called *group*. On each node, a special actor called DS proxy is created for each group that the blob is written to. This actor is responsible for performing all operations related to the group. The actor is created automatically through the NodeWarden service that will be described below. + +Physically, a group is a set of multiple physical devices (OS block devices) that are located on different nodes so that the failure of one device correlates as little as possible with the failure of another device. These devices are usually located in different racks or different datacenters. On each of these devices, some space is allocated for the group, which is managed by a special service called *VDisk*. Each VDisk runs on top of a block device from which it is separated by another service called *PDisk*. Blobs are broken into fragments based on *erasure coding* with these fragments written to VDisks. Before splitting into fragments, optional encryption of the data in the group can be performed. + +This scheme is shown in the figure below. + + + +VDisks from different groups are shown as multicolored squares; one color stands for one group. + +A group can be treated as a set of VDisks: + + + +Each VDisk within the group has a sequence number, the disks are numbered from 0 to N-1, where N is the number of disks in the group. + +In addition, the group disks are combined into fail domains, while the fail domains are combined into fail realms. As a rule, each fail domain has exactly one disk inside (although, in theory, it may have more, but this has found no application in practice), and multiple fail realms are only used for groups that host their data in three datacenters at once. In addition to the sequence number in the group, each VDisk is assigned an ID that consists of a fail realm index, fail domain index inside the fail realm, and the VDisk index inside the fail domain. In string form, this ID is written as `VDISK[GroupId:GroupGeneration:FailRealm:FailDomain:VDisk]`. + +All the fail realms have the same number of fail domains and all the fail domains have the same number of disks inside. The number of the fail realms, the number of the fail domains inside the fail realm, and the number of the disks inside the fail domain make up the geometry of the group. The geometry depends on the way the data is encoded in the group. For example, for block-4-2 numFailRealms = 1, numFailDomainsInFailRealm >= 8 (only 8 fail realms are used in practice), numVDisksInFailDomain >= 1 (strictly 1 fail domain is used in practice). For mirror-3-dc numFailRealms >= 3, numFailDomainsInFailRealm >= 3, and numVDisksInFailDomain >= 1 (3x3x1 are used). + +Each PDisk has an ID that consists of the number of the node that it is running on and the internal number of the PDisk inside this node. This ID is usually written as NodeId:PDiskId. For example, 1:1000. If you know the PDisk ID, you can calculate the service ActorId of this disk and send it a message. + +Each VDisk runs on top of a specific PDisk and has a *slot ID* consisting of three fields (NodeID:PDiskId:VSlotId) and the above-mentioned VDisk ID. Strictly speaking, there are different concepts: a slot is the space reserved on the PDisk and occupied by the VDisk, and the VDisk is a group component that occupies a certain slot and performs operations on it. Similarly to PDisks, if you know the slot ID, you can calculate the service ActorId of the running VDisk and send it a message. To send messages from the DS proxy to the VDisk, an intermediate actor called *BS_QUEUE* is used. + +The composition of each group is not fixed: it may change while using the system. For this purpose, the concept of "group generation" is introduced. Each "GroupId:GroupGeneration" pair corresponds to a fixed set of slots (a vector that consists of N slot IDs, where N is the size of the group), storing the data of the entire group. *Please note that a group generation and a tablet generation are not related in any way*. + +As a rule, groups of two adjacent generations differ by no more than one slot. + +### Subgroups + +For each blob, a special concept of a *subgroup* is introduced, which is an ordered subset of group disks with a strictly fixed number of elements, depending on the type of encoding (the number of elements in the group must be at least the same), where the data of this blob will be stored. For single-datacenter groups with conventional encoding, this subset is selected as the first N elements of a cyclic disk permutation in the group, where the permutation depends on the BlobId hash. + +Each disk in the subgroup corresponds to a disk in the group, but is limited by the allowed number of stored blobs. For example, to encode block-4-2 with four data parts and two parity parts, the functional purpose of disks in the subgroup is as follows: + +| Number in the subgroup | Possible PartIds | +| ------------------- | ------------------- | +| 0 | 1 | +| 1 | 2 | +| 2 | 3 | +| 3 | 4 | +| 4 | 5 | +| 5 | 6 | +| 6 | 1,2,3,4,5,6 | +| 7 | 1,2,3,4,5,6 | + +In this case, PartID=1..4 corresponds to the data parts (which are obtained by splitting the original blob into 4 equal parts), and PartID=5..6 are parity parts. Disks numbered 6 and 7 in the subgroup are called *handoff disks*. Any part, either one or more, can be written to them. Disks 0..5 can only store the corresponding blob parts. + +In practice, when performing writes, the system tries to write 6 parts to the first 6 disks of the subgroup and, in the vast majority of cases, these attempts are successful. However, if any of the disks is not available, the write operation fails and the system uses handoff disks where parts of the disks that did not respond in time are sent. This may result in a situation when multiple parts of the same blob are written to one handoff disk. This is acceptable, although it makes no sense in terms of storage: each part must have its own unique disk. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md new file mode 100644 index 0000000000..05d8f54d26 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md @@ -0,0 +1,4 @@ +### Garbage collection + +### Tablet suspension + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md new file mode 100644 index 0000000000..27f0431c91 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md @@ -0,0 +1,6 @@ +# Disk subsystem of a cluster aka {{ ydb-short-name }} BlobStorage + +{{ ydb-short-name }} BlobStorage is a {{ ydb-short-name }} subsystem that is responsible for reliable data storage. + +Lets you store *blobs* (binary fragments from 1 byte to 10 megabytes in size) with a unique identifier. + diff --git a/ydb/docs/en/core/concepts/cluster/common_scheme_ydb.md b/ydb/docs/en/core/concepts/cluster/common_scheme_ydb.md new file mode 100644 index 0000000000..1d03e7af51 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/common_scheme_ydb.md @@ -0,0 +1,5 @@ +{% include [concepts/index/intro.md](_includes/common_scheme_ydb/intro.md) %} + +{% include [concepts/index/when_use.md](_includes/common_scheme_ydb/nodes.md) %} + +{% include [concepts/index/when_use.md](_includes/common_scheme_ydb/tablets.md) %} diff --git a/ydb/docs/en/core/concepts/cluster/distributed_storage.md b/ydb/docs/en/core/concepts/cluster/distributed_storage.md new file mode 100644 index 0000000000..5b31a3ff61 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/distributed_storage.md @@ -0,0 +1,3 @@ +{% include [concepts/index/intro.md](_includes/distributed_storage/intro.md) %} + +{% include [concepts/index/when_use.md](_includes/distributed_storage/distributed_storage_interface.md) %} diff --git a/ydb/docs/en/core/concepts/cluster/index.md b/ydb/docs/en/core/concepts/cluster/index.md new file mode 100644 index 0000000000..489074e143 --- /dev/null +++ b/ydb/docs/en/core/concepts/cluster/index.md @@ -0,0 +1,8 @@ +# Cluster {{ ydb-short-name }} + +The information in this section is mainly intended for {{ ydb-short-name }} administrators and developers. + +In ["General {{ ydb-short-name }} schema"](common_scheme_ydb.md), you can find an overview about nodes and tablets. + +From ["Cluster's disk subsystem"](distributed_storage.md), you can learn about the {{ ydb-short-name }} distributed storage system. + diff --git a/ydb/docs/en/core/concepts/connect.md b/ydb/docs/en/core/concepts/connect.md new file mode 100644 index 0000000000..c0cae8a17b --- /dev/null +++ b/ydb/docs/en/core/concepts/connect.md @@ -0,0 +1,2 @@ + +{% include [connect.md](_includes/connect.md) %} diff --git a/ydb/docs/en/core/concepts/datamodel.md b/ydb/docs/en/core/concepts/datamodel.md index 934fb31496..36d7fde82c 100644 --- a/ydb/docs/en/core/concepts/datamodel.md +++ b/ydb/docs/en/core/concepts/datamodel.md @@ -2,7 +2,7 @@ {% include [Table](_includes/datamodel/table.md) %} -{% include [Folder](_includes/datamodel/folder.md) %} +{% include [Folder](_includes/datamodel/dir.md) %} {% include [Persistent Queue](_includes/datamodel/pq.md) %} diff --git a/ydb/docs/en/core/concepts/datatypes.md b/ydb/docs/en/core/concepts/datatypes.md index 1d5263968a..7676dc957a 100644 --- a/ydb/docs/en/core/concepts/datatypes.md +++ b/ydb/docs/en/core/concepts/datatypes.md @@ -1,3 +1,2 @@ -# Data types {{ ydb-name }} +This page has been deleted, for information about data types, see[YQL Data Types](../yql/reference/types/index.md). -{{ ydb-short-name }} uses YQL data types. diff --git a/ydb/docs/en/core/concepts/distributed_storage.md b/ydb/docs/en/core/concepts/distributed_storage.md new file mode 100644 index 0000000000..8288ec7f92 --- /dev/null +++ b/ydb/docs/en/core/concepts/distributed_storage.md @@ -0,0 +1,5 @@ +{% include [concepts/index/intro.md](_includes/distributed_storage/intro.md) %} + +{% include [concepts/index/when_use.md](_includes/distributed_storage/common_scheme_ydb.md) %} + +{% include [concepts/index/when_use.md](_includes/distributed_storage/detailed_distributed_storage.md) %} diff --git a/ydb/docs/en/core/concepts/toc_i.yaml b/ydb/docs/en/core/concepts/toc_i.yaml index 5c05a1854f..fa3795c630 100644 --- a/ydb/docs/en/core/concepts/toc_i.yaml +++ b/ydb/docs/en/core/concepts/toc_i.yaml @@ -1,11 +1,20 @@ items: -- { name: Overview, href: index.md } -- { name: Terms and definitions, href: databases.md } -- { name: Data model and schema, href: datamodel.md } +- { name: Overview, href: index.md } +- { name: Terms and definitions, href: databases.md } +- { name: Connecting to and authenticating with a database, href: connect.md } +- { name: Data model and schema, href: datamodel.md } - { name: Serverless and Dedicated operation modes, href: serverless_and_dedicated.md } -- { name: Data types, href: datatypes.md } -- { name: Transactions, href: transactions.md } -- { name: Secondary indexes, href: secondary_indexes.md } -- { name: Time to Live (TTL), href: ttl.md } -- { name: Scan queries, href: scan_query.md } -- { name: Database limits, href: limits-ydb.md } +- { name: Data types, href: datatypes.md, hidden: true } # Deprecated +- { name: Transactions, href: transactions.md } +- { name: Secondary indexes, href: secondary_indexes.md } +- { name: Time to Live (TTL), href: ttl.md } +- { name: Scan queries, href: scan_query.md } +- { name: Database limits, href: limits-ydb.md } +- name: YDB cluster + items: + - name: Overview + href: cluster/index.md + - name: General YDB schema + href: cluster/common_scheme_ydb.md + - name: Disk subsystem of a cluster + href: cluster/distributed_storage.md
\ No newline at end of file diff --git a/ydb/docs/en/core/deploy/configuration/config.md b/ydb/docs/en/core/deploy/configuration/config.md new file mode 100644 index 0000000000..a37154ecc9 --- /dev/null +++ b/ydb/docs/en/core/deploy/configuration/config.md @@ -0,0 +1,210 @@ +# Creating a configuration for deploying a cluster + +To deploy a {{ ydb-short-name }} cluster, you need to create a cluster configuration. +This section describes how to create a {{ ydb-short-name }} cluster configuration in YAML format. + +## Description of storage host configurations + +Create and describe a host configuration. For each host configuration, specify the sequence number and a list of paths to disks and their types. +The following disk types are available: `SSD`, `NVME`, and `ROT` (in this case, `ROT` disks are `HDD`). + +For example: + +```bash +host_configs: +- drive: + - path: /dev/disk/by-partlabel/ydb_disk_ssd_01 + type: SSD + host_config_id: 1 +``` + +In this example, we can find exactly one type of host whose sequence number is 1. In this host configuration, exactly one disk is specified, its type is `SSD` and path is `/dev/disk/by-partlabel/ydb_disk_ssd_01`. + +Below is another configuration example. Let's assume that we have two types of host configurations, one of them with 2 disks available on the host and the other one with 3 disks. +This configuration can be specified as follows: + +```bash +host_configs: +- drive: + - path: /dev/disk/by-partlabel/ydb_ssd_01 + type: SSD + - path: /dev/disk/by-partlabel/ydb_ssd_02 + type: SSD + host_config_id: 1 +- drive: + - path: /dev/disk/by-partlabel/ydb_ssd_01 + type: SSD + - path: /dev/disk/by-partlabel/ydb_ssd_02 + type: SSD + - path: /dev/disk/by-partlabel/ydb_ssd_03 + type: SSD + host_config_id: 2 +``` + +## Description of cluster hosts + +List the hosts to run a cluster on. For each host, specify the sequence number and the port where `Interconnect` will be run on this host. +You also need to specify the physical location of the host and the unique ID of the host configuration. + +For example, + +```bash +hosts: +- host: hostname1 + host_config_id: 1 + port: 19001 + location: + unit: '1' + data_center: '1' + rack: '1' +- host: hostname2 + host_config_id: 1 + node_id: 2 + port: 19001 + location: + unit: '1' + data_center: '1' + rack: '1' +``` + +## Description of a cluster domain configuration + +Describe a cluster domain configuration. Specify the domain name, storage types, the numbers of the hosts that will be included in `StateStorage`, and the `nto_select` parameter. +In the storage configuration, specify the type of storage and the type of storage fault tolerance (`erasure`), which will be used to initialize the database storage. +You should also specify what types of disks this storage type will correspond to. The following models of storage fault tolerance are available: + +* The `block-4-2` configuration assumes deployment in 8 fail domains (by default, fail domains are racks) and can withstand a failure of no more than 2 fail domains. +* The `mirror-3-dc` configuration assumes deployment in 3 data centers, each of which has 3 fault tolerance domains and can withstand a failure of one data center and one more fault tolerance domain (rack). +* The `none` configuration doesn't provide fault tolerance but is convenient for functional testing. + +The `StateStorage` fault tolerance is defined by the `nto_select` parameter. The `StateStorage` configuration is fault-tolerant if any subset of `nto_select` servers that are part of `StateStorage` is fault-tolerant. +`StateStorage` remains available if most hosts are available for any subset of `nto_select` servers that are part of `StateStorage`. + +For example, if a cluster uses the `block-4-2` fault tolerance model for its storage, to make `StateStorage` fault-tolerant, set the `nto_select` parameter to 5 and place the hosts in 5 different availability domains. +If the cluster uses the `mirror-3-dc` fault tolerance model, to make `StateStorage` fault-tolerant, set the `nto_select` parameter to 9 and place the hosts in 3 data centers with 3 availability domains in each of them. + +{% note warning %} + +Make sure the NToSelect value is odd. For the proper operation of `StateStorage`, make sure that most of the NToSelect replicas are available for an arbitrary set of NToSelect hosts in `StateStorage`. + +{% endnote %} + +For example, you can use the following configuration for functional testing on a single server: + +```bash +domains_config: + domain: + - name: Root + storage_pool_types: + - kind: ssd + pool_config: + box_id: 1 + erasure_species: none + kind: ssd + pdisk_filter: + - property: + - type: SSD + vdisk_kind: Default + state_storage: + - ring: + node: + - 1 + nto_select: 1 + ssid: 1 +``` + +In this case, a domain is named `Root` and storage of the `SSD` type is created in it. This type of storage corresponds to disks with the `type` parameter set to `SSD`. +The `erasure_species: none` line in the parameter indicates that storage is created without fault tolerance. + +If a cluster is located in three availability zones with 3 servers available in each of them, it may have the following configuration: + +```bash +domains_config: + domain: + - name: global + storage_pool_types: + - kind: ssd + pool_config: + box_id: 1 + erasure_species: mirror-3-dc + kind: ssd + pdisk_filter: + - property: + - type: SSD + vdisk_kind: Default + state_storage: + - ring: + node: [1, 2, 3, 4, 5, 6, 7, 8, 9] + nto_select: 9 + ssid: 1 +``` + +In this case, a domain is named `global` and storage of the `SSD` type is also created in it. The `erasure_species: mirror-3-dc` line indicates that storage is created with the `mirror-3-dc` fault tolerance model. `StateStorage` will include 9 servers with the `nto_select` parameter set to 9. + +## Description of an actor system configuration + +Create an actor system configuration. Specify how processor cores will be distributed across the pools of cores available in the system. + +```bash +actor_system_config: + executor: + - name: System + spin_threshold: 0 + threads: 2 + type: BASIC + - name: User + spin_threshold: 0 + threads: 3 + type: BASIC + - name: Batch + spin_threshold: 0 + threads: 2 + type: BASIC + - name: IO + threads: 1 + time_per_mailbox_micro_secs: 100 + type: IO + - name: IC + spin_threshold: 10 + threads: 1 + time_per_mailbox_micro_secs: 100 + type: BASIC + scheduler: + progress_threshold: 10000 + resolution: 256 + spin_threshold: 0 +``` + +{% note warning %} + +Make sure the total number of cores assigned to the IC, Batch, System, and User pools does not exceed the number of available system cores. + +{% endnote %} + +## Description of a static cluster group + +Specify a static cluster group's configuration. A static group is necessary for the operation of the basic cluster tablets, including `Hive`, `SchemeShard`, and `BlobstorageContoller`. +As a rule, these tablets do not store a lot of data, so we don't recommend creating more than one static group. + +For a static group, specify the disks and nodes that the static group will be placed on. For example, a configuration for the `erasure: none` model can be as follows: + +```bash +blob_storage_config: + service_set: + groups: + - erasure_species: none + rings: + - fail_domains: + - vdisk_locations: + - node_id: 1 + path: /dev/disk/by-partlabel/ydb_ssd_02 + pdisk_category: SSD +.... +``` + +For a configuration located in 3 availability zones, specify 3 rings. For a configuration within a single availability zone, specify exactly one ring. + +## Sample cluster configurations + +The [repository](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/yaml_config_examples/) provides model examples of cluster configurations for self-deployment. Check them out before deploying a cluster. + diff --git a/ydb/docs/en/core/deploy/configuration/toc_i.yaml b/ydb/docs/en/core/deploy/configuration/toc_i.yaml new file mode 100644 index 0000000000..5daa365aec --- /dev/null +++ b/ydb/docs/en/core/deploy/configuration/toc_i.yaml @@ -0,0 +1,3 @@ +items: +- name: Creating a cluster configuration + href: config.md
\ No newline at end of file diff --git a/ydb/docs/en/core/deploy/configuration/toc_p.yaml b/ydb/docs/en/core/deploy/configuration/toc_p.yaml new file mode 100644 index 0000000000..82343f6140 --- /dev/null +++ b/ydb/docs/en/core/deploy/configuration/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml } diff --git a/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md b/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md new file mode 100644 index 0000000000..e32f44ebcd --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md @@ -0,0 +1,111 @@ +## Create TLS certificates using OpenSSL {# generate-tls} + +{% note info %} + +You can use existing TLS certificates. It's important that certificates support both server and client authentication (`extendedKeyUsage = serverAuth,clientAuth`). + +{% endnote %} + +## Create a CA key {# generate-ca} + +Create a directory named `secure` to store the CA key and a directory named `certs` for certificates and node keys: + +```bash +mkdir secure +mkdir certs +``` + +Create a `ca.cnf` configuration file with the following content: + +```text +[ ca ] +default_ca = CA_default + +[ CA_default ] +default_days = 365 +database = index.txt +serial = serial.txt +default_md = sha256 +copy_extensions = copy +unique_subject = no + +[ req ] +prompt=no +distinguished_name = distinguished_name +x509_extensions = extensions + +[ distinguished_name ] +organizationName = YDB +commonName = YDB CA + +[ extensions ] +keyUsage = critical,digitalSignature,nonRepudiation,keyEncipherment,keyCertSign +basicConstraints = critical,CA:true,pathlen:1 + +[ signing_policy ] +organizationName = supplied +commonName = optional + +[ signing_node_req ] +keyUsage = critical,digitalSignature,keyEncipherment +extendedKeyUsage = serverAuth,clientAuth + +# Used to sign client certificates. +[ signing_client_req ] +keyUsage = critical,digitalSignature,keyEncipherment +extendedKeyUsage = clientAuth +``` + +Create a CA key by running the command: + +```bash +openssl genrsa -out secure/ca.key 2048 +``` + +Save this key separately, you'll need it for issuing certificates. If it's lost, you'll have to reissue all certificates. + +Create a private Certificate Authority (CA) certificate by running the command: + +```bash +openssl req -new -x509 -config ca.cnf -key secure/ca.key -out ca.crt -days 365 -batch +``` + +### Creating keys and certificates for cluster nodes {# generate-node-keypair} + +Create a `node.conf` configuration file with the following content: + +```text +# OpenSSL node configuration file +[ req ] +prompt=no +distinguished_name = distinguished_name +req_extensions = extensions + +[ distinguished_name ] +organizationName = YDB + +[ extensions ] +subjectAltName = DNS:<node>.<domain> +``` + +Create a certificate key by running the command: + +```bash +openssl genrsa -out node.key 2048 +``` + +Create a Certificate Signing Request (CSR) by running the command: + +```bash +openssl req -new -sha256 -config node.cnf -key certs/node.key -out node.csr -batch +``` + +Create a node certificate with the following command: + +```bash +openssl ca -config ca.cnf -keyfile secure/ca.key -cert certs/ca.crt -policy signing_policy \ +-extensions signing_node_req -out certs/node.crt -outdir certs/ -in node.csr -batch +``` + +Create similar certificate-key pairs for each node. + diff --git a/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md b/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md new file mode 100644 index 0000000000..e907b0c060 --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md @@ -0,0 +1,45 @@ +## Prepare a configuration file for {{ ydb-short-name }} {# preprate-configs} + +Download a sample config for the appropriate failure model of your cluster: + +* [block-4-2](https://github.com/ydb-platform/ydb/blob/main/ydb/deploy/yaml_config_examples/block-4-2.yaml): For a single-datacenter cluster. +* [mirror-3dc](https://github.com/ydb-platform/ydb/blob/main/ydb/deploy/yaml_config_examples/mirror-3dc-9-nodes.yaml): For a cross-datacenter cluster consisting of 9 nodes. +* [mirror-3dc](https://github.com/ydb-platform/ydb/blob/main/ydb/deploy/yaml_config_examples/mirror-3dc-3-nodes.yaml): For a cross-datacenter cluster consisting of 3 nodes. + +1. In the host_configs section, specify all disks and their types on each cluster node. Possible disk types: + +```text +host_configs: +- drive: + - path: /dev/disk/by-partlabel/ydb_disk_01 + type: SSD + host_config_id: 1 +``` + +* ROT (rotational): HDD. +* SSD: SSD or NVMe. + +2. In the `hosts` section, specify the FQDN of each node, their configuration and location in a `data_center` or `rack`. + +```text +hosts: +- host: node1.ydb.tech + host_config_id: 1 + walle_location: + body: 1 + data_center: 'zone-a' + rack: '1' +- host: node2.ydb.tech + host_config_id: 1 + walle_location: + body: 2 + data_center: 'zone-b' + rack: '1' +- host: node3.ydb.tech + host_config_id: 1 + walle_location: + body: 3 + data_center: 'zone-c' + rack: '1' +``` + diff --git a/ydb/docs/en/core/deploy/manual/concepts.md b/ydb/docs/en/core/deploy/manual/concepts.md new file mode 100644 index 0000000000..e26c6e4df8 --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/concepts.md @@ -0,0 +1,9 @@ +## Manual deployment + +### Terms and definitions + +YDB consists of two components: + +* Static nodes that provide the data storage layer. +* Dynamic nodes that implement data access and processing. Each database requires one or more separate dynamic nodes. + diff --git a/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md b/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md new file mode 100644 index 0000000000..cc61e87793 --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md @@ -0,0 +1,295 @@ +## Deploy YDB On-Premises {# deploy-on-premise} + +This document describes how to deploy a multi-tenant YDB cluster on multiple servers. + +## Before you start {# before-start} + +### Prerequisites {# requirements} + +Make sure you have SSH access to all servers. This is necessary to install artifacts and run the YDB binary file. +Your network configuration must allow TCP connections on the following ports (by default): + +* 2135, 2136: GRPC for client-cluster interaction. +* 19001, 19002: Interconnect for intra-cluster node interaction. +* 8765, 8766: The HTTP interface for cluster monitoring. + +Check out the [Production checklist](../production_checklist.md) and the recommended cluster topology and select the servers and disks to be used for data storage: + +* Use the `block-4-2` fault tolerance model for cluster deployment in one availability zone (AZ). To survive the loss of 2 nodes, use at least 8 nodes. +* Use the `mirror-3-dc` fault tolerance model for cluster deployment in three availability zones (AZ). To survive the loss of 1 AZ and 1 node in another AZ, use at least 9 nodes. The number of nodes in each AZ should be the same. + +Run each static node on a separate server. + +## Create a system user and a group to run {{ ydb-short-name }} under {# create-user} + +On each server where YDB will be running, execute: + +```bash +sudo groupadd ydb +sudo useradd ydb -g ydb +``` + +To make sure the {{ ydb-short-name }} server has access to block store disks to run, add the user to start the process under to the disk group. + +```bash +sudo usermod -aG disk ydb +``` + +## Prepare and format disks on each server {# prepare-disks} + +{% note warning %} + +We don't recommend using disks that are used by other processes (including the OS) for data storage. + +{% endnote %} + +1. Create a partition on the selected disk + +{% note alert %} + +Be careful! The following operation will delete all partitions on the specified disks! +Make sure that you specified the disks that have no other data! + +{% endnote %} + +```bash +sudo parted /dev/nvme0n1 mklabel gpt -s +sudo parted -a optimal /dev/nvme0n1 mkpart primary 0% 100% +sudo parted /dev/nvme0n1 name 1 ydb_disk_01 +sudo partx --u /dev/nvme0n1 +``` + +As a result, a disk labeled as `/dev/disk/by-partlabel/ydb_disk_01` will appear in the system. + +If you plan to use more than one disk on each server, specify a label that is unique for each of them instead of `ydb_disk_01`. You'll need to use these disks later in the configuration files. + +Download an archive with the `ydbd` executable file and the libraries necessary for working with YDB: + +```bash +curl https://binaries.ydb.tech/ydbd-main-linux-amd64.tar.gz | tar -xz +``` + +Create directories to run: + +```bash +mkdir -p /opt/ydb +chown ydb:ydb /opt/ydb +mkdir /opt/ydb/bin +mkdir /opt/ydb/cfg +``` + +2. Copy the binary file, libraries, and configuration file to the appropriate directories: + +```bash +sudo cp -i ydbd-main-linux-amd64/bin/ydbd /opt/ydb/bin/ +sudo cp -i ydbd-main-linux-amd64/lib/libaio.so /opt/ydb/lib/ +sudo cp -i ydbd-main-linux-amd64/lib/libiconv.so /opt/ydb/lib/ +sudo cp -i ydbd-main-linux-amd64/lib/libidn.so /opt/ydb/lib/ +``` + +3. Format the disk with the built-in command + +```bash +sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib /opt/ydb/bin/ydbd admin bs disk obliterate /dev/disk/by-partlabel/ydb_disk_01 +``` + +Perform this operation for each disk that will be used for data storage. + +Prepare the configuration files: + +{% list tabs %} + +- Unprotected mode + + In this mode, traffic between cluster nodes and between the client and cluster uses an unencrypted connection. Use this mode for testing purposes. + + {% include [prepare-configs.md](_includes/prepare-configs.md) %} + + Save the {{ ydb-short-name }} configuration file as `/opt/ydb/cfg/config.yaml` + +- Protected mode + + In this mode, traffic between cluster nodes and between the client and cluster is encrypted using the TLS protocol. + + {% include [generate-ssl.md](_includes/generate-ssl.md) %} + + Create directories for certificates on each node + + ```bash + mkdir /opt/ydb/certs + chmod 0750 /opt/ydb/certs + ``` + + Copy the node certificates and keys + + ```bash + sudo -u ydb cp certs/ca.crt certs/node.crt certs/node.key /opt/ydb/certs/ + ``` + + {% include [prepare-configs.md](_includes/prepare-configs.md) %} + + 3. In the `interconnect_config` and `grpc_config` sections, specify the path to the certificate, key, and CA certificates: + + ```text + interconnect_config: + start_tcp: true + encryption_mode: OPTIONAL + path_to_certificate_file: "/opt/ydb/certs/node.crt" + path_to_private_key_file: "/opt/ydb/certs/node.key" + path_to_ca_file: "/opt/ydb/certs/ca.crt" + + grpc_config: + cert: "/opt/ydb/certs/node.crt" + key: "/opt/ydb/certs/node.key" + ca: "/opt/ydb/certs/ca.crt" + ``` + + Save the configuration file as `/opt/ydb/cfg/config.yaml` + +{% endlist %} + +## Start static nodes {# start-storage} + +{% list tabs %} + +- Manual + + 1. Run {{ ydb-short-name }} storage on each node: + + ```bash + sudo su - ydb + cd /opt/ydb + export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib + /opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp --yaml-config /opt/ydb/cfg/config.yaml \ + --grpc-port 2135 --ic-port 19001 --mon-port 8765 --node static + ``` + + TBD: how and where to write logs? Log rotation + +- Using systemd + + 1. On each node, create a configuration file named `/etc/systemd/system/ydbd-storage.service` with the following content: + + ```text + [Unit] + Description=YDB storage node + After=network-online.target rc-local.service + Wants=network-online.target + StartLimitInterval=10 + StartLimitBurst=15 + + [Service] + Restart=always + RestartSec=1 + User=ydb + PermissionsStartOnly=true + StandardOutput=syslog + StandardError=syslog + SyslogIdentifier=ydbd + SyslogFacility=daemon + SyslogLevel=err + Environment=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib + ExecStart=/opt/ydb/bin/ydbd server --log-level 3 --syslog --tcp --yaml-config /opt/ydb/cfg/config.yaml --grpc-port 2135 --ic-port 19001 --mon-port 8765 --node static + LimitNOFILE=65536 + LimitCORE=0 + LimitMEMLOCK=3221225472 + + [Install] + WantedBy=multi-user.target + ``` + 2. Run {{ ydb-short-name }} storage on each node: + + ```bash + sudo systemctl start ydbd-storage + ``` + +{% endlist %} + +## Initialize a cluster {# initialize-cluster} + +On one of the cluster nodes, run the command: + +```bash +LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd admin blobstorage config init --yaml-file /opt/ydb/cfg/config.yaml ; echo $? +``` + +The command execution code should be null. + +## Creating the first database {# create-fist-db} + +To work with tables, you need to create at least one database and run a process serving this database (a dynamic node). + +```bash +LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd admin database /Root/testdb create ssd:1 +``` + +## Start the DB dynamic node {# start-dynnode} + +{% list tabs %} + +- Manual + + 1. Start the {{ ydb-short-name }} dynamic node for the /Root/testdb database: + + ```bash + sudo su - ydb + cd /opt/ydb + export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib + /opt/ydbd/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml \ + --tenant /Root/testdb --node-broker --node-broker --node-broker + ``` + + Run additional dynamic nodes on other servers to ensure database availability. + +- Using systemd + + 1. Create a configuration file named `/etc/systemd/system/ydbd-testdb.service` with the following content: + + ```text + [Unit] + Description=YDB testdb dynamic node + After=network-online.target rc-local.service + Wants=network-online.target + StartLimitInterval=10 + StartLimitBurst=15 + + [Service] + Restart=always + RestartSec=1 + User=ydb + PermissionsStartOnly=true + StandardOutput=syslog + StandardError=syslog + SyslogIdentifier=ydbd + SyslogFacility=daemon + SyslogLevel=err + Environment=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib + ExecStart=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml --tenant /Root/testdb --node-broker --node-broker --node-broker + LimitNOFILE=65536 + LimitCORE=0 + LimitMEMLOCK=32212254720 + + [Install] + WantedBy=multi-user.target + ``` + 2. Start the {{ ydb-short-name }} dynamic node for the /Root/testdb database: + + ```bash + sudo systemctl start ydbd-testdb + ``` + 3. Run additional dynamic nodes on other servers to ensure database availability. + +{% endlist %} + +## Test the created database {# try-first-db} + +1. Install the YDB CLI as described in [Installing the YDB CLI](../../reference/ydb-cli/install.md) +2. Create a `test_table`: + +```bash +ydb -e grpc://<node1.domain>:2136 -d /Root/testdb scripting yql \ +--script 'CREATE TABLE `testdir/test_table` (id Uint64, title Utf8, PRIMARY KEY (id));' +``` + +Where node.domain is the FQDN of the server where the dynamic nodes serving the `/Root/testdb` database are running. + diff --git a/ydb/docs/en/core/deploy/manual/toc_i.yaml b/ydb/docs/en/core/deploy/manual/toc_i.yaml new file mode 100644 index 0000000000..868e6d2912 --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/toc_i.yaml @@ -0,0 +1,5 @@ +items: +- name: Overview + href: concepts.md +- name: Local deployment + href: deploy-ydb-on-premises.md
\ No newline at end of file diff --git a/ydb/docs/en/core/deploy/manual/toc_p.yaml b/ydb/docs/en/core/deploy/manual/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/deploy/manual/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/deploy/orchestrated/_includes/ydb-kubernetes-operator.md b/ydb/docs/en/core/deploy/orchestrated/_includes/ydb-kubernetes-operator.md index bff2205f99..25d0f75698 100644 --- a/ydb/docs/en/core/deploy/orchestrated/_includes/ydb-kubernetes-operator.md +++ b/ydb/docs/en/core/deploy/orchestrated/_includes/ydb-kubernetes-operator.md @@ -1,8 +1,8 @@ # Installing a cluster -## Install the {{ ydb-name }} controller in the {#install-ydb-controller} cluster +## Install the {{ ydb-short-name }} controller in the {#install-ydb-controller} cluster -Install {{ ydb-name }} in the standard configuration: +Install {{ ydb-short-name }} in the standard configuration: {% list tabs %} @@ -29,7 +29,7 @@ Install {{ ydb-name }} in the standard configuration: {% endlist %} -## Create a {{ ydb-name }} cluster {#create-cluster} +## Create a {{ ydb-short-name }} cluster {#create-cluster} Apply the manifest for creating a {{ ydb-short-name }} cluster: @@ -139,7 +139,7 @@ Apply the manifest for creating a database: ## Test the controller {#test-ydb} -Test how {{ ydb-name }} works: +Test how {{ ydb-short-name }} works: {% list tabs %} @@ -195,7 +195,7 @@ Test how {{ ydb-name }} works: └─────────┘ ``` - Learn more about {{ ydb-short-name }} CLI commands in the [documentation](https://cloud.yandex.ru/docs/ydb/oss/public/reference/ydb-cli/commands/). + Learn more about {{ ydb-short-name }} CLI commands in the [documentation](https://cloud.yandex.com/en/docs/ydb/oss/public/reference/ydb-cli/commands/). {% endlist %} @@ -220,7 +220,7 @@ If you no longer need the created resources, delete them: kubectl delete pvc -l app.kubernetes.io/name=ydb ``` - 1. To remove the {{ ydb-name }} controller from the {{ k8s }} cluster, delete the release created by Helm: + 1. To remove the {{ ydb-short-name }} controller from the {{ k8s }} cluster, delete the release created by Helm: ```bash helm delete ydb-operator diff --git a/ydb/docs/en/core/deploy/orchestrated/aws_eks.md b/ydb/docs/en/core/deploy/orchestrated/aws_eks.md index a4b7b7140a..0ac18840af 100644 --- a/ydb/docs/en/core/deploy/orchestrated/aws_eks.md +++ b/ydb/docs/en/core/deploy/orchestrated/aws_eks.md @@ -1,6 +1,6 @@ -# Deploying {{ ydb-name }} in AWS EKS +# Deploying {{ ydb-short-name }} in AWS EKS -To create a [{{ ydb-name }}](https://cloud.yandex.ru/docs/ydb/) cluster using [{{ k8s }}](https://kubernetes.io/), follow these steps. +To create a [{{ ydb-short-name }}](https://cloud.yandex.com/en/docs/ydb/) cluster using [{{ k8s }}](https://kubernetes.io/), follow these steps. ## Before you start {#before-begin} diff --git a/ydb/docs/en/core/deploy/orchestrated/concepts.md b/ydb/docs/en/core/deploy/orchestrated/concepts.md index 904e1dcf45..8f119df782 100644 --- a/ydb/docs/en/core/deploy/orchestrated/concepts.md +++ b/ydb/docs/en/core/deploy/orchestrated/concepts.md @@ -1,6 +1,6 @@ # Overview -Deploying {{ ydb-name }} in {{ k8s }} is a simple and easy way to install {{ ydb-name }}. With {{ k8s }}, you can use a universal approach to managing your application in any cloud provider. This guide provides instructions on how to deploy {{ ydb-short-name }} in [{{ managed-k8s-full-name}}](yc_managed_kubernetes.md) and [AWS EKS](aws_eks.md). +Deploying {{ ydb-short-name }} in {{ k8s }} is a simple and easy way to install {{ ydb-short-name }}. With {{ k8s }}, you can use a universal approach to managing your application in any cloud provider. This guide provides instructions on how to deploy {{ ydb-short-name }} in [{{ managed-k8s-full-name}}](yc_managed_kubernetes.md) and [AWS EKS](aws_eks.md). {{ ydb-short-name }} is delivered as a Helm chart that is a package with templates of {{ k8s }} structures. You can deploy charts in the following environment: @@ -11,7 +11,7 @@ Deploying {{ ydb-name }} in {{ k8s }} is a simple and easy way to install {{ ydb For more information about Helm, see the [documentation](https://helm.sh/ru/docs/). -A Helm chart installs a controller in the {{ k8s }} cluster. It implements the logic required for deploying {{ ydb-name }} components. The controller is based on the [Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) pattern. +A Helm chart installs a controller in the {{ k8s }} cluster. It implements the logic required for deploying {{ ydb-short-name }} components. The controller is based on the [Operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/) pattern. {{ydb-short-name}} consists of two components: diff --git a/ydb/docs/en/core/deploy/orchestrated/operate.md b/ydb/docs/en/core/deploy/orchestrated/operate.md index 87290bcfb7..bcc6094a62 100644 --- a/ydb/docs/en/core/deploy/orchestrated/operate.md +++ b/ydb/docs/en/core/deploy/orchestrated/operate.md @@ -1,4 +1,4 @@ -# Using {{ ydb-name }} in {{ k8s }} +# Using {{ ydb-short-name }} in {{ k8s }} ## Monitoring diff --git a/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md b/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md index 78a6b12dda..9a265f2f77 100644 --- a/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md +++ b/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md @@ -1,12 +1,12 @@ -# Deploying {{ ydb-name }} in {{ managed-k8s-name }} +# Deploying {{ ydb-short-name }} in {{ managed-k8s-name }} -To create a [{{ ydb-name }}](https://cloud.yandex.ru/docs/ydb/) cluster using [{{ k8s }}](https://kubernetes.io/), follow these steps. +To create a [{{ ydb-short-name }}](https://cloud.yandex.com/en/docs/ydb/) cluster using [{{ k8s }}](https://kubernetes.io/), follow these steps. ## Before you start {#before-begin} 1. Create a {{ k8s }} cluster. - You can use an existing {{ k8s }} cluster or [create](https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create) a new one. + You can use an existing {{ k8s }} cluster or [create](https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create) a new one. {% note warning %} @@ -16,7 +16,7 @@ To create a [{{ ydb-name }}](https://cloud.yandex.ru/docs/ydb/) cluster using [{ 1. Install the {{ k8s }} CLI [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl). -1. [Define](https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials) a kubectl configuration. +1. [Define](https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials) a kubectl configuration. 1. Install the {{ k8s }} [Helm 3](https://helm.sh/docs/intro/install/) package manager. diff --git a/ydb/docs/en/core/deploy/production_checklist.md b/ydb/docs/en/core/deploy/production_checklist.md new file mode 100644 index 0000000000..a8ccfef6eb --- /dev/null +++ b/ydb/docs/en/core/deploy/production_checklist.md @@ -0,0 +1,26 @@ +## Checklist for deploying your product environment + +This section provides recommendations for deploying {{ ydb-full-name }} in product environments. + +## Topology + +When planning a deployment, it's important to choose the right cluster topology based on the required fault tolerance: + +* Mirror-3dc-3-nodes: This mode requires at least 3 servers with 3 disks in each. To ensure maximum fault tolerance, each server must be located in an independent datacenter. In this mode, a cluster keeps running if no more than 1 server fails. +* Mirror-3dc: This mode requires at least 9 servers. To ensure maximum fault tolerance, servers must be located in three independent data centers and different server racks in each of them. In this mode, a cluster keeps running if 1 datacenter fails completely and 1 server in another datacenter is out of service. +* Block-4-2: This mode requires at least 8 servers in one datacenter. To ensure maximum fault tolerance, servers must be located in 8 independent racks. + +TBD. mirror-3dc vs block-4-2: Whether we want the following in the doc: + +- Much more network traffic +- Higher storage overhead + +## Hardware configuration + +For correct operation, we recommend using the x86_64 CPU architecture with support for the following instructions: + +- SSE +- ??? + +## Software configuration + diff --git a/ydb/docs/en/core/downloads/downloads.md b/ydb/docs/en/core/downloads/downloads.md new file mode 100644 index 0000000000..e064f52c08 --- /dev/null +++ b/ydb/docs/en/core/downloads/downloads.md @@ -0,0 +1,64 @@ +# Downloads {# downloads} + +## {{ ydb-short-name }} CLI + +Read the documentation on how to work with the [{{ ydb-short-name }} CLI](../reference/ydb-cli/index.md) + +{% list tabs %} + +- Linux + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.1.6.0 | 2020-06-01 | [Binary file](https://storage.yandexcloud.net/yandexcloud-ydb/release/1.6.0/linux/amd64/ydb) | + +- macOS (Intel) + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.1.6.0 | 2020-06-01 | [Binary file](https://storage.yandexcloud.net/yandexcloud-ydb/release/1.6.0/darwin/amd64/ydb) | + +- macOS (M1 arm) + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.1.6.0 | 2020-06-01 | [Binary file](https://storage.yandexcloud.net/yandexcloud-ydb/release/1.6.0/darwin/arm64/ydb) | + +- Windows + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.1.6.0 | 2020-06-01 | [Binary file](https://storage.yandexcloud.net/yandexcloud-ydb/release/1.6.0/windows/amd64/ydb.exe) | + +{% endlist %} + +## {{ ydb-short-name }} server + +{% list tabs %} + +- Linux + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.21.4.62 | 2022-02-12 | [Binary file](https://binaries.ydb.tech/ydbd-main-linux-amd64.tar.gzx) | + +- Mac OS (Intel) + + Under development. + +- Mac OS (M1 arm) + + Under development. + +- Windows + + Under development. + +- Docker + + | Version | Date of issue | Download | + | :--- | :--- | :--- | + | v.24.4.62 | 2022-02-12 | `cr.yandex/crpl7ipeu79oseqhcgn2/ydb-oss:main` | + +{% endlist %} + diff --git a/ydb/docs/en/core/downloads/toc_i.yaml b/ydb/docs/en/core/downloads/toc_i.yaml new file mode 100644 index 0000000000..6813f7c6c3 --- /dev/null +++ b/ydb/docs/en/core/downloads/toc_i.yaml @@ -0,0 +1,3 @@ +items: +- name: Downloads + href: downloads.md
\ No newline at end of file diff --git a/ydb/docs/en/core/downloads/toc_p.yaml b/ydb/docs/en/core/downloads/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/downloads/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/faq/_includes/common.md b/ydb/docs/en/core/faq/_includes/common.md index 0289917842..d63dfbc107 100644 --- a/ydb/docs/en/core/faq/_includes/common.md +++ b/ydb/docs/en/core/faq/_includes/common.md @@ -2,20 +2,19 @@ title: "Yandex Database. FAQ" description: "What is Yandex Database? For what tasks is it worth using Yandex Database, and for which virtual machines with databases? What part of the management and maintenance of databases does Yandex Database take on? Answers to these and other questions in this article." --- - -# General questions about {{ ydb-name }} +# General questions about {{ ydb-short-name }} #### What is {{ ydb-short-name }}? {#what-is-ydb} -{{ ydb-short-name }} is a distributed fault-tolerant SQL DBMS. YDB provides high availability and scalability while simultaneously ensuring strict consistency and ACID transaction support. Queries are made using an SQL dialect (YQL). +{{ ydb-short-name }} is a distributed fault-tolerant SQL DBMS. {{ ydb-short-name }} provides high availability and scalability while simultaneously ensuring strict consistency and ACID transaction support. Queries are made using an SQL dialect (YQL). {{ ydb-short-name }} is a fully managed database. DB instances are created through the {{ ydb-short-name }} database management service. #### What features does {{ ydb-short-name }} provide? {#ydb-features} -{{ ydb-short-name }} provides high availability and data security through synchronous replication in three availability zones. {{ ydb-short-name }} also ensures even load distribution across available hardware resources. This means you don't need to order resources, Yandex Database automatically provisions and releases resources based on the user load. +{{ ydb-short-name }} provides high availability and data security through synchronous replication in three availability zones. {{ ydb-short-name }} also ensures even load distribution across available hardware resources. This means you don't need to order resources, {{ ydb-short-name }} automatically provisions and releases resources based on the user load. -#### What consistency model does YDB use? {#ydb-consistency} +#### What consistency model does {{ ydb-short-name }} use? {#ydb-consistency} To read data, {{ ydb-short-name }} uses a model of strict data consistency. diff --git a/ydb/docs/en/core/faq/_includes/index.md b/ydb/docs/en/core/faq/_includes/index.md index d58d5c375a..3f923685a6 100644 --- a/ydb/docs/en/core/faq/_includes/index.md +++ b/ydb/docs/en/core/faq/_includes/index.md @@ -1,4 +1,4 @@ -# Questions and answers about {{ ydb-name }} +# Questions and answers about {{ ydb-short-name }} * [General](../common.md) {% if oss %} diff --git a/ydb/docs/en/core/faq/_includes/serverless.md b/ydb/docs/en/core/faq/_includes/serverless.md index e0316a18a0..b46c12093b 100644 --- a/ydb/docs/en/core/faq/_includes/serverless.md +++ b/ydb/docs/en/core/faq/_includes/serverless.md @@ -2,7 +2,7 @@ #### How do secondary indexes affect the request cost? -Operations with indexes are estimated according to the same rules as operations with tables. They are reflected in the request statistics and included in the total indicators that are used to calculate the cost in Request Units (RU). For more information, see the [pricing policy for the serverless {{ ydb-short-name }} API](https://cloud.yandex.ru/docs/ydb/pricing/request_units_yql). +Operations with indexes are estimated according to the same rules as operations with tables. They are reflected in the request statistics and included in the total indicators that are used to calculate the cost in Request Units (RU). For more information, see the [pricing policy for the serverless {{ ydb-short-name }} API](https://cloud.yandex.com/en/docs/ydb/pricing/request_units_yql). When reading data from a table using an index, the request statistics will show the number of rows read from the index and their volume. diff --git a/ydb/docs/en/core/getting_started/_assets/embedded_query.png b/ydb/docs/en/core/getting_started/_assets/embedded_query.png Binary files differnew file mode 100644 index 0000000000..7b79eacbe1 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_assets/embedded_query.png diff --git a/ydb/docs/en/core/getting_started/_includes/auth.md b/ydb/docs/en/core/getting_started/_includes/auth.md new file mode 100644 index 0000000000..44c79df376 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/auth.md @@ -0,0 +1,10 @@ +# Authentication - Getting started + +A local YDB database with self-deployment does not require authentication, that is, runs in [anonymous mode](../../concepts/connect.md#auth-modes). + +Full information about all available authentication methods is given in the [DB connection](../../concepts/connect.md) article in the "Concepts" section. + +## Learn more about YDB {#next} + +Proceed to the [YDB CLI - Getting started](../cli.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/auth_sdk_overlay.md b/ydb/docs/en/core/getting_started/_includes/auth_sdk_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/auth_sdk_overlay.md diff --git a/ydb/docs/en/core/getting_started/_includes/cli.md b/ydb/docs/en/core/getting_started/_includes/cli.md new file mode 100644 index 0000000000..6799ebf215 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/cli.md @@ -0,0 +1,108 @@ +# {{ ydb-short-name }} CLI - Getting started + +## Prerequisites {#prerequisites} + +To execute commands via the CLI, you need the database connection parameters you've got when [creating](../create_db.md) it: + +- [Endpoint](../../concepts/connect.md#endpoint) +- [Database name](../../concepts/connect.md#database). + +You may also need a token or login/password if the database requires [authentication](../auth.md). To execute the below scenario, you need to select an option for saving them in an environment variable. + +## Installing the CLI {#install} + +Install the {{ ydb-short-name }} CLI as described in [Installing the {{ ydb-short-name }} CLI](../../reference/ydb-cli/install.md). + +To check that the YDB CLI has been installed, run it with `--help`: + +```bash +{{ ydb-cli }} --help +``` + +The response includes a welcome message, a brief description of the syntax, and a list of available commands: + +```text +YDB client + +Usage: ydb [options...] <subcommand> + +Subcommands: +ydb +├─ config Manage YDB CLI configuration +│ └─ profile Manage configuration profiles +│ ├─ activate Activate specified configuration profile (aliases: set) +... +``` + +All the features of the {{ ydb-short-name }} CLI built-in help are described in [Built-in help](../../reference/ydb-cli/commands/service.md#help) of the {{ ydb-short-name }} CLI reference. + +## Check the connection {#ping} {#scheme-ls} + +To test connection, you can use the command for [listing objects](../../reference/ydb-cli/commands/scheme-ls.md) in the database, `scheme ls`: + +```bash +{{ ydb-cli }} -e <endpoint> -d <database> scheme ls +``` + +If the command is successful, a list of objects in the database is shown in response. If you haven't created anything in the database yet, the output will only contain the `.sys` and `.sys_health` system directories with [diagnostic representations of YDB](../../troubleshooting/system_views.md). + +{% include [cli/ls_examples.md](cli/ls_examples.md) %} + +## Creating a connection profile {#profile} + +To avoid specifying connection parameters every time you call the YDB CLI, use the [profile](../../reference/ydb-cli/profile/index.md). Creating the profile described below will also let you copy subsequent commands through the clipboard without editing them, regardless of what database you're using to complete the "Getting started" step. + +[Create the profile](../../reference/ydb-cli/profile/create.md) `db1` using the following command: + +```bash +{{ ydb-cli }} config profile create db1 +``` + +You will be interactively prompted for connection parameters to be linked with the profile. Use for them the values verified in the [previous step](#ping). + +Check that the profile is OK with the `scheme ls` command: + +```bash +{{ ydb-cli }} --profile db1 scheme ls +``` + +## Executing an YQL script {#yql} + +The {{ ydb-short-name }} CLI `scripting yql` command lets you execute any command (both DDL and DML) in [YQL](../../yql/reference/index.md), an SQL dialect supported by {{ ydb-short-name }}: + +```bash +{{ ydb-cli }} --profile <profile_name> yql -s <yql_request> +``` + +For example: + +- Creating a table + + ```bash + {{ ydb-cli }} --profile db1 yql -s "create table t1(id uint64, primary key(id))" + ``` + +- Adding a record + + ```bash + {{ ydb-cli }} --profile db1 yql -s "insert into t1(id) values (1)" + ``` + +- Data selections + + ```bash + {{ ydb-cli }} --profile db1 yql -s "select * from t1" + ``` + +If you get the error `Profile db1 does not exist`, it means that you failed to create it in the [previous step](#profile). + +## Specialized CLI сommands {#ydb-api} + +Executing commands via `ydb yql` is a nice and easy way to get started. However, the YQL interface supports a part of the function set provided by the YDB API, and has to sacrifice efficiency for universality. + +The YDB CLI supports individual commands with complete sets of options for any existing YDB API. For a full list of commands, see the [YDB CLI reference](../../reference/ydb-cli/index.md). + +## Learn more about YDB {#next} + +Proceed to the [YQL - Getting started](../yql.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/cli/ls_example_generic.md b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_generic.md new file mode 100644 index 0000000000..6417d3b6d8 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_generic.md @@ -0,0 +1,19 @@ +For example, if: + +* Endpoint: grpc://ydb.example.com:2136 +* Database name: /john/db1 +* The database doesn't require authentication, or the proper environment variable has been set, as described [here](../../auth.md) article. +* The database has just been created and contains no objects + +Then the command and its execution result will look like this: + +```bash +{{ ydb-cli }} -e grpc://ydb.example.com:2136 -d /john/db1 scheme ls +``` + +The command execution result on the created empty database: + +```text +.sys_health .sys +``` + diff --git a/ydb/docs/en/core/getting_started/_includes/cli/ls_example_local.md b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_local.md new file mode 100644 index 0000000000..578a6fc688 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_local.md @@ -0,0 +1,8 @@ +### Connecting to a local database + +If you deployed a local YDB using a scenario for self-deployment [in Docker](../../ydb_docker.md) with the suggested configuration, you can check a YDB connection using the command: + +```bash +{{ ydb-cli }} -e grpc://localhost:2136 -d /local scheme ls +``` + diff --git a/ydb/docs/en/core/getting_started/_includes/cli/ls_example_overlay.md b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/cli/ls_example_overlay.md diff --git a/ydb/docs/en/core/getting_started/_includes/cli/ls_examples.md b/ydb/docs/en/core/getting_started/_includes/cli/ls_examples.md new file mode 100644 index 0000000000..6ce3f0ea16 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/cli/ls_examples.md @@ -0,0 +1,6 @@ +{% include [ls_example_generic.md](ls_example_generic.md) %} + +{% include [ls_example_overlay.md](ls_example_overlay.md) %} + +{% include [ls_example_local.md](ls_example_local.md) %} + diff --git a/ydb/docs/en/core/getting_started/_includes/create_db.md b/ydb/docs/en/core/getting_started/_includes/create_db.md new file mode 100644 index 0000000000..0edfdee09c --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/create_db.md @@ -0,0 +1,18 @@ +# Creating a database - Getting started + +Detailed information about databases can be found in the article [Terms and definitions - Database](../../concepts/databases.md#database) in the "Concepts" section. + +{% include [create_db_overlay.md](create_db_overlay.md) %} + +## Self-deployment {#self-hosted} + +There are three methods you can use to deploy {{ ydb-short-name }}: + +* [Using Docker](../ydb_docker.md). +* [In Kubernetes](../../deploy/orchestrated/concepts.md). +* [Locally, from a downloaded executable file](../ydb_local.md). + +## Learn more about YDB {#next} + +After creating your database, proceed to the [Authentication - Getting started](../auth.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/create_db_overlay.md b/ydb/docs/en/core/getting_started/_includes/create_db_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/create_db_overlay.md diff --git a/ydb/docs/en/core/getting_started/_includes/index.md b/ydb/docs/en/core/getting_started/_includes/index.md new file mode 100644 index 0000000000..e80bce9695 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index.md @@ -0,0 +1,14 @@ +{% include [intro.md](index/intro.md) %} + +{% include [create_db.md](index/create_db.md) %} + +{% include [network_access.md](index/network.md) %} + +{% include [auth.md](index/auth.md) %} + +{% include [cli.md](index/cli.md) %} + +{% include [yql.md](index/yql.md) %} + +{% include [sdk.md](index/sdk.md) %} + diff --git a/ydb/docs/en/core/getting_started/_includes/index/auth.md b/ydb/docs/en/core/getting_started/_includes/index/auth.md new file mode 100644 index 0000000000..d704b4463c --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/auth.md @@ -0,0 +1,2 @@ +* [Authentication ](../../auth.md): Learn what you need to access the database. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/cli.md b/ydb/docs/en/core/getting_started/_includes/index/cli.md new file mode 100644 index 0000000000..fd6d6bc205 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/cli.md @@ -0,0 +1,2 @@ +* [Using the command line interface (CLI)](../../cli.md): Install the {{ ydb-short-name }} CLI, connect to the DB, and view the usage examples. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/create_db.md b/ydb/docs/en/core/getting_started/_includes/index/create_db.md new file mode 100644 index 0000000000..d65b99d89c --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/create_db.md @@ -0,0 +1,2 @@ +- [Creating a database ](../../create_db.md): Create your own open-source {{ ydb-short-name }} database + diff --git a/ydb/docs/en/core/getting_started/_includes/index/intro.md b/ydb/docs/en/core/getting_started/_includes/index/intro.md new file mode 100644 index 0000000000..5526958d49 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/intro.md @@ -0,0 +1,6 @@ +# Getting started with {{ ydb-short-name }} + +The articles in this section will tell you how to quickly get started with {{ ydb-short-name }}. + +Each article provides one or more simple scenarios of doing an action, and additionally includes a link to other documents describing all the available {{ ydb-short-name }} features. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/network.md b/ydb/docs/en/core/getting_started/_includes/index/network.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/network.md diff --git a/ydb/docs/en/core/getting_started/_includes/index/sdk.md b/ydb/docs/en/core/getting_started/_includes/index/sdk.md new file mode 100644 index 0000000000..8f551767ed --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/sdk.md @@ -0,0 +1,2 @@ +* [{{ ydb-short-name }} SDK](../../sdk.md): Get familiar with {{ ydb-short-name }} SDKs for different programming languages and run simple DB queries from sample apps. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/toc_network.yaml b/ydb/docs/en/core/getting_started/_includes/index/toc_network.yaml new file mode 100644 index 0000000000..1dce705aca --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/toc_network.yaml @@ -0,0 +1,4 @@ +# To be supplied in overlay if needed +items: +- name: Dummy + hidden: true diff --git a/ydb/docs/en/core/getting_started/_includes/index/yql.md b/ydb/docs/en/core/getting_started/_includes/index/yql.md new file mode 100644 index 0000000000..2c2095a05e --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/index/yql.md @@ -0,0 +1,2 @@ +* [YQL](../../yql.md): Get familiar with YQL, the main query language for {{ ydb-short-name }}, an SQL dialect. + diff --git a/ydb/docs/en/core/getting_started/_includes/sdk.md b/ydb/docs/en/core/getting_started/_includes/sdk.md new file mode 100644 index 0000000000..7cd0b38c37 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/sdk.md @@ -0,0 +1,14 @@ +# {{ ydb-short-name }} SDK - Getting started + +{{ ydb-short-name }} SDKs are sets of software components to support {{ ydb-short-name }} databases for apps written in different programming languages. + +Follow these steps to launch a simple app using {{ ydb-short-name }}: + +1. [Install the {{ ydb-short-name }} SDK](../../reference/ydb-sdk/install.md) for the appropriate programming language. + +2. Download and run a sample test app using the SDK for the installed programming language and study its code: + - [Python](../../reference/ydb-sdk/example/python/index.md) + - [Go](../../reference/ydb-sdk/example/go/index.md) + +For more information about the {{ ydb-short-name }} SDK, see [Working with the {{ ydb-short-name }} SDK](../../reference/ydb-sdk/index.md). + diff --git a/ydb/docs/en/core/getting_started/_includes/useful_links.md b/ydb/docs/en/core/getting_started/_includes/useful_links.md index 50751e8f7a..6a151bcf0d 100644 --- a/ydb/docs/en/core/getting_started/_includes/useful_links.md +++ b/ydb/docs/en/core/getting_started/_includes/useful_links.md @@ -2,6 +2,7 @@ {% if oss %} -* [YDB OpenSource website](https://ydb.tech){% endif %} +* [{{ ydb-short-name }} OpenSource website](https://ydb.tech){% endif %} * [Yandex.Cloud management console](https://console.cloud.yandex.com) * [GitHub account ydb-platform](https://github.com/ydb-platform) + diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/01_intro.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/01_intro.md index 3ae2ef2f0f..5e3ab4fb0e 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/01_intro.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/01_intro.md @@ -1,15 +1,20 @@ -# Using the YDB Docker container +# Using the {{ ydb-short-name }} Docker container -For debugging or testing, you can run a {{ ydb-full-name }} instance in a Docker container. Docker containers use the current {{ ydb-short-name }} build version, but the build revision may differ. +For debugging or testing, you can run the YDB [Docker](https://docs.docker.com/get-docker/) container. -In local launch mode, you can only interact with a database using the {{ ydb-short-name }} API. The API is available at the `grpcs://localhost:2135` endpoint. The database name is `/local`. To work with your database, you can use the {{ ydb-short-name }} command-line client ([YDB CLI](../../../reference/ydb-cli/index.md)) built into the Docker image. +As a result of completing the instructions below, you'll get a local YDB database that can be accessed using the following: -{{ ydb-full-name }} in a Docker container accepts incoming TLS-encrypted connections. Certificates are generated automatically. To use certificates, you need to mount the `/ydb_cert` directory on the host system of a Docker container. +{% list tabs %} -To save the database state locally, mount the `/ydb_data` directory on the host system of a Docker container. +- gRPC + - [Endpoint ](../../../concepts/connect.md#endpoint): `grpc://localhost:2136` + - [Database location](../../../concepts/connect.md#database): `/local` + - [Authentication](../../../concepts/connect.md#auth-modes): Anonymous (without authentication) -{% note warning %} +- gRPCs/TLS + - [Endpoint ](../../../concepts/connect.md#endpoint): `grpcs://localhost:2135` + - [Database location](../../../concepts/connect.md#database): `/local` + - [Authentication](../../../concepts/connect.md#auth-modes): Anonymous (without authentication) -When a local database is running, some tasks may require a significant portion of the host system resources. +{% endlist %} -{% endnote %} diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/02_install.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/02_install.md index 41bff9637c..8069f28edd 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/02_install.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/02_install.md @@ -1,20 +1,20 @@ -## Pull a {{ ydb-short-name }} Docker image {#install} +## Installation {#install} -Pull the current public version of the Docker image: +Upload the current public version of the Docker image: ```bash -docker pull cr.yandex/yc/yandex-docker-local-ydb:latest +docker pull {{ ydb_local_docker_image }}:{{ ydb_local_docker_image_tag }} ``` Make sure the Docker image has been pulled: ```bash -docker image list +docker image list | grep {{ ydb_local_docker_image }} ``` Output: ```bash -REPOSITORY TAG IMAGE ID CREATED SIZE -cr.yandex/yc/yandex-docker-local-ydb latest b73c5c1441af 2 months ago 793MB -```
\ No newline at end of file +{{ ydb_local_docker_image }} {{ ydb_local_docker_image_tag }} b73c5c1441af 2 months ago 793MB +``` + diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/03_start.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/03_start.md index 33eca0cbc7..49bb8670bc 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/03_start.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/03_start.md @@ -1,36 +1,49 @@ -## Run a YDB Docker container {#start} +## Starting {#start} -Run a {{ ydb-short-name }} Docker container and mount the necessary directories: +The YDB Docker container uses the host system resources (CPU, RAM) within the limits allocated by the Docker settings. + +The YDB Docker container stores data in its file system whose sections are reflected in the host system directory. The start container command given below will create files in the current directory, so first create a working directory and then start the container from it: ```bash -docker run -d \ - --rm \ - --name ydb-local \ - -h localhost \ - -p 2135:2135 -p 8765:8765 \ +docker run -d --rm --name ydb-local -h localhost \ + -p 2135:2135 -p 8765:8765 -p 2136:2136 \ -v $(pwd)/ydb_certs:/ydb_certs -v $(pwd)/ydb_data:/ydb_data \ - -e YDB_LOCAL_SURVIVE_RESTART=true \ - cr.yandex/yc/yandex-docker-local-ydb:latest + -e YDB_DEFAULT_LOG_LEVEL=NOTICE \ + -e GRPC_TLS_PORT=2135 -e GRPC_PORT=2136 -e MON_PORT=8765 \ + {{ ydb_local_docker_image}}:{{ ydb_local_docker_image_tag }} ``` -Where: +If started successfully, you'll see the ID of the created container. + +### Startup parameters {#start-pars} + +`-d` means running the Docker container in the background. + +`--rm` means removing the container after its operation is completed. + +`--name` is the container name. Specify `ydb-local` to be able to perform the below instructions for stopping the container by copying text through the clipboard. + +`-h` is the container host name. Be sure to pass the `localhost` value. Otherwise, the container will be started with a random hostname. -* `-d` means running the Docker container in the background. -* `--rm` means removing the container after its operation is completed. -* `--name` is the container name. -* `-h` is the container host name. -* `-p` means publishing container ports on the host system. -* `-v` means mounting the host system directories into a container. -* `-e` means setting environment variables. +`-v` means mounting the host system directories into a container like `<host system directory>:<mount directory in the container>`. The YDB container uses the following mount directories: + +- `/ydb_data` for storing data. If this directory is not mounted, the container will be started without saving data to the host system disk. +- `/ydb_certs` for storing certificates to establish a TLS connection. The started container will write to this directory the certificates to be used for a TLS client connection. If this directory is not mounted, you won't be able to connect via TLS due to having no certificate information. + +`-e`: means setting environment variables in `<name>=<value>` format. The YDB container uses the following environment variables: + +- `YDB_DEFAULT_LOG_LEVEL`: The logging level. Acceptable values: `CRIT`, `ERROR`, `WARN`, `NOTICE`, and `INFO`. Defaults to `NOTICE`. +- `GRPC_PORT`: The port for unencrypted connections. Defaults to 2136. +- `GRPC_TLS_PORT`: The port for TLS connections. Defaults to 2135. +- `MON_PORT`: The port for the built-in web UI with [monitoring and introspection tools](../../../maintenance/embedded_monitoring/ydb_monitoring.md). Defaults to 8765. +- `YDB_PDISK_SIZE`: The size of the disk for storing data in `<NUM>GB` format (for example, `YDB_PDISK_SIZE=128GB`). Acceptable values: `64GB` and higher. Defaults to 64GB. +- `YDB_USE_IN_MEMORY_PDISKS`: Using disks in memory. Acceptable values are `true` and `false`, defaults to `false`. If enabled, the container's file system is not used for working with data, all data is only stored in the memory of a process and is lost when it's stopped. Currently, you can start the container on Apple M1 in this mode only. + +`-p` means publishing container ports on the host system. All applicable ports must be explicitly listed even if default values are used. {% note info %} -It may take several minutes to initialize the Docker container, depending on the capacity of the host system. The database will not be available until the container is initialized. +It may take several minutes to initialize the Docker container, depending on the allocated resources. The database will not be available until the container is initialized. {% endnote %} -{{ ydb-short-name }} Docker containers support some additional options that can be set through environment variables: - -* `YDB_LOCAL_SURVIVE_RESTART=true` enables you to restart a container without losing data. -* `YDB_USE_IN_MEMORY_PDISKS=true` enables you to store all your data in memory. If this option is enabled, restarting the container with a local {{ ydb-short-name }} instance will result in complete data loss. -* `YDB_DEFAULT_LOG_LEVEL=<level>` enables you to set the logging level.. Valid levels: `CRIT`, `ERROR`, `WARN`, `NOTICE`, `INFO`. diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/04_request.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/04_request.md index aae9233774..4c2b3919a1 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/04_request.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/04_request.md @@ -1,37 +1,24 @@ -## Make a query to the database {#request} +## Making queries {#request} -Make a query to the {{ ydb-short-name }} database in a Docker container: +Install the YDB CLI and execute queries as described in [YDB CLI - Getting started](../../cli.md), using the endpoint and database location specified at the beginning of this article. For example: ```bash -ydb \ - -e grpcs://localhost:2135 \ - --ca-file $(pwd)/ydb_certs/ca.pem \ - -d /local table query execute -q 'select 1;' +ydb -e grpc://localhost:2136 -d /local scheme ls ``` -Where: +To ensure a connection using TLS is successful, add the name of the file with the certificate to the connection parameters. The query in the example below should be executed from the same working directory that you used to start the container: -* `-e`: Database endpoint. -* `--ca-file`: Path to the TLS certificate. -* `-d`: DB name and query parameters. - -Output: - -```text -┌─────────┐ -| column0 | -├─────────┤ -| 1 | -└─────────┘ +```bash +ydb -e grpcs://localhost:2135 --ca-file ydb_certs/ca.pem -d /local scheme ls ``` A precompiled version of the [YDB CLI](../../../reference/ydb-cli/index.md) is also available within the image: ```bash -sudo docker exec <CONTAINER-ID> /ydb -e localhost:2135 -d /local table query execute -q 'select 1;' -┌─────────┐ -| column0 | -├─────────┤ -| 1 | -└─────────┘ -```
\ No newline at end of file +docker exec <container_id> /ydb -e localhost:2136 -d /local scheme ls +``` + +, where + +`<container_id>`: The container ID output when you [start](#start) it. + diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/05_stop.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/05_stop.md index 301b0fdf92..7a102068b4 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/05_stop.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/05_stop.md @@ -1,4 +1,4 @@ -## Stop a Docker container {#stop} +## Stopping {#stop} When everything is done, stop a Docker container: diff --git a/ydb/docs/en/core/getting_started/_includes/ydb_docker/06_license.md b/ydb/docs/en/core/getting_started/_includes/ydb_docker/06_license.md index b5d7598b93..4b6be5ea22 100644 --- a/ydb/docs/en/core/getting_started/_includes/ydb_docker/06_license.md +++ b/ydb/docs/en/core/getting_started/_includes/ydb_docker/06_license.md @@ -5,11 +5,12 @@ The Docker container root includes a file with the license agreement (`LICENSE`) Read the license agreement: ```bash -sudo docker run --rm -it --entrypoint cat cr.yandex/yc/yandex-docker-local-ydb LICENSE +docker run --rm -it --entrypoint cat {{ ydb_local_docker_image }} LICENSE ``` View all the included components and their licenses: ```bash -sudo docker run --rm -it --entrypoint cat cr.yandex/yc/yandex-docker-local-ydb THIRD_PARTY_LICENSES +docker run --rm -it --entrypoint cat {{ ydb_local_docker_image }} THIRD_PARTY_LICENSES ``` + diff --git a/ydb/docs/en/core/getting_started/_includes/yql.md b/ydb/docs/en/core/getting_started/_includes/yql.md new file mode 100644 index 0000000000..b8d7f11361 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql.md @@ -0,0 +1,216 @@ +# YQL - Getting started + +## Introduction {#intro} + +YQL is a {{ ydb-short-name }} query language, a dialect of SQL. Specifics of its syntax let you use it when executing queries on clusters. + +For more information about the YQL syntax, see the [YQL reference](../../yql/reference/index.md). + +The examples below show how to get started with YQL and assume sequential execution of the steps described: queries in the ["Working with data"](#dml) section access data in tables created in the ["Working with a data schema"](#ddl) section. Follow the steps one by one so that the examples copied through the clipboard are executed successfully. + +The {{ ydb-short-name }} YQL basic interface accepts a script that may consist of multiple commands and not a single command as input. + +## YQL query execution tools {#tools} + +In {{ ydb-short-name }}, you can make YQL queries to a database using: + +{% include [yql/ui_prompt.md](yql/ui_prompt.md) %} + +- [{{ ydb-short-name }} CLI](#cli) + +- [{{ ydb-short-name }} SDK](../sdk.md) + +{% include [yql/ui_execute.md](yql/ui_execute.md) %} + +### {{ ydb-short-name }} CLI {#cli} + +To execute scripts using the {{ ydb-short-name }} CLI, first do the following: + +- [Install the CLI](../cli.md#install). +- Define and check [DB connection parameters](../cli#scheme-ls). +- [Create a `db1` profile](../cli.md#profile) configured to connect to your database. + +Save the text of the scripts below to a file. Name it `script.yql` to be able to run the statements given in the examples by simply copying them through the clipboard. Next, run `{{ ydb-cli }} yql` indicating the use of the `db1` profile and reading the script from the `script.yql` file: + +```bash +{{ ydb-cli }} --profile db1 yql -f script.yql +``` + +## Working with a data schema {#ddl} + +### Creating a table {#create-table} + +A table with the specified columns is created [using the YQL `CREATE TABLE`](../../yql/reference/syntax/create_table.md) statement. Make sure the primary key is defined in the table. Column data types are described in [YQL data types](../../yql/reference/types/index.md). + +Currently, {{ ydb-short-name }} doesn't support the `NOT NULL` constraint, all columns allow null values, including the primary key columns. In addition, {{ ydb-short-name }} doesn't support the `FOREIGN KEY` constraint. + +Create series directory tables named `series`, `seasons`, and `episodes` by running the following script: + +```sql +CREATE TABLE series ( + series_id Uint64, + title Utf8, + series_info Utf8, + release_date Uint64, + PRIMARY KEY (series_id) +); + +CREATE TABLE seasons ( + series_id Uint64, + season_id Uint64, + title Utf8, + first_aired Uint64, + last_aired Uint64, + PRIMARY KEY (series_id, season_id) +); + +CREATE TABLE episodes ( + series_id Uint64, + season_id Uint64, + episode_id Uint64, + title Utf8, + air_date Uint64, + PRIMARY KEY (series_id, season_id, episode_id) +); +``` + +For a description of all features of working with tables, see the following sections of the YQL documentation: + +- [`CREATE TABLE`](../../yql/reference/syntax/create_table.md): Creating a table and defining initial parameters. +- [`ALTER TABLE`](../../yql/reference/syntax/alter_table.md): Changing the composition of table columns and parameters. +- [`DROP TABLE`](../../yql/reference/syntax/drop_table.md): Deleting a table. + +To execute the script via the {{ ydb-short-name }} CLI, follow the instructions given in the ["Executing YQL scripts in the {{ ydb-short-name }} CLI"](#cli) section in this article. + +### Getting a list of existing DB tables {#scheme-ls} + +Check that the tables are actually created in the database. + +{% include [yql/ui_scheme_ls.md](yql/ui_scheme_ls.md) %} + +To get a list of existing DB tables via the {{ ydb-short-name }} CLI, be sure to meet the prerequisites of the ["Executing YQL scripts in the {{ ydb-short-name }} CLI"](#cli) section of this article and run [the `scheme ls` statement](../cli.md#ping): + +```bash +{{ ydb-cli }} --profile db1 scheme ls +``` + +## Operations with data {#dml} + +Commands for executing YQL queries and scripts in the YDB CLI and web interface are run in **Autocommit** mode, meaning that a transaction is committed automatically after a transaction is completed. + +### UPSERT : Adding data {#upsert} + +The most efficient way to add data to {{ ydb-short-name }} is through the [`UPSERT`](../../yql/reference/syntax/upsert_into.md) statement. It inserts new data by primary keys regardless of whether data by these keys previously existed in the table. As a result, unlike regular `INSERT` and `UPDATE` statements, its execution does not require a pre-fetch of data on the server to verify if a key is unique. When working with {{ ydb-short-name }}, always consider `UPSERT` as the main way to add data and only use other statements when absolutely necessary. + +All statements that write data to {{ ydb-short-name }} support working with both samples and multiple entries passed directly in a query. + +Let's add data to the previously created tables: + +```yql +UPSERT INTO series (series_id, title, release_date, series_info) +VALUES + ( + 1, + "IT Crowd", + CAST(Date("2006-02-03") AS Uint64), + "The IT Crowd is a British sitcom produced by Channel 4, written by Graham Linehan, produced by Ash Atalla and starring Chris O'Dowd, Richard Ayoade, Katherine Parkinson, and Matt Berry."), + ( + 2, + "Silicon Valley", + CAST(Date("2014-04-06") AS Uint64), + "Silicon Valley is an American comedy television series created by Mike Judge, John Altschuler and Dave Krinsky. The series focuses on five young men who founded a startup company in Silicon Valley." + ) + ; + +UPSERT INTO seasons (series_id, season_id, title, first_aired, last_aired) +VALUES + (1, 1, "Season 1", CAST(Date("2006-02-03") AS Uint64), CAST(Date("2006-03-03") AS Uint64)), + (1, 2, "Season 2", CAST(Date("2007-08-24") AS Uint64), CAST(Date("2007-09-28") AS Uint64)), + (2, 1, "Season 1", CAST(Date("2014-04-06") AS Uint64), CAST(Date("2014-06-01") AS Uint64)), + (2, 2, "Season 2", CAST(Date("2015-04-12") AS Uint64), CAST(Date("2015-06-14") AS Uint64)) +; + +UPSERT INTO episodes (series_id, season_id, episode_id, title, air_date) +VALUES + (1, 1, 1, "Yesterday's Jam", CAST(Date("2006-02-03") AS Uint64)), + (1, 1, 2, "Calamity Jen", CAST(Date("2006-02-03") AS Uint64)), + (2, 1, 1, "Minimum Viable Product", CAST(Date("2014-04-06") AS Uint64)), + (2, 1, 2, "The Cap Table", CAST(Date("2014-04-13") AS Uint64)) +; +``` + +To execute the script via the {{ ydb-short-name }} CLI, follow the instructions given in the ["Executing YQL scripts in the {{ ydb-short-name }} CLI"](#cli) section in this article. + +To learn more about the commands for writing data, see the YQL reference: + +- [`INSERT`](../../yql/reference/syntax/insert_into.md): Adds entries. +- [`REPLACE`](../../yql/reference/syntax/replace_into.md): Adds/updates entries. +- [`UPDATE`](../../yql/reference/syntax/update.md): Updates the specified fields. +- [`UPSERT`](../../yql/reference/syntax/upsert_into.md): Adds entries/updates the specified fields. + +### SELECT : Data selection {#select} + +Make a select of the data added in the previous step: + +```sql +SELECT + series_id, + title AS series_title, + DateTime::ToDate(DateTime::FromDays(release_date)) AS release_date +FROM series; +``` + +or + +```sql +SELECT * FROM episodes; +``` + +If there are several `SELECT` statements in the YQL script, its execution will return several samples, each of which can be accessed separately. Run the above `SELECT` statements as a single script. + +To execute the script via the {{ ydb-short-name }} CLI, follow the instructions given in the ["Executing YQL scripts in the {{ ydb-short-name }} CLI"](#cli) section in this article. + +To learn more about the commands for selecting data, see the YQL reference: + +- [`SELECT`](../../yql/reference/syntax/select.md): Selects data. +- [`SELECT ... JOIN`](../../yql/reference/syntax/join.md): Joins tables when making a select. +- [`SELECT ... GROUP BY`](../../yql/reference/syntax/group_by.md): Groups data when making a select. + +### Parameterized queries {#param} + +Transactional applications working with a database are characterized by the execution of multiple similar queries that only differ in parameters. Like most databases, {{ ydb-short-name }} will work more efficiently if you define updateable parameters and their types and then initiate the execution of a query by passing the parameter values separately from its text. + +To define parameters in the text of a YQL query, use the [DECLARE](../../yql/reference/syntax/declare.md) statement. + +Methods for executing parameterized queries in the {{ ydb-short-name }} SDK are described in the [Test case](../../reference/ydb-sdk/example/index.md) section under "Parameterized queries" for the appropriate programming language. + +When debugging a parameterized query in the {{ ydb-short-name }} SDK, you can test it by calling the {{ ydb-short-name }} CLI, copying the full text of the query without any edits, and setting parameter values. + +Save the script for executing the parameterized query in a text file named `script.yql`: + +```sql +DECLARE $seriesId AS Uint64; +DECLARE $seasonId AS Uint64; + +SELECT sa.title AS season_title, sr.title AS series_title +FROM seasons AS sa +INNER JOIN series AS sr ON sa.series_id = sr.series_id +WHERE sa.series_id = $seriesId AND sa.season_id = $seasonId; +``` + +To make a parameterized select query, be sure to meet the prerequisites of the ["Executing YQL scripts in the {{ ydb-short-name }} CLI"](#cli) section of this article and run: + +```bash +{{ ydb-cli }} --profile db1 yql -f script.yql -p '$seriesId=1' -p '$seasonId=1' +``` + +For a full description of the ways to pass parameters, see [the {{ ydb-short-name }} CLI reference](../../reference/ydb-cli/index.md). + +## YQL tutorial {#tutorial} + +You can learn more about YQL use cases by completing tasks from the [YQL tutorial](../../yql/tutorial/index.md). + +## Learn more about YDB {#next} + +Proceed to the [YDB SDK - Getting started](../sdk.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_dml_autocommit.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_dml_autocommit.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_dml_autocommit.md diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_execute.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_execute.md new file mode 100644 index 0000000000..8f7e3ea6bd --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_execute.md @@ -0,0 +1,10 @@ +### Built-in YDB web interface {#embedded} + +To execute YQL queries and scripts on self-deployed YDB databases, you can use the [built-in YDB web interface](../../../maintenance/embedded_monitoring/overview.md). For a local deployment [using Docker](../../ydb_docker.md) with the default parameters, it is available at [http://localhost:8765](http://localhost:8765). + +Select **Databases** in the menu on the left, click on the database in the list, and switch to the **Query** tab: + + + +To execute a YQL script, click **Run Script**. + diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_prompt.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_prompt.md new file mode 100644 index 0000000000..151432baf3 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_prompt.md @@ -0,0 +1,2 @@ +- [Built-in web interface](#embedded) + diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_scheme_ls.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_scheme_ls.md new file mode 100644 index 0000000000..e5fd20f672 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_embedded_scheme_ls.md @@ -0,0 +1,2 @@ +In the built-in YDB web interface, the list of tables is shown on the left of the database page as a hierarchy. On the **Info** tab, you can see detailed information about the object selected in the hierarchy. + diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_execute.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_execute.md new file mode 100644 index 0000000000..16e7f9d06c --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_execute.md @@ -0,0 +1,2 @@ + +{% include [ui_embedded_execute.md](ui_embedded_execute.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_prompt.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_prompt.md new file mode 100644 index 0000000000..9550e1b18f --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_prompt.md @@ -0,0 +1,2 @@ + +{% include [ui_embedded_prompt.md](ui_embedded_prompt.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/getting_started/_includes/yql/ui_scheme_ls.md b/ydb/docs/en/core/getting_started/_includes/yql/ui_scheme_ls.md new file mode 100644 index 0000000000..b1d3988cc8 --- /dev/null +++ b/ydb/docs/en/core/getting_started/_includes/yql/ui_scheme_ls.md @@ -0,0 +1 @@ +{% include [ui_embedded_scheme_ls.md](ui_embedded_scheme_ls.md) %} diff --git a/ydb/docs/en/core/getting_started/auth.md b/ydb/docs/en/core/getting_started/auth.md new file mode 100644 index 0000000000..e7f2395443 --- /dev/null +++ b/ydb/docs/en/core/getting_started/auth.md @@ -0,0 +1,2 @@ + +{% include [auth.md](_includes/auth.md) %} diff --git a/ydb/docs/en/core/getting_started/cli.md b/ydb/docs/en/core/getting_started/cli.md new file mode 100644 index 0000000000..d737e7a176 --- /dev/null +++ b/ydb/docs/en/core/getting_started/cli.md @@ -0,0 +1 @@ +{% include [cli.md](_includes/cli.md) %} diff --git a/ydb/docs/en/core/getting_started/create_db.md b/ydb/docs/en/core/getting_started/create_db.md new file mode 100644 index 0000000000..51d7c75381 --- /dev/null +++ b/ydb/docs/en/core/getting_started/create_db.md @@ -0,0 +1,2 @@ + +{% include [create_db.md](_includes/create_db.md) %} diff --git a/ydb/docs/en/core/getting_started/index.md b/ydb/docs/en/core/getting_started/index.md index 28128865c0..12643df073 100644 --- a/ydb/docs/en/core/getting_started/index.md +++ b/ydb/docs/en/core/getting_started/index.md @@ -1,6 +1 @@ ---- -title: Getting started with Yandex Database (YDB). Overview ---- - -# Getting started with {{ ydb-name }} - +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/getting_started/sdk.md b/ydb/docs/en/core/getting_started/sdk.md new file mode 100644 index 0000000000..18bd227b1a --- /dev/null +++ b/ydb/docs/en/core/getting_started/sdk.md @@ -0,0 +1 @@ +{% include [sdk.md](_includes/sdk.md) %} diff --git a/ydb/docs/en/core/getting_started/toc_i.yaml b/ydb/docs/en/core/getting_started/toc_i.yaml index b5021438a4..7dbd11c22b 100644 --- a/ydb/docs/en/core/getting_started/toc_i.yaml +++ b/ydb/docs/en/core/getting_started/toc_i.yaml @@ -1,10 +1,31 @@ # common internal + external "Getting started" items items: +- name: Creating a database + href: create_db.md +- include: { mode: link, path: _includes/index/toc_network.yaml } +- name: Authentication + href: auth.md +- name: YDB command line interface (CLI) + href: cli.md +- name: YQL query language + href: yql.md +- name: YDB SDK + href: sdk.md - name: Self-deployment items: - name: Docker href: ydb_docker.md + - name: Starting a local cluster + href: ydb_local.md + # when: audience == "tech" + - name: Creating a cluster configuration + include: { mode: link, path: ../deploy/configuration/toc_i.yaml } + # when: audience == "tech" - name: Kubernetes - include: { mode: link, path: ../deploy/orchestrated/toc_i.yaml } + include: { mode: link, path: ../deploy/orchestrated/toc_p.yaml } + - name: Manual + include: { mode: link, path: ../deploy/manual/toc_p.yaml } + - name: Production checklist + href: ../deploy/production_checklist.md - name: Useful links href: useful_links.md
\ No newline at end of file diff --git a/ydb/docs/en/core/getting_started/toc_p.yaml b/ydb/docs/en/core/getting_started/toc_p.yaml index 94ce110868..7064fde1cb 100644 --- a/ydb/docs/en/core/getting_started/toc_p.yaml +++ b/ydb/docs/en/core/getting_started/toc_p.yaml @@ -1,4 +1,6 @@ items: - name: Overview href: index.md -- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file +- include: { mode: link, path: toc_i.yaml } +- name: How to update the documentation + include: { mode: link, path: ../how_to_edit_docs/toc_p.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/getting_started/useful_links.md b/ydb/docs/en/core/getting_started/useful_links.md index a0f91ce498..ad6685f23b 100644 --- a/ydb/docs/en/core/getting_started/useful_links.md +++ b/ydb/docs/en/core/getting_started/useful_links.md @@ -1,14 +1,2 @@ -{% include [useful_links.md](_includes/useful_links.md) %} - -{% if audience == "internal" or audience == "tech" %} -## Links to internal Yandex resources - -* [YandexDB web console](https://yc.yandex-team.ru/) -* [YDB club in Atushka](https://clubs.at.yandex-team.ru/ydb/) -* [ydb@](https://ml.yandex-team.ru/lists/ydb/): public mailing list for discussing YandexDB -* [ydb-announces@](https://ml.yandex-team.ru/lists/ydb-announces/): newsletter for announcements -* [st/YDBREQUESTS](https://st.yandex-team.ru/YDBREQUESTS/): Tracker queue for user requests -* [st/YDBOPS](https://st.yandex-team.ru/YDBOPS/): Tracker queue for service operation issues and resolving incidents -* [st/KIKIMR](https://st.yandex-team.ru/KIKIMR/): Tracker queue for YandexDB/KiKiMR development tasks -{% endif %}
\ No newline at end of file +{% include [useful_links.md](_includes/useful_links.md) %} diff --git a/ydb/docs/en/core/getting_started/ydb_local.md b/ydb/docs/en/core/getting_started/ydb_local.md new file mode 100644 index 0000000000..9d49cd69b4 --- /dev/null +++ b/ydb/docs/en/core/getting_started/ydb_local.md @@ -0,0 +1,88 @@ +# Starting a local cluster {{ ydb-short-name }} + +This section describes how to deploy a local {{ ydb-short-name }} cluster using a configuration in YAML format. + +{% note warning %} + +When a local database is running, some tasks may require a significant portion of the host system resources. + +{% endnote %} + +## Download {{ ydb-short-name }} {#download-ydb} + +Download and unpack an archive with the `ydbd` executable file and the libraries necessary for working with {{ ydb-short-name }}. Next, go to the directory with the artifacts: + +```bash +curl https://binaries.ydb.tech/ydbd-master-linux-amd64.tar.gz | tar -xz +cd ydbd-master-linux-amd64/ +export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:`pwd`/lib" +``` + +## Prepare a local cluster configuration {#prepare-configuration} + +Prepare a local cluster configuration that you want to deploy. To run a cluster with in-memory data storage, just copy the configuration. To deploy a cluster with data storage in a file, additionally create a 64GB file and specify the path to it in the configuration. + +{% list tabs %} + +- In memory + + ```bash + wget https://raw.githubusercontent.com/ydb-platform/ydb/main/ydb/deploy/yaml_config_examples/single-node-in-memory.yaml -O config.yaml + ``` + +- In a file + + ```bash + wget https://raw.githubusercontent.com/ydb-platform/ydb/main/ydb/deploy/yaml_config_examples/single-node-with-file.yaml -O config.yaml + dd if=/dev/zero of=/tmp/ydb-local-pdisk.data bs=1 count=0 seek=68719476736 + sed -i "s/\/tmp\/pdisk\.data/\/tmp\/ydb-local-pdisk\.data/g" config.yaml + ``` + +{% endlist %} + +## Start a static cluster node {#start-static-node} + +Start a static cluster node by running the command: + +```bash +./bin/ydbd server --yaml-config ./config.yaml --node 1 --grpc-port 2135 --ic-port 19001 --mon-port 8765 +``` + +Initialize storage with the command: + +```bash +./bin/ydbd admin blobstorage config init --yaml-file ./config.yaml +``` + +## Creating the first database {#create-db} + +To work with tables, you need to create at least one database and run a process serving this database. To do this, you need to run a set of commands: + +Create a database: + +```bash +./bin/ydbd admin database /<domain-name>/<db-name> create <storage-pool-kind>:<storage-unit-count> +``` + +For example, + +```bash +./bin/ydbd admin database /Root/test create ssd:1 +``` + +Start the process serving the database: + +```bash +./bin/ydbd server --yaml-config ./config.yaml --tenant /<domain-name>/<db-name> --node-broker <address>:<port> --grpc-port 31001 --ic-port 31003 --mon-port 31002 +``` + +For example, to run the process for the database created above, execute the following command. + +```bash +./bin/ydbd server --yaml-config ./config.yaml --tenant /Root/test --node-broker localhost:2135 --grpc-port 31001 --ic-port 31003 --mon-port 31002 +``` + +## Working with the database via the Web UI {#web-ui} + +To view the database structure and execute a YQL query, use the web interface embedded in the `ydbd` process. To do this, open your browser and go to `http://localhost:8765`. For more details about the embedded web interface, see [Embedded UI](../maintenance/embedded_monitoring/ydb_monitoring.md). + diff --git a/ydb/docs/en/core/getting_started/yql.md b/ydb/docs/en/core/getting_started/yql.md new file mode 100644 index 0000000000..1bba08b228 --- /dev/null +++ b/ydb/docs/en/core/getting_started/yql.md @@ -0,0 +1 @@ +{% include [yql.md](_includes/yql.md) %} diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/build/intro.md b/ydb/docs/en/core/how_to_edit_docs/_includes/build/intro.md index 1ca8ee09b0..a90632e91b 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/build/intro.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/build/intro.md @@ -1,2 +1,2 @@ -# Сборка документации +# Building documentation diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md b/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md index 94a9ecb9dc..8c8e8f00fe 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md @@ -1,34 +1,36 @@ -## Локальная сборка OpenSource-инструментами +## Local build with OpenSource tools -Сборка документации осуществляется утилитой [YFM-Docs](https://github.com/yandex-cloud/yfm-docs). +The documentation is built using the [YFM-Docs](https://github.com/yandex-cloud/yfm-docs) utility. -Порядок установки YFM-Docs описан на [вводной странице документации по этой утилите](https://ydocs.tech/ru/tools/docs/). +Learn how to install YFM-Docs on the [introductory page of the utility documentation](https://ydocs.tech/tools/docs/). -Для сборки OpenSource документации YDB нужно выполнить команду: +To build the {{ ydb-short-name }} OpenSource documentation, run the command: -``` bash +```bash $ yfm -i <source_dir> -o <output_dir> --allowHTML ``` -Где: -- `source_dir` - директория, куда склонировано содержимое [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) -- `output_dir` - директория, куда будет выполнена генерация HTML-файлов +Where: -Сборка занимает несколько секунд, и не должна выводить сообщений об ошибках в лог (stdout). +- `source_dir` is the directory where the contents of [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) is cloned. +- `output_dir` is the output directory for HTML files. -В качестве `source_dir` можно задавать `.` (точку), если команда yfm вызывается непосредственно из каталога `source_dir`, например: +Building the documentation takes a few seconds and there should be no errors logged to the stdout log. -``` bash +You can specify `.` (a dot) as `source_dir` if the yfm command is called directly from `source_dir`. For example: + +```bash docs $ yfm -i . -o ~/docs/ydboss --allowHTML ``` -Для просмотра собранной локально документации можно открыть каталог из браузера, или воспользоваться простым web-сервером, встроенным в Python: +To view the documentation built locally, you can open the directory from your browser or use a simple web server built into Python: -``` bash +```bash python3 -m http.server 8888 -d ~/docs/ydboss ``` -При запущенном таким образом сервере собранная локально документация доступна по ссылкам: -- [http://localhost:8888/ru](http://localhost:8888/ru) - на русском языке -- [http://localhost:8888/en](http://localhost:8888/en) - на английском языке +With the server run in this way, the locally built documentation is available at the links: + +- [http://localhost:8888/ru](http://localhost:8888/ru) (in Russian) +- [http://localhost:8888/en](http://localhost:8888/en) (in English) diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md b/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md index c873ac70b6..f671bc04a3 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md @@ -1,21 +1,22 @@ -## Технология сборки +## Build technology -### Наложение слоев +### Overlaying layers -Сборка любого варианта документации осуществляется по одном шаблону: +Any version of the documentation is built using a common template: -1. В базовой директории сборки размещаются два каталога: `core` и `overlay`. -2. В каталоге `core` размещается базовая документация, которая может быть подключена как внешняя зависимость (например, git submodule). -2. В каталоге `overlay` размещается специфический для данной сборки контент. -2. В рабочую директорию копируется содержимое core. -3. В рабочую директорию копируется содержимое overlay, перезаписывая все файлы, относительный путь которых внутри overlay совпадет с каким-либо файлом из core. +1. The basic build directory contains two directories: `core` and `overlay`. +1. The `core` directory stores the basic documentation that can be added as an external dependency (such as git submodule). +1. The `overlay` directory stores content that is specific for this build. +1. The contents of the core directory are copied to the working directory. +1. The contents of the overlay directory are copied to the working directory, replacing any file whose relative path in the overlay directory matches that of any file from the core directory. -В результате в рабочей директории сборки получается контент из core, в котором заменены файлы, присутствующие в overlay. Если в `overlay` файла нет -- в сборку попадет файл из `core`. +As a result, the working directory of the build contains the content from the core directory, in which the files present in the overlay directory are replaced. If a certain file is missing in `overlay`, a file from `core` gets to the build. -Такое наложение позволяет переопределить контент, сохранив размещение статей, а значит -- все ссылки на них их других статей, неважно идут они из статьей `core`, или из статей `overlay`. В результате, адаптация контента может быть локализована в рамках той статьи, которая требует адаптации, без необходимости делать технические корректировки в других статьях. +This overlay allows redefining the content, preserving the placement of articles, that is, keeping any links to them from other articles, whether they come from articles placed in `core` or `overlay`. As a result, content adaptation can be carried out within the article that requires adaptation, without the need to make technical adjustments in other articles. {% note warning %} -Так как наложение директорий `core` и `overlay` производится только при сборке, ссылки из статей в `overlay` на статьи в `core` не будут кликабельны в IDE, которое ничего не знает о том что какие-то директории будут при сборке оказываться наложенными друг поверх друга. +Since the overlay of the `core` and `overlay` directories is only done when building the documentation, links from the articles in `overlay` to the articles in `core` won't be clickable in an integrated development environment (IDE) that knows nothing about the fact that some directories will be overlaid on top of each other during the build. + +{% endnote %} -{% endnote %}
\ No newline at end of file diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/content.md b/ydb/docs/en/core/how_to_edit_docs/_includes/content.md index 2dcd0cb467..94917ffd05 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/content.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/content.md @@ -1,311 +1,324 @@ -# Руководство по созданию контента +# Content creation guide -## Введение {#intro} +## Introduction {#intro} -Исходный код документации создается в формате [Markdown](https://ru.wikipedia.org/wiki/Markdown), с расширениями [YFM (Yandex Flavoured Markdown)](https://ydocs.tech/ru/). Эффективным способом быстро изучить эти форматы является знакомство с исходным кодом имеющейся документации YDB. +The source code of the documentation is created in [Markdown](https://en.wikipedia.org/wiki/Markdown) format with [YFM (Yandex Flavoured Markdown)](https://ydocs.tech/) extensions. An effective way to quickly learn these formats is to study the source code of the available {{ ydb-short-name }} documentation. -Кроме описанных здесь формальных правил, всегда действует неявное дополнение: если на какую-то тему правило формально не описано, необходимо посмотреть как уже сделано ранее в других местах документации, и сделать по аналогии. +In addition to the formal rules described here, always keep in mind the following: if a rule is not formally described for some topic, you should look at how this has been done elsewhere in the documentation and do it by analogy. -Также нужно помнить о том, что из любых правил бывают исключения, и их невозможно описать все. +Also remember that there are exceptions to any rules and it is impossible to describe them all. -## Ядро и кастомизация документации +## Documentation core and customization -OpenSource документация по YDB поддерживает создание на её базе кастомизированных документаций для enterprise-окружений где эксплуатируется YDB, или для облачных провайдеров, предоставляющих сервис YDB в своих экосистемах. +The OpenSource {{ ydb-short-name }} documentation lets you create customized documentation based on it for enterprise environments using {{ ydb-short-name }} or for cloud providers that provide the {{ ydb-short-name }} service in their ecosystems. -При добавлении контента вам придется в первую очередь выбрать куда его добавлять: в OpenSource ядро, или в какую-либо кастомизацию. Общая рекомендация - добавлять контент в OpenSource блок таким образом, чтобы он мог читаться в разных контекстах без необходимости его специфичной адаптации. Только когда это сделать не удается, или размещение в OpenSource ядре какого-то контента явно запрещено (например, ссылок на ресурсы интранет в enterprise-контексте), можно переходить к добавлению контента в исходный код кастомизации документации. +When adding content, you should first choose where to add it: to the OpenSource core or to some customization. The general recommendation is to add content to the OpenSource block in such a way that it can be read in different contexts without the need for its specific adaptation. Only when this can't be done or the placement of some content in the OpenSource core is explicitly prohibited (for example, links to intranet resources in an enterprise context), you can proceed to adding content to the source code of the documentation customization. {% include [custom_extension.md](content/custom_extension.md) %} -В тех разделах, где требования отличаются для базового контента и кастомизации, приведены две соответствующих закладки: +In the sections where the requirements differ for basic content and customization, there are two corresponding bookmarks: -- **Core**: ядро любой документации -- базовый контент -- **Overlay**: накладываемый поверх ядра контент, адаптирующий его к кастомной сборке +- **Core**: The core of any documentation, the basic content. +- **Overlay**: Content that is overlaid on top of the core and adapting it to a custom build. -Весь контент OpenSource сборки входит в состав ядра, и при его сборке применяется нулевая кастомизация, поэтому при его изменениях вам потребуется только закладка "Core". +All the content of the OpenSource build is part of the core, and zero customization is applied when building it, so when you change it, you'll only need the "Core" bookmark. -## Директории {#placement} +## Directories {#placement} -Базовая структура директорий исходного кода основывается на тематике контента, а не типе файлов или их технической роли при сборке. Технические файлы и директории размещаются внутри тематических. +The basic structure of the source code directories is based on the subject of the content and not on the type of files or their technical role during the build. Technical files and directories are placed inside the subject ones. -### Статья "Обзор" {#index} +### "Overview" article {#index} -Каждая тематическиая директория обязана содержать статью "Обзор" с именем файла `index.md`. В статье "Обзор": +Each subject directory must contain the "Overview" article with the `index.md` filename. The "Overview" article: -- Описывается о чем рассказывают статьи в данной директории -- Может даваться перечень ссылок на все или отдельные наиболее важные статьи -- Может даваться перечень ссылок "Смотрите также" на связанные другие статьи или разделы +- Describes what the articles in this directory tell about. +- Optionally, provides a list of links to all or some of the most important articles. +- Optionally, provides a set of "See also" links to other related articles and sections. -Наличие статьи "Обзор" позволяет ссылаться на директорию в целом, а не на конкретную статью, а также без потери ссылочной целостности преобразовывать статьи в директории. +The presence of the "Overview" article lets you refer to the entire directory rather than a specific article and convert articles into directories without losing the referential integrity. -## Статьи {#articles} +## Articles {#articles} {% list tabs %} - Core - Любая статья в базовой OpenSource документации создается в расчете на то, что она может быть расширена или скорректирована в кастомизированной документации, создаваемой на её основе. При этом, доработки базовой документации должны применяться в кастомизированной без ручного merge изменений контента, иначе ведение множества производных документаций становится очень трудозатратной задачей, а контроль полноты ручных адаптаций становится невозможным. + Any article in the basic OpenSource documentation is created with a view to extend or adjust it in the customized documentation created on its basis. At the same time, improvements to the basic documentation should be applied in a customized version with no manual merge of content changes, otherwise maintaining a lot of derived documentation versions becomes a very labor-intensive task and monitoring the completeness of manual adaptations becomes impossible. - Для достижения этой цели существующими инструментами файл со статьей внутри тематического каталога не содержит непосредственно контента, и предназначен для переопределения при кастомизации документации. Контент находится в одном нескольких файлах с блоками контента, включаемых в файл статьи директивами include. + To achieve this goal using the existing tools, a file with an article inside a subject directory does not directly contain any content and is designed to be redefined when customizing the documentation. The content is located in one or more files with content blocks included in the article file through include directives. - Вставляемые в статью блоки контента находятся в технической директории `_includes`, вложенной непосредственно в тематическую директорию. + The content blocks inserted into the article are located in a technical directory named `_includes` and nested directly in the subject directory. - **Статья из одного блока** + **A single-block article** - В простейшем случае статья состоит из одного блока, вставляемого директивой `{% include ...%}`: + In the simplest case, an article consists of a single block inserted by the `{% include ...%}` directive: `article1.md`: - ``` md + ```md + {% include [article1.md](_includes/article1.md) %} - ``` - В директории `_includes` создается файл с контентом, одноименный с файлом статьи: + In the `_includes` directory, a content file with the same name as the article file is created: `_includes/article1.md`: - ``` md - # Статья - Текст статьи. + + ```md + # Article + Article text. ``` - **Статья из нескольких блоков** + **A multi-block article** - Статья из одного блока позволяет при кастомизации документации добавить произвольный контент перед или после контента из базовой OpenSource документации. Этого может оказаться недостаточно, и тогда контент может быть разделен на большее количество блоков, позволяя при кастомизации документации как вставить произвольный контент между ними, так и исключить неприменимые блоки. + A single-block article lets you add arbitrary content before or after the content from the basic OpenSource documentation when customizing the documentation. This may not be enough, in which case the content can be divided into more blocks, letting you insert arbitrary content between them and exclude non-applicable blocks when customizing the documentation. - В этом случае, под статью создается одноименная директория внутри `_includes`. Например, если мы находимся внутри тематического каталога `subject1`, и хотим сделать статью `article1` из двух блоков, то должны сделать три файла со следующим содержимым: + In this case, a directory with the same name is created for the article inside `_includes`. For example, if you're inside the subject directory `subject1` and want to create an article named `article1` with two blocks, you should create three files with the following contents: `subject1/article1.md`: - ``` markdown + ```markdown + {% include [definition.md](_includes/article1/definition.md) %} - + {% include [examples.md](_includes/article1/examples.md) %} - ``` `subject1/_includes/article1/definition.md`: - ``` md - # Статья документации - ## Определение {#definition} - Статья документации -- это набор контента, раскрывающий определенную тему, интересную целевой аудитории. + + ```md + # Documentation article + ## Definition {#definition} + A documentation article is a set of content that reveals a specific topic that is interesting to the target audience. ``` `subject1/_includes/article1/examples.md`: - ``` md - ## Примеры {#examples} - - Кошки не похожи на людей. Кошки -- это кошки. - - Куст -- это совокупность веток, торчащих из одного места. + + ```md + ## Examples {#examples} + - Cats aren't like people. Cats are cats. + - A bush is a collection of branches sticking out of one place. ``` - Overlay - В кастомизации документации можно добавлять статьи, тематические каталоги, а также переопределять содержимое статей из `core`, размещая файл со статьей по тому же относительному пути внутри `overlay`, что он размещен внутри `core`. + When customizing the documentation, you can add articles and subject directories and redefine the content of articles from `core`, placing the article file at the same relative path inside `overlay` as it is placed inside `core`. - При переопределении можно: + When redefining the content, you can: - - Добавить дополнительный контент в начало или конец статьи + - Add additional content to the beginning or end of an article. `subject1/article1.md`: - ``` md + ```md + {% include [article1](_includes/article1.md) %} - - В дополнение к базовым способам авторизации, в нашей компании применяется авторизация на основе нанотрубок. + + In addition to the basic authorization methods, our company uses nanotube-based authorization. ``` - - Вставить дополнительный контент между директивами include + - Insert additional content between include directives. `subject1/article1.md`: - ``` markdown + ```markdown + {% include [definition.md](_includes/article1/definition.md) %} - - В нашей компании объем данных в БД ограничен 150ZB. - + + In our company, the amount of DB data is limited to 150ZB. + {% include [examples.md](_includes/article1/examples.md) %} - - Пример 2: + + Example 2: The quick brown fox jumps over the lazy dog. ``` - - Убрать из сборки часть контента исходной статьи, удалив соответствующую директиву include + - Remove some of the content of the original article from the build by removing the corresponding include directive. `subject1/article1.md`: - ``` markdown + ```markdown + {% include [definition.md](_includes/article1/definition.md) %} - - В нашей компании объем данных в БД ограничен 150ZB. - - Пример: + + In our company, the amount of DB data is limited to 150ZB. + + Example: The quick brown fox jumps over the lazy dog. ``` - При наличии множество блоков, и значительного количества добавляемого контента, может оказаться полезным его оформление также в виде блоков include, в созданной внутри тематической директории overlay поддиректории _includes. В этом случае, файл со статьей сохранит понятную структуру, охватываемую взглядом. + If there are multiple blocks and a significant amount of content to add, it may be useful to design it in the form of include blocks as well, in the _includes subdirectory created inside the overlay subject directory. In this case, the article file will retain a clear and visible structure. - Технически переопределенная статья может содержать полностью свой контент, вообще не включая что-либо из базовой документации. Однако, скорее всего это означает, что необходимо сделать рефакторинг контента в базовой документации с тем, чтобы он мог эффективно переопределяться, а не заменяться. + Technically, a redefined article may contain its entire content, without including anything from the basic documentation at all. However, most likely this means that it is necessary to refactor the content in the basic documentation so that it can be effectively redefined rather than replaced. {% endlist %} -### Особенности использования директивы include {#include-hints} +### Specifics of using the include directive {#include-hints} -- Содержимое `_includes` не публикуется, поэтому на него бесполезно и запрещено ссылаться, оно может быть только included в cтатьи директивой `{% include ... %}`. К сожалению, при локальной сборке содержимое _includes остается доступным и работает, и ошибка проявляется только при развертывании документации на ферме. -- Вокруг директивы `{% include ... %}` необходимо оставлять пустые строки, иначе она не будет исполнена при сборке. -- В квадратных скобках нужно писать имя файла, чтобы было на что кликать при просмотре в default viewer github (который не понимает что такое include), и чтобы один include в нём визуально отличался от другого. -- После `{%` в начале и перед `%}` в конце обязателен пробел, иначе include не будет исполнен при сборке. +- The `_includes` contents are not published, so it is useless and forbidden to refer to them, it can only be included in articles by the `{% include ... %}` directive. Unfortunately, when building documentation locally, the contents of _includes remain available and operational, and the error only occurs when deploying the documentation on the farm. +- Make sure to leave empty lines around the `{% include ... %}` directive, otherwise, it won't be executed during the build. +- Write a file name in square brackets so that there is something to click on when viewing it in the default viewer on github (which does not understand what include is), and so that one include in it visually differs from the other. +- After `{%` at the beginning and before `%}`, a space is required at the end, otherwise the include won't be executed during the build. -### Другие применения include {#include-reuse} +### Other use cases for include {#include-reuse} -Директива `{% include ... %}` может технически применяться не только в целях поддержки создания производных документаций, но и просто для переиспользования контента в нескольких статьях. Однако, такое применение является нежелательным, так как появление одинакового контента в разных статьях запутывает пользователя, а редактирование included контента без понимания контекста, в котором он приводится, может легко приводить к потере смысла. +The `{% include ... %}` directive can technically be used both to support the creation of derivative documentation versions and just to reuse content in several articles. However, this is undesirable, since the appearance of the same content in different articles confuses the user, and editing the included content without understanding the context it is given in can easily lead to loss of meaning. -Допустимым и хорошим вариантом применения переиспользуемого контента является предоставление одной и той же информации в разных вариантах группировки в пределах одного контекста. Например, раздел FAQ содержит как статьи по темам, так и статью "Все вопросы по всем темам". Аналогично информация по ценам может приводиться как в специализированных статьях по каким-то тарифицируемым функциям, там и в общей статье прайс-листа. +An acceptable and good way to use the reused content is to provide the same information in different grouping options within the same context. For example, the FAQ section contains both articles on certain topics and the "All questions on all topics" article. Similarly, information about prices can be given both in specialized articles on some chargeable functions and in the general article of the price list. -## Оглавление {#toc} +## Table of contents {#toc} -Каждая статья должна быть упомянута в файле оглавления (TOC - Table Of Contents), иначе она не будет опубликована. Оглавление является основным инструментом начала навигации по документации, и размещается в левой части страниц сайта документации. +Each article must be mentioned in the Table Of Contents (TOC) file, otherwise it won't be published. The table of contents is the main tool for starting navigation through the documentation and is placed on the left of the documentation website pages. {% list tabs %} - Core - Внутри каждой тематической директории находятся два файла TOC: + There are two TOC files inside each subject directory: - 1. `toc_i.yaml`: содержит перечень статей для включения в оглавление, находящихся непосредственно в данной директории, за исключением особой обязательной статьи "Обзор", которая должна обязательно присутствовать в любом подразделе TOC: + 1. `toc_i.yaml`: Contains a list of articles to be included in the table of contents and located directly in this directory, with the exception of a special "Overview" article that is mandatory and must be present in any subsection of the TOC: - ``` yaml + ```yaml items: - - name: Статья 1 + - name: Article 1 href: article1.md - - name: Статья 2 + - name: Article 2 href: article2.md ... ``` - 2. `toc_p.yaml`: содержит фиксированный контент, включающий статью "Обзор" и ссылку на `toc_i.yaml`: - ``` yaml + 1. `toc_p.yaml`: Contains fixed content including the "Overview" article and a link to `toc_i.yaml`: + + ```yaml items: - - name: Обзор + - name: Overview href: index.md - include: { mode: link, path: toc_i.yaml } ``` - Если у директории существуют дочерние директории, то в `toc_i.yaml` родительской директории явно указываются относительные ссылки на их `toc_p.yaml`: + If a directory has child directories, then the `toc_i.yaml` of the parent directory explicitly specifies relative references to their `toc_p.yaml`: - ``` yaml + ```yaml ... - - name: Подтема 1 + - name: Subtopic 1 include: { mode: link, path: subject1/toc_p.yaml } ... ``` - Непосредственно в директории `core` находится файл с корневым оглавлением `toc_p.yaml`, содержащий включение оглавлений `toc_p.yaml` из всех тематических директорий. + In the `core` directory proper, there is a file with the root table of contents named `toc_p.yaml`, including `toc_p.yaml` tables of contents from all subject directories. - Overlay - Если в кастомизированной документации не требуется изменения состава оглавления в некоторой тематической директории относительно базовой документации, то никаких файлов toc в `overlay` не создается. В сборку попадет файл toc из базовой документации. + If the customized documentation does not require changing the contents of the TOC in some subject directory as compared to the basic documentation, no toc files are created in `overlay`. The toc file from the basic documentation will be included in the build. - При необходимости скорректировать состав статей в overlay включается файл `toc_p.yaml`, изначально скопированный из `core`, в котором можно добавить статьи или подразделы в начало или конец перечня статей из базовой документации: + If you need to adjust the list of articles, the `toc_p.yaml` file is included in overlay, which is originally copied from `core`, and you can add articles or subsections to the beginning or end of the list of articles from the basic documentation there: - ``` yaml + ```yaml items: - # Включение статьи "обзор" в начало перечня - - name: Обзор + # Adding the "Overview" article at the beginning of the list + - name: Overview href: index.md - # Включение дополнительных пунктов перед пунктами из базовой документации - - name: Статья кастомизации 1 + # Adding additional items before the items from the basic documentation + - name: Customization article 1 href: cust_article1.md - # Включение пунктов TOC базовой документации + # Including TOC items from the basic documentation - include: { mode: link, path: toc_i.yaml } - # Включение дополнительных пунктов после пунктов из базовой документации - - name: Дополнительный подраздел + # Adding additional items after the items from the basic documentation + - name: Additional subsection include: { mode: link, path: cust_subdir/toc_p.yaml } - - name: Статья кастомизации 2 + - name: Customization article 2 href: cust_article2.md ``` {% endlist %} +## Article section headings and anchors {#headers-and-anchors} -## Заголовки разделов статей и якоря {#headers-and-anchors} - -Каждая статья должна в начале содержать заголовок первого уровня, обозначаемый одним символом `#`. Этот заголовок содержится в первом блоке контента. +Each article should contain a first-level heading at the beginning that is marked with a single `#`. This heading is contained in the first content block. -Заголовки следующих уровней могут быть технически произвольным образом размещены в разных блоках контента, однако имеет размещать заголовки в начале блоков. Необходимо помнить, что в результате сборки статьи получится одна HTML-страница с заголовками. +Next level headings can be placed in different content blocks technically in an arbitrary way. However, it makes sense to place them at the beginning of the blocks. Keep in mind that building an article results in a single HTML page with headings. -Каждый заголовок второго и более уровня снабжается якорем (anchor), который может быть использован в ссылках вида `host:path/article#anchor`. При проходе по такой ссылке браузер прокрутит содержимое страницы до нужного места. Якоря задается в заголовке следующим образом: +Each heading of the second and further level is provided with an anchor that can be used in links like `host:path/article#anchor`. When you click on this link, the browser scrolls the contents of the page to the desired location. An anchor is set in a heading as follows: -``` md -## Заголовок {#anchor} +```md +## Heading {#anchor} ``` -Допускается указание нескольких якорей, однако такое применение должно иметь под собой веские основания вроде сохранения совместимости с ранее существовавшими якорями при рефакторинге контента, например при объединении разделов: +It is allowed to specify several anchors, but you should have a good reason for that, such as maintaining the compatibility with previous anchors when refactoring the content, for example, when merging sections: -``` md -## Заголовок объединенного раздела по теме X и Y {#X} {#Y} {#XY} +```md +## Heading of a merged section on topic X and Y {#X} {#Y} {#XY} ``` -Указание якорей является расширением синтаксиса YFM по отношению к Markdown, поэтому общедоступные инструменты отображения Markdown (вроде preview в IDE) их не понимают. +Specifying anchors is an extension of the YFM syntax in relation to Markdown, so publicly available Markdown display tools (like preview in an IDE) do not understand them. + +## Links {#links} + +Links in the text of an article to other articles are the main way to guide the user to other topics that are associated with the content they're reading just like traffic signs. Ideally, a documentation developer should understand how the user will get to the desired information. If some article is not linked to other topics in any way and is only posted somewhere in the table of contents in the "Other" section, the user may never reach it, and we'll keep responding to support requests saying "it's all described here, why don't you read the documentation?". -## Ссылки {#links} +No one ever reads documentation as a work of art from beginning to end, users always read it to fix an issue, they have to find information about how to fix it, and links between articles are the main navigation tool once the user has started reading an article by selecting it in the TOC. -Ссылки в тексте статей на другие статьи являются основным средством направления пользователя на другие темы, ассоциативно связанные с читаемым в данный момент контентом, подобно указателям на дороге. В идеале, разработчик документации должен понимать, каким образом пользователь доберется до нужной информации. Если некоторая статья никак не привязана ссылками к другим темам, а только размещена где-то в оглавлении в разделе "Прочее", то пользователь может никогда до неё не дойти, а мы будем регулярно отвечать на запросы в поддержку "так это же все описано вот тут, почему вы не читаете документацию"? +There are two types of links: -Никто и никогда не читает документацию как художественное произведение с начала до конца, пользователи всегда приходят для решения какой-то проблемы, они должны это решение найти, и ссылки между статьями являются основным инструментом навигации после того, как пользователь начал чтение, выбрав статью в TOC. +1. To the source code of the documentation. These are always relative links to some .md file. The more transitions "to a level up" or entries to nested directories this link has, the more likely it is that the documentation is poorly structured. Ideally, links should not navigate more than one level up or one directory down the hierarchy: -Ссылки бывают двух видов: -1. На исходный код документации. Это всегда относительные ссылки на какой-то .md файл. При этом, чем больше в такой ссылке переходов "на уровень выше" или входов во вложенные директории, тем больше вероятность что документация плохо структурирована. В идеале, ссылки не должны выходить более чем на уровень вверх или на одну директорию вниз по иерархии: - ``` md - Для поддержания работоспособности котенка его [нужно кормить](../feeding/index.md). + ```md + To keep a kitten active, you [need to feed it](../feeding/index.md). ``` -2. На внешние по отношению к документации ресурсы. Это fully-qualified URLs, в которых никогда не добавляется index: - ``` md - Хорошие корма для котов можно найти в [магазине Pushok](http://www.pushok.ru/catalog). +1. To resources external to the documentation. These are fully-qualified URLs in which no index is ever added: + + ```md + You can find a good choice of cat food at the [Pushok store](http://www.pushok.ru/catalog). ``` -Текст внутри квадратных скобок, отображаемый при рендеринге документации, должен быть достаточно длинным, чтобы в него можно было легко попасть мышью или пальцем при клике. +Text inside the square brackets displayed when rendering documentation should be long enough to be easily clicked on with a mouse or tapped with a finger. -Существуют ситуации, ктогда URL ресурса имеет самостоятельную ценность, и должен быть отображен в документации, например, в случае публикации ссылок на репозиторий в github. В таких случаях его необходимо дублировать как внутри квадратных скобок, так и внутри обычных, так как YFM, в отличае от стандартного Markdown, не распознает автоматом URL в тексте: +There are situations when the URL of a resource has its own value and should be displayed in the documentation, for example, when publishing links to a repository on Github. In this case, it must be duplicated inside both square and regular brackets, since YFM, unlike standard Markdown, does not automatically recognize URLs in text: -``` md -Github репозиторий YDB: [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) +```md + {{ ydb-short-name }} repository on Github: [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) ``` -## Картинки {#pictures} +## Images {#pictures} -Картинки являются частью контента определенной статьи, и размещаются в технической поддиректории `_assets` в тематической директории, где размещена статья. Вставка картинки в текст статьи осуществляется ссылкой, перед которой указан восклицательный знак `!`: +Images are part of an article's content and placed in the `_assets` technical subdirectory of the subject directory where the article is posted. Images are inserted in the article text using a link preceded by an exclamation mark `!`: -``` md +```md  ``` -Указываемый в квадратных скобках текст показывается в браузере пока не загружена сама картинка. Он может повторять имя файла из ссылки без расширения. +The text specified in square brackets is displayed in the browser until the image itself is loaded. It can repeat the file name from the link with no extension. + +Desirable image formats: + +- Charts: [.SVG](https://en.wikipedia.org/wiki/SVG) +- Screenshots: [.PNG](https://en.wikipedia.org/wiki/Portable_Network_Graphics) -Желательные форматы для изображений: -- Диаграммы: [.SVG](https://ru.wikipedia.org/wiki/SVG) -- Скриншоты: [.PNG](https://ru.wikipedia.org/wiki/PNG) +Since an image is part of an article, it is impossible to post the image without this article. If there is no text of the article yet, determine the subject directory for its placement and the file name of the future article and only specify a link to the image in the text (keep in mind that the [text of the article is placed in the `_includes` subdirectory](#articles)!), and do not include the article itself in the TOC until it is ready for publication. -Так как картинка является частью какой-то статьи, то размещение картинки без статьи невозможно. Если текста статьи пока нет, то необходимо определить тематический каталог размещения и имя файла будущей статьи, в тексте указать только ссылку на картинку (не забыть про то что [текст статьи размещается в подкаталоге `_includes`](#articles)!), и не включать саму статью в TOC до тех пор, пока он не будет готова к публикации. +When inserting images, you can also specify: -Дополнительно при вставке картинки могут быть указаны: -- Hint для отображения при наведении мышью: ``. Пользоваться не рекомендуется, так как многие устройства сегодня не имеют курсора мыши. -- Размер изображения: ``. Рекомендуется пользоваться в варианте указания XSIZE для картинок в формате SVG, чтобы они были растянуты корректно по ширине экрана документации, независимо от того с каким разрешением были сохранены: ``. При указании таким образом только размера по X размер по Y подбирается автоматически с сохранением пропорций. +- A hint to be shown when hovering over an image: ``. We don't recommend using it, since lots of modern devices have no mouse pointer. +- Image size: ``. We recommend that you use it when specifying the XSIZE for SVG images so that they are expanded properly by the width of the documentation screen, regardless of the resolution they were saved with: ``. If you only specify the X-size in this way, the Y-size is selected automatically with the proportions maintained. -## Обратная совместимость {#compatibility} +## Backward compatibility {#compatibility} -Развитие документации не должно приводить к тому, что у её пользователей перестанут работать сохраненные ссылки: как закладки в браузерах, так и зафиксированные на множестве неподконтрольных разработчикам документации ресурах вроде wiki-страниц. +The development of documentation should not lead to a situation when users encounter saved links that are broken: both in browser tabs and those fixed in a variety of resources that are not controlled by the documentation developers, such as wiki pages. -Это означает, что переименовывать статьи, или переносить по директориям в общем случае нельзя. +This means that you generally can't rename articles or move them across directories. -Если необходимо статью преобразовать в набор статей в пределах новой директории, то для сохранения совместимсти данная директория должна получить то же имя, что раньше было у статьи, а внутри должна появиться статья `index.md`. В результате, по старой ссылке на статью пользователи начнут попадать на страницу "Обзор" новосозданной директории. +If you need to convert an article into a set of articles within a new directory, to maintain the compatibility, this directory should be named as the original article and an `index.md` article should appear inside. As a result, following the old link to the article, users will get to the "Overview" page of the newly created directory. -Если без переноса никак не обойтись, то можно применить следующий механизм: +If you cannot but move an article, do the following: -1. По старому месту расположения публикуется статья с контентом "Данная статья больше не существует, новое место размещения <здесь>, обновите ссылку", или вариации на тему этого контента. Главная задача -- чтобы при проходе по старой ссылке пользователь не получил HTTP 404 Not found, а смог понять куда оно переместилось, на что разбилось, или что вообще правда удалено и почему. -2. В TOC в старом месте расположения указывается ссылка на эту статью с включенным флагом `hidden`: - ``` yaml +1. At the previous location, publish an article saying "This article no longer exists, a new location is <here>, please update the link" or something like that. The main thing is to ensure that, when following the old link, the user does not receive the HTTP 404 Not found error and can understand where it has been moved, what it has been broken into, or that it has been actually deleted and why. +1. At the old location of the article in the TOC, give a link to it with the `hidden` flag enabled: + + ```yaml - name: Deprecated article href: old_article.md hidden: true ``` -В результате, данная статья не будет перечислена где-либо в TOC, но будет доступна при проходе по прямой ссылке, если она осталась где-то сохранена. + +As a result, this article won't be listed anywhere in the TOC, but will be available when following a direct link provided that you keep it somewhere. diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md b/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md index 400edabf47..8a19c28e9f 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md @@ -1,8 +1,8 @@ -# Как участвовать в разработке документации +# How to participate in the development of documentation -Исходный код документации YDB открыт и опубликован на [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs). +The source code of the {{ ydb-short-name }} documentation is open and published at [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs). -Команда YDB принимает запросы на изменения и дополнения в документации YDB, который может предложить каждый желающий через механизм [Pull Requests на github.com](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/working-with-your-remote-repository-on-github-or-github-enterprise/creating-an-issue-or-pull-request). +The {{ ydb-short-name }} team accepts requests for edits and updates to the {{ ydb-short-name }} documentation, which anyone can offer through [Pull Requests on github.com](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/working-with-your-remote-repository-on-github-or-github-enterprise/creating-an-issue-or-pull-request). -Прежде чем создавать такой запрос, необходимо ознакомиться с требованиями по разработке документации, описанными в статьях данного раздела, и [проверить сборку локально](../build.md). +Before creating a pull request, see the documentation development requirements described in the articles of this section and [check the build locally](../build.md). diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md b/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md index 5354d11134..8519355902 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md @@ -1,56 +1,59 @@ -# Создание кастомизированной документации по YDB +# Creating customized documentation for {{ ydb-short-name }} -Содержимое подкаталога `core` подключается в разные окружения сборки документации как библиотека, для создания кастомизированной документации. +The contents of the `core` subdirectory are linked to various documentation build environments as a library for creating customized documentation. -Подкаталог `overlay` предназначен для кастомизирующего контента, и для OpenSource сборки является техническим, так как OpenSource сборка не содержит кастомизаций ядра. +The `overlay` subdirectory is designed for customizing content and is technical for an OpenSource build, since the OpenSource build does not contain core customizations. -### Прокси-статьи +### Proxy articles -Файл со статьей в `core` никогда не содержит контента непосредственно, а только одну или несколько инструкций include блоков этого контента, размещенных в подкаталоге `_includes`. В результате, замена этого файла в `overlay` не приводит к необходимости копирования контента, но позволяет сделать следующие виды его адаптации: -- Добавить дополнительный контент в начало или нонец статьи -- Вставить дополнительный контент между директивами include -- Убрать из сборки часть контента исходной статьи, удалив соответствующую директиву include +A file with an article in `core` never contains content directly, it only contains one or more instructions of include blocks of this content stored in the `_includes` subdirectory. As a result, replacing this file in `overlay` does not lead to the need to copy the content, but lets you make the following types of its adaptation: -Инструкции по оформлению статей в core и overlay приведены в [Руководстве по созданию контента - Статьи](content.md#articles). +- Add additional content to the beginning or end of an article. +- Insert additional content between include directives. +- Remove some of the content of the original article from the build by removing the corresponding include directive. -### TOC (Table of Contents) - содержание +Instructions for placing articles in core and overlay are given in the [Content creation guide - Articles](content.md#articles). -TOC в документации YDB собирается из множества файлов, размещенных в директориях со статьями, которые иерархически включаются друг в друга директивой include. В результате, каждый из этих файлов может быть переопределен по-отдельности в адаптированной документации. +### TOC (Table of Contents) -Как и для статей, для файлов TOC применяется подход с прокси, со следующей реализацией: +The TOC in the {{ ydb-short-name }} documentation is built from a set of files placed in directories with articles that are hierarchically included in each other by the include directive. As a result, each of these files can be redefined separately in the adapted documentation. -1. Файл с пунктами меню для core называется `toc_i.yaml`. Он никогда не переопределяется в `overlay`. -2. Рядом с файлом `toc_i.yaml` в core находится файл `toc_p.yaml`, содержащий ссылку на `toc_i.yaml`, и предназначенный для переопределения в `overlay`. -3. Включание TOC других разделов делается ссылкой на файл `toc_p.yaml`. -4. Если в некоторой директории не требуется корректировать содержимое TOC, то никакие файлы toc* в `overlay` не создаются, что приводит к использованию toc_p --> toc_i из core при сборке. -5. Если в некоторой директории требуется скорректировать содержимое toc, то в `overlay` создается файл `toc_p.yaml`, в контент которого включается существовавший в core `- include: { mode: link, path: toc_i.yaml }`, а выше или ниже -- дополнительные пункты для адаптированной сборки. +As with articles, a proxy approach is used for TOC files as follows: -Хорошей практикой является исключение из toc_i.yaml пункта "Озбор", и включения его непосредственно в toc_p.yaml. Данная статья обязана быть первой в каждом подменю, и всегда имеет одно имя статьи (index.md). Включение отдельным пунктом в toc_p позволяет добавить новые статьи в адаптированной документации перед статьями из core, но с сохранением "Озбор" на первом месте: +1. A file with menu items for core is named `toc_i.yaml`. It is never redefined in `overlay`. +1. Next to the `toc_i.yaml` file in core, there is a file named `toc_p.yaml`. It contains a link to `toc_i.yaml` and is designed to be redefined in `overlay`. +1. Other sections are included in the TOC via a link to the `toc_p.yaml` file. +1. If there is no need to adjust the contents of the TOC in a certain directory, no toc* files are created in `overlay`. This results in using toc_p --> toc_i from core in the build. +1. If the contents of the TOC need to be adjusted in a certain directory, a file named `toc_p.yaml` is created in `overlay` with the `- include: { mode: link, path: toc_i.yaml }` from core added to its content and additional items for the adapted build added above or below. -toc_p.yaml в некотором корпоративном overlay: -``` yaml +It is a good practice to exclude the "Overview" item from toc_i.yaml and include it directly in toc_p.yaml. This article must be the first in each submenu and always has the same article name (index.md). Including a separate item in toc_p lets you add new articles to the adapted documentation before the articles from core, but keeping the "Overview" in the first place: + +toc_p.yaml in some corporate overlay: + +```yaml items: -- name: Обзор +- name: Overview href: index.md -- name: Роли сотруников +- name: Employee roles href: corp_roles.md - include: { mode: link, path: toc_i.yaml } -- name: Корпоративная авторизация +- name: Corporate authorization href: corp_auth.md ``` -Как и для статей, в core может быть заготовлено несколько файлов toc_i*.yaml, включаемых в toc_p отдельными include. +As with articles, core may have several toc_i*.yaml files included in toc_p with separate includes. -В корневом каталоге также есть пара файлов toc_i и toc_p, представляющих верхний уровень оглавления. +There are also a couple of toc_i and toc_p files in the root directory, representing the top level of the table of contents. -Так как включение файлов производится ссылками на них из родительского toc, то имя файла на самом деле технически может быть любым, и описанное выше -- только соглашение о наименованиях. Но есть один запрет: +Since the inclusion of files is implemented through references to them from the parent toc, a file name may actually be technically anything and the above is only a naming convention. However, there is one restriction: {% note warning %} -Имя файла с TOC не может быть `toc.yaml`, так как инструмент сборки ищет файлы с такими именами по всем подкаталогам, и пытается их сам добавить в TOC. Этом вариант добавления в документации YDB не используется, любое включение всегда явное директивой include из другого TOC. +A TOC file name may not be `toc.yaml`, since the build tool searches for files with this name across all subdirectories and tries to add them to the TOC on its own. This include option is not used in the YDB documentation, any inclusion is always explicit by the include directive from another TOC. {% endnote %} -Единственное исключение -- это начальный `toc.yaml`, который размещается в корне сборки документации, и всегда содержит команду копирования содержимого core и перехода к обработке TOC в core: `include: { mode: merge, path: core/toc_m.yaml }`. +The only exception is the initial `toc.yaml` that is placed in the documentation build root and always contains the command to copy the core content and proceed to TOC handling in core: `include: { mode: merge, path: core/toc_m.yaml }`. + +Instructions for making a TOC in core and overlay are given in the [Content creation guide - TOC](content.md#toc). -Инструкции по оформлению TOC в core и overlay приведены в [Руководстве по созданию контента - TOC](content.md#toc). diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md index 7f80702cb3..6508653300 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md @@ -1,3 +1,4 @@ -## Статьи для продвинутых читателей +## Articles for advanced readers + +- [Creating customized documentation](../customize.md) -- [Создание кастомизированной документации](../customize.md) diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md index 39b1b24ce6..4db6754f88 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md @@ -1,6 +1,6 @@ -## Основные статьи +## Main articles -- [Руководство по созданию контента](../../content.md) - как нужно технически офрормять статьи -- [Структура тематических каталогов](../../subjects.md) - где нужно размещать тот или иной контент -- [Сборка документации](../../build.md) - как собирать документацию +- [Content creation guide](../../content.md): Describes how to create articles in technical terms. +- [Structure of subject directories](../../subjects.md): Describes where to place certain content. +- [Building documentation](../../build.md): Describes how to build documentation. diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md index 7338aa067f..13385e4398 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md @@ -1,5 +1,6 @@ -# Как редактировать документацию +# How to edit the documentation -## Введение +## Introduction + +The [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) repository contains the open source code of the {{ ydb-short-name }} documentation in Russian and English. Based on it, an OpenSource version of the {{ ydb-short-name }} documentation is built and published at [http://ydb.tech](http://ydb.tech). It is also used to create customized {{ ydb-short-name }} documentation versions for enterprise environments and cloud services. -В репозитории [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) размещен публично доступный исходный код документации по YDB на русском и английском языках. На его основании собирается и публикуется документация по OpenSource версии YDB на сайте [http://ydb.tech](http://ydb.tech), а также он используется для создания кастомизированной документации по YDB для enterprise-окружений и облачных сервисов. diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md b/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md index aa65f08683..9efb9f5c3b 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md @@ -1,33 +1,34 @@ -# Структура тематических каталогов документации YDB +# Structure of {{ ydb-short-name }} documentation subject directories -## Введение {#intro} +## Introduction {#intro} -Статьи документации размещаются в иерархической структуре тематических директорий. Тематическая директория объединяет набор статей на определенную общую тематику. +Documentation articles are arranged in a hierarchical structure of subject directories. A subject directory combines a set of articles on a certain general topic. -Описание тематики включает в себя: -1. Целевую аудиторию, то есть для кого написана статья. Аудитория определяется ролью потенциального читателя при работе с YDB, со следующими тремя базовыми аудиториями: - - Разработчик приложений на YDB - - Администратор YDB - - Разработчик/контрибьютор YDB -2. Назначение -- задачу или проблему читателя, решение которой описывается в контенте. +A topic description includes: -В основном, структура директорий напрямую отражается на структуре оглавления документации. Типовыми исключениями являются: -- Промежуточные стадии создания документации. Новая директория формируется сразу, для сохранения ссылочной целостности в дальнейшем, но её контент еще недостаточен для того, чтобы быть оформленным отдельным подменю в оглавлении. В этом случае статьи могут быть временно включены в существующие подменю. -- Исторически сложившиеся директории, перенос которых нежелателен из-за потери ссылочной целостности. +1. The target audience of the article. The audience depends on the role of a potential reader when working with {{ ydb-short-name }}, with the following three basic audiences: + - Developer of applications for {{ ydb-short-name }} + - {{ ydb-short-name }} administrator + - {{ ydb-short-name }} developer/contributor +1. Purpose: The reader's task or problem the solution of which is described in the content. -## Структура 1-го уровня +Basically, the directory structure directly affects the structure of the documentation table of contents. Typical exceptions are: -На 1-м уровне размещены следующие тематические директории: +- Intermediate stages of documentation creation. A new directory is created immediately to preserve the referential integrity in the future, but its content is still insufficient to be designed as a separate submenu in the table of contents. In this case, articles can be temporarily included in existing submenus. +- Historically established directories whose transfer is undesirable due to the loss of referential integrity. -| Имя | Аудитория | Назначение | -| --- | ----------| ---------- | -| getting_started | Все | Предоставить быстрые решения типовых задач, которые возникают при начале работы с новой БД: что это, как поставить, как соединиться, как выполнить простейшие операции с данными, куда дальше по каким вопросам ходить в документации | -| concepts | Разработчик приложений | Ознакомить с основными структурными компонентами YDB, с которыми придется иметь дело при разработке приложений под YDB. Дать понимание роли каждого такого компонента при решении задач разработки приложений. Предоставить расширенную информацию по вариантам конфигурации компонентов, доступных разработчик приложений. | -| reference, yql | Разработчик приложений | Справочник для ежедневного использования по инструментам доступа к функциям YDB: CLI, YQL, SDK | -| best_practices | Разработчик приложений | Типовые подходы к решению основных задач, возникающих при разработке приложений | -| maintenance | Разработчик приложений | Как обслуживать свои базы данных YDB на администрируемом кем-то большом кластере, или при самостоятельном развертывании небольших БД для разработки: резервные копии, мониторинг, логи | -| troubleshooting | ? | Инструменты локализации причин проблем | -| how_to_edit_docs | Разработчики YDB | Как дорабатывать документацию по YDB | -| faq | Все | Вопросы и ответы | +## Level 1 structure +The following subject directories are placed on level 1: + +| Name | Audience | Purpose | +| --- | ---------- | ---------- | +| getting_started | All | Provide quick solutions to standard problems that arise when starting to work with a new database: what is it, how to install it, how to connect, how to perform elementary operations with data, where to go further in the documentation and on what issues. | +| concepts | Application developer | Describe the main structural components of {{ ydb-short-name }} that one will deal with when developing applications for {{ ydb-short-name }}. Provide an insight into the role of each of these components in developing applications. Provide detailed information about component configuration options available to application developers. | +| reference, yql | Application developer | A reference guide for daily use on tools for accessing {{ ydb-short-name }} functions: CLI, YQL, and SDK | +| best_practices | Application developer | Common approaches to addressing the main challenges that arise when developing applications | +| maintenance | Application developer | How to maintain your {{ ydb-short-name }} databases deployed on a large cluster administered by someone, or when independently deploying small databases for development: backups, monitoring, logs, and more. | +| troubleshooting | ? | Tools for identifying the cause of issues | +| how_to_edit_docs | Developers {{ ydb-short-name }} | How to update the {{ ydb-short-name }} documentation | +| faq | All | Questions and answers | diff --git a/ydb/docs/en/core/how_to_edit_docs/informal.md b/ydb/docs/en/core/how_to_edit_docs/informal.md index 67c719ba6d..e986f00243 100644 --- a/ydb/docs/en/core/how_to_edit_docs/informal.md +++ b/ydb/docs/en/core/how_to_edit_docs/informal.md @@ -1,5 +1,5 @@ +## Informal rules -## Неформальные правила +1. `DRY`. If you have to copy and paste some content, most likely you are doing something wrong. +1. `../../../...` If you have to add relative references that point to many levels up, most likely you are doing something wrong. -1. `DRY`. Если вам приходится копипастить контент -- скорее всего вы делаете что-то неправильно -1. `../../../...` Если вам приходится делать относительные ссылки с выходом на много уровней вверх -- скорее всего вы делаете что-то неправильно. diff --git a/ydb/docs/en/core/how_to_edit_docs/toc_i.yaml b/ydb/docs/en/core/how_to_edit_docs/toc_i.yaml index 3c028fd16f..374533807b 100644 --- a/ydb/docs/en/core/how_to_edit_docs/toc_i.yaml +++ b/ydb/docs/en/core/how_to_edit_docs/toc_i.yaml @@ -1,13 +1,11 @@ items: -- name: Руководство по созданию контента +- name: Content creation guide href: content.md -- name: Структура тематических каталогов +- name: Structure of subject directories href: subjects.md -- name: Сборка документации +- name: Building documentation href: build.md -- name: Как участвовать в разработке документации +- name: How to participate in the development of documentation href: contribute.md -- name: Создание кастомизированной документации - href: customize.md - -
\ No newline at end of file +- name: Creating customized documentation + href: customize.md
\ No newline at end of file diff --git a/ydb/docs/en/core/how_to_edit_docs/toc_p.yaml b/ydb/docs/en/core/how_to_edit_docs/toc_p.yaml index 3e62ad228b..94ce110868 100644 --- a/ydb/docs/en/core/how_to_edit_docs/toc_p.yaml +++ b/ydb/docs/en/core/how_to_edit_docs/toc_p.yaml @@ -1,4 +1,4 @@ items: -- name: Обзор +- name: Overview href: index.md - include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/index.yaml b/ydb/docs/en/core/index.yaml index fa96d3c389..60d31139eb 100644 --- a/ydb/docs/en/core/index.yaml +++ b/ydb/docs/en/core/index.yaml @@ -1,11 +1,11 @@ -title: Yandex Database +title: YDB description: - >- - <i>Yandex Database</i> (YDB) is a fault-tolerant Distributed SQL DBMS. YDB provides high availability, horizontal scalability, strict consistency, and ACID transaction support. Queries are made using an SQL dialect (<a href="yql/reference/overview">YQL</a>). + <i>YDB</i> is a fault-tolerant distributed SQL DBMS. YDB provides high availability, horizontal scalability, strict consistency, and ACID transaction support. Queries are made using an SQL dialect (<a href="yql/reference/overview">YQL</a>). nav: - title: Yandex Database + title: YDB meta: - title: Yandex Database + title: YDB links: - title: Getting started description: Creating a database, connecting to it, setting up access permissions, and performing basic operations with it diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/01_intro.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/01_intro.md index 804c23c231..4aae62c250 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/01_intro.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/01_intro.md @@ -2,10 +2,10 @@ title: "Yandex Database (YDB) backups" description: "This section describes supported methods for creating YDB database backups and restoring data from previously created backups. YDB lets you use the following destinations to create backups: CSV files on the file system and AWS S3-compatible storage." --- - # Backups -This section describes supported methods for creating YDB database backups and restoring data from previously created backups. {{ ydb-short-name }} lets you use the following destinations to create backups: +This section describes supported methods for creating {{ ydb-short-name }} database backups and restoring data from previously created backups. {{ ydb-short-name }} lets you use the following destinations to create backups: + * CSV files on the file system. * AWS S3-compatible storage. @@ -13,4 +13,5 @@ This section describes supported methods for creating YDB database backups and r Backups may negatively affect the database interaction indicators. Queries may take longer to execute. Before performing a database backup under load in production databases, test the process in the testing environment. -{% endnote %}
\ No newline at end of file +{% endnote %} + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/02_prerequisites.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/02_prerequisites.md index 61fd13dec5..d3ac294189 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/02_prerequisites.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/02_prerequisites.md @@ -1,6 +1,6 @@ ## Prerequisites -1. Install the [YDB console client](../../../reference/ydb-cli/index.md). +1. Install the [{{ ydb-short-name }} console client](../../../reference/ydb-cli/index.md). 1. Get the DB connection parameters: **Endpoint** and **Database**. 1. Save the **Endpoint** and **Database** parameter values in the `$YDB_ENDPOINT` and `$YDB_DB_PATH` environment variables. diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/03_limitations.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/03_limitations.md index 061d70fd00..065179b014 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/03_limitations.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/03_limitations.md @@ -1,3 +1,4 @@ ## Limitations {#limitations} -In the current implementation, a maximum of 1000 tables can be backed up in YDB at a time.
\ No newline at end of file +In the current implementation, a maximum of 1000 tables can be backed up in {{ ydb-short-name }} at a time. + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_1_header.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_1_header.md index bf9d5e98af..8eaf4ac215 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_1_header.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_1_header.md @@ -7,3 +7,4 @@ Saving the structure of the `backup` directory in the `$YDB_DB_PATH` database to ``` For each directory in the database, a file system directory is created. For each table, a directory is created in the file system to place the structure description file in. Table data is saved to one or more `CSV` files, one file line per table row. + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_2_body.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_2_body.md index b0279e77cb..7d8e38f8f5 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_2_body.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/04_fs_backup_2_body.md @@ -16,7 +16,7 @@ my_backup_of_basic_example/ 3 directories, 6 files ``` -The structure of each table is saved in a file named `scheme.pb`. For example, the `episodes/scheme.pb` file stores the schema of the `episodes` table. The data of each table is saved in one or more files named like `data_xx.csv`, where xx is the file's sequence number. The name of the first file is `data_00.csv`. The file size limit is 100 MB. +The structure of each table is saved in a file named `scheme.pb`. For example, the `episodes/scheme.pb` file stores the schema of the `episodes` table. The data of each table is saved in one or more files named like `data_xx.csv`, where xx is the file's sequence number. The name of the first file is `data_00.csv`. The file size limit is 100 MB.
### Saving table schemas @@ -25,3 +25,4 @@ Running the `{{ ydb-cli }} tools dump` command with the `--scheme-only` option o ``` {{ ydb-cli }} -e $YDB_ENDPOINT -d $YDB_DB_PATH tools dump -p $YDB_DB_PATH/examples -o my_backup_of_basic_example/ --scheme-only ``` + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/05_fs_restore.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/05_fs_restore.md index d493753424..471977fe8e 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/05_fs_restore.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/05_fs_restore.md @@ -15,3 +15,4 @@ The command below checks that all tables saved in the `my_backup_of_basic_exampl ``` {{ ydb-cli }} -e $YDB_ENDPOINT -d $YDB_DB_PATH tools restore -p $YDB_DB_PATH/restored_basic_example -i my_backup_of_basic_example/ --dry-run ``` + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_1_header.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_1_header.md index ba0ba18d6d..f3702cb2c0 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_1_header.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_1_header.md @@ -1,4 +1,4 @@ ## Backups to S3-compatible storage {#s3_backup} -YDB lets you save database backups to storage that supports the [Amazon Simple Storage Service (AWS S3) API](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html). +{{ ydb-short-name }} lets you save database backups to storage that supports the [Amazon Simple Storage Service (AWS S3) API](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html). diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_3_access_keys.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_3_access_keys.md index 17428130db..4534f812c3 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_3_access_keys.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_3_access_keys.md @@ -1,9 +1,10 @@ ### Creating access keys {#s3_create_access_keys} -Access keys are used for authentication and authorization in S3-compatible storage. The [YDB console client](../../../reference/ydb-cli/index.md) provides three ways to pass keys: +Access keys are used for authentication and authorization in S3-compatible storage. The [{{ ydb-short-name }} console client](../../../reference/ydb-cli/index.md) provides three ways to pass keys: * Through the `--access-key` and `--secret-key` command line options. * Using the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables. * Through the `~/.aws/credentials` file created and used by the [AWS CLI](https://aws.amazon.com/cli/). The settings apply in the order described above. For example, if you use all three ways to pass the access_key or secret_key value at the same time, the values passed through the command-line options will be used. + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_4_export.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_4_export.md index 7cbf80fffa..3c3d294972 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_4_export.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_4_export.md @@ -2,7 +2,7 @@ The commands given in the examples below are based on the assumption that access key data is saved to the `~/.aws/credentials` file. -Running the operation for exporting data from the `$YDB_DB_PATH/backup/episodes`, `$YDB_DB_PATH/backup/seasons`, and `$YDB_DB_PATH/backup/series` tables in the YDB `$YDB_DB_PATH` database to files prefixed with `20200601/` in the `testdbbackups` bucket in {{ objstorage-name }}. +Running the operation for exporting data from the `$YDB_DB_PATH/backup/episodes`, `$YDB_DB_PATH/backup/seasons`, and `$YDB_DB_PATH/backup/series` tables in the {{ ydb-short-name }} `$YDB_DB_PATH` database to files prefixed with `20200601/` in the `testdbbackups` bucket in {{ objstorage-name }}. ``` {{ ydb-cli }} -e $YDB_ENDPOINT -d $YDB_DB_PATH export s3 --s3-endpoint {{ s3-storage-host }} --bucket testdbbackups\ @@ -52,7 +52,7 @@ aws --endpoint-url=https://{{ s3-storage-host }} s3 ls testdbbackups/20200601/ {% note info %} -To back up all tables in the YDB directory, specify the path to the directory as the source. +To back up all tables in the {{ ydb-short-name }} directory, specify the path to the directory as the source. ``` {{ ydb-cli }} -e $YDB_ENDPOINT -d $YDB_DB_PATH export s3 \ diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_5_restore.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_5_restore.md index 3eab33f47a..95a14c7978 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_5_restore.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_5_restore.md @@ -1,5 +1,6 @@ ### Restoring data from S3-compatible storage {#s3_restore} -First, connect the bucket with the YDB database backup files in S3-compatible storage to make the database available through operations on the file system. +First, connect the bucket with the {{ ydb-short-name }} database backup files in S3-compatible storage to make the database available through operations on the file system. + +When done, upload the data from the backup in {{ ydb-short-name }} following the instructions given in [Restoring data from backups in the file system](#filesystem_restore). -When done, upload the data from the backup in YDB following the instructions described in [Restoring data from backups in the file system](#filesystem_restore).
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_6_forget.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_6_forget.md index a04b8c6362..48c7350946 100644 --- a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_6_forget.md +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/06_s3_6_forget.md @@ -1,9 +1,10 @@ ### Terminating the backup operation {#s3_forget} -To minimize the impact of the backup process on user load indicators, data is sent from table copies. Before sending data from YDB, the backup process creates consistent copies of all the tables being sent to YDB. As they're made using the copy-on-write technique, there is almost no change in the size of the DB at the time of their creation. Once the data is sent, the created copies remain in the DB. +To minimize the impact of the backup process on user load indicators, data is sent from table copies. Before sending data from {{ ydb-short-name }}, the backup process creates consistent copies of all the tables being sent to {{ ydb-short-name }}. As they're made using the copy-on-write technique, there is almost no change in the size of the DB at the time of their creation. Once the data is sent, the created copies remain in the DB. To delete the table copies from the DB and the completed operation from the list of operations, run the command: ``` {{ ydb-cli }} -e $YDB_ENDPOINT -d $YDB_DB_PATH operation forget 'ydb://export/6?id=283824558378666&kind=s3' -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/charts.md b/ydb/docs/en/core/maintenance/embedded_monitoring/charts.md index a0f21dbf54..e3f52b2be8 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/charts.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/charts.md @@ -8,3 +8,4 @@ The main metrics of the system are displayed on the dashboard: * **Memory Usage**: RAM utilization by nodes. * **Disk Space Usage**: Disk space utilization by nodes. * **SelfPing**: The highest actual delivery time of deferred messages in the actor system over the measurement interval. Measured for messages with a 10 ms delivery delay. If this value grows, it might indicate microbursts of workload, high CPU utilization, or displacement of the {{ ydb-short-name }} process from CPU cores by other processes. + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md b/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md new file mode 100644 index 0000000000..238a7cbb5b --- /dev/null +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md @@ -0,0 +1,70 @@ +# Hive web-viewer + +The Hive web-viewer provides an interface for working with Hive. +Hive can be shared by a cluster or be tenant. +You can go to the Hive web-viewer from the {{ ydb-short-name }} Monitoring. + +## Home page + +The home page provides information about the distribution and usage of resources by tablets on each node in the form of a table. + +The table is preceded by the following brief information: + +* **Tenant**: The tenant that Hive is responsible for. +* **Tablets**: The percentage of tablets started and the number of running tablets to their total number. +* **Boot Queue**: The number of tablets in the boot queue. +* **Wait Queue**: The number of tablets that can't be started. +* **Resource Total**: Resource utilization by tablets (cpu, net). +* **Resource StDev**: Standard deviation of resource utilization (cnt, cpu, mem, net). + +Then there is a table where each row represents a node managed by Hive. It has the following columns: + +* **Node**: The node number. +* **Name**: The node FQDN and ic-port. +* **DC**: The datacenter where the node resides. +* **Domain**: The node tenant. +* **Uptime**: The node uptime. +* **Unknown**: The number of tablets whose state is unknown. +* **Starting**: The number of tablets being started. +* **Running**: The number of tablets running. +* **Types**: Tablet distribution by type. +* **Usage**: A normalized dominant resource. +* **Resources** : + * **cnt**: The number of tablets that are not using any resources. + * **cpu**: CPU usage by tablets. + * **mem**: RAM usage by tablets. + * **net**: Bandwidth usage by tablets. +* **Active**: Enables/disables the node to move tablets to this node. +* **Freeze**: Disables tablets to be deployed on other nodes. +* **Kick**: Moves all tablets from the node at once. +* **Drain**: Smoothly moves all tablets from the node. + +Additional pages are presented below the table: + +* **Bad tablets**: A list of tablets having issues or errors. +* **Heavy tablets**: A list of tablets utilizing a lot of resources. +* **Waiting tablets**: A list of tablets that can't be started. +* **Resources**: Resource utilization by each tablet. +* **Tenants**: A list of tenants indicating their local Hive tablets. +* **Nodes**: A list of nodes. +* **Storage**: A list of storage group pools. +* **Groups**: A list of storage groups for each tablet. +* **Settings**: The Hive configuration page. +* **Reassign Groups**: The page for reassigning storage groups across tablets. + +You can also view what tablets use a particular group and, vice versa, what groups are used on a particular tablet. + +## Reassign Groups {#reassign_groups} + +Click **Reassign Groups** to open the window with parameters for balancing: + +* **Storage pool**: Pool of storage groups for balancing. +* **Storage group**: If the previous item is not specified, you can specify only one group separately. +* **Type**: Type of tablets that balancing will be performed for. +* **Channels**: Range of channels that balancing will be performed for. +* **Percent**: Percentage of the total number of tablet channels that will be moved as a result of balancing. +* **Inflight**: The number of tablets being moved to other groups at the same time. + +After specifying all the parameters, click "Query" to get the number of channels moved and unlock the "Reassign" button. +Clicking this button starts reassigning. + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/interconnect_overview.md b/ydb/docs/en/core/maintenance/embedded_monitoring/interconnect_overview.md index 8a0cda4147..a2df6e35a5 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/interconnect_overview.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/interconnect_overview.md @@ -12,3 +12,4 @@ Shows, for every other node: * system clock difference * connection availability * last written error + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/logs.md b/ydb/docs/en/core/maintenance/embedded_monitoring/logs.md index 1e3d4d9b30..a68bdda591 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/logs.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/logs.md @@ -37,3 +37,4 @@ To change the logging level: 1. To change the logging level for individual components, use the table under `Component log settings`. In the line with the name of the component whose logging level you want to change, in the `Component` column, select the desired logging level from the drop-down list in the `Log level` column. 1. To save changes, click `Submit` + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md b/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md index b46b8df08b..9764f99f49 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md @@ -1,9 +1,9 @@ # Overview of introspection tools -YDB provides tools for monitoring and determining system health: +{{ ydb-short-name }} provides tools for monitoring and determining system health: -* [YDB Monitoring](ydb_monitoring.md): The main monitor of the cluster. It shows the health of nodes and storage groups. +* [{{ ydb-short-name }} Monitoring](ydb_monitoring.md): The main monitor of the cluster. It shows the health of nodes and storage groups. * [Interconnect overview](interconnect_overview.md): The state of cluster interconnects. -* [Logs](logs.md): Each YDB component writes messages to logs of different levels. They can be used to detect severe issues or identify the root causes of issues. -* [Charts](charts.md): YDB collects a range of metrics that can be used to determine the health of the entire system or a specific component. +* [Logs](logs.md): Each {{ ydb-short-name }} component writes messages to logs of different levels. They can be used to detect severe issues or identify the root causes of issues. +* [Charts](charts.md): {{ ydb-short-name }} collects a range of metrics that can be used to determine the health of the entire system or a specific component. diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md b/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md index fec59ef2fe..b80620b059 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md @@ -77,7 +77,6 @@ http://<endpoint>:8765/monitoring/node/<node-id>/ Information about the node is presented in the following sections: * **Pools**: CPU utilization broken down by the internal stream pools, with roughly the following pool functions: - * **System**: The tasks of critical system components. * **User**: User tasks, queries executed by tablets. * **Batch**: Long-running background tasks. @@ -87,14 +86,12 @@ Information about the node is presented in the following sections: High pool utilization might degrade performance and increase the system response time. * **Common info**: Basic information about the node: - * **Version**: The {{ ydb-short-name }} version. * **Uptime**: The node uptime. * **DC**: The availability zone where the node resides. * **Rack**: The ID of the rack where the node resides. * **Load average**: Average host CPU utilization for different time intervals: - * 1 minute. * 5 minutes. * 15 minutes. @@ -111,14 +108,14 @@ For each group, the following information is provided: * The ID of the storage group. * The performance indicator of the group. -* The number of vdisks in the group. +* The number of VDisks in the group. * The data storage topology. -Each storage group can also be expanded into a list of vdisks, with the following information provided for each vdisk: +Each storage group can also be expanded into a list of VDisks, with the following information provided for each VDisk: -* The ID of the vdisk. -* The unique (within the node) ID of the block store volume where the vdisk resides. -* The ID of the node where the vdisk resides. +* VDiskID. +* The unique (within the node) PDiskID where the VDisk resides. +* The ID of the node where the VDisk resides. * Free/available space on the block store volume. * The path used to access the block store volume. @@ -152,7 +149,6 @@ In the `Tenant Info` section, you can see the following information: * **Pools**: The total CPU utilization by the tenant nodes broken down by internal stream pools (for more information about pools, see the [tenant page](#tenant_page)). * **Metrics**: Data about tablet utilization for this tenant: - * **Memory**: The RAM utilized by tablets. * **CPU**: CPU utilized by tablets. * **Storage**: The amount of data stored by tablets. @@ -165,7 +161,7 @@ In the `Tenant Info` section, you can see the following information: The tenant page also includes the following tabs: * **HealthCheck**: The report regarding cluster issues, if any. -* **Storage**: The [list of storage groups](#tenant_storage_page) that includes information about which vdisks reside on which nodes and devices. +* **Storage**: The [list of storage groups](#tenant_storage_page) that includes information about which VDisks reside on which nodes and block store volumes. * **Compute**: The [list of nodes](#tenant_compute_page), which includes the nodes and tablets running on them. * **Schema**: The [tenant's schema](#tenant_scheme) that lets you view tables, execute YQL queries, view a list of the slowest queries and the most loaded shards. * **Network**: The [health of the cluster network](#tenant_network). @@ -227,3 +223,4 @@ The indicator colors have the following meaning: * **Red**: There are critical problems, the component is down (or runs with limitations). If a component includes other components, then in the absence of its own issues, the state is determined by aggregating the states of its parts. + diff --git a/ydb/docs/en/core/maintenance/manual/_includes/cluster_decomision.md b/ydb/docs/en/core/maintenance/manual/_includes/cluster_decomision.md new file mode 100644 index 0000000000..44d4faef56 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/_includes/cluster_decomision.md @@ -0,0 +1,4 @@ +# Decommissioning nodes and disks + +**WILL BE SOON** + diff --git a/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md b/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md new file mode 100644 index 0000000000..8ab7c7660a --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md @@ -0,0 +1,34 @@ +# Adding storage groups + +To add storage groups, you need to redefine the config of the pool you want to extend. + +Before that, you need to get the config of the desired pool. You can do this with the following command: + +```proto +Command { + ReadStoragePool{ + BoxId: <box-id> + // StoragePoolId: <storage-pool-id> + Name: <pool name> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file ReadStoragePool.txt +``` + +Insert the obtained pool config into the protobuf below and edit the **NumGroups** field value in it. + +```proto +Command { + DefineStoragePool { + <pool config> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file DefineStoragePool.txt +``` + diff --git a/ydb/docs/en/core/maintenance/manual/balancing_load.md b/ydb/docs/en/core/maintenance/manual/balancing_load.md new file mode 100644 index 0000000000..245bbcf236 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/balancing_load.md @@ -0,0 +1,44 @@ +# Disk load balancing + +{{ ydb-short-name }} supports two methods for the disk load distribution: + +## Distribute the load evenly across groups + +At the bottom of the [Hive web-viewer](../embedded_monitoring/hive.md#reassign_groups) page there is a button named "Reassign Groups". + +## Distribute VDisks evenly across block store volumes + +If VDisks aren't evenly distributed across block store volumes, you can [move them](moving_vdisks.md#moving_vdisk) one at a time from overloaded store volumes. + +## Changing the number of slots for VDisks on PDisks + +To add storage groups, redefine the host config by increasing the number of slots on PDisks. + +Before that, you need to get the config to be changed. You can do this with the following command: + +```proto +Command { + TReadHostConfig{ + HostConfigId: <host-config-id> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file ReadHostConfig.txt +``` + +Insert the obtained config into the protobuf below and edit the **PDiskConfig/ExpectedSlotCount** field value in it. + +```proto +Command { + TDefineHostConfig { + <host config> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file DefineHostConfig.txt +``` + diff --git a/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md b/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md new file mode 100644 index 0000000000..3fd05f34d5 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md @@ -0,0 +1,36 @@ +# Changing an actor system's configuration + +## On static nodes + +Static nodes take the configuration of the actor system from the kikimr/cfg/sys.txt file. + +After changing the configuration, restart the node. + +## On dynamic nodes + +Dynamic nodes take the configuration from the CMS. To change it, you can use the following command. + +```proto +ConfigureRequest { + Actions { + AddConfigItem { + ConfigItem { + // UsageScope: { ... } + Config { + ActorSystemConfig { + <actor system config> + } + } + MergeStrategy: 3 + } + } + } +} +``` + +kikimr -s <endpoint> admin console execute --domain=<domain> --retry=10 actorsystem.txt + +``` + + + diff --git a/ydb/docs/en/core/maintenance/manual/cluster_decomision.md b/ydb/docs/en/core/maintenance/manual/cluster_decomision.md new file mode 100644 index 0000000000..44d4faef56 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/cluster_decomision.md @@ -0,0 +1,4 @@ +# Decommissioning nodes and disks + +**WILL BE SOON** + diff --git a/ydb/docs/en/core/maintenance/manual/cluster_expansion.md b/ydb/docs/en/core/maintenance/manual/cluster_expansion.md new file mode 100644 index 0000000000..0c0d21a27a --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/cluster_expansion.md @@ -0,0 +1,79 @@ +# Cluster extension {#expand_cluster} + +1) Add information about new nodes to NameserviceConfig in the names.txt file + + ``` + Node { + NodeId: 1 + Port: <ic-port> + Host: "<existing-host>" + InterconnectHost: "<existing-host>" + Location { + DataCenter: "DC1" + Module: "M1" + Rack: "R1" + Unit: "U1" + } + } + Node { + NodeId: 2 + Port: <ic-port> + Host: "<new-host>" + InterconnectHost: "<new-host>" + Location { + DataCenter: "DC1" + Module: "M2" + Rack: "R2" + Unit: "U2" + } + } + ClusterUUID: "<cluster-UUID>" + AcceptUUID: "<cluster-UUID>" + ``` + +2) Update the NameserviceConfig via CMS + +3) Add new nodes to DefineBox + + Sample protobuf for DefineBox + + ``` + Command { + DefineHostConfig { + HostConfigId: 1 + Drive { + Path: "<device-path>" + Type: SSD + PDiskConfig { + ExpectedSlotCount: 2 + } + } + } + } + Command { + DefineBox { + BoxId: 1 + Host { + Key { + Fqdn: "<existing-host>" + IcPort: <ic-port> + } + HostConfigId: 1 + } + Host { + Key { + Fqdn: "<new-host>" + IcPort: <ic-port> + } + HostConfigId: 1 + } + } + } + ``` + + Using the command + + ``` + kikimr -s <endpoint> admin bs config invoke --proto-file DefineBox.txt + ``` + diff --git a/ydb/docs/en/core/maintenance/manual/cms.md b/ydb/docs/en/core/maintenance/manual/cms.md new file mode 100644 index 0000000000..625399ae45 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/cms.md @@ -0,0 +1,44 @@ +# Updating configurations via CMS + +## Get the current settings + +The following command will let you get the current settings for a cluster or tenant. + +``` +./kikimr -s <endpoint> admin console configs load --out-dir <config-folder> +``` + +``` +./kikimr -s <endpoint> admin console configs load --out-dir <config-folder> --tenant <tenant-name> +``` + +## Update the settings + +First, you need to pull the desired config as indicated above and then prepare a protobuf file with an update request. + +``` +Actions { + AddConfigItem { + ConfigItem { + Cookie: "<cookie>" + UsageScope { + TenantAndNodeTypeFilter { + Tenant: "<tenant-name>" + } + } + Config { + <config-name> { + <full-config> + } + } + } + } +} +``` + +The UsageScope field is optional and is needed to use settings for a specific tenant. + +``` +./kikimr -s <endpoint> admin console configs update <protobuf-file> +``` + diff --git a/ydb/docs/en/core/maintenance/manual/disk_end_space.md b/ydb/docs/en/core/maintenance/manual/disk_end_space.md new file mode 100644 index 0000000000..48a511a90b --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/disk_end_space.md @@ -0,0 +1,62 @@ +# Methods to free up space on physical devices + +When the disk space is used up, the database may start responding to all queries with an error. To keep the database healthy, we recommend deleting a part of the data or adding block store volumes to extend the cluster. + +Below are instructions that can help you add or free up disk space. + +## Defragment a VDisk + +When working with the DB, internal VDisk fragmentation is done. You can find out the percentage of fragmentation on the VDisk monitoring page. We do not recommend that you perform defragmentation of VDisks that are fragmented by 20% or less. + +According to the failure model, the cluster survives the loss of two VDisks in the same group without data loss. If all VDisks in the group are up and there are no VDisks with the error or replication status, then deleting data from one VDisk will result in the VDisk recovering it in a compact format. Please keep in mind that data storage redundancy will be decreased until automatic data replication is complete. + +During data replication, the load on all the group's VDisks increases, and response times may deteriorate. + +1. View the fragmentation coefficient on the VDisk page in the viewer. + + If its value is more than 20%, defragmentation can help free up VDisk space. + +2. Check the status of the group that hosts the VDisk. There should be no VDisks that are unavailable or in the error or replication status in the group. + + You can view the status of the group in the viewer. + +3. Run the wipe command for the VDisk. + + All data stored by the VDisk will be permanently deleted, and after that, the VDisk will begin restoring the data by reading it from the other VDisks in the group. + + ```bash + kikimr admin blobstorage group reconfigure wipe --domain <Domain number> --node <Node ID> --pdisk <pdisk-id> --vslot <Slot number> + ``` + + You can view the details for the command in the viewer. + +If the block store volume is running out of space, you can apply defragmentation to the entire block store volume. + +1. Check the health of the groups in the cluster. There shouldn't be any problem groups on the same node with the problem block store volume. + +2. Log in via SSH to the node hosting this block store volume. + +3. Check if you can [restart the process](node_restarting.md#restart_process). + +4. Stop the process + + ```bash + sudo systemctl stop kikimr + ``` + +5. Format the block store volume + + ```bash + sudo kikimr admin blobstorage disk obliterate <path to the store volume part label> + ``` + +6. Run the process + + ```bash + sudo systemctl start kikimr + ``` + +## Moving individual VDisks from full block store volumes + +If defragmentation can't help freeing up space on the block store volume, you can [move](moving_vdisks.md#moving_disk) individual VDisks. + diff --git a/ydb/docs/en/core/maintenance/manual/failure_model.md b/ydb/docs/en/core/maintenance/manual/failure_model.md new file mode 100644 index 0000000000..fd567cd2de --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/failure_model.md @@ -0,0 +1,19 @@ +# How to stay within the failure model + +## One VDisk in the storage group failed {#storage_group_lost_one_disk} + +With SelfHeal enabled, this situation is considered normal. SelfHeal will move the VDisk over the time specified in the settings, then data replication will start on a different PDisk. + +If SelfHeal is disabled, you'll have to manually move the VDisk. Before moving it, make sure that **only one VDisk** in the storage group has failed. +Then follow the [instructions](moving_vdisks.md#removal_from_a_broken_device). + +## More than one VDisk in the same storage group have failed without going beyond the failure model {#storage_group_lost_more_than_one_disk} + +In this kind of failure, no data is lost, the system maintains operability, and read and write queries are executed successfully. Performance might degrade because of the load handover from the failed disks to the operable ones. + +If multiple VDisks have failed in the group, SelfHeal stops moving VDisks. If the maximum number of failed VDisks for the failure model has been reached, recover at least one VDisk before you start [moving the VDisks](moving_vdisks.md#removal_from_a_broken_device). You may also need to be more careful when [moving VDisks one by one](moving_vdisks.md#moving_vdisk). + +## The number of failed VDisks has exceeded the failure model {#exceeded_the_failure_model} + +The availability and operability of the system might be lost. Make sure to revive at least one VDisk without losing the data stored on it. + diff --git a/ydb/docs/en/core/maintenance/manual/index.md b/ydb/docs/en/core/maintenance/manual/index.md new file mode 100644 index 0000000000..e35a8c7ded --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/index.md @@ -0,0 +1,25 @@ +# Maintaining a cluster's disk subsystem + +## Tackling cluster failures + +The cluster may become inoperable for a number of reasons: + +* It may [go beyond the failure model](failure_model.md), stopping data reads and writes to the storage group completely. + +* Unbalanced workload on disks may strongly affect the request processing latency. Load balancing methods are described in this [article](balancing_load.md). + +* Writing can also stop if multiple physical disks run out of space. This can be solved [by freeing up space](disk_end_space.md) or adding block store volumes to [expand the cluster](cluster_expansion.md). + +Unauthorized withdrawal of nodes can result in issues described above. To prevent the issues, make sure to [drain the nodes correctly for maintenance](node_restarting.md). + +Enabling [Scrubbing](scrubbing.md) and [SelfHeal](selfheal.md) would also be a good preventative measure. + +## Editing the cluster configuration + +A {{ ydb-short-name }} cluster lets you: + +* [Expand](cluster_expansion.md) block store volumes and nodes. +* [Configure](change_actorsystem_configs.md) the actor system on your nodes. +* Edit configs via [CMS](cms.md). +* [Add](adding_storage_groups.md) new storage groups. + diff --git a/ydb/docs/en/core/maintenance/manual/moving_vdisks.md b/ydb/docs/en/core/maintenance/manual/moving_vdisks.md new file mode 100644 index 0000000000..c9aab4bd81 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/moving_vdisks.md @@ -0,0 +1,42 @@ +# Moving VDisks + +## Move a VDisk from a block store volume {#moving_vdisk} + +To move a VDisk from a block store volume, log in to the node via SSH and run the command: + +```bash +kikimr admin bs config invoke --proto 'Command { ReassignGroupDisk { GroupId: <Storage group ID> GroupGeneration: <Storage group generation> FailRealmIdx: <FailRealm> FailDomainIdx: <FailDomain> VDiskIdx: <Slot number> } }' +``` + +You can find the parameters for the command in the viewer (link). + +## Move VDisks from a broken/missing block store volume {#removal_from_a_broken_device} + +If SelfHeal is disabled or fails to move VDisks, you'll have to run this operation manually. + +1. Make sure that the VDisk has actually failed. + + Write down the node's FQDN, ic-port, VDisk path, and pdisk-id + +2. Go to any cluster node + +3. Move the VDisk + + ```bash + kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<host>" IcPort: <ic-port>} Path: "<Path to the storage volume part label>" PDiskId: <pdisk-id> Status: BROKEN } }' + ``` + +## Enable the VDisk back after reassignment {#return_a_device_to_work} + +1. In Monitoring, make sure that the PDisk is actually operable + + Write down the node's FQDN, ic-port, store path, and pdisk-id + +2. Go to any cluster node + +3. Enable the PDisk back + + ```bash + kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<host>" IcPort: <ic-port>} Path: "<Path to the storage volume part label>" PDiskId: <pdisk-id> Status: ACTIVE } }' + ``` + diff --git a/ydb/docs/en/core/maintenance/manual/node_restarting.md b/ydb/docs/en/core/maintenance/manual/node_restarting.md new file mode 100644 index 0000000000..433c336512 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/node_restarting.md @@ -0,0 +1,41 @@ +# Safe restart and shutdown of nodes + +## Stopping/restarting a YDB process on a node {#restart_process} + +To make sure that the process is stoppable, follow these steps. + +1. Access the node via SSH. + +1. Execute the command + + ```bash + kikimr cms request restart host {node_id} --user {user} --duration 60 --dry --reason 'some-reason' + ``` + + If the process is stoppable, you'll see `ALLOW`. + +1. Stop the process + + ```bash + sudo service kikimr stop + ``` + +1. Restart the process if needed + + ```bash + sudo service kikimr start + ``` + +## Replacing equipment {#replace_hardware} + +Before replacing equipment, make sure that the YDB process is [stoppable](#restart_process). +If the replacement is going to take a long time, first move all the VDisks from this node and wait until replication is complete. +After replication is complete, you can safely shut down the node. + +To make sure that disabling the dynamic node doesn't affect query handling, drain the tablets from this node first. + +Go to the [Hive web-viewer](../embedded_monitoring/hive.md) page. +Click "View Nodes" to see a list of all nodes. + +Before disabling the node, first disable the transfer of tablets through the Active button, then click Drain, and wait for all the tablets to be moved away. + diff --git a/ydb/docs/en/core/maintenance/manual/scrubbing.md b/ydb/docs/en/core/maintenance/manual/scrubbing.md new file mode 100644 index 0000000000..4da321e362 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/scrubbing.md @@ -0,0 +1,7 @@ +# Enabling/disabling Scrubbing + +The Scrub settings let you adjust the interval from the beginning of the previous disk scrubbing cycle to that of the next one and the maximum number of disks that can be scrubbed simultaneously. The default value is 1 month. +`$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { ScrubPeriodicitySeconds: 86400 MaxScrubbedDisksAtOnce: 1 } }'` + +If ScrubPeriodicitySeconds is 0, Scrubbing is disabled. + diff --git a/ydb/docs/en/core/maintenance/manual/selfheal.md b/ydb/docs/en/core/maintenance/manual/selfheal.md new file mode 100644 index 0000000000..0e95492974 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/selfheal.md @@ -0,0 +1,110 @@ +# Enabling/disabling SelfHeal {#selfheal} + +During cluster operation, individual block store volumes used by YDB or entire nodes may fail. To maintain the cluster's uptime and fault tolerance when it's impossible to promptly fix the failed nodes or volumes, YDB provides SelfHeal. + +SelfHeal includes two parts. Detecting faulty disks and moving them carefully to avoid data loss and disintegration of storage groups. + +SelfHeal is enabled by default. +Below are instructions how to enable or disable SelfHeal. + +1. Enabling detection + + Open the page + + ```http://localhost:8765/cms#show=config-items-25``` + + It can be enabled via the viewer -> Cluster Management System -> CmsConfigItems + + Status field: Enable + + Or via the CLI + + * Go to any node + + * Create a file with modified configurations + + Sample config.txt file + + ``` + Actions { + AddConfigItem { + ConfigItem { + Config { + CmsConfig { + SentinelConfig { + Enable: true + } + } + } + } + } + } + ``` + + * Update the config on the cluster + + ```bash + kikimr admin console configs update config.txt + ``` + +2. Enable SelfHeal + + ```bash + kikimr -s <endpoint> admin bs config invoke --proto 'Command{EnableSelfHeal{Enable: true}}' + ``` + +Disabled in a similar way by setting the value to false. + +### SelfHeal settings + +viewer -> Cluster Management System -> CmsConfigItems +If there are no settings yet, click Create, if there are, click the "pencil" icon in the corner. + +* **Status**: Enables/disables Self Heal in the CMS. +* **Dry run**: Enables/disables the mode in which the CMS doesn't change the BSC setting. +* **Config update interval (sec.)**: BSC config update interval. +* **Retry interval (sec.)**: Config update retry interval. +* **State update interval (sec.)**: PDisk state update interval, the State is what we're monitoring (through a whiteboard, for example) +* **Timeout (sec.)**: PDisk state update timeout. +* **Change status retries**: The number of retries to change the PDisk Status in the BSC, the Status is what is stored in the BSC (ACTIVE, FAULTY, BROKEN, and so on). +* **Change status retry interval (sec.)**: Interval between retries to change the PDisk Status in the BSC. The CMS is monitoring the disk state with the **State update interval**. If the disk remains in the same state for several **Status update interval** cycles, the CMS changes its Status in the BSC. +Next are the settings for the number of update cycles through which the CMS will change the disk Status. If the disk State is Normal, the disk is switched to the ACTIVE Status. Otherwise, the disk is switched to the FAULTY status. The 0 value disables changing the Status for the state (this is done for Unknown by default). +For example, with the default settings, if the CMS is monitoring the state of the Initial disk for 5 Status update interval cycles of 60 seconds each, the disk Status will be changed to FAULTY. +* **Default state limit**: For States with no setting specified, this default value can be used. For unknown PDisk States that have no setting, this value is used, too. This value is used if no value is set for States such as Initial, InitialFormatRead, InitialSysLogRead, InitialCommonLogRead, and Normal. +* **Initial**: PDisk starts initializing. Transition to FAULTY. +* **InitialFormatRead**: PDisk is reading its format. Transition to FAULTY. +* **InitialFormatReadError**: PDisk has received an error when reading its format. Transition to FAULTY. +* **InitialSysLogRead**: PDisk is reading the system log. Transition to FAULTY. +* **InitialSysLogReadError**: PDisk has received an error when reading the system log. Transition to FAULTY. +* **InitialSysLogParseError**: PDisk has received an error when parsing and checking the consistency of the system log. Transition to FAULTY. +* **InitialCommonLogRead**: PDisk is reading the common VDisk log. Transition to FAULTY. +* **InitialCommonLogReadError**: PDisk has received an error when reading the common VDisk log. Transition to FAULTY. +* **InitialCommonLogParseError**: PDisk has received an error when parsing and checking the consistency of the common log. Transition to FAULTY. +* **CommonLoggerInitError**: PDisk has received an error when initializing internal structures to be logged to the common log. Transition to FAULTY. +* **Normal**: PDisk has completed initialization and is running normally. Transition to ACTIVE will occur after this number of Cycles (that is, by default, if the disk is Normal for 5 minutes, it's switched to ACTIVE). +* **OpenFileError**: PDisk has received an error when opening a disk file. Transition to FAULTY. +* **Missing**: The node responds, but this PDisk is missing from its list. Transition to FAULTY. +* **Timeout**: The node didn't respond within the specified timeout. Transition to FAULTY. +* **NodeDisconnected**: The node has disconnected. Transition to FAULTY. +* **Unknown**: Something unexpected, for example, the TEvUndelivered response to the state request. Transition to FAULTY. + +## Enabling/disabling donor disks + +If donor disks are disabled, when moving the VDisk, its data is lost and has to be restored according to the selected erasure. + +The recovery operation is more expensive than regular data transfers. Data loss also occurs, which may lead to data loss when going beyond the failure model. + +To prevent the above problems, there are donor disks. + +When transferring disks with donor disks enabled, the old VDisk remains alive until the new one transfers all the data from it to itself. + +The donor disk is the old VDisk after the transfer, which continues to store its data and only responds to read requests from the new VDisk. + +When receiving a request to read data that the new VDisk has not yet transferred, it redirects the request to the donor disk. + +To enable the donor disks, run the following command: + +`$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { EnableDonorMode: true } }'` + +Similarly, when changing the setting to `false`, the command disables the mode. + diff --git a/ydb/docs/en/core/maintenance/toc_i.yaml b/ydb/docs/en/core/maintenance/toc_i.yaml index cbea249f63..5779af0e59 100644 --- a/ydb/docs/en/core/maintenance/toc_i.yaml +++ b/ydb/docs/en/core/maintenance/toc_i.yaml @@ -10,10 +10,38 @@ items: when: false - name: YDB Monitoring href: embedded_monitoring/ydb_monitoring.md + - name: Hive web-viewer + href: embedded_monitoring/hive.md - name: Connections overview href: embedded_monitoring/interconnect_overview.md # when: false - name: Logs href: embedded_monitoring/logs.md - name: Charts - href: embedded_monitoring/charts.md
\ No newline at end of file + href: embedded_monitoring/charts.md + - name: Maintaining a cluster's disk subsystem + items: + - name: Overview + href: manual/index.md + - name: How to stay within the failure model + href: manual/failure_model.md + - name: Disk load balancing + href: manual/balancing_load.md + - name: Methods to free up space on physical devices + href: manual/disk_end_space.md + - name: Cluster extension + href: manual/cluster_expansion.md + - name: Adding storage groups + href: manual/adding_storage_groups.md + - name: Safe restart and shutdown of nodes + href: manual/node_restarting.md + - name: Enabling/disabling SelfHeal + href: manual/selfheal.md + - name: Enabling/disabling Scrubbing + href: manual/scrubbing.md + - name: Moving VDisks + href: manual/moving_vdisks.md + - name: Updating configurations via CMS + href: manual/cms.md + - name: Updating configuration of the actor system + href: manual/change_actorsystem_configs.md
\ No newline at end of file diff --git a/ydb/docs/en/core/operations/manual/operation.md b/ydb/docs/en/core/operations/manual/operation.md new file mode 100644 index 0000000000..92d3c5b8b2 --- /dev/null +++ b/ydb/docs/en/core/operations/manual/operation.md @@ -0,0 +1,525 @@ +# Maintaining a cluster's disk subsystem + +## Extending a cluster with static nodes {#expand_cluster} + +1. Add information about new nodes to NameserviceConfig in the names.txt file. + + ``` + Node { + NodeId: 1 + Port: <IR port> + Host: "<old-host>" + InterconnectHost: "<old-host>" + Location { + DataCenter: "DC1" + Module: "M1" + Rack: "R1" + Unit: "U1" + } + } + Node { + NodeId: 2 + Port: <IR port> + Host: "<new-host>" + InterconnectHost: "<new-host>" + Location { + DataCenter: "DC1" + Module: "M2" + Rack: "R2" + Unit: "U2" + } + } + ClusterUUID: "<cluster-UUID>" + AcceptUUID: "<cluster-UUID>" + ``` + +1. Update the NameserviceConfig via CMS. + +1. Add new nodes to DefineBox. + + Sample proto file for DefineBox. + + ``` + Command { + DefineHostConfig { + HostConfigId: 1 + Drive { + Path: "<device-path>" + Type: SSD + PDiskConfig { + ExpectedSlotCount: 2 + } + } + } + } + Command { + DefineBox { + BoxId: 1 + Host { + Key { + Fqdn: "<old-host>" + IcPort: <IR port> + } + HostConfigId: 1 + } + Host { + Key { + Fqdn: "<new-host>"| + IcPort: <IR port> + } + HostConfigId: 1 + } + } + } + ``` + + Using the command + + ``` + kikimr -s <endpoint> admin bs config invoke --proto-file DefineBox.txt + ``` + +## Decommissioning nodes and disks + +**WILL BE SOON** + +## SelfHeal {#selfheal} + +During cluster operation, individual block store volumes used by YDB or entire nodes may fail. To maintain the cluster's uptime and fault tolerance when it's impossible to promptly fix the failed nodes or volumes, YDB provides SelfHeal. + +SelfHeal includes two parts. Detecting faulty disks and moving them carefully to avoid data loss and disintegration of storage groups. + +SelfHeal is enabled by default. +Below are instructions how to enable or disable SelfHeal. + +1. Enabling detection. + + Open the page + + ```http://localhost:8765/cms#show=config-items-25``` + + It can be enabled via the viewer -> Cluster Management System -> CmsConfigItems + + Status field: Enable + + Or via the CLI + + * Go to any node. + + * Create a file with modified configs. + + Sample config.txt file. + + ``` + Actions { + AddConfigItem { + ConfigItem { + Config { + CmsConfig { + SentinelConfig { + Enable: true + } + } + } + } + } + } + ``` + + * Update the config on the cluster. + + ```bash + kikimr admin console configs update config.txt + ``` + +1. Enable SelfHeal. + + ```bash + kikimr -s <endpoint> admin bs config invoke --proto 'Command{EnableSelfHeal{Enable: true}}' + ``` + +### SelfHeal settings + +viewer -> Cluster Management System -> CmsConfigItems +If there are no settings yet, click Create, if there are, click the "pencil" icon in the corner. + +* **Status**: Enables/disables Self Heal in the CMS. +* **Dry run**: Enables/disables the mode in which the CMS doesn't change the BSC setting. +* **Config update interval (sec.)**: BSC config update interval. +* **Retry interval (sec.)**: Config update retry interval. +* **State update interval (sec.)**: PDisk state update interval, the State is what we're monitoring (through a whiteboard, for example). +* **Timeout (sec.)**: PDisk state update timeout. +* **Change status retries**: The number of retries to change the PDisk Status in the BSC, the Status is what is stored in the BSC (ACTIVE, FAULTY, BROKEN, and so on). +* **Change status retry interval (sec.)**: Interval between retries to change the PDisk Status in the BSC. The CMS is monitoring the disk state with the **State update interval**. If the disk remains in the same state for several **Status update interval** cycles, the CMS changes its Status in the BSC. +Next are the settings for the number of update cycles through which the CMS will change the disk Status. If the disk State is Normal, the disk is switched to the ACTIVE Status. Otherwise, the disk is switched to the FAULTY status. The 0 value disables changing the Status for the state (this is done for Unknown by default). +For example, with the default settings, if the CMS is monitoring the state of the Initial disk for 5 Status update interval cycles of 60 seconds each, the disk Status will be changed to FAULTY. +* **Default state limit**: For States with no setting specified, this default value can be used. For unknown PDisk States that have no setting, this value is used, too. This value is used if no value is set for States such as Initial, InitialFormatRead, InitialSysLogRead, InitialCommonLogRead, and Normal. +* **Initial**: PDisk starts initializing. Transition to FAULTY. +* **InitialFormatRead**: PDisk is reading its format. Transition to FAULTY. +* **InitialFormatReadError**: PDisk has received an error when reading its format. Transition to FAULTY. +* **InitialSysLogRead**: PDisk is reading the system log. Transition to FAULTY. +* **InitialSysLogReadError**: PDisk has received an error when reading the system log. Transition to FAULTY. +* **InitialSysLogParseError**: PDisk has received an error when parsing and checking the consistency of the system log. Transition to FAULTY. +* **InitialCommonnLogRead**: PDisk is reading the common VDisk log. Transition to FAULTY. +* **InitialCommonnLogReadError**: PDisk has received an error when reading the common Vdisk log. Transition to FAULTY. +* **InitialCommonnLogParseError**: PDisk has received an error when parsing and checking the consistency of the common log Transition to FAULTY. +* **CommonLoggerInitError**: PDisk has received an error when initializing internal structures to be logged to the common log. Transition to FAULTY. +* **Normal**: PDisk has completed initialization and is running normally. Transition to ACTIVE will occur after this number of Cycles (that is, by default, if the disk is Normal for 5 minutes, it's switched to ACTIVE). +* **OpenFileError**: PDisk has received an error when opening a disk file. Transition to FAULTY. +* **Missing**: The node responds, but this PDisk is missing from its list. Transition to FAULTY. +* **Timeout**: The node didn't respond within the specified timeout. Transition to FAULTY. +* **NodeDisconnected**: The node has disconnected. Transition to FAULTY. +* **Unknown**: Something unexpected, for example, the TEvUndelivered response to the state request. Transition to FAULTY. + +## Enabling/disabling donor disks + +If donor disks are disabled, when transferring the Vdisk, its data is lost and has to be restored according to the selected erasure. + +The recovery operation is more expensive than regular data transfers. Data loss also occurs, which may lead to data loss when going beyond the failure model. + +To prevent the above problems, there are donor disks. + +When transferring disks with donor disks enabled, the old VDisk remains alive until the new one transfers all the data from it to itself. + +The donor disk is the old VDisk after the transfer, which continues to store its data and only responds to read requests from the new VDisk. + +When receiving a request to read data that the new VDisk has not yet transferred, it redirects the request to the donor disk. + +To enable the donor disks, run the following command: + +`$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { EnableDonorMode: true } }'` + +Similarly, when changing the setting to `false`, the command to disable the mode. + +## Scrubbing + +### Enabling/Disabling + +**WILL BE SOON** + +### Scrubbing settings + +The Scrub settings let you adjust the interval from the beginning of the previous disk scrubbing cycle to that of the next one and the maximum number of disks that can be scrubbed simultaneously. The default value is 1 month. +`$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { ScrubPeriodicitySeconds: 86400 MaxScrubbedDisksAtOnce: 1 } }'` + +**WILL BE SOON** + +## Move one vdisk from a block store volume {#moving_vdisk} + +To move a disk from a block store volume, log in to the node via ssh and execute the following command: + +```bash +kikimr admin bs config invoke --proto 'Command { ReassignGroupDisk { GroupId: <Storage group ID> GroupGeneration: <Storage group generation> FailRealmIdx: <FailRealm> FailDomainIdx: <FailDomain> VDiskIdx: <Slot number> } }' +``` + +You can find the parameters for the command in the viewer (link). + +## Move vdisks from a broken/missing block store volume {#removal_from_a_broken_device} + +If SelfHeal is disabled or fails to move vdisks, you'll have to run this operation manually. + +1. Make sure that the disk has actually failed. + + Write down the node's fqdn, ic-port, disk path, and pdiskId. + +1. Go to any cluster node. + +1. Move the disk. + + ```bash + kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<Host>" IcPort: <IC Port>} Path: "<Path to the device part label>" PDiskId: <ID PDisk> Status: BROKEN } }'|||UNTRANSLATED_CONTENT_END||| + ``` + +## Enable the disk back after reassignment {#return_a_device_to_work} + +1. In Monitoring, make sure that the disk is actually operable. + + Write down the node's fqdn, ic-port, disk path, and pdiskId. + +1. Go to any cluster node. + +1. Re-enable the disk. + + ```bash + kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<Host>" IcPort: <IC Port>} Path: "<Path to the storage volume part label>" PDiskId: <PDisk ID> Status: ACTIVE } }' + ``` + +## Stopping/restarting a YDB process on a node {#restart_process} + +To make sure that the process is stoppable, follow these steps. + +1. Access the node via ssh. + +1. Execute the command + + ```bash + kikimr cms request restart host {node_id} --user {user} --duration 60 --dry --reason 'some-reason' + ``` + + If the process is stoppable, you'll see `ALLOW`. + +1. Stop the process + + ```bash + sudo service kikimr stop + ``` + +1. Restart the process if needed + + ```bash + sudo service kikimr start + ``` + +## Replacing equipment {#replace_hardware} + +Before replacing equipment, make sure that the YDB process is [stoppable](#restart_process). +If the replacement is going to take a long time, first move all the vdisks from this node and wait until replication is complete. +After replication is complete, you can safely shut down the node. + +To disable a dynamic node, you may also need to drain the tablets to avoid the effect on running queries. + +Go to a hive or tenant hive's web monitoring page. +Click "View Nodes" to see a list of all nodes running under this hive. + +It provides various information about running tablets and resources used. +On the right of the list, there are buttons with the following actions for each node: + +* **Active**: Enables/disables the node to move tablets to this node. +* **Freeze**: Disables tablets to be deployed on other nodes. +* **Kick**: Moves all tablets from the node at once. +* **Drain**: Smoothly moves all tablets from the node. + +Before disabling the node, first disable the transfer of tablets through the Active button, then click Drain, and wait for all the tablets to be moved away. + +## Adding storage groups + +To add storage groups, you need to redefine the config of the pool you want to extend. + +Before that, you need to get the config of the desired pool. You can do this with the following command: + +```proto +Command { + ReadStoragePool{ + BoxId: <box-id> + // StoragePoolId: <storage-pool-id> + Name: <pool name> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file ReadStoragePool.txt +``` + +Insert the obtained pool config into the protobuf below and edit the **NumGroups** field value in it. + +```proto +Command { + DefineStoragePool { + <pool config> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file DefineStoragePool.txt +``` + +## Changing the number of slots for VDisks on the PDisk + +To add storage groups, redefine the host config by increasing the number of disk slots for it. + +Before that, you need to get the config to be changed. You can do this with the following command: + +```proto +Command { + TReadHostConfig{ + HostConfigId: <host-config-id> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file ReadHostConfig.txt +``` + +Insert the obtained config into the protobuf below and edit the **PDiskConfig/ExpectedSlotCount** field value in it. + +```proto +Command { + TDefineHostConfig { + <host config> + } +} +``` + +``` +kikimr -s <endpoint> admin bs config invoke --proto-file DefineHostConfig.txt +``` + +## Cluster health issues {#cluster_liveness_issues} + +### No more than 2 disks belonging to the block-4-2 storage group failed {#storage_group_lost_two_disk} + +In this kind of failure, no data is lost, the system maintains operability, and read and write queries are executed successfully. Performance might degrade because of the load handover from the failed disks to the operable ones. + +If 2 disks are unavailable at the same time, we recommend reviving at least one of them or replacing one disk to start the replication process. This will provide some room for maneuver in the event a third disk fails before replication completes. + +### More than 2 disks belonging to the block-4-2 storage group failed{#exceeded_the_failure_modele} + +The availability and operability of the system might be lost. Make sure to revive at least one disk without losing the data stored on it. + +## Disk subsystem issues {#storage_issues} + +When the disk space is used up, the database may start responding to all queries with an error. To keep the database healthy, we recommend deleting a part of the data or adding block store volumes to extend the cluster. + +Below are instructions that can help you add or free up disk space. + +### Defragment vdisk + +As disks run, their data becomes fragmented. You can find out the disk's fragmentation rate on the vdisk monitoring page. We don't recommend defragmenting disks with a fragmentation ratio of 20% or less. + +According to the failure model, the cluster survives the loss of two vdisks in the same group without data loss. If all vdisks in the group are up and there are no vdisks with the error or replication status, then deleting data from one vdisk will result in the vdisk recovering it in a compact format. Please keep in mind that data storage redundancy will be decreased until automatic data replication is complete. + +During data replication, the load on all the group's vdisks increases, and response times may deteriorate. + +1. View the fragmentation coefficient on the vdisk page in the viewer (link). + + If its value is more than 20%, defragmentation can help free up disk space. + +1. Check the status of the group that hosts the vdisk. There should be no vdisks that are unavailable or in the error or replication status in the group. + + You can view the status of the group in the viewer (link). + +1. Run the wipe command for the vdisk. + + All data stored by the vdisk will be permanently deleted, and after that, the vdisk will begin restoring the data by reading it from the other vdisks in the group. + + ```bash + kikimr admin blobstorage group reconfigure wipe --domain <Domain number> --node <Node ID> --pdisk <PDisk ID> --vslot <Slot number> + ``` + + You can view the details for the command in the viewer (link). + +If the block store volume is running out of space, you can apply defragmentation to the entire block store volume. + +1. Check the health of the groups in the cluster. There shouldn't be any problem groups on the same node with the problem block store volume. + +1. Log in via SSH to the node hosting this disk. + +1. Check if it is possible to restart the process (link to the maintenance file). + +1. Stop the process. + + ```bash + sudo systemctl stop kikimr + ``` + +1. Format the disk. + + ```bash + sudo kikimr admin blobstorage disk obliterate <path to the store volume part label> + ``` + +1. Run the process. + + ```bash + sudo systemctl start kikimr + ``` + +### Distribute vdisks evenly across block store volumes + +If vdisks aren't evenly distributed across block store volumes, you can [move them](#moving_vdisks) one at a time from overloaded store volumes. + +### Distribute the load evenly across groups + +At the bottom of the hive web monitoring page, there is a button named "Reassign Groups". +Click it to open the window with parameters for balancing: + +* **Storage pool**: Pool of storage groups for balancing. +* **Storage group**: If the previous item is not specified, you can specify only one group separately. +* **Type**: Type of tablets that balancing will be performed for. +* **Channels**: Range of channels that balancing will be performed for. +* **Percent**: Percentage of the total number of tablet channels that will be moved as a result of balancing. +* **Inflight**: The number of tablets being moved to other groups at the same time. + +After specifying all the parameters, click "Query" to get the number of channels moved and unlock the "Reassign" button. +Clicking this button starts balancing. + +## Changing settings in the CMS + +### Get the current settings + +The following command will let you get the current settings for a cluster or tenant. + +``` +./kikimr -s <endpoint> admin console configs load --out-dir <directory-for-configs> +``` + +``` +./kikimr -s <endpoint> admin console configs load --out-dir <directory-for-configs> --tenant <tenant-name> +``` + +### Update the settings + +First, you need to pull the desired config as indicated above and then prepare a protobuf file with an update request. + +``` +Actions { + AddConfigItem { + ConfigItem { + Cookie: "<cookies>" + UsageScope { + TenantAndNodeTypeFilter { + Tenant: "<tenant-name>" + } + } + Config { + <config-name> { + <full config> + } + } + } + } +} +``` + +The UsageScope field is optional and is needed to use settings for a specific tenant. + +``` +./kikimr -s <endpoint> admin console configs update <file-with-settings> +``` + +## Changing an actor system's configuration + +### On static nodes + +Static nodes take the configuration of the actor system from a file located at kikimr/cfg/sys.txt. + +After changing the configuration, restart the node. + +### On dynamic nodes + +Dynamic nodes take the configuration from the CMS. To change it, you can use the following command. + +```proto +ConfigureRequest { + Actions { + AddConfigItem { + ConfigItem { + // UsageScope: { ... } + Config { + ActorSystemConfig { + <actor system config> + } + } + MergeStrategy: 3 + } + } + } +} +``` + +kikimr -s <endpoint> admin console execute --domain=<domain> --retry=10 actorsystem.txt + +``` diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env.md new file mode 100644 index 0000000000..5608098d2b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env.md @@ -0,0 +1,5 @@ +{% include [env_cloud.md](env_cloud.md) %} + +{% include [env_overlay.md](env_overlay.md) %} + +{% include [env_static.md](env_static.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md new file mode 100644 index 0000000000..0660f7c088 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md @@ -0,0 +1,5 @@ +- If the value of the `IAM_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. +- Otherwise, if the value of the `YC_TOKEN` environment variable is set, the **Refresh Token** authentication mode is used and the token to transfer to the IAM endpoint is taken from this variable value when repeating the request. +- Otherwise, if the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. +- Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **Service Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_overlay.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md new file mode 100644 index 0000000000..fbed4967db --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md @@ -0,0 +1,2 @@ +- If the value of the `YDB_USER` or `YDB_PASSWORD` environment variable is set, the username and password based authentication mode is used. The username is read from the `YDB_USER` variable. If it is not set, an error saying `User password was provided without user name` is returned. The password is read from the `YDB_PASSWORD` variable. If it is not set, then, depending on whether the `--no-password` command-line option is specified, either an empty password is used or a password is requested interactively. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options.md new file mode 100644 index 0000000000..01f8b08f36 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options.md @@ -0,0 +1,13 @@ + +{% include [options_header.md](options_header.md) %} + +{% include [options_overlay.md](options_overlay.md) %} + +{% include [options_cloud.md](options_cloud.md) %} + +{% include [options_static.md](options_static.md) %} + +{% include [options_footer.md](options_multiple.md) %} + +{% include [options_cloud.md](options_cloud_additional.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md new file mode 100644 index 0000000000..837fdae778 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md @@ -0,0 +1,5 @@ +- `--iam-token-file <filepath>` : The **Access Token** authentication mode is used based on the contents of the file specified in this option value. +- `--yc-token-file <filepath>` : The **Refresh Token** authentication mode is used based on the contents of the file specified in this option value. +- `--use-metadata-credentials` : The **Metadata** authentication mode is used. +- `--sa-key-file <filepath>` : The **Service Account Key** authentication mode is used with the key and other parameters taken from the JSON file specified in this option value. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md new file mode 100644 index 0000000000..f1937dc14e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md @@ -0,0 +1,4 @@ +When using authentication modes that involve token rotation along with regularly re-requesting them from IAM (**Refresh Token** and **Service Account Key**), a special parameter can be set to indicate where the IAM service is located: + +- `--iam-endpoint <URL>` : Sets the URL of the IAM service to request new tokens in authentication modes with token rotation. The default value is `"iam.api.cloud.yandex.net"`. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md new file mode 100644 index 0000000000..1d65d46fc7 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md @@ -0,0 +1,2 @@ +The authentication mode and parameters are selected by setting one of the following options: + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md new file mode 100644 index 0000000000..5a525922d9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md @@ -0,0 +1,8 @@ +If several of the above options are set simultaneously in the command line, the CLI returns an error asking you to specify only one: + +``` +$ {{ ydb-cli }} --use-metadata-credentials --iam-token-file ~/.ydb/token scheme ls +More than one auth method were provided via options. Choose exactly one of them +Try "--help" option for more info. +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_overlay.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md new file mode 100644 index 0000000000..b830d88295 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md @@ -0,0 +1,4 @@ +- `--user <username>` : The username and password based authentication mode is used with the username set in this option value. Additionally , you can specify: + - `--password-file <filename>` : The password is read from the specified file. + - `--no-password` : Defines an empty password. The password will be requested interactively if none of the password identification options listed above are specified in the command line parameters. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md b/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md new file mode 100644 index 0000000000..b44999c1ad --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md @@ -0,0 +1,76 @@ +# {{ ydb-short-name }} CLI commands + +General syntax for calling {{ ydb-short-name }} CLI commands: + +```bash +{{ ydb-cli }} [global options] <command> [<subcommand> ...] [command options] +``` + +, where: + +- `{{ ydb-cli}}` is the command to run the {{ ydb-short-name }} CLI from the OS command line. +- `[global options]` are [global options](../commands/global-options.md) that are common for all {{ ydb-short-name }} CLI commands. +- `<command>` is the command. +- `[<subcomand> ...]` are subcommands specified if the selected command contains subcommands. +- `[command options]` are command options specific to each command and subcommands. + +## Commands {#list} + +You can learn about the necessary commands by selecting the subject section in the menu on the left or using the alphabetical list below. + +Any command can be run from the command line with the `--help` option to get help on it. You can get a list of all supported {{ ydb-short-name }} CLI of commands by running the {{ ydb-short-name }} CLI with the `--help` option [with no command specified](../commands/service.md). + +| Command / subcommand | Brief description | +| --- | --- | +| [config profile activate](../profile/activate.md) | Activating a [profile](../profile/index.md) | +| [config profile create](../profile/create.md) | Creating a [profile](../profile/index.md) | +| [config profile delete](../profile/create.md) | Deleting a [profile](../profile/index.md) | +| [config profile get](../profile/list-and-get.md) | Getting parameters of a [profile](../profile/index.md) | +| [config profile list](../profile/list-and-get.md) | List of [profiles](../profile/index.md) | +| [config profile set](../profile/activate.md) | Activating a [profile](../profile/index.md) | +| [discovery list](../commands/discovery-list.md) | List of endpoints | +| [discovery whoami](../commands/discovery-whoami.md) | Authentication | +| export s3 | Exporting data to S3 storage | +| import file csv | Importing data from a CSV file | +| import file tsv | Importing data from a TSV file | +| import s3 | Importing data from S3 storage | +| [init](../profile/create.md) | Initializing the CLI, creating a [profile](../profile/index.md) | +| operation cancel | Interrupting a long-running operation | +| operation forget | Deleting a long-running operation from the history | +| operation get | Getting the status of a long-running operation | +| operation list | List of long-running operations | +| [scheme describe](../commands/scheme-describe.md) | Description of a data schema object | +| [scheme ls](../commands/scheme-ls.md) | List of data schema objects | +| [scheme mkdir](../commands/dir.md#mkdir) | Creating a directory | +| scheme permissions add | Granting permissions | +| scheme permissions chown | Changing the owner of an object | +| scheme permissions clear | Clearing permissions | +| scheme permissions grant | Granting permissions | +| scheme permissions remove | Removing a permission | +| scheme permissions revoke | Revoking a permission | +| scheme permissions set | Setting permissions | +| [scheme rmdir](../commands/dir.md#rmdir) | Deleting a directory | +| scripting yql | Executing a YQL script | +| table attribute add | Adding a table attribute | +| table attribute drop | Deleting a table attribute | +| table drop | Deleting a table | +| [table index add global](../commands/index-ops.md) | Adding a synchronous index | +| [table index add global-async](../commands/index-ops.md) | Adding an asynchronous index | +| [table index add global-sync](../commands/index-ops.md) | Adding a synchronous index | +| [table index drop](../commands/index-ops.md) | Deleting an index | +| [table query execute](../commands/query.md) | Executing a YQL query | +| [table query explain](../commands/explain-plan.md) | YQL query execution plan | +| [table readtable](../commands/readtable.md) | Streaming table reads | +| table ttl drop | Deleting TTL parameters | +| table ttl set | Setting TTL parameters | +| tools copy | Copying tables | +| tools dump | Dumping a directory or table to the file system | +| [tools rename](../commands/tools/rename.md) | Renaming tables | +| tools restore | Restoring data from the file system | + +{% if ydb-cli == "ydb" %} +[update](../commands/service.md) | Updating the {{ ydb-short-name }} CLI +[version](../commands/service.md) | Displaying the version of the {{ ydb-short-name }} CLI +{% endif %} +[workload](../commands/workload/index.md) | Generating YQL load | Running a YQL script (with streaming support) + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md b/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md new file mode 100644 index 0000000000..bb1aba7b43 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md @@ -0,0 +1,70 @@ +# Connecting the CLI to and authenticating with a database + +Most of the {{ ydb-short-name }} CLI commands relate to operations on a {{ ydb-short-name }} database and require establishing a connection to it to be executed. + +The {{ ydb-short-name }} CLI determines what DB to connect to and what [authentication mode](../../../concepts/connect.md#auth-modes) to use based on information from the following sources (in order of priority): + +1. The command line. +2. The profile set in the `--profile` command-line option. +3. Environment variables. +4. The activated profile. + +For the {{ ydb-short-name }} CLI to try connecting to the DB after you completed these steps, make sure to specify the [Endpoint](../../../concepts/connect.md#endpoint) and [Database location](../../../concepts/connect.md#database). + +If all the steps are completed, but the {{ ydb-short-name }} CLI did not determine the authentication mode, requests will be sent to the {{ ydb-short-name }} server without adding authentication data. This may let you successfully work with locally deployed {{ ydb-short-name }} clusters that require no authentication. For all databases available over the network, such requests will be rejected by the server with an authentication error returned. + +For possible situations when the {{ ydb-short-name }} CLI will not try to connect to a database, see the [Error messages](#errors) section below. + +## Command line parameters {#command-line-pars} + +DB connection options in the command line are specified before defining the command and its parameters: + +```bash +{{ ydb-cli }} <connection_options> <command> <command_options> +``` + +- `-e, --endpoint <endpoint>` : [Endpoint](../../../concepts/connect.md#endpoint): The main connection parameter that allows finding the {{ ydb-short-name }} server on the network. If no port is specified, port 2135 is used. If no protocol is specified, gRPCs (with encryption) is used in {{ ydb-short-name }} CLI public builds. +- `-d, --database <database>` : [DB location](../../../concepts/connect.md#database). + +{% include [auth/options.md](auth/options.md) %} + +## Parameters from the profile set by the command-line option {#profile} + +If a certain connection parameter is not specified in the command line when calling the {{ ydb-short-name }} CLI, it tries to determine it by the [profile](../profile/index.md) set in the `--profile` command-line option. + +The profile may define most variables similar to the options from the [Command line parameters](#command-line-pars) section. Their values are processed in the same way as command line parameters. + +## Parameters from environment variables {#env} + +If the profile is not explicitly specified in the command line or the authentication parameters are not set in it, the {{ ydb-short-name }} CLI tries to determine the authentication mode and parameters by the {{ ydb-short-name }} CLI environment as follows: + +{% include [env.md](auth/env.md) %} + +## Parameters from the activated profile {#activated-profile} + +If any connection parameter could not be determined in the previous steps and the profile was not explicitly specified in the command line with the `--profile` option, the {{ ydb-short-name }} CLI tries to use the connection parameters from the [activated profile](../profile/activate.md). + +## Error messages {#errors} + +### Errors before attempting to establish a DB connection + +If all the steps described in the beginning of this article are completed, but the [Endpoint](../../../concepts/connect.md#endpoint) is not determined, the command is aborted and an error message saying `Missing required option 'endpoint'` is returned. + +If all the steps described in the beginning of this article are completed, but the [DB location](../../../concepts/connect.md#database) is not identified, the command is aborted and an error message saying `Missing required option 'database'` is returned. + +If the authentication mode is known, but the necessary additional parameters are not, the command is aborted and an error message describing the issue is returned: + +- `(No such file or directory) util/system/file.cpp:857: can't open "<filepath>" with mode RdOnly|Seq (0x00000028)`: Couldn't open and read the `<filepath>` file specified in one of the parameters with the file name and path. + +## Additional parameters {#additional} + +When using gRPCs (with encryption), you may need to [select a root certificate](../../../concepts/connect.md#tls-cert): + +- `--ca-file <filepath>` : Root certificate PEM file for a TLS connection. + +Currently, root certificates are not stored in profiles and can only be defined by command line options. + +## Authentication {#whoami} + +The {{ ydb-short-name }} CLI [`discovery whoami`](../commands/discovery-whoami.md) auxiliary command lets you check the account that you actually used to authenticate with the server. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/_includes/index.md new file mode 100644 index 0000000000..6d34a1e7db --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/index.md @@ -0,0 +1,21 @@ +# {{ ydb-short-name }} CLI + +The {{ ydb-short-name }} CLI provides software for managing your data in {{ ydb-short-name }}. + +To learn how to use the {{ ydb-short-name }} CLI, see [Installing the {{ ydb-short-name }} CLI](../install.md). + +When connecting to a database, you need to authenticate. For the first connection, you can use the quick recipe from the [Authentication](../../../getting_started/auth.md) article in the "Getting started" section. + +Full information about defining DB connection and authentication parameters is given in the [Connecting to a database and authenticating with the {{ ydb-short-name }} CLI](../connect.md) article in this section. + +For a full description of {{ ydb-short-name }} CLI commands, see the following articles of this section: + +* [List of objects](../commands/scheme-ls.md). +* [Getting information about schema objects](../commands/scheme-describe.md). +* [Working with directories](../commands/dir.md). +* [Making a DB query](../commands/query.md). +* [Streaming table reads](../commands/readtable.md). +* [Working with secondary indexes](../commands/index-ops.md). +* [Getting a list of DB endpoints](../commands/discovery-list.md). +* [Load testing](../commands/workload/index.md). + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/install.md b/ydb/docs/en/core/reference/ydb-cli/_includes/install.md new file mode 100644 index 0000000000..87c5f29882 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/install.md @@ -0,0 +1,78 @@ +# Installing the {{ ydb-short-name }} CLI + +{% include [install_overlay.md](install_overlay.md) %} + +{% list tabs %} + +- Linux + + To install the {{ ydb-short-name }} CLI, run the command: + + ```bash + curl https://storage.yandexcloud.net/yandexcloud-ydb/install.sh | bash + ``` + + The script will install the {{ ydb-short-name }} CLI and add the executable file path to the `PATH` environment variable. + + {% note info %} + + The script will update the `PATH` variable only if you run it in the bash or zsh command shell. If you run the script in a different shell, add the path to the CLI to the `PATH` variable yourself. + + {% endnote %} + + To update the environment variables, restart the command shell. + +- macOS + + To install the {{ ydb-short-name }} CLI, run the command: + + ```bash + curl https://storage.yandexcloud.net/yandexcloud-ydb/install.sh | bash + ``` + + The script will install the {{ ydb-short-name }} CLI and add the executable file path to the `PATH` environment variable. + + To update the environment variables, restart the command shell. + +- Windows + + You can install the {{ ydb-short-name }} CLI using: + + * PowerShell. To do this, run the command: + + ```powershell + iex (New-Object System.Net.WebClient).DownloadString('https://storage.yandexcloud.net/yandexcloud-ydb/install.ps1') + ``` + + Specify whether to add the executable file path to the `PATH` environment variable: + + ```text + Add ydb installation dir to your PATH? [Y/n] + ``` + + * cmd. To do this, run the command: + + ```cmd + @"%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://storage.yandexcloud.net/yandexcloud-ydb/install.ps1'))" + ``` + + Specify whether to add the executable file path to the `PATH` environment variable: + + ```text + Add ydb installation dir to your PATH? [Y/n] + ``` + + To update the environment variables, restart the command shell. + + {% note info %} + + The {{ ydb-short-name }} CLI uses Unicode characters in the output of some commands. If these characters aren't displayed correctly in the Windows console, switch the encoding to UTF-8: + + ```cmd + chcp 65001 + ``` + + {% endnote %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/install_overlay.md b/ydb/docs/en/core/reference/ydb-cli/_includes/install_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/install_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-cli/commands.md b/ydb/docs/en/core/reference/ydb-cli/commands.md new file mode 100644 index 0000000000..67f9942c09 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands.md @@ -0,0 +1,2 @@ +{% include [commands.md](_includes/commands.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/commands/all-output.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/commands/all-output.md index 46cb9435e4..4f110fab3d 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/commands/all-output.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/commands/all-output.md @@ -85,14 +85,14 @@ Options: --ca-file PATH Path to a file containing the PEM encoding of the server root certificates for tls connections. If this parameter is empty, the default roots will be used. --iam-token-file PATH IAM token file. Note: IAM tokens expire in 12 hours. - For more info go to: cloud.yandex.ru/docs/iam/concepts/authorization/iam-token + For more info go to: cloud.yandex.com/en/docs/iam/concepts/authorization/iam-token Token search order: 1. This option 2. Profile specified with --profile option 3. "IAM_TOKEN" environment variable 4. Active configuration profile --yc-token-file PATH YC token file. It should contain OAuth token of a Yandex Passport user to get IAM token with. - For more info go to: cloud.yandex.ru/docs/iam/concepts/authorization/oauth-token + For more info go to: cloud.yandex.com/en/docs/iam/concepts/authorization/oauth-token Token search order: 1. This option 2. Profile specified with --profile option @@ -100,14 +100,14 @@ Options: 4. Active configuration profile --use-metadata-credentials Use metadata service on a virtual machine to get credentials - For more info go to: cloud.yandex.ru/docs/compute/operations/vm-connect/auth-inside-vm + For more info go to: cloud.yandex.com/en/docs/compute/operations/vm-connect/auth-inside-vm Definition priority: 1. This option 2. Profile specified with --profile option 3. "USE_METADATA_CREDENTIALS" environment variable 4. Active configuration profile (default: 0) --sa-key-file PATH Service account key file - For more info go to: cloud.yandex.ru/docs/iam/operations/iam-token/create-for-sa + For more info go to: cloud.yandex.com/en/docs/iam/operations/iam-token/create-for-sa Definition priority: 1. This option 2. Profile specified with --profile option diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md new file mode 100644 index 0000000000..54b58c7c1e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md @@ -0,0 +1,2 @@ +where [connection options] are [database connection options](../../connect.md#command-line-pars) + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md new file mode 100644 index 0000000000..bd19db854c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md @@ -0,0 +1,85 @@ +# Directories + +The {{ ydb-short-name }} database maintains an internal hierarchical structure of [directories](../../../../concepts/datamodel.md#dir) that can host database objects. + +{{ ydb-short-name }} CLI supports operations to change the directory structure and to access schema objects by their directory name. + +## Creating a directory {#mkdir} + +The `scheme mkdir` command creates the directories: + +```bash +{{ ydb-cli }} [connection options] scheme mkdir <path> +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +In the `path` parameter, specify the relative path to the directory being created, from the root database directory. This command creates all the directories that didn't exist at the path when the command was called. + +If the destination directory had already existed at the path, then the command execution will be completed successfully (result code 0) with a warning that no changes have been made: + +```text +Status: SUCCESS +Issues: +<main>: Error: dst path fail checks, path: /<database>/<path>: path exist, request accepts it, +pathId: [OwnerId: <some>, LocalPathId: <some>], path type: EPathTypeDir, path state: EPathStateNoChanges +``` + +The full path syntax starting with a `/` character is also supported. The full path must begin with the [database location](../../../../concepts/connect.md#database) specified in the connection parameters or with which operations are allowed via the established connection to the cluster. + +Examples: + +- Creating a directory at the database root + + ```bash + {{ ydb-cli }} --profile db1 scheme mkdir dir1 + ``` + +- Creating directories at the specified path from the database root + + ```bash + {{ ydb-cli }} --profile db1 scheme mkdir dir1/dir2/dir3 + ``` + +## Deleting a directory {#rmdir} + +The `scheme rmdir` command deletes a directory: + +```bash +{{ ydb-cli }} [connection options] scheme rmdir <path> +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +In the `path` parameter, specify the relative path to the directory to be deleted. This directory must not contain objects (including tables and subdirectories), otherwise the command will fail with an error: + +```text +Status: SCHEME_ERROR +Issues: +<main>: Error: path table fail checks, path: /<database>/<path>: path has children, request +doesn't accept it, pathId: [OwnerId: <some>, LocalPathId: <some>], path type: +EPathTypeDir, path state: EPathStateNoChanges, alive children: <count> +``` + +## Using directories in other CLI commands {#use} + +In all CLI commands to which the object name is passed by the parameter, it can be specified with a directory, for example, in [`scheme describe`](../scheme-describe.md): + +```bash +{{ ydb-cli }} --profile db1 scheme describe dir1/table_a +``` + +The [`scheme ls`](../scheme-ls.md) command supports passing the path to the directory as a parameter: + +```bash +{{ ydb-cli }} --profile db1 scheme ls dir1/dir2 +``` + +## Using directories in YQL {#yql} + +Names of objects issued in [YQL](../../../../yql/reference/index.md) queries may contain a path to the object directory. This path will be concatenated with the path prefix from the [`TablePathPrefix` pragma](../../../../yql/reference/syntax/pragma.md#table-path-prefix). If the pragma is omitted, the object name is resolved relative to the database root. + +## Implicit creation of directories during import {#import} + +The data import command creates a directory tree mirroring the original imported catalog. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md new file mode 100644 index 0000000000..2f1b93166e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md @@ -0,0 +1,27 @@ +# List of endpoints + +The `discovery list` information command lets you get a list of [endpoints](../../../../concepts/connect.md#endpoint) for the {{ ydb-short-name }} cluster that you can establish connections with in order to access your database: + +```bash +{{ ydb-cli }} [connection options] discovery list +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +The output rows in the response contain the following information: + +1. Endpoint, including protocol and port +2. Availability zone (in square brackets) +3. The `#` character is used for the list of {{ ydb-short-name }} services available on this endpoint + +An endpoint discovery request to the {{ ydb-short-name }} cluster is executed in the {{ ydb-short-name }} SDK at driver initialization so that you can use the `discovery list` CLI command to localize connection issues. + +## Example + +```bash +$ ydb --profile db1 discovery list +grpcs://vm-etn01q5-ysor.etn01q5k.ydb.mdb.yandexcloud.net:2135 [sas] #table_service #scripting #discovery #rate_limiter #locking #kesus +grpcs://vm-etn01q5-arum.etn01ftr.ydb.mdb.yandexcloud.net:2135 [vla] #table_service #scripting #discovery #rate_limiter #locking #kesus +grpcs://vm-etn01q5beftr.ydb.mdb.yandexcloud.net:2135 [myt] #table_service #scripting #discovery #rate_limiter #locking #kesus +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md new file mode 100644 index 0000000000..a3b4ede641 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md @@ -0,0 +1,25 @@ +# Authentication + +The `discovery whoami` information command lets you check the account on behalf of which the server actually accepts requests: + +```bash +{{ ydb-cli }} [connection options] discovery whoami [-g] +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +The response includes the account name (User SID) and, if the `-g` option is specified, the information whether the account belongs to groups. + +If authentication is not enabled on the {{ ydb-short-name }} server (for example, in the case of an independent local deployment), the command will fail with an error. + +Support for the `-g` option depends on the server configuration. If disabled, you'll receive `User has no groups` in response, regardless of the actual inclusion of your account in any groups. + +## Example + +```bash +$ ydb --profile db1 discovery whoami -g +User SID: aje5kkjdgs0puc18976co@as + +User has no groups +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md new file mode 100644 index 0000000000..44690bd9b8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md @@ -0,0 +1,11 @@ +# Global parameters + +## DB connection options {#connection-options} + +DB connection options are described in [Connecting to and authenticating with a database](../../connect.md#command-line-pars). + +## Service options {#service-options} + +- `--profile <name>` : Indicates the use of the DB connection profile with the specified name when executing a {{ ydb-short-name }} CLI command. Most connection parameters can be stored in the profile. +- `-v, --verbose` : Outputs detailed information about all operations being executed. Specifying this option is helpful when locating DB connection issues. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md new file mode 100644 index 0000000000..3063baf9b6 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md @@ -0,0 +1,73 @@ +# List of objects + +The `scheme ls` command lets you get a list of objects in the database: + +```bash +{{ ydb-cli }} [connection options] scheme ls [path] [-lR] +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +Executing the command without parameters produces a compressed list of object names in the database's root directory. + +In the `path` parameter, you can specify the [directory](../dir.md) you want to list objects in. + +The following options are available for the command: + +- `-l` : Full details about attributes of each object +- `-R` : Recursive traversal of all subdirectories + +**Examples** + +- Getting objects from the root database directory in a compressed format + +```bash +{{ ydb-cli }} --profile db1 scheme ls +``` + +- Getting objects in all database directories in a compressed format + +```bash +{{ ydb-cli }} --profile db1 scheme ls -R +``` + +- Getting objects from the given database directory in a compressed format + +```bash +{{ ydb-cli }} --profile db1 scheme ls dir1 +{{ ydb-cli }} --profile db1 scheme ls dir1/dir2 +``` + +- Getting objects in all subdirectories in the given directory in a compressed format + +```bash +{{ ydb-cli }} --profile db1 scheme ls dir1 -R +{{ ydb-cli }} --profile db1 scheme ls dir1/dir2 -R +``` + +- Getting complete information on objects in the root database directory + +```bash +{{ ydb-cli }} --profile db1 scheme ls -l +``` + +- Getting complete information about objects in a given database directory + +```bash +{{ ydb-cli }} --profile db1 scheme ls dir1 -l +{{ ydb-cli }} --profile db1 scheme ls dir2/dir3 -l +``` + +- Getting complete information about objects in all database directories + +```bash +{{ ydb-cli }} --profile db1 scheme ls -lR +``` + +- Getting complete information on objects in all subdirectories of a given database directory + +```bash +{{ ydb-cli }} --profile db1 scheme ls dir1 -lR +{{ ydb-cli }} --profile db1 scheme ls dir2/dir3 -lR +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls/intro.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls/intro.md index 40464ef169..3aa35df5e2 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls/intro.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls/intro.md @@ -1,6 +1,6 @@ # Listing objects -To get a listing of objects, use the `scheme ls <Path>`. If you don't specify the path, a listing of the DB root will be output: +To get a listing of objects, use the `scheme ls <Path>` subcommand. If you don't specify the path, a listing of the DB root will be output: ```bash {{ ydb-cli }} scheme ls diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md new file mode 100644 index 0000000000..1ca21de5be --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md @@ -0,0 +1,14 @@ +# Service commands + +These commands relate to the {{ ydb-short-name }} CLI itself and do not involve establishing a DB connection. They can be expressed either as a parameter or as an option. + +| Name | Description | +| --- | --- | +| `-?`, `-h`, `--help` | Output the {{ ydb-short-name }} CLI syntax help | +| `Version` | Output the {{ ydb-short-name }} CLI version (for public builds) | +| `update` | Update the {{ ydb-short-name }} CLI to the latest version (for public builds) | +| `--license` | Show the license (for public builds) | +| `--credits` | Show third-party product licenses (for public builds) | + +If it is not known whether the used {{ ydb-short-name }} CLI build is public, you can find out if a particular service command is supported through help output. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/copy.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/copy.md index 483618514f..b4912403e9 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/copy.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/copy.md @@ -24,7 +24,7 @@ View a description of the command to copy a table: | `--client-timeout <value>` | Client side operation timeout, ms. | | `--operation-timeout <value>` | Server side operation timeout, ms. | | `--cancel-after <value>` | Operation lifetime, after which the operation is canceled, ms. | -| `--item <value>=<value>,...` | Operation parameters. Possible values:<br/><ul><li>`destination`, `dst`, or `d` — required parameter, the path of the destination table. If the destination path contains folders, they must be created in advance. The table with the destination name should not exist.</li><li>`source`, `src`, or `s` — required parameter, the path of the source table.</li></ul> | +| `--item <property>=<value>,...` | Operation parameters. Possible values:<br/><ul><li>`destination`, `dst`, or `d` — required parameter, the path of the destination table. If the destination path contains folders, they must be created in advance. The table with the destination name should not exist.</li><li>`source`, `src`, or `s` — required parameter, the path of the source table.</li></ul> | ## Examples {#examples} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md index 08af088704..ee82fd9b59 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md @@ -1,11 +1,11 @@ # Renaming a table -Using the `tools rename` subcommand, you can rename one or more tables at the same time, move a table to a different directory within the same database, or replace one table with another within the same transaction. +Using the `tools rename` subcommand, you can [rename](../../../../../concepts/datamodel.md#rename) one or more tables at the same time, move a table to another directory within the same database, replace one table with another one within the same transaction. General command format: ```bash -ydb [global options...] tools rename [options...] +{{ ydb-cli }} [global options...] tools rename [options...] ``` * `global options`: [Global parameters](../../../commands/global-options.md). @@ -14,147 +14,58 @@ ydb [global options...] tools rename [options...] View a description of the command to rename a table: ```bash -ydb tools rename --help +{{ ydb-cli }} tools rename --help ``` ## Subcommand parameters {#options} +A single run of the `tools rename` command executes a single rename transaction that may include one or more operations to rename different tables. + | Parameter name | Parameter description | | --- | --- | +| `--item <property>=<value>,...` | Description of the rename operation. Can be specified multiple times if multiple rename operations need to be executed within a single transaction.</br></br>Required properties:</br><ul><li>`source`, `src`, and `s`: Path to the source table.</li><li>`destination`, `dst`, and `d`: Path to the destination table. If the destination path contains folders, they must be [created in advance](../../dir.md#mkdir).</li></ul>Advanced properties:</br><ul> <li>`replace`, `force`: Overwrite the destination table. If `True`, the destination table is overwritten with its data deleted. `False`: If the destination table exists, an error is returned and the entire rename transaction is canceled. Default value: `False`.</li></ul> | | `--timeout <value>` | Operation timeout, ms. | -| `--item <value>=<value>,...` | Operation parameters. Possible values:<br/><ul><li>`destination`, `dst`, or `d`: Required parameter, the path to the destination table. If the destination path contains directories, they must be created in advance.</li> <li>`source`, `src`, or `s`: Required parameter, the path of the source table.</li><li>`replace`, `force`: Optional parameter. If the value is `True`, the destination table must exist and will be overwritten. `False`: The destination table must not exist. Default value: `False`.</li></ul> | - -## Examples {#examples} - -### Renaming a table {#rename} - -The database contains a `new-project` directory with tables such as `main_table`, `second_table`, and `third_table`. - -Rename the tables: - -```bash -{{ ydb-cli }} tools rename \ - --item source=new-project/main_table,destination=new-project/episodes \ - --item source=new-project/second_table,destination=new-project/seasons \ - --item source=new-project/third_table,destination=new-project/series -``` - -Get a listing of objects: - -```bash -ydb scheme ls new-project -``` - -Result: - -```text -episodes seasons series -``` - -### Moving tables {#move} - -The database contains a `new-project` directory with tables such as `main_table`, `second_table`, and `third_table`. - -Create a directory: - -```bash -{{ ydb-cli }} scheme mkdir cinema -``` - -Rename the tables and move them to the created directory: - -```bash -{{ ydb-cli }} tools rename \ - --item source=new-project/main_table,destination=cinema/episodes \ - --item source=new-project/second_table,destination=cinema/seasons \ - --item source=new-project/third_table,destination=cinema/series -``` - -Get a listing of objects in the created directory: - -```bash -ydb scheme ls cinema -``` - -Result: - -```text -episodes seasons series -``` -### Replacing a table {#replace} +When including multiple rename operations in a single `tools rename` call, they're executed in the specified order, but within a single transaction. This lets you rotate the table under load without data loss: the first operation is renaming the working table to the backup one and the second is renaming the new table to the working one. -The database contains a `prod-project` directory with tables such as `main_table` and `other_table`, and a `pre-prod-project` directory with tables such as `main_table` and `other_table`. - -Replace the `main_table` in the `prod-project` directory with the same-name table from the `pre-prod-project` directory: - -```bash -{{ ydb-cli }} tools rename \ - --item replace=True,source=pre-prod-project/main_table,destination=prod-project/main_table -``` - -Get a listing of `prod-project` directory objects: - -```bash -ydb scheme ls prod-project -``` - -Result: - -```text -main_table other_table -``` - -Get a listing of `pre-prod-project` directory objects: - -```bash -ydb scheme ls pre-prod-project -``` - -Result: - -```text -other_table -``` - -The `pre-prod-project` directory no longer contains the `main_table`. - -### Replacing a table without deleting it {#switch} - -The database contains a `prod-project` directory with tables such as `main_table` and `other_table`, and a `pre-prod-project` directory with tables such as `main_table` and `other_table`. - -Move the `prod-project/main_table` to the `prod-project/main_table.backup` and the `pre-prod-project/main_table` to the `prod-project/main_table`: - -```bash +## Examples {#examples} -{{ ydb-cli }} tools rename \ - --item source=prod-project/main_table,destination=prod-project/main_table.backup \ - --item source=pre-prod-project/main_table,destination=prod-project/main_table -``` +- Renaming a single table: -Get a listing of `prod-project` directory objects: + ```bash + {{ ydb-cli }} tools rename --item src=old_name,dst=new_name + ``` -```bash -ydb scheme ls prod-project -``` +- Renaming multiple tables within a single transaction: -Result: + ```bash + {{ ydb-cli }} tools rename \ + --item source=new-project/main_table,destination=new-project/episodes \ + --item source=new-project/second_table,destination=new-project/seasons \ + --item source=new-project/third_table,destination=new-project/series + ``` -```text -main_table other_table main_table.backup -``` +- Moving tables to a different directory: -Get a listing of `pre-prod-project` directory objects: + ```bash + {{ ydb-cli }} tools rename \ + --item source=new-project/main_table,destination=cinema/main_table \ + --item source=new-project/second_table,destination=cinema/second_table \ + --item source=new-project/third_table,destination=cinema/third_table + ``` -```bash -ydb scheme ls pre-prod-project -``` +- Replacing a table -Result: + ```bash + {{ ydb-cli }} tools rename \ + --item replace=True,source=pre-prod-project/main_table,destination=prod-project/main_table + ``` -```text -other_table -``` +- Rotating a table -The `pre-prod-project` directory no longer contains the `main_table`. + ```bash + {{ ydb-cli }} tools rename \ + --item source=prod-project/main_table,destination=prod-project/main_table.backup \ + --item source=pre-prod-project/main_table,destination=prod-project/main_table + ``` diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md index d8875bd13f..42cd5c6aa3 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md @@ -19,14 +19,15 @@ View a description of the command to restore data from a dump: ## Parameters of the subcommand {#options} -Parameter name | Parameter description ---- | --- -`-p`<br/>`--path` | The path to the folder or table to dump.<br/>Default value is `.`, a full database dump will be performed. -`-o`<br/>`--output` | Required parameter.<br/>The path on the local file system where the dump objects will be placed. The dump folder must not exist or must be empty. -`--scheme-only` | Make a dump of the DB schema only. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. -`--avoid-copy` | Avoid copying. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. -`--save-partial-result` | Save the results of an incomplete dump. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. -`--consistency-level` | Consistency level. Possible values:<br/><ul><li>`database`: Consistency at the DB level.</li><li>`table`: Consistency at the table level.</li>Default value is `database`. +| Parameter name | Parameter description | +| --- | --- | +| `-p`<br/>`--path` | The path to the folder or table to dump.<br/>Default value is `.`, a full database dump will be performed. | +| `-o`<br/>`--output` | Required parameter.<br/>The path on the local file system where the dump objects will be placed. The dump folder must not exist or must be empty. | +| `--scheme-only` | Make a dump of the DB schema only. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | +| `--avoid-copy` | Avoid copying. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | +| `--save-partial-result` | Save the results of an incomplete dump. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | +| `--consistency-level` | Consistency level. Possible values:<br/><ul><li>`database`: Consistency at the DB level.</li><li>`table`: Consistency at the table level.</li>Default value is `database`. | + `-p <value>`<br/>`--path <value>` | Required parameter.<br/>The path in the DB by which the folder or table is restored. `-o <value>`<br/>`--output <value>` | Required parameter.<br/>The path on the local file system where the dump objects are located. `--dry-run` | Do not restore tables. Make sure that:<br/><ul><li>— All dump tables are present in the DB.</li><li>— Schemas of all dump tables correspond to schemas of DB tables. diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/dir.md b/ydb/docs/en/core/reference/ydb-cli/commands/dir.md new file mode 100644 index 0000000000..adee551a77 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/dir.md @@ -0,0 +1 @@ +{% include [dir.md](_includes/dir.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/discovery-list.md b/ydb/docs/en/core/reference/ydb-cli/commands/discovery-list.md index d73a84974b..37ab1db809 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/discovery-list.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/discovery-list.md @@ -1 +1 @@ -{% include [intro.md](_includes/discovery-list/intro.md) %} +{% include [discovery-list.md](_includes/discovery-list.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/discovery-whoami.md b/ydb/docs/en/core/reference/ydb-cli/commands/discovery-whoami.md new file mode 100644 index 0000000000..0060ce70d9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/discovery-whoami.md @@ -0,0 +1 @@ +{% include [discovery-whoami.md](_includes/discovery-whoami.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/global-options.md b/ydb/docs/en/core/reference/ydb-cli/commands/global-options.md index e69de29bb2..674bfa2bc5 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/global-options.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/global-options.md @@ -0,0 +1 @@ +{% include [global-options.md](_includes/global-options.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/index-ops.md b/ydb/docs/en/core/reference/ydb-cli/commands/index-ops.md new file mode 100644 index 0000000000..ea72a9c399 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/index-ops.md @@ -0,0 +1,13 @@ +{% include [intro.md](_includes/operations-index/intro.md) %} + +{% include [operation-index.md](_includes/operations-index/operation-index.md) %} + +{% include [operations-index-exp.md](_includes/operations-index/operations-index-exp.md) %} + +{% include [one-get.md](_includes/operations-index/one-get.md) %} + +{% include [one-get-exp.md](_includes/operations-index/one-get-exp.md) %} + +{% include [all-get.md](_includes/operations-index/all-get.md) %} + +{% include [delete-index.md](_includes/operations-index/delete-index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md index d00a69f08e..e5cacf169f 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md @@ -1 +1 @@ -{% include [intro.md](_includes/scheme-ls/intro.md) %} +{% include [intro.md](_includes/scheme-ls.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md index 0f08eef277..180e27a7ca 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md @@ -1 +1,2 @@ -{% include [intro.md](_includes/scheme-mkdir/intro.md) %} +This page has been deleted, the content has been moved to a [new section](dir.md). + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/service.md b/ydb/docs/en/core/reference/ydb-cli/commands/service.md new file mode 100644 index 0000000000..40d156acf2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/service.md @@ -0,0 +1 @@ +{% include [service.md](_includes/service.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md new file mode 100644 index 0000000000..e1430ca139 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md @@ -0,0 +1,25 @@ +# Load testing + +You can use the `workload` command to run different types of workload against your DB. + +General command format: + +```bash +{{ ydb-cli }} [global options...] workload [subcommands...] +``` + +* `global options`: [Global parameters](../../../commands/global-options.md). +* `subcommands`: [Subcommands](#subcomands). + +See the description of the command to run the data load: + +```bash +{{ ydb-cli }} workload --help +``` + +## Available subcommands {#subcommands} + +The following types of load tests are supported at the moment: + +* [Stock](../stock.md): An online store warehouse simulator. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md new file mode 100644 index 0000000000..9ef2a6d666 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md @@ -0,0 +1,367 @@ +# Stock load + +Simulates a warehouse of an online store: creates multi-product orders, gets a list of orders per customer. + +## Types of load {#workload_types} + +This load test runs 5 types of load: + +* [getCustomerHistory](#getCustomerHistory) reads the specified number of orders for the customer with id = 10000. This creates a workload to read the same rows from different threads. +* [getRandomCustomerHistory](#getRandomCustomerHistory) reads the specified number of orders made by a randomly selected customer. A load that reads data from different threads is created. +* [insertRandomOrder](#insertRandomOrder) creates a random order. For example, a customer has created an order of 2 products, but hasn't yet paid for it, hence the quantities in stock aren't decreased for the products. The database writes the data about the order and products. The read/write load is created (the INSERT checks for an existing entry before inserting the data). +* [submitRandomOrder](#submitRandomOrder) creates and processes a randomly generated order. For example, a customer has created and paid an order of 2 products. The data about the order and products is written to the database, product availability is checked and quantities in stock are decreased. A mixed data load is created. +* [submitSameOrder](#submitSameOrder): Creates orders with the same set of products. For example, all customers buy the same set of products (a newly released phone and a charger). This creates a workload of competing updates of the same rows in the table. + +## Load test initialization + +To get started, create tables and populate them with data: + +```bash +{{ ydb-cli }} workload stock init [init options...] +``` + +* `init options`: [Initialization options](#init_options). + +See the description of the command to run the data load: + +```bash +{{ ydb-cli }} workload init --help +``` + +### Available parameters {#init_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--products <value>` | `-p <value>` | Number of products. Valid values: between 1 and 500000. The default value is 100. | +| `--quantity <value>` | `-q <value>` | Quantity of each product in stock. Default value: 1000. | +| `--orders <value>` | `-o <value>` | Initial number of orders in the database. The default value is 100. | +| `--min-partitions <value>` | - | Minimum number of shards for tables. Default value: 40. | +| `--auto-partition <value>` | - | Enabling/disabling auto-sharding. Possible values: 0 or 1. Default: 1. | + +3 tables are created using the following DDL statements: + +```sql +CREATE TABLE `stock`(product Utf8, quantity Int64, PRIMARY KEY(product)) WITH (AUTO_PARTITIONING_BY_LOAD = ENABLED, AUTO_PARTITIONING_MIN_PARTITIONS_COUNT = <min-partitions>); +CREATE TABLE `orders`(id Uint64, customer Utf8, created Datetime, processed Datetime, PRIMARY KEY(id), INDEX ix_cust GLOBAL ON (customer, created)) WITH (READ_REPLICAS_SETTINGS = "per_az:1", AUTO_PARTITIONING_BY_LOAD = ENABLED, AUTO_PARTITIONING_MIN_PARTITIONS_COUNT = <min-partitions>, UNIFORM_PARTITIONS = <min-partitions>, AUTO_PARTITIONING_MAX_PARTITIONS_COUNT = 1000); +CREATE TABLE `orderLines`(id_order Uint64, product Utf8, quantity Int64, PRIMARY KEY(id_order, product)) WITH (AUTO_PARTITIONING_BY_LOAD = ENABLED, AUTO_PARTITIONING_MIN_PARTITIONS_COUNT = <min-partitions>, UNIFORM_PARTITIONS = <min-partitions>, AUTO_PARTITIONING_MAX_PARTITIONS_COUNT = 1000); +``` + +### Load initialization examples {#init-stock-examples} + +Creating a database with 1000 products, 10000 items of each product, and no orders: + +```bash +{{ ydb-cli }} workload stock init -p 1000 -q 10000 -o 0 +``` + +Creating a database with 10 products, 100 items of each product, 10 orders, and a minimum number of shards equal 100: + +```bash +{{ ydb-cli }} workload stock init -p 10 -q 100 -o 10 ----min-partitions 100 +``` + +## Running a load test + +To run the load, execute the command: + +```bash +{{ ydb-cli }} workload stock run [workload type...] [global workload options...] [specific workload options...] +``` + +During this test, workload statistics for each time window are displayed on the screen. + +* `workload type`: The [types of workload](#workload_types). +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: Options of a specific load type. + +See the description of the command to run the data load: + +```bash +{{ ydb-cli }} workload run --help +``` + +### Global parameters for all types of load {#global_workload_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--seconds <value>` | `-s <value>` | Duration of the test, in seconds. Default value: 10. | +| `--threads <value>` | `-t <value>` | The number of parallel threads creating the load. Default value: 10. | +| `--quiet` | - | Outputs only the final test result. | +| `--print-timestamp` | - | Print the time together with the statistics of each time window. | + +## getCustomerHistory load {#getCustomerHistory} + +This type of load reads the specified number of orders for the customer with id = 10000. + +YQL query: + +```sql +DECLARE $cust AS Utf8; +DECLARE $limit AS UInt32; + +SELECT id, customer, created FROM orders view ix_cust + WHERE customer = 'Name10000' + ORDER BY customer DESC, created DESC + LIMIT $limit; +``` + +To run this type of load, execute the command: + +```bash +{{ ydb-cli }} workload stock run getCustomerHistory [global workload options...] [specific workload options...] +``` + +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: [Parameters of a specific type of load](#customer_history_options). + +### Parameters for getCustomerHistory {#customer_history_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--limit <value>` | `-l <value>` | The required number of orders. Default value: 10. | + +## getRandomCustomerHistory load {#getRandomCustomerHistory} + +This type of load reads the specified number of orders from randomly selected customers. + +YQL query: + +```sql +DECLARE $cust AS Utf8; +DECLARE $limit AS UInt32; + +SELECT id, customer, created FROM orders view ix_cust + WHERE customer = $cust + ORDER BY customer DESC, created DESC + LIMIT $limit; +``` + +To run this type of load, execute the command: + +```bash +{{ ydb-cli }} workload stock run getRandomCustomerHistory [global workload options...] [specific workload options...] +``` + +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: [Parameters of a specific type of load](#random_customer_history_options). + +### Parameters for getRandomCustomerHistory {#random_customer_history_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--limit <value>` | `-l <value>` | The required number of orders. Default: 10. | + +## insertRandomOrder load {#insertRandomOrder} + +This type of load creates a randomly generated order. The order includes several different products, 1 item per product. The number of products in the order is generated randomly based on an exponential distribution. + +YQL query: + +```sql +DECLARE $ido AS UInt64; +DECLARE $cust AS Utf8; +DECLARE $lines AS List<Struct<product:Utf8,quantity:Int64>>; +DECLARE $time AS DateTime; + +INSERT INTO `orders`(id, customer, created) VALUES + ($ido, $cust, $time); +UPSERT INTO `orderLines`(id_order, product, quantity) + SELECT $ido, product, quantity FROM AS_TABLE( $lines ); +``` + +To run this type of load, execute the command: + +```bash +{{ ydb-cli }} workload stock run insertRandomOrder [global workload options...] [specific workload options...] +``` + +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: [Parameters of a specific type of load](#insert_random_order_options). + +### Parameters for insertRandomOrder {#insert_random_order_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--products <value>` | `-p <value>` | Number of products in the test. The default value is 100. | + +## submitRandomOrder load {#submitRandomOrder} + +This type of load creates a randomly generated order and processes it. The order includes several different products, 1 item per product. The number of products in the order is generated randomly based on an exponential distribution. Order processing consists in decreasing the number of ordered products in stock. + +YQL query: + +```sql +DECLARE $ido AS UInt64; +DECLARE $cust AS Utf8; +DECLARE $lines AS List<Struct<product:Utf8,quantity:Int64>>; +DECLARE $time AS DateTime; + +INSERT INTO `orders`(id, customer, created) VALUES + ($ido, $cust, $time); + +UPSERT INTO `orderLines`(id_order, product, quantity) + SELECT $ido, product, quantity FROM AS_TABLE( $lines ); + +$prods = SELECT * FROM orderLines AS p WHERE p.id_order = $ido; + +$cnt = SELECT COUNT(*) FROM $prods; + +$newq = + SELECT + p.product AS product, + COALESCE(s.quantity, 0) - p.quantity AS quantity + FROM $prods AS p + LEFT JOIN stock AS s + ON s.product = p.product; + +$check = SELECT COUNT(*) AS cntd FROM $newq as q WHERE q.quantity >= 0; + +UPSERT INTO stock + SELECT product, quantity FROM $newq WHERE $check=$cnt; + +$upo = SELECT id, $time AS tm FROM orders WHERE id = $ido AND $check = $cnt; + +UPSERT INTO orders SELECT id, tm AS processed FROM $upo; + +SELECT * FROM $newq AS q WHERE q.quantity < 0 +``` + +To run this type of load, execute the command: + +```bash +{{ ydb-cli }} workload stock run submitRandomOrder [global workload options...] [specific workload options...] +``` + +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: [Parameters of a specific type of load](#submit_random_order_options). + +### Parameters for submitRandomOrder {#submit_random_order_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--products <value>` | `-p <value>` | Number of products in the test. The default value is 100. | + +## submitSameOrder load {#submitSameOrder} + +This type of load creates an order with the same set of products and processes it. Order processing consists in decreasing the number of ordered products in stock. + +YQL query: + +```sql +DECLARE $ido AS UInt64; +DECLARE $cust AS Utf8; +DECLARE $lines AS List<Struct<product:Utf8,quantity:Int64>>; +DECLARE $time AS DateTime; + +INSERT INTO `orders`(id, customer, created) VALUES + ($ido, $cust, $time); + +UPSERT INTO `orderLines`(id_order, product, quantity) + SELECT $ido, product, quantity FROM AS_TABLE( $lines ); + +$prods = SELECT * FROM orderLines AS p WHERE p.id_order = $ido; + +$cnt = SELECT COUNT(*) FROM $prods; + +$newq = + SELECT + p.product AS product, + COALESCE(s.quantity, 0) - p.quantity AS quantity + FROM $prods AS p + LEFT JOIN stock AS s + ON s.product = p.product; + +$check = SELECT COUNT(*) as cntd FROM $newq AS q WHERE q.quantity >= 0; + +UPSERT INTO stock + SELECT product, quantity FROM $newq WHERE $check=$cnt; + +$upo = SELECT id, $time AS tm FROM orders WHERE id = $ido AND $check = $cnt; + +UPSERT INTO orders SELECT id, tm AS processed FROM $upo; + +SELECT * FROM $newq AS q WHERE q.quantity < 0 +``` + +To run this type of load, execute the command: + +```bash +{{ ydb-cli }} workload stock run submitSameOrder [global workload options...] [specific workload options...] +``` + +* `global workload options`: The [global options for all types of load](#global_workload_options). +* `specific workload options`: [Parameters of a specific type of load](#submit_same_order_options) + +### Parameters for submitSameOrder {#submit_same_order_options} + +| Parameter name | Short name | Parameter description | +| --- | --- | --- | +| `--products <value>` | `-p <value>` | Number of products per order. The default value is 100. | + +## Examples of running the loads + +* Run the load `insertRandomOrder` for 5 seconds across 10 threads with 1000 products. + +```bash +{{ ydb-cli }} workload stock run insertRandomOrder -s 5 -t 10 -p 1000 +``` + +Possible result: + +```text +Elapsed Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) +1 132 0 0 69 108 132 157 +2 157 0 0 63 88 97 104 +3 156 0 0 62 84 104 120 +4 160 0 0 62 77 90 94 +5 174 0 0 61 77 97 100 + +Txs Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) +779 155.8 0 0 62 89 108 157 +``` + +* Run the `submitSameOrder` load for 5 seconds across 5 threads with 2 products per order, printing out only final results. + +```bash +{{ ydb-cli }} workload stock run submitSameOrder -s 5 -t 5 -p 1000 --quiet +``` + +Possible result: + +```text +Txs Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) +16 3.2 67 3 855 1407 1799 1799 +``` + +* Run the `getRandomCustomerHistory` load for 5 seconds across 100 threads, printing out time for each time window. + +```bash +{{ ydb-cli }} workload stock run getRandomCustomerHistory -s 5 -t 10 --print-timestamp +``` + +Possible result: + +```text +Elapsed Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) Timestamp +1 1046 0 0 7 16 25 50 2022-02-08T17:47:26Z +2 1070 0 0 7 17 22 28 2022-02-08T17:47:27Z +3 1041 0 0 7 17 22 28 2022-02-08T17:47:28Z +4 1045 0 0 7 17 23 31 2022-02-08T17:47:29Z +5 998 0 0 8 18 23 42 2022-02-08T17:47:30Z + +Txs Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) +5200 1040 0 0 8 17 23 50 +``` + +## Interpretation of results + +* `Elapsed`: Time window ID. By default, a time window is 1 second. +* `Txs/sec`: Number of successful load transactions in the time window. +* `Retries`: The number of repeat attempts to execute the transaction by the client in the time window. +* `Errors`: The number of errors that occurred in the time window. +* `p50(ms)`: 50th percentile of request latency, in ms. +* `p95(ms)`: 95th percentile of request latency, in ms. +* `p99(ms)`: 99th percentile of request latency, in ms. +* `pMax(ms)`: 100th percentile of request latency, in ms. +* `Timestamp`: Timestamp of the end of the time window. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/index.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/index.md new file mode 100644 index 0000000000..12643df073 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/index.md @@ -0,0 +1 @@ +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/stock.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/stock.md new file mode 100644 index 0000000000..906d59ba44 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/stock.md @@ -0,0 +1 @@ +{% include [stock.md](_includes/stock.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/connect.md b/ydb/docs/en/core/reference/ydb-cli/connect.md new file mode 100644 index 0000000000..c0cae8a17b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/connect.md @@ -0,0 +1,2 @@ + +{% include [connect.md](_includes/connect.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/index.md b/ydb/docs/en/core/reference/ydb-cli/index.md index b297642e85..b3cf50bcae 100644 --- a/ydb/docs/en/core/reference/ydb-cli/index.md +++ b/ydb/docs/en/core/reference/ydb-cli/index.md @@ -1,7 +1,2 @@ -# {{ ydb-short-name }} CLI -The {{ ydb-short-name }} command-line interface (CLI) provides downloadable software for managing your data in {{ ydb-short-name }}. - -To get started with the {{ ydb-short-name }} CLI: - -1. [Take a look at {{ ydb-short-name }} CLI commands](commands/index.md). +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/install.md b/ydb/docs/en/core/reference/ydb-cli/install.md new file mode 100644 index 0000000000..898729dbbe --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/install.md @@ -0,0 +1,2 @@ + +{% include [install.md](_includes/install.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md index b89dbfd237..51e03e3a69 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md @@ -1,16 +1,83 @@ -# Activating a profile +# Activated profile -{% include [profile-list](profile-list.md) %} +Executing {{ ydb-short-name }} CLI commands on a database require establishing a connection to the database. If the {{ ydb-short-name }} CLI couldn't identify a certain connection parameter by [command-line parameters and environment variables](../../connect.md), it's taken from the activated profile. -Activate the profile with the `example` name: +Profile activation is an easy way to get started with the {{ ydb-short-name }} CLI, since the connection parameters that are set once will be applied automatically to any command, without the need to specify any connection parameters in the command line. + +However, this simplicity may lead to undesirable behavior in further operation, as soon as you need to work with multiple databases: + +- The activated profile is applied implicitly, meaning that it can be applied by mistake when a certain connection parameter is missing in the command line. +- The activated profile is applied implicitly, meaning that it can be applied by mistake when a typo is made in the name of an environment variable. +- The activated profile cannot be used in scripts, since it is saved in a file and its change in one terminal window will affect all other windows, possibly leading to an unexpected change of the DB in the middle of the loop being executed in the script. + +When you need to connect to any new database other than the initial one for the first time, we recommend that you deactivate the profile and always select it explicitly using the `--profile` option. + +## Activating a profile with a command {#activate} + +Profile activation is performed by running the command ```bash -{{ ydb-cli }} config profile activate example +{{ ydb-cli }} config profile activate [profile_name] ``` -Result: +, where `[profile_name]` is an optional profile name. + +If the profile name is specified, it is activated. If a profile with the specified name does not exist, an error is returned prompting you to view the list of available profiles: ```text -Profile "example" was activated. +No existing profile "<profile_name>". Run "ydb config profile list" without arguments to see existing profiles ``` +If the profile name is not specified, you'll be asked to choose between the following options in interactive mode: + +```text +Please choose profile to activate: + [1] Don't do anything, just exit + [2] Deactivate current active profile (if any) + [3] <profile_name_1> (active) + [4] <profile_name_2> + ... +Please enter your numeric choice: +``` + +- `1` terminates the command execution and keeps the currently activated profile activated. It's marked as `(active)` in the list of existing profiles starting from item 3. +- `2` deactivates the currently activated profile. If no profile has been activated before, nothing changes. +- `3` and so on activates the selected profile. The currently activated profile is marked as `(active)`. + +If the profile is successfully activated, the execution ends with a message saying + +```text +Profile "<profile_name>" was activated. +``` + +### Example + +Activating a profile named `mydb1`: + +```bash +$ {{ ydb-cli }} config profile activate mydb1 +Profile "mydb1" was activated. +``` + +## Activating a profile during its initialization {#init} + +As the last step during the interactive execution of the [command to create or update](../create.md) the profile `{{ ydb-cli }} config profile create`, you're prompted to activate the created (or updated) profile: + +```text +Activate profile "<profile_name>" to use by default? (current active profile is not set) y/n: +``` + +Choose `y` (Yes) to activate the profile. + +## Deactivating a profile {#deactivate} + +Currently, the {{ ydb-short-name }} CLI only supports profile deactivation in interactive mode, when calling the activation command without specifying the profile (choosing item `2` in the above [activation command](#activate)). + +If necessary, you can use input redirection in your OS to automatically select option `2` in interactive input: + +```bash +echo 2 | {{ ydb-cli }} config profile activate +``` + +The efficiency of this method is not guaranteed in any way. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md new file mode 100644 index 0000000000..1455fbb0e5 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md @@ -0,0 +1,111 @@ +# Creating and updating profiles + +Currently you can only create and update profiles interactively using the following commands: + +```bash +{{ ydb-cli }} init +``` + +or + +```bash +{{ ydb-cli }} config profile create [profile_name] +``` + +where `[profile_name]` is an optional name of the profile being created or updated. + +The first step of the interactive scenario is different in the `init` and `profile create` commands: + +{% list tabs %} + +- Init + + Outputs a list of existing profiles (if any) and prompts you to make a choice: Create a new or update the configuration of an existing profile: + + ```text + Please choose profile to configure: + [1] Create a new profile + [2] test + [3] local + ``` + + If no profiles exist or you select option `1` in the previous step, the name of a profile to create is requested: + + ```text + Please enter name for a new profile: + ``` + + If you enter the name of an existing profile at this point, the {{ ydb-short-name }} CLI proceeds to updating its parameters as if an option with the name of this profile was selected at once. + +- Profile Create + + If no profile name is specified in the command line, it is requested: + + ```text + Please enter configuration profile name to create or re-configure: + ``` + +{% endlist %} + +Next, you'll be prompted to sequentially perform the following actions with each connection parameter that can be saved in the profile: + +- Don't save +- Set a new value or Use <value> +- Use current value (this option is available when updating an existing profile) + +## Example + +Creating a new `mydb1` profile: + +1. Run the command: + + ```bash + {{ ydb-cli }} config profile create mydb1 + ``` + +1. Specify the [endpoint](../../../../concepts/connect.md#endpoint) or don't save this parameter for the profile: + + ```text + Pick desired action to configure endpoint: + [1] Set a new endpoint value + [2] Don't save endpoint for profile "mydb1" + Please enter your numeric choice: + ``` + +1. Enter the [DB name](../../../../concepts/connect.md#database) or don't save this parameter for the profile: + + ```text + Pick desired action to configure database: + [1] Set a new database value + [2] Don't save database for profile "mydb1" + Please enter your numeric choice: + ``` + +1. Select the authentication mode or don't save this parameter for the profile: + + ```text + Pick desired action to configure authentication method: + [1] Use IAM token (iam-token) cloud.yandex.com/en/docs/iam/concepts/authorization/iam-token + [2] Use OAuth token of a Yandex Passport user (yc-token) cloud.yandex.com/en/docs/iam/concepts/authorization/oauth-token + [3] Use metadata service on a virtual machine (use-metadata-credentials) cloud.yandex.com/en/docs/compute/operations/vm-connect/auth-inside-vm + [4] Use security account key file (sa-key-file) cloud.yandex.com/en/docs/iam/operations/iam-token/create-for-sa + [5] Don't save authentication data for profile "mydb1" + Please enter your numeric choice: + ``` + + If you are not sure what authentication mode to choose, see the [Authentication](../../../../getting_started/auth.md) article in the "Getting started" section. + + All available authentication methods are described in [Connecting to and authenticating with a database](../../../../concepts/connect.md#auth-modes). The set of methods and text of the hints may differ from those given in this example. + + If the method you choose involves specifying an additional parameter, you'll be prompted to enter it. For example, if you select `4` (Use service account key file): + + ```text + Please enter Path to service account key file (sa-key-file): + ``` + +1. In the last step, you'll be prompted to activate the created profile to be used by default. Choose 'n' (No) until you read the article about [Activating a profile and using the activated profile](../activate.md): + + ```text + Activate profile "mydb1" to use by default? (current active profile is not set) y/n: n + ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md index 3192599c7d..e28817bfa4 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md @@ -1,22 +1,38 @@ # Deleting a profile -{% include [profile-list](profile-list.md) %} - -Delete the `example` profile: +Currently, you can only delete profiles interactively with the following command: ```bash -{{ ydb-cli }} config profile delete example +{{ ydb-cli }} config profile delete <profile_name> ``` -Result: +, where `<profile_name>` is the profile name. + +The {{ ydb-short-name }} CLI will request confirmation to delete the profile: ```text -Profile "example" will be permanently removed. Continue? (y/n): +Profile "<profile_name>" will be permanently removed. Continue? (y/n): ``` -Confirm the deletion. Result: +Choose `y` (Yes) to delete the profile. -```text -Profile "example" was removed. +## Example {#example} + +Deleting the `mydb1` profile: + +```bash +$ {{ ydb-cli }} config profile delete mydb1 +Profile "mydb1" will be permanently removed. Continue? (y/n): y +Profile "mydb1" was removed. ``` +## Deleting a profile without interactive input {#non-interactive} + +Although this mode is not supported by the {{ ydb-short-name }} CLI, if necessary, you can use input redirection in your OS to automatically respond `y` to the request to confirm the deletion: + +```bash +echo y | {{ ydb-cli }} config profile delete my_profile +``` + +The efficiency of this method is not guaranteed in any way. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md new file mode 100644 index 0000000000..7c5f9e51fe --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md @@ -0,0 +1,34 @@ +# Managing profiles + +A profile is a saved and locally named configuration of DB connection parameters. With profiles, you can reuse data about DB location and authentication parameters, making a CLI call much shorter: + +- Calling the `scheme ls` command without a profile: + + ```bash + {{ ydb-cli }} \ + -e grpsc://some.host.in.some.domain:2136 \ + -d /some_long_identifier1/some_long_identifier2/database_name \ + --yc-token-file ~/secrets/token_database1 \ + scheme ls + ``` + +- Calling the same `scheme ls` command using a profile: + + ```bash + {{ ydb-cli }} --profile db1 scheme ls + ``` + +## Profile management commands {#commands} + +- [Creating a profile](../create.md) +- [Using a profile](../use.md) +- [Getting a list of profiles and profile parameters](../list-and-get.md) +- [Deleting a profile](../delete.md) +- [Activating a profile and using the activated profile](../activate.md) + +## Where profiles are stored {#location} + +Profiles are stored locally in a file named `~/ydb/config/config.yaml`. + +{% include [location_overlay.md](location_overlay.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md index e43f0df480..91966f6974 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md @@ -1,26 +1,45 @@ # Getting profile information -To specify a profile, use its name. You can get the profile name from the list of profiles. +## Getting a list of profiles {#list} -## Getting a list of profiles {#profile-list} - -Get the list of profiles: +Getting a list of profiles: ```bash {{ ydb-cli }} config profile list ``` -Result: +If there is a currently [activated profile](../activate.md), it will be marked as `(active)` in the output list, for example: ```text -example (active) +prod +test (active) +local ``` -## Getting detailed profile information {#profile-get} +## Getting detailed profile information {#get} + +Getting parameters saved in the specified profile: + +```bash +{{ ydb-cli }} config profile get <profile_name> +``` -Get details about the profile named `example`: +For example: ```bash -{{ ydb-cli }} config profile get example +$ {{ ydb-cli }} config profile get local1 + endpoint: grpcs://ydb.serverless.yandexcloud.net:2135 + database: /rul1/b1g8skp/etn02099 + sa-key-file: /Users/username/secrets/sa_key_test.json ``` +## Getting profiles with content {#get-all} + +Full information on all profiles and parameters stored in them: + +```bash +{{ ydb-cli }} config profile list --with-content +``` + +The output of this command combines the output of the command to get a list of profiles (with the active profile marked) and the parameters of each profile in the lines following its name. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/location_overlay.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/location_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/location_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md new file mode 100644 index 0000000000..cda6440a88 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md @@ -0,0 +1,38 @@ +# Using a profile + +## Connection based on a selected profile {#explicit} + +A profile can be applied when running a {{ ydb-short-name }} CLI command with the `--profile <profile_name>` option specified: + +```bash +{{ ydb-cli }} --profile <profile_name> <command and command options> +``` + +For example: + +```bash +{{ ydb-cli }} --profile mydb1 scheme ls -l +``` + +In this case, all DB connection parameters are taken from the profile. At the same time, if the authentication parameters are not specified in the profile, the {{ ydb-short-name }} CLI will try to define them based on environment variables, as described in [Connecting to and authenticating with a database - Environment variable](../../connect.md#env). + +## Connection based on a selected profile and specified command line parameters {#explicit-and-pars} + +Apart from the `--profile` option, there may be other connection parameters specified in the command line. For example: + +```bash +{{ ydb-cli }} --profile mydb1 -d /local2 scheme ls -l +``` + +```bash +{{ ydb-cli }} --profile mydb1 --user alex scheme ls -l +``` + +In this case, the connection parameters specified in the command line have priority over those stored in the profile. This format lets you reuse profiles to connect to different databases or under different accounts. In addition, specifying the authentication parameter in the command line (such as `--user alex` in the example above) disables environment variable checks regardless of their presence in the profile. + +## Connection based on an activated profile {#implicit} + +If the `--profile` option is not specified in the command line, the {{ ydb-short-name }} CLI will try to take from the currently activated profile all the connection parameters that it couldn't define in other ways (from command-line options or environment variables, as described in [Connecting to and authenticating with a database](../../connect.md)). + +Implicit use of the activated profile may cause errors, so we recommend that you read the [Activated profile](../activate.md) article before using this mode. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/create.md b/ydb/docs/en/core/reference/ydb-cli/profile/create.md new file mode 100644 index 0000000000..6cef4e9340 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/create.md @@ -0,0 +1 @@ +{% include [create.md](_includes/create.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/index.md b/ydb/docs/en/core/reference/ydb-cli/profile/index.md new file mode 100644 index 0000000000..12643df073 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/index.md @@ -0,0 +1 @@ +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/profile/toc_i.yaml new file mode 100644 index 0000000000..a0c3071712 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/toc_i.yaml @@ -0,0 +1,11 @@ +items: +- name: Creating a profile + href: create.md +- name: Using a profile in requests + href: use.md +- name: Getting profile information + href: list-and-get.md +- name: Deleting a profile + href: delete.md +- name: Activated profile + href: activate.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/toc_p.yaml b/ydb/docs/en/core/reference/ydb-cli/profile/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/use.md b/ydb/docs/en/core/reference/ydb-cli/profile/use.md new file mode 100644 index 0000000000..2afc80d711 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/profile/use.md @@ -0,0 +1 @@ +{% include [index.md](_includes/use.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml index 1f77c8f30e..9759a8b45c 100644 --- a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml @@ -1,41 +1,55 @@ items: - - name: Overview - href: index.md - - name: Profile management + - name: Install + href: install.md + - name: Structure of YDB CLI commands + href: commands.md + - name: Service commands + href: commands/service.md + - name: Connecting to and authenticating with a database + href: connect.md + - name: Global parameters + href: commands/global-options.md + - name: Working with the DB schema items: - - name: Getting profile information - href: profile/list-and-get.md - - name: Activating a profile - href: profile/activate.md - - name: Deleting a profile - href: profile/delete.md - - name: Commands + - name: List of objects + href: commands/scheme-ls.md + - name: Information about the object + href: commands/scheme-describe.md + - name: Directories + href: commands/dir.md + - name: Renaming tables + href: commands/tools/rename.md + - name: Operations with data items: - - name: Overview - href: commands/index.md - - name: Structure and description of YDB CLI commands - href: commands/commands.md - - name: Global parameters - href: commands/global-options.md - - name: Listing objects - href: commands/scheme-ls.md - - name: Getting information about schema objects - href: commands/scheme-describe.md - - name: Getting a list of DB endpoints - href: commands/discovery-list.md - - name: Making a DB query - href: commands/query.md - - name: Streaming table reads - href: commands/readtable.md - - name: Working with secondary indexes - href: commands/operations-index.md - - name: Working with directories - href: commands/scheme-mkdir.md - - name: Scan queries - href: commands/scan-query.md - - name: Utilities - items: - # - name: Copy tables - # href: commands/tools/copy.md - - name: Renaming tables - href: commands/tools/rename.md
\ No newline at end of file + - name: Making a DB query + href: commands/query.md + - name: Query execution plan + href: commands/explain-plan.md + - name: Streaming table reads + href: commands/readtable.md + - name: Working with secondary indexes + href: commands/index-ops.md + - name: Scan queries + href: commands/scan-query.md +# - name: Utilities +# items: + # - name: Copy tables + # href: commands/tools/copy.md + # - name: Making backup + # href: commands/tools/dump.md + # - name: Restore backup + # href: commands/tools/restore.md + - name: Managing profiles + include: { mode: link, path: profile/toc_p.yaml } + - name: Information services + items: + - name: List of endpoints + href: commands/discovery-list.md + - name: Authentication + href: commands/discovery-whoami.md + - name: Load testing + items: + - name: Overview + href: commands/workload/index.md + - name: Stock load + href: commands/workload/stock.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/toc_p.yaml b/ydb/docs/en/core/reference/ydb-cli/toc_p.yaml index 5bfec4365d..94ce110868 100644 --- a/ydb/docs/en/core/reference/ydb-cli/toc_p.yaml +++ b/ydb/docs/en/core/reference/ydb-cli/toc_p.yaml @@ -1,2 +1,4 @@ items: +- name: Overview + href: index.md - include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md new file mode 100644 index 0000000000..1c9a4a6e0a --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md @@ -0,0 +1,88 @@ +# Authentication when establishing a connection between an SDK and database + +As described in the article on [connecting the {{ ydb-short-name }} client to and authenticating with the server](../../../concepts/connect.md), for successful authentication, the client should add an authentication token that is verified by the server to its request. + +The {{ ydb-short-name }} SDK uses an object that is responsible for generating these tokens. The SDK provides built-in methods for obtaining this object: + +1. Ad-hoc methods for different [authentication modes](../../../concepts/connect.md#auth-modes) with explicit parameter transfer. +2. A method that determines the authentication mode and the necessary parameters from the environment where the application is run. + +Usually, a token generation object is created before initializing the {{ ydb-short-name }} driver and passed as a parameter to its builder. The C++ and Go SDKs additionally let you work with multiple databases and token generation objects through a single driver. + +If a token generation object is not defined, the driver won't add any authentication information to requests. This may let you successfully connect to locally deployed {{ ydb-short-name }} clusters that require no authentication. For all databases available over the network, such requests will be rejected with an authentication error returned. + +## Methods for creating token generation objects {#auth-provider} + +You can click on any of the methods described below to go to the source code of the relevant example on github.com. You can also learn about the [authentication code recipes](../recipes/auth/index.md). + +{% list tabs %} + +- Python + + | [Mode](../../../concepts/connect.md#auth-modes) | Method | + | ----- | ----- | + | Anonymous | [`ydb.AnonymousCredentials()`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/anonymous-credentials) | + | Access Token | [`ydb.AccessTokenCredentials( token )`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/access-token-credentials) | + | Metadata | [`ydb.iam.MetadataUrlCredentials()`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/metadata-credentials) | + | Service Account Key | [`ydb.iam.ServiceAccountCredentials.from_file(`</br> `key_file, iam_endpoint=None, iam_channel_credentials=None )`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/service-account-credentials) | + | Determined by environment variables | `ydb.construct_credentials_from_environ()` | + +- Go + + | [Mode](../../../concepts/connect.md#auth-modes) | Package | Method | + | ----- | ----- | ---- | + | Anonymous | [ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk/blob/master/go.mod) | [`ydb.WithAnonymousCredentials()`](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth/anonymous_credentials) | + | Access Token | [ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk/blob/master/go.mod) | [`ydb.WithAccessTokenCredentials( token )`](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth/access_token_credentials) | + | Metadata | [ydb-go-yc](https://github.com/ydb-platform/ydb-go-yc/blob/master/go.mod) | [`yc.WithMetadataCredentials( ctx )`](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth/metadata_credentials) | + | Service Account Key | [ydb-go-yc](https://github.com/ydb-platform/ydb-go-yc/blob/master/go.mod) | [`yc.WithServiceAccountKeyFileCredentials( key_file )`](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth/service_account_credentials) | + | Determined by environment variables | [ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/go.mod) | [`environ.WithEnvironCredentials(ctx)`](https://github.com/ydb-platform/ydb-go-examples/tree/master/cmd/auth/environ) | + +- Java + + | [Mode](../../../concepts/connect.md#auth-modes) | Method | + | ----- | ----- | + | Anonymous | [`com.yandex.ydb.core.auth.NopAuthProvider.INSTANCE`](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/auth/anonymous_credentials) | + | Access Token | [`com.yandex.ydb.auth.iam.CloudAuthProvider.newAuthProvider(`</br> `yandex.cloud.sdk.auth.provider.IamTokenCredentialProvider`</br> `.builder()`</br> `.token(accessToken)`</br> `.build()`</br>`);`](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/auth/access_token_credentials) | + | Metadata | [`com.yandex.ydb.auth.iam.CloudAuthProvider.newAuthProvider(`</br> `yandex.cloud.sdk.auth.provider.ComputeEngineCredentialProvider`</br> `.builder()`</br> `.build()`</br>`);`](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/auth/metadata_credentials) | + | Service Account Key | [`com.yandex.ydb.auth.iam.CloudAuthProvider.newAuthProvider(`</br> `yandex.cloud.sdk.auth.provider.ApiKeyCredentialProvider`</br> `.builder()`</br> `.fromFile(Paths.get(saKeyFile))`</br> `.build()`</br>`);`](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/auth/service_account_credentials) | + | Determined by environment variables | [`com.yandex.ydb.auth.iam.CloudAuthHelper.getAuthProviderFromEnviron();`](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/auth/environ/src/main/java/com/yandex/ydb/example) | + +- Node.js + + | [Mode](../../../concepts/connect.md#auth-modes) | Method | + | ----- | ----- | + | Anonymous | [`new 'ydb-sdk'.AnonymousAuthService()`](https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/examples/auth/anonymous-credentials) | + | Access Token | [`new 'ydb-sdk'.TokenAuthService( accessToken, database )`](https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/examples/auth/access-token-credentials) | + | Metadata | [`new 'ydb-sdk'.MetadataAuthService( database )`](https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/examples/auth/metadata-credentials) | + | Service Account Key | [`new 'ydb-sdk'.getSACredentialsFromJson( saKeyFile )`](https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/examples/auth/service-account-credentials) | + | Determined by environment variables | [`new 'ydb-sdk'.getCredentialsFromEnv( entryPoint, database, logger )`](https://github.com/ydb-platform/ydb-nodejs-sdk/tree/main/examples/auth/environ) | + +{% endlist %} + +## Procedure for determining the authentication mode and parameters from the environment {#env} + +The following algorithm that is the same for all SDKs applies: + +1. If the value of the `YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS` environment variable is set, the **System Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. +2. Otherwise, if the value of the `YDB_ANONYMOUS_CREDENTIALS` environment variable is set to 1, the anonymous authentication mode is used. +3. Otherwise, if the value of the `YDB_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. +4. Otherwise, if the value of the `YDB_ACCESS_TOKEN_CREDENTIALS` environment variable is set, the **Access token** authentication mode is used, where the this variable value is passed. +5. Otherwise, the **Metadata** authentication mode is used. + +If the last step of the algorithm is selecting the **Metadata** mode, you can deploy a working application on VMs and in Yandex Cloud Functions without setting any environment variables. + +## Python SDK specifics + +{% note warning %} + +The behavior of the Python SDK differs from the one described above. + +{% endnote %} + +1. The algorithm for determining the authentication mode and the necessary parameters from the environment variables in the `construct_credentials_from_environ()` method differs from the one used in other SDKs: + - If the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. + - Otherwise, if the value of the `YDB_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. + - Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **System Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. + - Or else, no authentication information is added to requests. +2. If no object responsible for generating tokens is passed when initializing the driver, the [general procedure](#env) for reading environment variable values applies. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md new file mode 100644 index 0000000000..e0664c8244 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md @@ -0,0 +1,24 @@ +# YDB SDK + +{% include [index_intro_overlay.md](index_intro_overlay.md) %} + +OpenSource SDKs in the following programming languages are available to work with YDB: + +{% if oss %} + +- C++ [https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp) +{% endif %} +- C# (.NET) [https://github.com/ydb-platform/ydb-dotnet-sdk](https://github.com/ydb-platform/ydb-dotnet-sdk) +- Go [https://github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) (archived versions: [v1](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) and [v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2)) +- Java [https://github.com/yandex-cloud/ydb-java-sdk](https://github.com/yandex-cloud/ydb-java-sdk) +- Node.js [https://github.com/yandex-cloud/ydb-nodejs-sdk](https://github.com/yandex-cloud/ydb-nodejs-sdk) +- PHP [https://github.com/yandex-cloud/ydb-php-sdk](https://github.com/yandex-cloud/ydb-php-sdk) +- Python [https://github.com/yandex-cloud/ydb-python-sdk](https://github.com/yandex-cloud/ydb-python-sdk) + +The SDK documentation contains the following sections: + +- [Installation](../install.md) +- [Authentication](../auth.md) +- [Test app](../example/index.md) +- [Code recipes](../recipes/index.md) + diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/index/examples.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/index/examples.md index ed8c879a42..e69de29bb2 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/_includes/index/examples.md +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/index/examples.md @@ -1 +0,0 @@ -The [Test app](../../example/index.md) section describes the code of test apps implemented using the YDB SDKs for different programming languages. diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/index/intro.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/index/intro.md index 1d51bc9210..e69de29bb2 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/_includes/index/intro.md +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/index/intro.md @@ -1,12 +0,0 @@ -# YDB SDK - -OpenSource SDKs in the following programming languages are available to work with YDB: - -{% if cpp_oss_link %} -- C++ [{{ cpp_oss_link }}]({{ cpp_oss_link }}){% endif %} -- Python [https://github.com/yandex-cloud/ydb-python-sdk](https://github.com/yandex-cloud/ydb-python-sdk) -- Go [https://github.com/yandex-cloud/ydb-go-sdk](https://github.com/yandex-cloud/ydb-go-sdk) -- Node.js [https://github.com/yandex-cloud/ydb-nodejs-sdk](https://github.com/yandex-cloud/ydb-nodejs-sdk) -- PHP [https://github.com/yandex-cloud/ydb-php-sdk](https://github.com/yandex-cloud/ydb-php-sdk) -- Java [https://github.com/yandex-cloud/ydb-java-sdk](https://github.com/yandex-cloud/ydb-java-sdk) -- C# (.NET) [https://github.com/ydb-platform/ydb-dotnet-sdk](https://github.com/ydb-platform/ydb-dotnet-sdk) diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/index_intro_overlay.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/index_intro_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/index_intro_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md new file mode 100644 index 0000000000..ec44d5c9b9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md @@ -0,0 +1,38 @@ +# Installing the SDK + +Follow the instructions below to quickly install the OpenSource SDK. Make sure to preinstall and configure tools for working with the selected programming language and package managers on your workstation. + +The build process using the source code is described in the source code repositories on GitHub. Follow the links given on the [YDB SDK - Overview](../index.md) page. + +{% list tabs %} + +- Python + + Run the command from the command line: + + {% include [install/cmd_python.md](install/cmd_python.md) %} + + If the command fails, make sure your environment has [Python](https://www.python.org/downloads/) 3.8 or newer installed with the [pip](https://pypi.org/project/pip/) package manager enabled. + +- Go + + Run the command from the command line: + + {% include [install/cmd_go.md](install/cmd_go.md) %} + + To ensure the installation is successful, first install [Go](https://go.dev/doc/install) 1.13 or higher in your environment. + +- C# (.NET) + + {% include [install/cmd_dotnet.md](install/cmd_dotnet.md) %} + +- Java + + Add dependencies to the Maven project as described in the ["Install the SDK"](https://github.com/yandex-cloud/ydb-java-sdk#install-the-sdk) step of the `readme.md` file in the source code repository. + +- PHP + + {% include [install/cmd_php.md](install/cmd_php.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_dotnet.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_dotnet.md new file mode 100644 index 0000000000..01f626af6b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_dotnet.md @@ -0,0 +1,3 @@ +``` bash +dotnet add package Ydb.Sdk +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_go.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_go.md new file mode 100644 index 0000000000..b3a4cb8812 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_go.md @@ -0,0 +1,3 @@ +``` bash +go get -u github.com/ydb-platform/ydb-go-sdk/v3 +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_php.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_php.md new file mode 100644 index 0000000000..ea964a35db --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_php.md @@ -0,0 +1,3 @@ +``` bash +composer require yandex-cloud/ydb-php-sdk +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_python.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_python.md new file mode 100644 index 0000000000..f0b6edf2ea --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install/cmd_python.md @@ -0,0 +1,3 @@ +``` bash +python3 -m pip install ydb +```
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/auth.md b/ydb/docs/en/core/reference/ydb-sdk/auth.md new file mode 100644 index 0000000000..e7f2395443 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/auth.md @@ -0,0 +1,2 @@ + +{% include [auth.md](_includes/auth.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md index 225e85e098..22977978e8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md @@ -3,3 +3,4 @@ The article is being updated. {% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md index bc1ad1a3a7..6760da6cf3 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md @@ -6,3 +6,4 @@ SELECT * FROM episodes; ``` For more information about PRAGMA YQL, see the [YQL documentation](../../../../../yql/reference/index.md). + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-cpp.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-cpp.md index 7be5df695a..a9f0329c9c 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-cpp.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-cpp.md @@ -1,6 +1,6 @@ # App in C++ -This page contains a detailed description of the code of a [test app](https://a.yandex-team.ru/arc/trunk/arcadia/kikimr/public/sdk/cpp/examples/basic_example) that is available as part of the {{ ydb-short-name }} [C++ SDK](https://a.yandex-team.ru/arc/trunk/arcadia/kikimr/public/sdk/cpp). +This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp/examples/basic_example) that is available as part of the {{ ydb-short-name }} [C++ SDK](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp). {% include [init.md](steps/01_init.md) %} @@ -21,7 +21,7 @@ App code snippet for creating a client: TClient client(driver); ``` -{% include [create_table.md](steps/02_create_table.md) %} +{% include [steps/02_create_table.md](steps/02_create_table.md) %} To create tables, use the `CreateTable` method: @@ -71,9 +71,28 @@ Column, name: series_info, type: Utf8? Column, name: release_date, type: Uint64? ``` +{% include [steps/03_write_queries.md](steps/03_write_queries.md) %} + +Code snippet for inserting and updating data: + +```c++ +//! Shows basic usage of mutating operations. +static TStatus UpsertSimpleTransaction(TSession session, const TString& path) { + auto query = Sprintf(R"( + PRAGMA TablePathPrefix("%s"); + + UPSERT INTO episodes (series_id, season_id, episode_id, title) VALUES + (2, 6, 1, "TBD"); + )", path.c_str()); + + return session.ExecuteDataQuery(query, + TTxControl::BeginTx(TTxSettings::SerializableRW()).CommitTx()).GetValueSync(); +} +``` + {% include [pragmatablepathprefix.md](auxilary/pragmatablepathprefix.md) %} -{% include [query_processing.md](steps/03_query_processing.md) %} +{% include [steps/04_query_processing.md](steps/04_query_processing.md) %} To execute YQL queries, use the `ExecuteDataQuery` method. The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `TTxControl` class. @@ -111,7 +130,7 @@ static TStatus SelectSimpleTransaction(TSession session, const TString& path, } ``` -{% include [results_processing.md](steps/04_results_processing.md) %} +{% include [steps/05_results_processing.md](steps/05_results_processing.md) %} The `TResultSetParser` class is used for processing query results. The code snippet below shows how to process query results using the `parser` object: @@ -134,25 +153,6 @@ The given code snippet outputs the following text to the console at startup: series, Id: 1, title: IT Crowd, Release date: 2006-02-03 ``` -{% include [write_queries.md](steps/05_write_queries.md) %} - -Code snippet for inserting and updating data: - -```c++ -//! Shows basic usage of mutating operations. -static TStatus UpsertSimpleTransaction(TSession session, const TString& path) { - auto query = Sprintf(R"( - PRAGMA TablePathPrefix("%s"); - - UPSERT INTO episodes (series_id, season_id, episode_id, title) VALUES - (2, 6, 1, "TBD"); - )", path.c_str()); - - return session.ExecuteDataQuery(query, - TTxControl::BeginTx(TTxSettings::SerializableRW()).CommitTx()).GetValueSync(); -} -``` - {% include [param_queries.md](steps/06_param_queries.md) %} The code snippet shows the use of parameterized queries and the `GetParamsBuilder` to generate parameters and pass them to the `ExecuteDataQuery` method. @@ -429,4 +429,3 @@ static TStatus ExplicitTclTransaction(TSession session, const TString& path, con } ``` -{% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md index b262272b9a..a02c2d6082 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md @@ -1,10 +1,10 @@ # App in C# (.NET) -This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb-dotnet-examples), that is available as part of the [C# (.NET) SDK](https://github.com/ydb-platform/ydb-dotnet-sdk) {{ ydb-short-name }}. +This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb-dotnet-examples), that uses the [C# (.NET) SDK](https://github.com/ydb-platform/ydb-dotnet-sdk) {{ ydb-short-name }}. {% include [addition.md](auxilary/addition.md) %} -{% include [init.md](steps/01_init.md) %} +{% include [steps/01_init.md](steps/01_init.md) %} App code snippet for driver initialization: @@ -34,7 +34,7 @@ App code snippet for creating a session: using var tableClient = new TableClient(driver, new TableClientConfig()); ``` -{% include [create_table.md](steps/02_create_table.md) %} +{% include [steps/02_create_table.md](steps/02_create_table.md) %} To create tables, use the `session.ExecuteSchemeQuery` method with a DDL (Data Definition Language) YQL query. @@ -73,9 +73,40 @@ var response = await tableClient.SessionExec(async session => response.Status.EnsureSuccess(); ``` +{% include [steps/03_write_queries.md](steps/03_write_queries.md) %} + +Code snippet for inserting and updating data: + +```c# +var response = await tableClient.SessionExec(async session => +{ + var query = @" + DECLARE $id AS Uint64; + DECLARE $title AS Utf8; + DECLARE $release_date AS Date; + + UPSERT INTO series (series_id, title, release_date) VALUES + ($id, $title, $release_date); + "; + + return await session.ExecuteDataQuery( + query: query, + txControl: TxControl.BeginSerializableRW().Commit(), + parameters: new Dictionary<string, YdbValue> + { + { "$id", YdbValue.MakeUint64(1) }, + { "$title", YdbValue.MakeUtf8("NewTitle") }, + { "$release_date", YdbValue.MakeDate(DateTime.UtcNow) } + } + ); +}); + +response.Status.EnsureSuccess(); +``` + {% include [pragmatablepathprefix.md](auxilary/pragmatablepathprefix.md) %} -{% include [query_processing.md](steps/03_query_processing.md) %} +{% include [steps/04_query_processing.md](steps/04_query_processing.md) %} To execute YQL queries, use the `Session.executeDataQuery()` method. The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `TxControl` class. In the code snippet below, a transaction with the `SerializableRW` mode and an automatic commit after executing the request is used. The values of the request parameters are passed in the form of a dictionary name-value in the `parameters` argument. @@ -108,7 +139,7 @@ var queryResponse = (ExecuteDataQueryResponse)response; var resultSet = queryResponse.Result.ResultSets[0]; ``` -{% include [results_processing.md](steps/04_results_processing.md) %} +{% include [steps/05_results_processing.md](steps/05_results_processing.md) %} The result of query execution (resultset) consists of an organized set of rows. Example of processing the query execution result: @@ -122,37 +153,6 @@ foreach (var row in resultSet.Rows) } ``` -{% include [write_queries.md](steps/05_write_queries.md) %} - -Code snippet for inserting and updating data: - -```c# -var response = await tableClient.SessionExec(async session => -{ - var query = @" - DECLARE $id AS Uint64; - DECLARE $title AS Utf8; - DECLARE $release_date AS Date; - - UPSERT INTO series (series_id, title, release_date) VALUES - ($id, $title, $release_date); - "; - - return await session.ExecuteDataQuery( - query: query, - txControl: TxControl.BeginSerializableRW().Commit(), - parameters: new Dictionary<string, YdbValue> - { - { "$id", YdbValue.MakeUint64(1) }, - { "$title", YdbValue.MakeUtf8("NewTitle") }, - { "$release_date", YdbValue.MakeDate(DateTime.UtcNow) } - } - ); -}); - -response.Status.EnsureSuccess(); -``` - {% include [scan_query.md](steps/08_scan_query.md) %} ```c# @@ -184,4 +184,3 @@ public void executeScanQuery() } ``` -{% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-go.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-go.md index a25c484298..e69de29bb2 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-go.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-go.md @@ -1,219 +0,0 @@ -# App in Go - -This page contains a detailed description of the code of a [test app](https://github.com/yandex-cloud/ydb-go-sdk/tree/master/example/basic_example_v1) that is available as part of the {{ ydb-short-name }} [Go SDK](https://github.com/yandex-cloud/ydb-go-sdk). - -{% include [addition.md](auxilary/addition.md) %} - -{% include [init.md](steps/01_init.md) %} - -App code snippet for driver initialization: - -```go -func (cmd *Command) Run(ctx context.Context, params cli.Parameters) error { - dialer := &ydb.Dialer{ - DriverConfig: cmd.config(params), - TLSConfig: cmd.tls(), - Timeout: time.Second, - } - driver, err := dialer.Dial(ctx, params.Endpoint) - if err != nil { - return fmt.Errorf("dial error: %v", err) - } - defer driver.Close() -``` - -App code snippet for creating a session: - -```go -tableClient := table.Client{ - Driver: driver, -} -sp := table.SessionPool{ - IdleThreshold: time.Second, - Builder: &tableClient, -} -defer sp.Close(ctx) -``` - -{% include [create_table.md](steps/02_create_table.md) %} - -To create tables, use the `Session.CreateTable()` method: - -```go -func createTables(ctx context.Context, sp *table.SessionPool, prefix string) (err error) { - err = table.Retry(ctx, sp, - table.OperationFunc(func(ctx context.Context, s *table.Session) error { - return s.CreateTable(ctx, path.Join(prefix, "series"), - table.WithColumn("series_id", ydb.Optional(ydb.TypeUint64)), - table.WithColumn("title", ydb.Optional(ydb.TypeUTF8)), - table.WithColumn("series_info", ydb.Optional(ydb.TypeUTF8)), - table.WithColumn("release_date", ydb.Optional(ydb.TypeUint64)), - table.WithColumn("comment", ydb.Optional(ydb.TypeUTF8)), - table.WithPrimaryKeyColumn("series_id"), - ) - }), - ) -``` - -You can use the `Session.DescribeTable()` method to output information about the table structure and make sure that it was properly created: - -```go -func describeTable(ctx context.Context, sp *table.SessionPool, path string) (err error) { - err = table.Retry(ctx, sp, - table.OperationFunc(func(ctx context.Context, s *table.Session) error { - desc, err := s.DescribeTable(ctx, path) - if err != nil { - return err - } - log.Printf("\n> describe table: %s", path) - for _, c := range desc.Columns { - log.Printf("column, name: %s, %s", c.Type, c.Name) - } - return nil - }), - ) -``` - -{% include [pragmatablepathprefix.md](auxilary/pragmatablepathprefix.md) %} - -{% include [query_processing.md](steps/03_query_processing.md) %} - -To execute YQL queries, use the `Session.Execute()` method. -The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the ```TxControl``` class. - -```go -func selectSimple(ctx context.Context, sp *table.SessionPool, prefix string) (err error) { - query := render( - template.Must(template.New("").Parse(` - PRAGMA TablePathPrefix("not_var{{ .TablePathPrefix }}"); - DECLARE $seriesID AS Uint64; - $format = DateTime::Format("%Y-%m-%d"); - SELECT - series_id, - title, - $format(DateTime::FromSeconds(CAST(DateTime::ToSeconds(DateTime::IntervalFromDays(CAST(release_date AS Int16))) AS Uint32))) AS release_date - FROM - series - WHERE - series_id = $seriesID; - `)), - templateConfig{ - TablePathPrefix: prefix, - }, - ) - readTx := table.TxControl( - table.BeginTx( - table.WithOnlineReadOnly(), - ), - table.CommitTx(), - ) - var res *table.Result - err = table.Retry(ctx, sp, - table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { - _, res, err = s.Execute(ctx, readTx, query, - table.NewQueryParameters( - table.ValueParam("$seriesID", ydb.Uint64Value(1)), - ), - table.WithQueryCachePolicy( - table.WithQueryCachePolicyKeepInCache(), - ), - table.WithCollectStatsModeBasic(), - ) - return - }), - ) - if err != nil { - return err - } - for res.NextSet() { - for res.NextRow() { - res.SeekItem("series_id") - id := res.OUint64() - - res.NextItem() - title := res.OUTF8() - - res.NextItem() - date := res.OString() - - log.Printf( - "\n> select_simple_transaction: %d %s %s", - id, title, date, - ) - } - } - if err := res.Err(); err != nil { - return err - } - return nil -} -``` - -{% include [results_processing.md](steps/04_results_processing.md) %} - -Query results: - -```go -for res.NextSet() { - for res.NextRow() { - res.SeekItem("series_id") - id := res.OUint64() - - res.NextItem() - title := res.OUTF8() - - res.NextItem() - date := res.OString() - - log.Printf( - "\n> select_simple_transaction: %d %s %s", - id, title, date, - ) - } -} -``` - -{% include [scan_query.md](steps/08_scan_query.md) %} - -```go -func executeScanQuery(ctx context.Context, sp *table.SessionPool, prefix string) (err error) { -query := ` - SELECT series_id, season_id, COUNT(*) AS episodes_count - FROM episodes - GROUP BY series_id, season_id - ORDER BY series_id, season_id;` - -var res *table.Result -err = table.Retry(ctx, sp, - table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { - res, err = s.StreamExecuteScanQuery(ctx, query, table.NewQueryParameters()) - return err - }), -) -if err != nil { - return err -} -var ( - seriesID uint64 - seasonID uint64 - count uint64 -) -log.Print("\n> scan_query_select:") -for res.NextResultSet(ctx, "series_id", "season_id", "episodes_count") { - for res.NextRow() { - err = res.ScanWithDefaults(&seriesID, &seasonID, &count) - if err != nil { - return err - } - log.Printf("# Season, SeriesId: %d, SeasonId: %d, Count: %d", seriesID, seasonID, count) - } -} -if err = res.Err(); err != nil { - return err -} -return nil -} -``` - -{% include [error_handling.md](steps/50_error_handling.md) %} - diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-java.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-java.md index 3e7baa65c0..c097e28e83 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-java.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-java.md @@ -95,9 +95,30 @@ private void describeTables() { } ``` +{% include [steps/03_write_queries.md](steps/03_write_queries.md) %} + +Code snippet for inserting and updating data: + +```java +private void upsertSimple() { + String query = String.format( + "PRAGMA TablePathPrefix(\"%s\");\n" + + "\n" + + "UPSERT INTO episodes (series_id, season_id, episode_id, title) VALUES\n" + + "(2, 6, 1, \"TBD\");", + database); + + TxControl txControl = TxControl.serializableRw().setCommitTx(true); + + execute(session -> session.executeDataQuery(query, txControl) + .join() + .toStatus()); +} +``` + {% include [pragmatablepathprefix.md](auxilary/pragmatablepathprefix.md) %} -{% include [query_processing.md](steps/03_query_processing.md) %} +{% include [steps/04_query_processing.md](steps/04_query_processing.md) %} To execute YQL queries, use the `Session.executeDataQuery()` method. The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `TxControl` class. @@ -127,17 +148,9 @@ private void selectSimple() { } ``` -{% include [results_processing.md](steps/04_results_processing.md) %} - -When the query is executed, `result.getResultSet(0)` is returned. -The code snippet below shows the output of query results using the `TablePrinter` helper class. +As a result of executing the query, an object of the `DataQueryResult` class is generated. It may contain several sets obtained using the `getResultSet( <index> )` method. Since there was only one `SELECT` statement in the query, the result contains only one selection indexed as `0`. -```java -System.out.println("\n--[ SelectSimple ]--"); -new TablePrinter(result.getResultSet(0)).print(); -``` - -The given code snippet outputs the following text to the console at startup: +It is output with semigraphics formatting using the `TablePrinter` helper class. The given code snippet outputs the following text to the console at startup: ```bash --[ SelectSimple ]-- @@ -148,28 +161,7 @@ The given code snippet outputs the following text to the console at startup: +-----------+------------------+--------------------+ ``` -{% include [write_queries.md](steps/05_write_queries.md) %} - -Code snippet for inserting and updating data: - -```java -private void upsertSimple() { - String query = String.format( - "PRAGMA TablePathPrefix(\"%s\");\n" + - "\n" + - "UPSERT INTO episodes (series_id, season_id, episode_id, title) VALUES\n" + - "(2, 6, 1, \"TBD\");", - database); - - TxControl txControl = TxControl.serializableRw().setCommitTx(true); - - execute(session -> session.executeDataQuery(query, txControl) - .join() - .toStatus()); -} -``` - -{% include [write_queries.md](steps/05_write_queries.md) %} +{% include [param_queries.md](steps/06_param_queries.md) %} The code snippet below shows the use of parameterized queries and the `Params` class to generate parameters and pass them to the `executeDataQuery` method. @@ -436,4 +428,3 @@ private Status explicitTcl(Session session) { } ``` -{% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-nodejs.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-nodejs.md index f43fa82ae7..86e9c68da4 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-nodejs.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-nodejs.md @@ -19,4 +19,3 @@ await session.streamExecuteScanQuery(` `, consumer, {'$value': Primitive.utf8('ttt')}); ``` -{% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-php.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-php.md index 680ea66658..0eaff5d21f 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-php.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-php.md @@ -18,7 +18,6 @@ $config = [ ]; $ydb = new Ydb($config); - ``` App code snippet for creating a session: @@ -31,4 +30,3 @@ $table = $ydb->table(); $session = $table->session(); ``` -{% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md index bea03512af..12647fb0df 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md @@ -1,28 +1,50 @@ # Test app -This section describes the code of test apps implemented using YDB SDK in different programming languages. +This section describes the code of same-type test apps implemented using {{ ydb-short-name }} SDKs in different programming languages: + +{% if oss %} + +- [C++](../example-cpp.md) +{% endif %} +- [C# (.NET)](../example-dotnet.md) +- [Go](../go/index.md) +- [Java](../example-java.md) +- [Node.js](../example-nodejs.md) +- [Python](../python/index.md) A test app performs the following steps: {% include [init.md](steps/01_init.md) %} +{% if oss %}[C++](../example-cpp.md#init) | {% endif %} [C# (.NET)](../example-dotnet.md#init) | [Go](../go/index.md#init) | [Java](../example-java.md#init) | Node.js | [PHP](../example-php.md#init) | [Python](../python/index.md#init) + {% include [create_table.md](steps/02_create_table.md) %} -{% include [query_processing.md](steps/03_query_processing.md) %} +{% if oss %}[C++](../example-cpp.md#create-table) | {% endif %} [C# (.NET)](../example-dotnet.md#create-table) | [Go](../go/index.md#create-table) | [Java](../example-java.md#create-table) | Node.js | PHP | [Python](../python/index.md#create-table) + +{% include [write_queries.md](steps/03_write_queries.md) %} -{% include [results_processing.md](steps/04_results_processing.md) %} +{% if oss %}[C++](../example-cpp.md#write-queries) | {% endif %} [C# (.NET)](../example-dotnet.md#write-queries) | Go | [Java](../example-java.md#write-queries) | Node.js | PHP | [Python](../python/index.md#write-queries) -{% include [write_queries.md](steps/05_write_queries.md) %} +{% include [query_processing.md](steps/04_query_processing.md) %} + +{% if oss %}[C++](../example-cpp.md#query-processing) | {% endif %} [C# (.NET)](../example-dotnet.md#query-processing) | [Go](../go/index.md#query-processing) | [Java](../example-java.md#query-processing) | Node.js | PHP | [Python](../python/index.md#query-processing) {% include [param_queries.md](steps/06_param_queries.md) %} -{% include [param_prep_queries.md](steps/07_param_prep_queries.md) %} +{% if oss %}[C++](../example-cpp.md#param-queries) | {% endif %} [C# (.NET)](../example-dotnet.md#param-queries) | [Go](../go/index.md#param-queries) | [Java](../example-java.md#param-queries) | Node.js | PHP | [Python](../python/index.md#param-queries) {% include [scan_query.md](steps/08_scan_query.md) %} +{% if oss %}C++ | {% endif %} [C# (.NET)](../example-dotnet.md#scan-query) | [Go](../go/index.md#scan-query) | [Java](../example-java.md#scan-query) | [Node.js](../example-nodejs.md#scan-query) | PHP | [Python](../python/index.md#scan-query) + {% include [multistep_transactions.md](steps/09_multistep_transactions.md) %} +{% if oss %}[C++](../example-cpp.md#multistep-transactions) | {% endif %} C# (.NET) | Go | [Java](../example-java.md#multistep-transactions) | Node.js | PHP | Python + {% include [transaction_control.md](steps/10_transaction_control.md) %} +{% if oss %}[C++](../example-cpp.md#tcl) | {% endif %} C# (.NET) | Go | [Java](../example-java.md#tcl) | Node.js | PHP | [Python](../python/index.md#tcl) + {% include [error_handling.md](steps/50_error_handling.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md new file mode 100644 index 0000000000..b31f1357c8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md @@ -0,0 +1,10 @@ +{% note info %} + +If you previously reviewed the articles of the "Getting started" section, you must have used the necessary parameters when [getting started with the YDB CLI](../../../../getting_started/cli.md) and can get them from the profile: + +```bash +{{ ydb-cli }} config profile get db1 +``` + +{% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md index 9e57dcbed3..b49aab5618 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md @@ -1,7 +1,8 @@ -## Creating a new driver, client, and session instance {#driver-client-session-init} +## Initializing a database connection {#init} -To interact with {{ ydb-short-name }}, you have to create an instance of the driver, client, and session: +To interact with {{ ydb-short-name }}, create an instance of the driver, client, and session: * The {{ ydb-short-name }} driver lets the app and {{ ydb-short-name }} interact at the transport layer. The driver must exist throughout the {{ ydb-short-name }} access lifecycle and be initialized before creating a client or session. * The {{ ydb-short-name }} client runs on top of the {{ ydb-short-name }} driver and enables the handling of entities and transactions. * The {{ ydb-short-name }} session contains information about executed transactions and prepared queries, and is part of the {{ ydb-short-name }} client context. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md index d942a7982c..e8771af94d 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md @@ -1 +1,10 @@ -## Creating tables using the CreateTable API {#create-table-api} +## Creating tables {#create-table} + +Creating tables to be used in operations on a test app. This step results in the creation of DB tables of the series directory data model: + +- `Series` +- `Seasons` +- `Episodes` + +Once the tables are created, the method for getting information about data schema objects is called and the result of its execution is output. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_query_processing.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_query_processing.md index 24107efeff..93886f1ac7 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_query_processing.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_query_processing.md @@ -1 +1,2 @@ ## Processing queries and transactions {#query-processing} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_write_queries.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_write_queries.md new file mode 100644 index 0000000000..8c8364a4b2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/03_write_queries.md @@ -0,0 +1,4 @@ +## Adding data {#write-queries} + +Adding data to the created tables using an [`UPSERT`](../../../../../yql/reference/syntax/upsert_into.md) statement of [YQL](../../../../../yql/reference/index.md). A data update request is sent within a single request to the server with transaction auto-commit mode enabled. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_query_processing.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_query_processing.md new file mode 100644 index 0000000000..a1632f77bc --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_query_processing.md @@ -0,0 +1,4 @@ +## Retrieving data with a Select {#query-processing} + +Retrieving data using a [`SELECT`](../../../../../yql/reference/syntax/select.md) statement in [YQL](../../../../../yql/reference/index.md). Handling the retrieved data selection in the app. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_results_processing.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_results_processing.md index ebe3e53b31..a6b3c132ad 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_results_processing.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/04_results_processing.md @@ -1 +1,2 @@ -## Processing execution results {#results-processing} +### Processing execution results {#results-processing} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_results_processing.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_results_processing.md new file mode 100644 index 0000000000..a6b3c132ad --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_results_processing.md @@ -0,0 +1,2 @@ +### Processing execution results {#results-processing} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_write_queries.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_write_queries.md index 02325e4da9..c01f55b97b 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_write_queries.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/05_write_queries.md @@ -1 +1,2 @@ ## Queries to insert or update data {#write-queries} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/06_param_queries.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/06_param_queries.md index 7943721319..cd6bde38a7 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/06_param_queries.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/06_param_queries.md @@ -1,7 +1,4 @@ ## Parameterized queries {#param-queries} -{% note warning %} +Querying data using parameters. This query execution option is preferable, as it allows the server to reuse the query execution plan for subsequent calls and protects against vulnerabilities like [SQL Injection](https://en.wikipedia.org/wiki/SQL_injection). -We strongly recommend that you use only parameterized queries. This lets your database prepare for parsing and optimization of multiple queries that differ only in parameter values just once. - -{% endnote %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md index fc04fcd5e8..4b2a1abeb5 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md @@ -3,3 +3,4 @@ Parameterized prepared queries are saved as templates where specially formatted names are replaced by relevant parameter values each time you execute the query. Use parameterized queries to improve performance by reducing how often queries that only differ in parameter values are compiled and recompiled. The prepared query is stored in the session context. Code snippet for parameterized prepared queries: + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md index fc278a401a..4a2e6e219b 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md @@ -1,3 +1,4 @@ ## Scan queries {#scan-query} -The result of executing a [scan query](../../../../../concepts/scan_query.md) is a data stream. Below is an example of processing the query execution result. +Making a [scan query](../../../../../concepts/scan_query.md) that results in a data stream. Streaming lets you read an unlimited number of rows and amount of data. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/09_multistep_transactions.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/09_multistep_transactions.md index bb19fd88a0..c5a94abc43 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/09_multistep_transactions.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/09_multistep_transactions.md @@ -1,3 +1,4 @@ -## Multi-step transactions with intermediate data processing on the client side {#multistep-transactions} +## Multistep transactions {#multistep-transactions} + +Multiple commands are executed within a single multistep transaction. The client-side code can be run between query executions. Using a transaction ensures that select queries made in its context are consistent with each other. -The code snippet below shows how to use multi-step transactions consisting of multiple queries. The client-side code can be run between query executions. diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/10_transaction_control.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/10_transaction_control.md index b76306445b..093026167e 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/10_transaction_control.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/10_transaction_control.md @@ -1,3 +1,6 @@ -## Explicit TCL Begin and Commit calls {#tcl-usage} +## Managing transactions {#tcl} + +Transactions are managed through [TCL](../../../../../concepts/transactions.md) Begin and Commit calls. + +In most cases, instead of explicitly using Begin and Commit calls, it's better to use transaction control parameters in execute calls. This helps you avoid unnecessary requests to {{ ydb-short-name }} and run your queries more efficiently. -In most cases, instead of explicitly using the [TCL](../../../../../concepts/transactions.md) Begin and Commit calls, it's better to use transaction control parameters in the execute calls. This helps you avoid unnecessary requests to {{ ydb-short-name }} and run your queries more efficiently. diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md index eb6c9370c3..41c92f22a4 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md @@ -1,3 +1,4 @@ ## Handling errors {#error-handling} For more information about error handling, see [Error handling in the API](../../../error_handling.md). + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md new file mode 100644 index 0000000000..fb4181771b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md @@ -0,0 +1,232 @@ +# App in Go (archived version 1) + +This page contains a detailed description of the code of a [test app](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1/example/basic_example_v1) that is available as part of the [v1 Go SDK](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) {{ ydb-short-name }}. + +{% include [init.md](../_includes/steps/01_init.md) %} + +To work with {{ ydb-short-name }} in `Go`, import the `ydb-go-sdk` driver package: + +```go +import ( + // general imports + "context" + "path" + + // imports of ydb-go-sdk packages + "github.com/yandex-cloud/ydb-go-sdk" + "github.com/yandex-cloud/ydb-go-sdk/table" // to work with the table service +) +``` + +App code snippet for driver initialization: + +```go +func (cmd *Command) Run(ctx context.Context, params cli.Parameters) error { + dialer := &ydb.Dialer{ + DriverConfig: cmd.config(params), + TLSConfig: cmd.tls(), + Timeout: time.Second, + } + driver, err := dialer.Dial(ctx, params.Endpoint) + if err != nil { + return fmt.Errorf("dial error: %v", err) + } + defer driver.Close() +``` + +App code snippet for creating a session: + +```go +tableClient := table.Client{ + Driver: driver, +} +sp := table.SessionPool{ + IdleThreshold: time.Second, + Builder: &tableClient, +} +defer sp.Close(ctx) +``` + +{% include [create_table.md](../_includes/steps/02_create_table.md) %} + +To create tables, use the `Session.CreateTable()` method: + +```go +func createTables(ctx context.Context, sp *table.SessionPool, prefix string) (err error) { + err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) error { + return s.CreateTable(ctx, path.Join(prefix, "series"), + table.WithColumn("series_id", ydb.Optional(ydb.TypeUint64)), + table.WithColumn("title", ydb.Optional(ydb.TypeUTF8)), + table.WithColumn("series_info", ydb.Optional(ydb.TypeUTF8)), + table.WithColumn("release_date", ydb.Optional(ydb.TypeUint64)), + table.WithColumn("comment", ydb.Optional(ydb.TypeUTF8)), + table.WithPrimaryKeyColumn("series_id"), + ) + }), + ) +``` + +You can use the `Session.DescribeTable()` method to output information about the table structure and make sure that it was properly created: + +```go +func describeTable(ctx context.Context, sp *table.SessionPool, path string) (err error) { + err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) error { + desc, err := s.DescribeTable(ctx, path) + if err != nil { + return err + } + log.Printf("\n> describe table: %s", path) + for _, c := range desc.Columns { + log.Printf("column, name: %s, %s", c.Type, c.Name) + } + return nil + }), + ) +``` + +{% include [query_processing.md](../_includes/steps/04_query_processing.md) %} + +To execute YQL queries, use the `Session.Execute()` method. +The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the ```TxControl``` class. + +```go +var ( + query = ` + DECLARE $seriesID AS Uint64; + $format = DateTime::Format("%Y-%m-%d"); + SELECT + series_id, + title, + $format(DateTime::FromSeconds(CAST(DateTime::ToSeconds(DateTime::IntervalFromDays(CAST(release_date AS Int16))) AS Uint32))) AS release_date + FROM + series + WHERE + series_id = $seriesID; + ` + res *table.Result +) +readTx := table.TxControl( + table.BeginTx( + table.WithOnlineReadOnly(), + ), + table.CommitTx(), +) +err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { + _, res, err = s.Execute(ctx, readTx, query, + table.NewQueryParameters( + table.ValueParam("$seriesID", ydb.Uint64Value(1)), + ), + table.WithQueryCachePolicy( + table.WithQueryCachePolicyKeepInCache(), + ), + table.WithCollectStatsModeBasic(), + ) + if err != nil { + return err + } + return res.Err() + }), +) +if err != nil { + // handling the query execution error +} +``` + +{% include [results_processing.md](../_includes/steps/05_results_processing.md) %} + +Query results: + +```go +log.Println("> select_simple_transaction:") +for res.NextSet() { + for res.NextRow() { + res.SeekItem("series_id") + id := res.OUint64() + + res.NextItem() + title := res.OUTF8() + + res.NextItem() + date := res.OString() + + log.Printf( + "# SeriesID: %d , Title: %s, Date: %s\n", + id, title, date, + ) + } +} +``` + +{% include [scan_query.md](../_includes/steps/08_scan_query.md) %} + +```go +var ( + query = ` + DECLARE $series AS List<UInt64>; + SELECT series_id, season_id, title, CAST(CAST(first_aired AS Date) AS String) AS first_aired + FROM seasons + WHERE series_id IN $series + ` + res *table.Result +) +err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { + res, err = s.StreamExecuteScanQuery(ctx, query, + table.NewQueryParameters( + table.ValueParam("$series", + ydb.ListValue( + ydb.Uint64Value(1), + ydb.Uint64Value(10), + ), + ), + ), + ) + if err != nil { + return err + } + return res.Err() + }), +) +if err != nil { + // handling the query execution error +} +``` + +{% include [results_processing.md](../_includes/steps/08_scan_query.md) %} + +Query results: + +```go +log.Println("> scan_query_select:") +for res.NextStreamSet(ctx) { + if err = res.Err(); err != nil { + return err + } + + for res.NextRow() { + res.SeekItem("series_id") + id := res.OUint64() + + res.SeekItem("season_id") + season := res.OUint64() + + res.SeekItem("title") + title := res.OUTF8() + + res.SeekItem("first_aired") + date := res.OString() + + log.Printf("# Season, SeriesId: %d, SeasonId: %d, Title: %s, Air date: %s\n", id, season, title, date) + } +} +if err = res.Err(); err != nil { + return err +} +return nil +``` + +{% include [error_handling.md](../_includes/steps/50_error_handling.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md new file mode 100644 index 0000000000..20dad36b27 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md @@ -0,0 +1,214 @@ +# App in Go (archived version 2) + +This page contains a detailed description of the code of a [test app](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2/example/basic_example_v1) that is available as part of the [v2 Go SDK](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2) {{ ydb-short-name }}. + +{% include [init.md](../_includes/steps/01_init.md) %} + +To work with {{ ydb-short-name }} in `Go`, import the `ydb-go-sdk` driver package: + +```go +import ( + // general imports + "context" + "path" + + // imports of ydb-go-sdk packages + "github.com/yandex-cloud/ydb-go-sdk/v2" + "github.com/yandex-cloud/ydb-go-sdk/v2/table" // to work with the table service +) +``` + +App code snippet for driver initialization: + +```go +func (cmd *Command) Run(ctx context.Context, params cli.Parameters) error { + dialer := &ydb.Dialer{ + DriverConfig: cmd.config(params), + TLSConfig: cmd.tls(), + Timeout: time.Second, + } + driver, err := dialer.Dial(ctx, params.Endpoint) + if err != nil { + return fmt.Errorf("dial error: %v", err) + } + defer driver.Close() +``` + +App code snippet for creating a session: + +```go +tableClient := table.Client{ + Driver: driver, +} +sp := table.SessionPool{ + IdleThreshold: time.Second, + Builder: &tableClient, +} +defer sp.Close(ctx) +``` + +{% include [create_table.md](../_includes/steps/02_create_table.md) %} + +To create tables, use the `Session.CreateTable()` method: + +```go +func createTables(ctx context.Context, sp *table.SessionPool, prefix string) (err error) { + err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) error { + return s.CreateTable(ctx, path.Join(prefix, "series"), + table.WithColumn("series_id", ydb.Optional(ydb.TypeUint64)), + table.WithColumn("title", ydb.Optional(ydb.TypeUTF8)), + table.WithColumn("series_info", ydb.Optional(ydb.TypeUTF8)), + table.WithColumn("release_date", ydb.Optional(ydb.TypeUint64)), + table.WithColumn("comment", ydb.Optional(ydb.TypeUTF8)), + table.WithPrimaryKeyColumn("series_id"), + ) + }), + ) +``` + +You can use the `Session.DescribeTable()` method to output information about the table structure and make sure that it was properly created: + +```go +func describeTable(ctx context.Context, sp *table.SessionPool, path string) (err error) { + err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) error { + desc, err := s.DescribeTable(ctx, path) + if err != nil { + return err + } + log.Printf("\n> describe table: %s", path) + for _, c := range desc.Columns { + log.Printf("column, name: %s, %s", c.Type, c.Name) + } + return nil + }), + ) +``` + +{% include [query_processing.md](../_includes/steps/04_query_processing.md) %} + +To execute YQL queries, use the `Session.Execute()` method. +The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the ```TxControl``` class. + +```go +var ( + query = `--!syntax_v1 + DECLARE $seriesID AS Uint64; + $format = DateTime::Format("%Y-%m-%d"); + SELECT + series_id, + title, + $format(DateTime::FromSeconds(CAST(DateTime::ToSeconds(DateTime::IntervalFromDays(CAST(release_date AS Int16))) AS Uint32))) AS release_date + FROM + series + WHERE + series_id = $seriesID;` + res *table.Result +) +readTx := table.TxControl( + table.BeginTx( + table.WithOnlineReadOnly(), + ), + table.CommitTx(), +) +err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { + _, res, err = s.Execute(ctx, readTx, query, + table.NewQueryParameters( + table.ValueParam("$seriesID", ydb.Uint64Value(1)), + ), + table.WithQueryCachePolicy( + table.WithQueryCachePolicyKeepInCache(), + ), + table.WithCollectStatsModeBasic(), + ) + return + }), +) +if err != nil { + return err +} +``` + +{% include [results_processing.md](../_includes/steps/05_results_processing.md) %} + +Query results: + +```go + var ( + id *uint64 + title *string + date *[]byte + ) + log.Println("> select_simple_transaction:") + for res.NextResultSet(ctx, "series_id", "title", "release_date") { + for res.NextRow() { + err = res.Scan(&id, &title, &date) + if err != nil { + return err + } + log.Printf( + "# SeriesID: %d , Title: %s, Date: %s\n", + *id, *title, *date, + ) + } + } + if err = res.Err(); err != nil { + return err + } + return nil +} +``` + +{% include [scan_query.md](../_includes/steps/08_scan_query.md) %} + +```go +var ( + query = ` + SELECT series_id, season_id, COUNT(*) AS episodes_count + FROM episodes + GROUP BY series_id, season_id + ORDER BY series_id, season_id;` + res *table.Result +) +err = table.Retry(ctx, sp, + table.OperationFunc(func(ctx context.Context, s *table.Session) (err error) { + res, err = s.StreamExecuteScanQuery(ctx, query, table.NewQueryParameters()) + return err + }), +) +if err != nil { + return err +} +``` + +{% include [results_processing.md](../_includes/steps/08_scan_query.md) %} + +Query results: + +```go + var ( + seriesID uint64 + seasonID uint64 + count uint64 + ) + log.Println("> scan_query_select:") + for res.NextResultSet(ctx, "series_id", "season_id", "episodes_count") { + for res.NextRow() { + err = res.ScanWithDefaults(&seriesID, &seasonID, &count) + if err != nil { + return err + } + log.Printf("# Season, SeriesId: %d, SeasonId: %d, Count: %d\n", seriesID, seasonID, count) + } + } + if err = res.Err(); err != nil { + return err + } + return nil +} +``` + +{% include [error_handling.md](../_includes/steps/50_error_handling.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/example-go.md b/ydb/docs/en/core/reference/ydb-sdk/example/example-go.md index 8194917be3..3ece27c3e8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/example-go.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/example-go.md @@ -1 +1,2 @@ {% include [example-go.md](_includes/example-go.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md new file mode 100644 index 0000000000..77f7845920 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md @@ -0,0 +1,27 @@ +To run the example using any available YDB database, you need to know the [Endpoint](../../../../../concepts/connect.md#endpoint) and [Database location](../../../../../concepts/connect.md#database). + +If authentication is enabled in the database, you also need to choose the [authentication mode](../../../../../concepts/connect.md#auth-modes) and obtain secrets: a token or username/password. + +Run the command as follows: + +```bash +( export <auth_mode_var>="<auth_mode_value>" && cd ydb-go-examples && \ +go run ./basic -ydb="<endpoint>?database=<database>" ) +``` + +, where + +- `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint). +- `<database>` is the [DB location](../../../../../concepts/connect.md#database). +- `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. +- `<auth_mode_value>` is the authentication parameter value for the selected mode. + +For example: + +```bash +( export YDB_ACCESS_TOKEN_CREDENTIALS="t1.9euelZqOnJuJlc..." && cd ydb-go-examples && \ +go run ./basic -ydb="grpcs://ydb.example.com:2135?database=/somepath/somelocation" ) +``` + +{% include [../../_includes/pars_from_profile_hint.md](../../_includes/pars_from_profile_hint.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md new file mode 100644 index 0000000000..26cb88a015 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md @@ -0,0 +1,7 @@ +To connect to a locally deployed YDB database according to the [Docker](../../../../../getting_started/ydb_docker.md) use case, run the following command in the default configuration: + +```bash +( export YDB_ANONYMOUS_CREDENTIALS=1 && cd ydb-go-examples && \ +go run ./basic -ydb="grpc://localhost:2136?database=/local" ) +``` + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md new file mode 100644 index 0000000000..dbb8af9b43 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md @@ -0,0 +1,12 @@ +{% list tabs %} + +- Local Docker + + {% include [run_docker.md](run_docker.md) %} + +- Any database + + {% include [run_custom.md](run_custom.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md new file mode 100644 index 0000000000..8e20c29b52 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md @@ -0,0 +1,271 @@ +# App in Go + +This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb-go-examples/tree/master/basic) that uses the {{ ydb-short-name }} [Go SDK](https://github.com/ydb-platform/ydb-go-sdk/v3). + +## Downloading and starting {#download} + +The startup script given below uses [git](https://git-scm.com/downloads) and [Go](https://go.dev/doc/install). Be sure to install the [YDB Go SDK](../../install.md) first. + +Create a working directory and use it to run from the command line the command to clone the GitHub repository: + +```bash +git clone https://github.com/ydb-platform/ydb-go-examples/ +``` + +Next, from the same working directory, run the command to start the test app. The command will differ depending on the database to connect to. + +{% include [run_options.md](_includes/run_options.md) %} + +{% include [init.md](../_includes/steps/01_init.md) %} + +To work with `YDB` in `Go`, import the `ydb-go-sdk` driver package: + +```go +import ( + // general imports + "context" + "path" + + // imports of ydb-go-sdk packages + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/table" // to work with the table service + "github.com/ydb-platform/ydb-go-sdk/v3/table/options" // to work with the table service + "github.com/ydb-platform/ydb-go-sdk/v3/table/result" // to work with the table service + "github.com/ydb-platform/ydb-go-sdk/v3/table/result/named" // to work with the table service + "github.com/ydb-platform/ydb-go-sdk-auth-environ" // for authentication using environment variables + "github.com/ydb-platform/ydb-go-yc" // to work with YDB in Yandex.Cloud +) +``` + +App code snippet for driver initialization: + +```go +ctx := context.Background() +// connection string +dsn := "grpcs://ydb.serverless.yandexcloud.net:2135/?database=/ru-central1/b1g8skpblkos03malf3s/etn01f8gv9an9sedo9fu" +// IAM token +token := "t1.9euelZrOy8aVmZKJm5HGjceMkMeVj-..." +// creating a DB connection object, which is the input point for YDB services +db, err := ydb.New( + ctx, + ydb.WithConnectionString(dsn), +// yc.WithInternalCA(), // using Yandex.Cloud certificates + ydb.WithAccessTokenCredentials(token), // token-based authentication +// ydb.WithAnonimousCredentials(token), // anonymous authentication (for example, in docker ydb) +// yc.WithMetadataCredentials(token), // authentication from inside a VM in Yandex.Cloud or a function in Yandex Functions +// yc.WithServiceAccountKeyFileCredentials("~/.ydb/sa.json"), // authentication in Yandex.Cloud using a service account file +// environ.WithEnvironCredentials(ctx), // authentication using environment variables +) +if err != nil { + // connection error handling +} +// closing the driver at the end of the program is mandatory +defer func() { + _ = db.Close(ctx) +} +``` + +The `db` object is an input point for working with `YDB` services. +To work with the table service, use the `db.Table()` client. +The client of the table service provides an `API` for making queries to tables. +The most popular method is `db.Table().Do(ctx, op)`. It implements background session creation and repeated attempts to perform the `op` user operation where the created session is passed to the user-defined code. +The session has an exhaustive `API` that lets you perform `DDL`, `DML`, `DQL`, and `TCL` requests. + +{% include [steps/02_create_table.md](../_includes/steps/02_create_table.md) %} + +To create tables, use the `table.Session.CreateTable()` method: + +```go +err = db.Table().Do( + ctx, + func(ctx context.Context, s table.Session) (err error) { + return s.CreateTable(ctx, path.Join(db.Name(), "series"), + options.WithColumn("series_id", types.Optional(types.TypeUint64)), + options.WithColumn("title", types.Optional(types.TypeUTF8)), + options.WithColumn("series_info", types.Optional(types.TypeUTF8)), + options.WithColumn("release_date", types.Optional(types.TypeDate)), + options.WithColumn("comment", types.Optional(types.TypeUTF8)), + options.WithPrimaryKeyColumn("series_id"), + ) + }, +) +if err != nil { + // handling the situation when the request failed +} +``` + +You can use the `table.Session.DescribeTable()` method to output information about the table structure and make sure that it was properly created: + +```go +err = db.Table().Do( + ctx, + func(ctx context.Context, s table.Session) (err error) { + desc, err := s.DescribeTable(ctx, path.Join(db.Name(), "series")) + if err != nil { + return + } + log.Printf("> describe table: %s\n", tableName) + for _, c := range desc.Columns { + log.Printf(" > column, name: %s, %s\n", c.Type, c.Name) + } + return + } +) +if err != nil { + // handling the situation when the request failed +} +``` + +{% include [steps/04_query_processing.md](../_includes/steps/04_query_processing.md) %} + +To execute YQL queries, use the `table.Session.Execute()` method. +The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `table.TxControl` structure. + +```go +var ( + readTx = table.TxControl( + table.BeginTx( + table.WithOnlineReadOnly(), + ), + table.CommitTx(), + ) +) +err := db.Table().Do( + ctx, + func(ctx context.Context, s table.Session) (err error) { + var ( + res result.Result + id *uint64 // pointer - for optional results + title *string // pointer - for optional results + date *time.Time // pointer - for optional results + ) + _, res, err = s.Execute( + ctx, + readTx, + ` + DECLARE $seriesID AS Uint64; + SELECT + series_id, + title, + release_date + FROM + series + WHERE + series_id = $seriesID; + `, + table.NewQueryParameters( + table.ValueParam("$seriesID", types.Uint64Value(1)), // substitution in the query condition + ), + options.WithQueryCachePolicy( + options.WithQueryCachePolicyKeepInCache(), // enabling the server cache of compiled queries + ), + ) + if err != nil { + return err + } + defer func() { + _ = res.Close() // making sure the result is closed + }() + log.Printf("> select_simple_transaction:\n") + for res.NextResultSet(ctx) { + for res.NextRow() { + // passing column names from the scanning line to ScanNamed, + // addresses (and data types) to assign query results to + err = res.ScanNamed( + named.Optional("series_id", &id), + named.Optional("title", &title), + named.Optional("release_date", &date), + ) + if err != nil { + return err + } + log.Printf( + " > %d %s %s\n", + *id, *title, *date, + ) + } + } + return res.Err() + }, +) +if err != nil { + // handling the query execution error +} +``` + +{% include [scan_query.md](../_includes/steps/08_scan_query.md) %} + +To execute scan queries, use the `table.Session.StreamExecuteScanQuery()` method. + +```go +var ( + query = ` + DECLARE $series AS List<UInt64>; + SELECT series_id, season_id, title, first_aired + FROM seasons + WHERE series_id IN $series + ` + res result.StreamResult +) +err = c.Do( + ctx, + func(ctx context.Context, s table.Session) (err error) { + res, err = s.StreamExecuteScanQuery(ctx, query, + table.NewQueryParameters( + table.ValueParam("$series", + types.ListValue( + types.Uint64Value(1), + types.Uint64Value(10), + ), + ), + ), + ) + if err != nil { + return err + } + defer func() { + _ = res.Close() // making sure the result is closed + }() + var ( + seriesID uint64 + seasonID uint64 + title string + date time.Time + ) + log.Print("\n> scan_query_select:") + for res.NextResultSet(ctx) { + if err = res.Err(); err != nil { + return err + } + for res.NextRow() { + // named.OptionalOrDefault lets you "deploy" optional + // results or use the default value of the go type + err = res.ScanNamed( + named.OptionalOrDefault("series_id", &seriesID), + named.OptionalOrDefault("season_id", &seasonID), + named.OptionalOrDefault("title", &title), + named.OptionalOrDefault("first_aired", &date), + ) + if err != nil { + return err + } + log.Printf("# Season, SeriesId: %d, SeasonId: %d, Title: %s, Air date: %s", seriesID, seasonID, title, date) + } + } + return res.Err() + }, +) +if err != nil { + // handling the query execution error +} +``` + +{% note info %} + +Sample code of a test app that uses archived of versions the Go SDK: + +- [github.com/yandex-cloud/ydb-go-sdk](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) is available at this [link](../archive/example-go-v1.md), +- [github.com/yandex-cloud/ydb-go-sdk/v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2) is available at this [link](../archive/example-go-v2.md). + +{% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md new file mode 100644 index 0000000000..7e86eb7c62 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md @@ -0,0 +1,27 @@ +To run the example using any available YDB database, you need to know the [Endpoint](../../../../../concepts/connect.md#endpoint) and [Database location](../../../../../concepts/connect.md#database). + +If authentication is enabled in the database, you also need to choose the [authentication mode](../../../../../concepts/connect.md#auth-modes) and obtain secrets: a token or username/password. + +Run the command as follows: + +```bash +<auth_mode_var>="<auth_mode_value>" \ +python3 ydb-python-sdk/examples/basic_example_v1/ -e <endpoint> -d <database> +``` + +, where + +- `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint). +- `<database>` is the [DB location](../../../../../concepts/connect.md#database). +- `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. +- `<auth_mode_value>` is the authentication parameter value for the selected mode. + +For example: + +```bash +YDB_ACCESS_TOKEN_CREDENTIALS="t1.9euelZqOnJuJlc..." \ +python3 ydb-python-sdk/examples/basic_example_v1/ -e grpcs://ydb.example.com:2135 -d /path/db ) +``` + +{% include [../../_includes/pars_from_profile_hint.md](../../_includes/pars_from_profile_hint.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md new file mode 100644 index 0000000000..104391121d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md @@ -0,0 +1,7 @@ +To connect to a locally deployed YDB database according to the [Docker](../../../../../getting_started/ydb_docker.md) use case, run the following command in the default configuration: + +```bash +YDB_ANONYMOUS_CREDENTIALS=1 \ +python3 ydb-python-sdk/examples/basic_example_v1/ -e grpc://localhost:2136 -d /local +``` + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md new file mode 100644 index 0000000000..dbb8af9b43 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md @@ -0,0 +1,12 @@ +{% list tabs %} + +- Local Docker + + {% include [run_docker.md](run_docker.md) %} + +- Any database + + {% include [run_custom.md](run_custom.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md new file mode 100644 index 0000000000..071112ecac --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md @@ -0,0 +1,230 @@ +# App in Python + +This page contains a detailed description of the code of a [test app](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/basic_example_v1) that is available as part of the {{ ydb-short-name }} [Python SDK](https://github.com/yandex-cloud/ydb-python-sdk). + +## Downloading and starting {#download} + +The start scenario given below uses [git](https://git-scm.com/downloads) and [Python3](https://www.python.org/downloads/). Be sure to install the [YDB Python SDK](../../install.md) first. + +Create a working directory and use it to run from the command line the command to clone the GitHub repository and install the necessary Python packages: + +```bash +git clone https://github.com/ydb-platform/ydb-python-sdk.git +python3 -m pip install iso8601 +``` + +Next, from the same working directory, run the command to start the test app. The command will differ depending on the database to connect to. + +{% include [run_options.md](_includes/run_options.md) %} + +{% include [init.md](../_includes/steps/01_init.md) %} + +App code snippet for driver initialization: + +```python +def run(endpoint, database, path): + driver_config = ydb.DriverConfig( + endpoint, database, credentials=ydb.construct_credentials_from_environ(), + root_certificates=ydb.load_ydb_root_certificate(), + ) + with ydb.Driver(driver_config) as driver: + try: + driver.wait(timeout=5) + except TimeoutError: + print("Connect failed to YDB") + print("Last reported errors by discovery:") + print(driver.discovery_debug_details()) + exit(1) +``` + +App code snippet for creating a session: + +```python +session = driver.table_client.session().create() +``` + +{% include [create_table.md](../_includes/steps/02_create_table.md) %} + +To create tables, use the `session.create_table()` method: + +```python +def create_tables(session, path): + session.create_table( + os.path.join(path, 'series'), + ydb.TableDescription() + .with_column(ydb.Column('series_id', ydb.OptionalType(ydb.PrimitiveType.Uint64))) + .with_column(ydb.Column('title', ydb.OptionalType(ydb.PrimitiveType.Utf8))) + .with_column(ydb.Column('series_info', ydb.OptionalType(ydb.PrimitiveType.Utf8))) + .with_column(ydb.Column('release_date', ydb.OptionalType(ydb.PrimitiveType.Uint64))) + .with_primary_key('series_id') + ) +``` + +You can use the `session.describe_table()` method to output information about the table structure and make sure that it was properly created: + +```python +def describe_table(session, path, name): + result = session.describe_table(os.path.join(path, name)) + print("\n> describe table: series") + for column in result.columns: + print("column, name:", column.name, ",", str(column.type.item).strip()) +``` + +The given code snippet outputs the following text to the console at startup: + +```bash +> describe table: series +('column, name:', 'series_id', ',', 'type_id: UINT64') +('column, name:', 'title', ',', 'type_id: UTF8') +('column, name:', 'series_info', ',', 'type_id: UTF8') +('column, name:', 'release_date', ',', 'type_id: UINT64') +``` + +{% include [steps/03_write_queries.md](../_includes/steps/03_write_queries.md) %} + +Code snippet for inserting and updating data: + +```python +def upsert_simple(session, path): + session.transaction().execute( + """ + PRAGMA TablePathPrefix("{}"); + UPSERT INTO episodes (series_id, season_id, episode_id, title) VALUES + (2, 6, 1, "TBD"); + """.format(path), + commit_tx=True, + ) +``` + +{% include [pragmatablepathprefix.md](../_includes/auxilary/pragmatablepathprefix.md) %} + +{% include [steps/04_query_processing.md](../_includes/steps/04_query_processing.md) %} + +To execute YQL queries, use the `session.transaction().execute()` method. +The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `TxControl` class. + +In the code snippet below, the transaction is executed using the `transaction().execute()` method. The transaction execution mode set is `ydb.SerializableReadWrite()`. When all the queries in the transaction are completed, the transaction is automatically committed by explicitly setting the flag: `commit_tx=True`. The query body is described using the YQL syntax and is passed to the `execute` method as a parameter. + +```python +def select_simple(session, path): + result_sets = session.transaction(ydb.SerializableReadWrite()).execute( + """ + PRAGMA TablePathPrefix("{}"); + $format = DateTime::Format("%Y-%m-%d"); + SELECT + series_id, + title, + $format(DateTime::FromSeconds(CAST(DateTime::ToSeconds(DateTime::IntervalFromDays(CAST(release_date AS Int16))) AS Uint32))) AS release_date + FROM series + WHERE series_id = 1; + """.format(path), + commit_tx=True, + ) + print("\n> select_simple_transaction:") + for row in result_sets[0].rows: + print("series, id: ", row.series_id, ", title: ", row.title, ", release date: ", row.release_date) + + return result_sets[0] +``` + +When the query is executed, `result_set` is returned whose iteration outputs the following text to the console: + +```bash +> SelectSimple: +series, Id: 1, title: IT Crowd, Release date: 2006-02-03 +``` + +{% include [param_prep_queries.md](../_includes/steps/07_param_prep_queries.md) %} + +```python +def select_prepared(session, path, series_id, season_id, episode_id): + query = """ + PRAGMA TablePathPrefix("{}"); + DECLARE $seriesId AS Uint64; + DECLARE $seasonId AS Uint64; + DECLARE $episodeId AS Uint64; + $format = DateTime::Format("%Y-%m-%d"); + SELECT + title, + $format(DateTime::FromSeconds(CAST(DateTime::ToSeconds(DateTime::IntervalFromDays(CAST(air_date AS Int16))) AS Uint32))) AS air_date + FROM episodes + WHERE series_id = $seriesId AND season_id = $seasonId AND episode_id = $episodeId; + """.format(path) + + prepared_query = session.prepare(query) + result_sets = session.transaction(ydb.SerializableReadWrite()).execute( + prepared_query, { + '$seriesId': series_id, + '$seasonId': season_id, + '$episodeId': episode_id, + }, + commit_tx=True + ) + print("\n> select_prepared_transaction:") + for row in result_sets[0].rows: + print("episode title:", row.title, ", air date:", row.air_date) + + return result_sets[0] +``` + +The given code snippet outputs the following text to the console at startup: + +```bash +> select_prepared_transaction: +('episode title:', u'To Build a Better Beta', ', air date:', '2016-06-05') +``` + +{% include [scan_query.md](../_includes/steps/08_scan_query.md) %} + +```python +def executeScanQuery(driver): + query = ydb.ScanQuery(""" + SELECT series_id, season_id, COUNT(*) AS episodes_count + FROM episodes + GROUP BY series_id, season_id + ORDER BY series_id, season_id + """, {}) + + it = driver.table_client.scan_query(query) + + while True: + try: + result = next(it) + print result.result_set.rows + except StopIteration: + break +``` + +{% include [transaction_control.md](../_includes/steps/10_transaction_control.md) %} + +Code snippet for `transaction().begin()` and `tx.Commit()` calls: + +```python +def explicit_tcl(session, path, series_id, season_id, episode_id): + query = """ + PRAGMA TablePathPrefix("{}"); + + DECLARE $seriesId AS Uint64; + DECLARE $seasonId AS Uint64; + DECLARE $episodeId AS Uint64; + + UPDATE episodes + SET air_date = CAST(CurrentUtcDate() AS Uint64) + WHERE series_id = $seriesId AND season_id = $seasonId AND episode_id = $episodeId; + """.format(path) + prepared_query = session.prepare(query) + + tx = session.transaction(ydb.SerializableReadWrite()).begin() + + tx.execute( + prepared_query, { + '$seriesId': series_id, + '$seasonId': season_id, + '$episodeId': episode_id + } + ) + + print("\n> explicit TCL call") + + tx.commit() +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/example/toc_i.yaml index 54f8e4e338..8f3c58e838 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-sdk/example/toc_i.yaml @@ -3,11 +3,10 @@ items: href: index.md - name: C++ href: example-cpp.md - when: oss - name: C# (.NET) href: example-dotnet.md - name: Go - href: example-go.md + href: go/index.md - name: Java href: example-java.md - name: Node.js @@ -15,4 +14,10 @@ items: - name: PHP href: example-php.md - name: Python - href: example-python.md
\ No newline at end of file + href: python/index.md +- name: Archive + items: + - name: Go v1 + href: archive/example-go-v1.md + - name: Go v2 + href: archive/example-go-v2.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/toc_p.yaml b/ydb/docs/en/core/reference/ydb-sdk/example/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/example/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/index.md b/ydb/docs/en/core/reference/ydb-sdk/index.md index 83b2b05fc1..53ac869580 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/index.md @@ -1,3 +1,3 @@ -{% include [intro.md](_includes/index/intro.md) %} +{% include [index.md](_includes/index.md) %} + -{% include [examples.md](_includes/index/examples.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/install.md b/ydb/docs/en/core/reference/ydb-sdk/install.md new file mode 100644 index 0000000000..898729dbbe --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/install.md @@ -0,0 +1,2 @@ + +{% include [install.md](_includes/install.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md new file mode 100644 index 0000000000..22977978e8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md @@ -0,0 +1,6 @@ +{% note info %} + +The article is being updated. + +{% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/index.md new file mode 100644 index 0000000000..f796721f42 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/index.md @@ -0,0 +1,6 @@ +# Code recipes + +{% include [work in progress message](addition.md) %} + +This section contains code recipes in different programming languages for a variety of tasks that are common when working with the {{ ydb-short-name }} SDK. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit.md new file mode 100644 index 0000000000..cb4507ed54 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit.md @@ -0,0 +1,20 @@ +# Setting the session pool size + +{% include [work in progress message](addition.md) %} + +The client's session pool size affects resource consumption (RAM, CPU) on the server side of YDB. +Simple math: if `1000` clients of the same DB have `1000` sessions each, `100000` actors (workers, session performers) are created on the server side. If you don't limit the number of sessions on the client, this may result in a slow cluster that is close to a failure. +By default, the `YDB SDK` limits the number of sessions to `50`. +A good recommendation is to set the limit on the number of client sessions to the minimum required for the normal operation of the client app. Keep in mind that sessions are single-threaded both on the server and client side. So if the application needs to make `1000` simultaneous (`inflight`) requests to `YDB` for its estimated load, the limit should be set to `1000` sessions. +Here it's necessary to distinguish between the estimated `RPS` (requests per second) and `inflight`. In the first case, this is the total number of requests to `YDB` completed within `1` second. For example, if `RPS`=`10000` and the average `latency` is `100`ms, it's sufficient to set the session limit to `1000`. This means that each session will perform an average of `10` consecutive requests for the estimated second. + +Below are examples of the code for setting the session pool limit in different `YDB SDKs`. + +{% list tabs %} + +- Go + + {% include [go.md](session_pool_limit/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit/go.md new file mode 100644 index 0000000000..ce1e15b6f3 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/session_pool_limit/go.md @@ -0,0 +1,23 @@ +```go +package main + +import ( + "context" + + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + db, err := ydb.New( + ctx, + ... + ydb.WithSessionPoolSizeLimit(500), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md new file mode 100644 index 0000000000..43c59e5f93 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md @@ -0,0 +1,6 @@ +{% note info %} + +The feature is not implemented or the documentation is being developed. + +{% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md new file mode 100644 index 0000000000..e71825b7da --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md @@ -0,0 +1,14 @@ +# Authentication using a token + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for authentication using a token in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](access_token/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token/go.md new file mode 100644 index 0000000000..a48a856189 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token/go.md @@ -0,0 +1,26 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + ydb.WithAccessTokenCredentials(os.Getenv("YDB_TOKEN")), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md new file mode 100644 index 0000000000..5e025a6733 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md @@ -0,0 +1,14 @@ +# Anonymous authentication + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for anonymous authentication in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](anonymous/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous/go.md new file mode 100644 index 0000000000..a8adc0e90b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous/go.md @@ -0,0 +1,25 @@ +```go +package main + +import ( + "context" + + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + ydb.WithAnonymousCredentials(), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md new file mode 100644 index 0000000000..e643f8857c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md @@ -0,0 +1,23 @@ +# Authentication using environment variables + +{% include [work in progress message](../../_includes/addition.md) %} + +When using this method, the authentication mode and its parameters are defined by the environment that an application is run in, [as described here](../../../auth.md#env). + +By setting one of the following environment variables, you can control the authentication method: + +* `YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS=<path/to/sa_key_file>`: Use a service account file in Yandex.Cloud. +* `YDB_ANONYMOUS_CREDENTIALS="1"`: Use anonymous authentication. Relevant for testing against a Docker container with {{ ydb-short-name }}. +* `YDB_METADATA_CREDENTIALS="1"`: Use the metadata service inside Yandex.Cloud (a Yandex function or a VM). +* `YDB_ACCESS_TOKEN_CREDENTIALS=<access_token>`: Use token-based authentication. + +Below are examples of the code for authentication using environment variables in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](env/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env/go.md new file mode 100644 index 0000000000..3f5b0aa053 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env/go.md @@ -0,0 +1,27 @@ +```go +package main + +import ( + "context" + "os" + + environ "github.com/ydb-platform/ydb-go-sdk-auth-environ" + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + environ.WithEnvironCredentials(ctx), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md new file mode 100644 index 0000000000..6b2b0ace42 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md @@ -0,0 +1,8 @@ +# Authentication + +{% include [work in progress message](../../_includes/addition.md) %} + +{{ ydb-short-name }} supports multiple authentication methods when connecting to the server side. Each of them is usually specific to a particular environment pair: where the client application is located (in the trusted {{ ydb-short-name }} zone or outside it) and the {{ ydb-short-name }} server side (a Docker container, Yandex.Cloud, data cloud, or deployment on a separate cluster). + +This section contains code recipes with authentication settings in different {{ ydb-short-name }} SDKs. For a general description of the SDK authentication principles, see the [Authentication in an SDK](../../../auth.md) article. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md new file mode 100644 index 0000000000..9241d3db47 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md @@ -0,0 +1,14 @@ +# Authentication using the metadata service + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for authentication using environment variables in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](metadata/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata/go.md new file mode 100644 index 0000000000..ac6c708642 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata/go.md @@ -0,0 +1,28 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + yc "github.com/ydb-platform/ydb-go-yc" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + yc.WithMetadataCredentials(ctx), + yc.WithInternalCA(), // append Yandex Cloud certificates + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md new file mode 100644 index 0000000000..1ffe2f6ab5 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md @@ -0,0 +1,14 @@ +# Authentication using a service account file + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for authentication using a service account file in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](service_account/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account/go.md new file mode 100644 index 0000000000..9672f1328e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account/go.md @@ -0,0 +1,30 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + yc "github.com/ydb-platform/ydb-go-yc" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + yc.WithServiceAccountKeyFileCredentials( + os.Getenv("YDB_SERVICE_ACCOUNT_KEY_FILE_CREDENTIALS"), + ), + yc.WithInternalCA(), // append Yandex Cloud certificates + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md new file mode 100644 index 0000000000..5d9a6a938d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md @@ -0,0 +1,10 @@ +# Username and password based authentication + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for authentication based on a username and token in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/access_token.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/access_token.md new file mode 100644 index 0000000000..606d617e7d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/access_token.md @@ -0,0 +1,3 @@ + +{% include [access_token.md](_includes/access_token.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/anonymous.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/anonymous.md new file mode 100644 index 0000000000..49b00e9286 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/anonymous.md @@ -0,0 +1,2 @@ + +{% include [anonymous.md](_includes/anonymous.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/env.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/env.md new file mode 100644 index 0000000000..004e4f5d39 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/env.md @@ -0,0 +1 @@ +{% include [index.md](_includes/env.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/index.md new file mode 100644 index 0000000000..bd4e95cb41 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/index.md @@ -0,0 +1,3 @@ + +{% include [index.md](_includes/index.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/metadata.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/metadata.md new file mode 100644 index 0000000000..d15b521add --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/metadata.md @@ -0,0 +1 @@ +{% include [index.md](_includes/metadata.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/service_account.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/service_account.md new file mode 100644 index 0000000000..70996edf71 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/service_account.md @@ -0,0 +1 @@ +{% include [index.md](_includes/service_account.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/static.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/static.md new file mode 100644 index 0000000000..6ee75b9186 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/static.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/static.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_i.yaml new file mode 100644 index 0000000000..e6b09bf483 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_i.yaml @@ -0,0 +1,15 @@ +items: +- name: Overview + href: index.md +- name: Using a token + href: access_token.md +- name: Anonymous + href: anonymous.md +- name: Service account file + href: service_account.md +- name: Metadata service + href: metadata.md +- name: Using environment variables + href: env.md +- name: Username and password based + href: static.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_p.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md new file mode 100644 index 0000000000..9dc2b596c2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md @@ -0,0 +1,9 @@ +# Balancing + +{% include [work in progress message](../../_includes/addition.md) %} + +{{ ydb-short-name }} uses client load balancing because it is more efficient when a lot of traffic from multiple client applications comes to a database. +In most cases, it just works in the {{ ydb-short-name }} SDK. However, sometimes specific settings for client load balancing are required, for example, to reduce server hops and request time or to distribute the load across availability zones. + +This section contains code recipes with client load balancing settings in different {{ ydb-short-name }} SDKs. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md new file mode 100644 index 0000000000..9a50784323 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md @@ -0,0 +1,14 @@ +# Prefer the nearest data center + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for setting the "prefer the nearest data center" balancing algorithm option in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](prefer_local/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local/go.md new file mode 100644 index 0000000000..44b5676a1c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local/go.md @@ -0,0 +1,31 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/balancers" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + ydb.WithBalancer( + balancers.PreferLocalDC( + balancers.RandomChoice(), + ), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md new file mode 100644 index 0000000000..b93efe00d8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md @@ -0,0 +1,14 @@ +# Prefer the availability zone + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for setting the "prefer the availability zone" balancing algorithm option in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](prefer_location/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location/go.md new file mode 100644 index 0000000000..440f9827af --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location/go.md @@ -0,0 +1,33 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/balancers" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + ydb.WithBalancer( + balancers.PreferLocations( + balancers.RandomChoice(), + "MAN", + "VLA", + ), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md new file mode 100644 index 0000000000..a42b83814f --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md @@ -0,0 +1,16 @@ +# Random choice + +{% include [work in progress message](../../_includes/addition.md) %} + +The {{ ydb-short-name }} SDK uses the `random_choice` algorithm by default. + +Below are examples of the code for forced setting of the "random choice" balancing algorithm in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](random_choice/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice/go.md new file mode 100644 index 0000000000..b97c7ea77c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice/go.md @@ -0,0 +1,29 @@ +```go +package main + +import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/balancers" +) + +func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + db, err := ydb.New( + ctx, + ... + ydb.WithBalancer( + balancers.RandomChoice(), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/index.md new file mode 100644 index 0000000000..e5687972c2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/index.md @@ -0,0 +1,3 @@ + +{% include [balancing.md](_includes/index.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_local.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_local.md new file mode 100644 index 0000000000..ba06e91a9a --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_local.md @@ -0,0 +1 @@ +{% include [index.md](_includes/prefer_local.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_location.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_location.md new file mode 100644 index 0000000000..825e8ccb44 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/prefer_location.md @@ -0,0 +1 @@ +{% include [index.md](_includes/prefer_location.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/random_choice.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/random_choice.md new file mode 100644 index 0000000000..ecd24b7afc --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/random_choice.md @@ -0,0 +1 @@ +{% include [index.md](_includes/random_choice.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_i.yaml new file mode 100644 index 0000000000..780d63a598 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_i.yaml @@ -0,0 +1,9 @@ +items: +- name: Overview + href: index.md +- name: Random choice + href: random_choice.md +- name: Prefer the nearest data center + href: prefer_local.md +- name: Prefer the availability zone + href: prefer_location.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_p.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md new file mode 100644 index 0000000000..5b91066120 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md @@ -0,0 +1,8 @@ +# Troubleshooting + +{% include [work in progress message](../../_includes/addition.md) %} + +When troubleshooting issues with {{ ydb-short-name }}, diagnostics tools such as logging, metrics, OpenTracing/Jaeger tracing are helpful. We strongly recommend that you enable them in advance before any problems occur. This will help see changes in the overall picture before, during, and after an issue when troubleshooting it. This greatly speeds up our investigation into incidents and lets us provide assistance much faster. + +This section contains code recipes for enabling diagnostics tools in different {{ ydb-short-name }} SDKs + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md new file mode 100644 index 0000000000..45e762184e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md @@ -0,0 +1,14 @@ +# Enabling tracing in Jaeger + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code enabling Jaeger tracing in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](jaeger/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger/go.md new file mode 100644 index 0000000000..564e9f5521 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger/go.md @@ -0,0 +1,64 @@ +```go +package main + +import ( + "context" + "time" + + "github.com/opentracing/opentracing-go" + jaegerConfig "github.com/uber/jaeger-client-go/config" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/trace" + + tracing "github.com/ydb-platform/ydb-go-sdk-opentracing" +) + +const ( + tracerURL = "localhost:5775" + serviceName = "ydb-go-sdk" +) + +func main() { + tracer, closer, err := jaegerConfig.Configuration{ + ServiceName: serviceName, + Sampler: &jaegerConfig.SamplerConfig{ + Type: "const", + Param: 1, + }, + Reporter: &jaegerConfig.ReporterConfig{ + LogSpans: true, + BufferFlushInterval: 1 * time.Second, + LocalAgentHostPort: tracerURL, + }, + }.NewTracer() + if err != nil { + panic(err) + } + + defer closer.Close() + + // set global tracer of this application + opentracing.SetGlobalTracer(tracer) + + span, ctx := opentracing.StartSpanFromContext(context.Background(), "client") + defer span.Finish() + + db, err := ydb.New( + ctx, + ... + ydb.WithTraceDriver(tracing.Driver( + tracing.WithDetails(trace.DetailsAll), + )), + ydb.WithTraceTable(tracing.Table( + tracing.WithDetails(trace.DetailsAll), + )), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md new file mode 100644 index 0000000000..e5311ad598 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md @@ -0,0 +1,14 @@ +# Enabling logging + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code that enables logging in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](logs/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md new file mode 100644 index 0000000000..22fad3b9af --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md @@ -0,0 +1,119 @@ +There are several ways to enable logs in an application that uses `ydb-go-sdk`: + +* Set the environment variable `YDB_LOG_SEVERITY_LEVEL=info` (possible values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `quiet`, defaults to `quiet`). +This environment variable enables the built-in `ydb-go-sdk` logger (synchronous, non-block) with output to the standard output stream. +* {% cut "Connect a outside logger `go.uber.org/zap`" %} + ```go + package main + + import ( + "context" + + "go.uber.org/zap" + + ydbZap "github.com/ydb-platform/ydb-go-sdk-zap" + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/trace" + ) + + func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var log *zap.Logger // zap-logger with init out of this scope + db, err := ydb.New( + ctx, + ... + ydbZap.WithTraces( + log, + ydbZap.WithDetails(trace.DetailsAll), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() + } + ``` + + {% endcut %} +* {% cut "Connect a outside logger `github.com/rs/zerolog`" %} + + ```go + package main + + import ( + "context" + "os" + + "github.com/rs/zerolog" + + ydbZerolog "github.com/ydb-platform/ydb-go-sdk-zerolog" + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/trace" + ) + + func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var log zerolog.Logger // zap-logger with init out of this scope + db, err := ydb.New( + ctx, + ydb.WithConnectionString(os.Getenv("YDB_CONNECTION_STRING")), + ydbZerolog.WithTraces( + log, + ydbZerolog.WithDetails(trace.DetailsAll), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() + } + ``` + + {% endcut %} +* {% cut "Enable your own logger implementation `github.com/ydb-platform/ydb-go-sdk/v3/log.Logger`" %} + + ```go + package main + + import ( + "context" + "os" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/log" + "github.com/ydb-platform/ydb-go-sdk/v3/trace" + ) + + func main() { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + var logger log.Logger // logger implementation with init out of this scope + db, err := ydb.New( + ctx, + ydb.WithConnectionString(os.Getenv("YDB_CONNECTION_STRING")), + ydb.WithLogger( + trace.DetailsAll, + ydb.WithExternalLogger(logger), + ), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() + } + ``` + + {% endcut %} + +{% include [overlay](go_appendix.md) %} + +* Implement your own logging package based on the `github.com/ydb-platform/ydb-go-sdk/v3/trace` packet. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go_appendix.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go_appendix.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go_appendix.md diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md new file mode 100644 index 0000000000..83f44762e0 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md @@ -0,0 +1,14 @@ +# Enabling metrics in Prometheus + +{% include [work in progress message](../../_includes/addition.md) %} + +Below are examples of the code for enabling metrics in Prometheus in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](prometheus/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus/go.md new file mode 100644 index 0000000000..8cf3ed707d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus/go.md @@ -0,0 +1,34 @@ +```go +package main + +import ( + "context" + + "github.com/prometheus/client_golang/prometheus" + metrics "github.com/ydb-platform/ydb-go-sdk-prometheus" + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + ctx := context.Background() + registry := prometheus.NewRegistry() + db, err := ydb.New( + ctx, + ... + ydb.WithTraceDriver(metrics.Driver( + registry, + metrics.WithSeparator("_"), + )), + ydb.WithTraceTable(metrics.Table( + registry, + metrics.WithSeparator("_"), + )), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/index.md new file mode 100644 index 0000000000..e5687972c2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/index.md @@ -0,0 +1,3 @@ + +{% include [balancing.md](_includes/index.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/jaeger.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/jaeger.md new file mode 100644 index 0000000000..4941b019de --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/jaeger.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/jaeger.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/logs.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/logs.md new file mode 100644 index 0000000000..c37ce7b35e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/logs.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/logs.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/prometheus.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/prometheus.md new file mode 100644 index 0000000000..33c41f2c80 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/prometheus.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/prometheus.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_i.yaml new file mode 100644 index 0000000000..fa5d09fcc6 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_i.yaml @@ -0,0 +1,9 @@ +items: +- name: Overview + href: index.md +- name: Enable logging + href: logs.md +- name: Enable metrics in Prometheus + href: prometheus.md +- name: Enable tracing in Jaeger + href: jaeger.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_p.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/index.md new file mode 100644 index 0000000000..bd4e95cb41 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/index.md @@ -0,0 +1,3 @@ + +{% include [index.md](_includes/index.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md new file mode 100644 index 0000000000..d8f8a19ec9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md @@ -0,0 +1,66 @@ +In the {{ ydb-short-name }} Go SDK, correct error handling is implemented by several programming interfaces: + +* The basic logic of error handling is implemented by the helper `retry.Retry` function +The details of the execution of repeated requests are hidden as much as possible. +The user can influence the logic of the function `retry.Retry` using two methods: + - Via the context (where you can set the deadline and cancel). + - Via the operation's idempotency flag `retry.WithIdempotent()`. By default, the operation is considered non-idempotent. + + The user passes a custom function to `retry.Retry` that returns an error by its signature. +If the custom function returns `nil`, then repeat queries stop. +If the custom function returns an error, the {{ ydb-short-name }} Go SDK tries to identify this error and executes retries depending on it. + + {% cut "Example code using the function `retry.Retry`:" %} + + ```go + package main + + import ( + "context" + "time" + + "github.com/ydb-platform/ydb-go-sdk/v3" + "github.com/ydb-platform/ydb-go-sdk/v3/retry" + ) + + func main() { + db, err := ydb.New( + ctx, + ... + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() + var cancel context.CancelFunc + // fix deadline for retries + ctx, cancel := context.WithTimeout(ctx, time.Second) + err = retry.Retry( + ctx, + func(ctx context.Context) error { + whoAmI, err := db.Discovery().WhoAmI(ctx) + if err != nil { + return err + } + fmt.Println(whoAmI) + }, + retry.WithIdempotent(), + retry.WithTrace(trace.Trace{}), + retry.WithID("my high-level call"), + ) + if err != nil { + panic(err) + } + } + ``` + + {% endcut %} + +* The `db.Table()` table query service immediately provides the `table.Client` programming interface that uses the `retry` package and tracks the lifetime of the {{ ydb-short-name }} sessions. +Two public functions are available to the user: `db.Table().Do(ctx, op)` (where `op` provides a session) and `db.Table().DoTx(ctx, op)` (where `op` provides a transaction). +As in the previous case, the user can affect the logic of repeat queries using the context and the idempotence flag, while the {{ ydb-short-name }} Go SDK interprets errors returned by `op`. + +* Queries to other {{ ydb-short-name }} services (`db.Scripting()`, `db.Scheme()`, `db.Coordination()`, `db.Ratelimiter()`, and `db.Discovery()`) also use the `retry.Retry` function internally to make repeat queries. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md new file mode 100644 index 0000000000..120aa1922c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md @@ -0,0 +1,20 @@ +# Retrying + +{% include [work in progress message](../_includes/addition.md) %} + +{{ ydb-short-name }} is a distributed database management system with automatic load scaling. +Routine maintenance can be carried out on the server side, with server racks or entire data centers temporarily shut down. +This may result in errors arising from {{ ydb-short-name }} operation. +There are different response scenarios depending on the error type. +{{ ydb-short-name }} To ensure high database availability, SDKs provide built-in tools for retries, accounting for error types and responses to them. + +Below are code examples showing the {{ ydb-short-name }} SDK built-in tools for retries: + +{% list tabs %} + +- Go + + {% include [go.md](_includes/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit.md new file mode 100644 index 0000000000..6f3b0c252d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/session_pool_limit.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/_includes/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/_includes/go.md new file mode 100644 index 0000000000..ce1e15b6f3 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/_includes/go.md @@ -0,0 +1,23 @@ +```go +package main + +import ( + "context" + + "github.com/ydb-platform/ydb-go-sdk/v3" +) + +func main() { + db, err := ydb.New( + ctx, + ... + ydb.WithSessionPoolSizeLimit(500), + ) + if err != nil { + panic(err) + } + defer func() { + _ = db.Close(ctx) + }() +} +``` diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md new file mode 100644 index 0000000000..2cc5bc47a9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md @@ -0,0 +1,20 @@ +# Setting the session pool size + +{% include [work in progress message](../_includes/addition.md) %} + +The client's session pool size affects resource consumption (RAM, CPU) on the server side of {{ ydb-short-name }}. +Simple math: if `1000` clients of the same DB have `1000` sessions each, `100000` actors (workers, session performers) are created on the server side. If you don't limit the number of sessions on the client, this may result in a slow cluster that is close to a failure. +By default, the {{ ydb-short-name }} SDK limits the number of sessions to `50`. +A good recommendation is to set the limit on the number of client sessions to the minimum required for the normal operation of the client app. Keep in mind that sessions are single-threaded both on the server and client side. So if the application needs to make `1000` simultaneous (`inflight`) requests to {{ ydb-short-name }} for its estimated load, the limit should be set to `1000` sessions. +Here it's necessary to distinguish between the estimated `RPS` (requests per second) and `inflight`. In the first case, this is the total number of requests to {{ ydb-short-name }} completed within `1` second. For example, if `RPS`=`10000` and the average `latency` is `100`ms, it's sufficient to set the session limit to `1000`. This means that each session will perform an average of `10` consecutive requests for the estimated second. + +Below are examples of the code for setting the session pool limit in different {{ ydb-short-name }} SDKs. + +{% list tabs %} + +- Go + + {% include [go.md](_includes/go.md) %} + +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_i.yaml new file mode 100644 index 0000000000..4443196364 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_i.yaml @@ -0,0 +1,13 @@ +items: +- name: Overview + href: index.md +- name: Authentication + include: { mode: link, path: auth/toc_p.yaml } +- name: Balancing + include: { mode: link, path: balancing/toc_p.yaml } +- name: Running repeat queries + href: retry/index.md +- name: Troubleshooting + include: { mode: link, path: debug/toc_p.yaml } +- name: Setting the session pool size + href: session_pool_limit/index.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_p.yaml b/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_p.yaml new file mode 100644 index 0000000000..5bfec4365d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/toc_p.yaml @@ -0,0 +1,2 @@ +items: +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml index 3f349b1c75..7a2fdca571 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml @@ -1,7 +1,14 @@ items: - name: Overview href: index.md + - name: Install + href: install.md + - name: Authentication + href: auth.md - name: Test app - include: { mode: link, path: example/toc_i.yaml } + include: { mode: link, path: example/toc_p.yaml } - name: Handling errors in the API - href: error_handling.md
\ No newline at end of file + href: error_handling.md + - name: Code recipes + include: { mode: link, path: recipes/toc_p.yaml } + diff --git a/ydb/docs/en/core/toc_p.yaml b/ydb/docs/en/core/toc_p.yaml index 34abacb52e..b3bc4402e0 100644 --- a/ydb/docs/en/core/toc_p.yaml +++ b/ydb/docs/en/core/toc_p.yaml @@ -1,29 +1,4 @@ -title: Yandex Database +title: YDB href: index.yaml items: -# Header -- name: Contents - href: index.yaml -- name: Getting started - items: - - name: Overview - href: getting_started/index.md - - include: { mode: link, path: getting_started/toc_i.yaml } - -# Main -# - { name: Step-by-step instructions, include: { mode: link, path: solutions/toc_i.yaml }, when: audience == "external" } -- { name: Concepts, include: { mode: link, path: concepts/toc_p.yaml } } -- { name: Step-by-step instructions, include: { mode: link, path: operations/toc_p.yaml } } - -- { name: Recommendations, include: { mode: link, path: best_practices/toc_p.yaml } } -- { name: Maintenance, include: { mode: link, path: maintenance/toc_p.yaml } } -- { name: Diagnostics, include: { mode: link, path: troubleshooting/toc_p.yaml } } - -# References -- { name: YQL, include: { mode: link, path: yql/toc_p.yaml } } -- { name: Working with the YDB CLI, include: { mode: link, path: reference/ydb-cli/toc_p.yaml } } -- { name: Working with the YDB SDK, include: { mode: link, path: reference/ydb-sdk/toc_p.yaml } } - -# Footer -- { name: Questions and answers, include: { mode: link, path: faq/toc_p.yaml } } - +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md b/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md index d5ce8b7052..e5ae359460 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md +++ b/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md @@ -1,10 +1,10 @@ -## {{ ydb-full-name }} {#ydb} +## {{ ydb-short-name }} {#ydb} ### Resource usage metrics {#resources} | Metric name<br/>Type, units of measurement | Description<br/>Labels | | ----- | ----- | -| `resources.storage.used_bytes`<br/>`IGAUGE`, bytes | The size of user and service data stored in distributed network storage. The service data includes the data of the primary and [secondary indexes](https://cloud.yandex.ru/docs/ydb/oss/public/develop/concepts/secondary_indexes). | +| `resources.storage.used_bytes`<br/>`IGAUGE`, bytes | The size of user and service data stored in distributed network storage. The service data includes the data of the primary and [secondary indexes](https://cloud.yandex.com/en/docs/ydb/oss/public/develop/concepts/secondary_indexes). | | `resources.storage.limit_bytes`<br/>`IGAUGE`, bytes | A limit on the size of user and service data that a database can store in distributed network storage. | ### API metrics {#api} @@ -16,9 +16,9 @@ | `api.grpc.request.inflight_count`<br/>`IGAUGE`, pieces | The number of requests that a database is simultaneously handling in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | | `api.grpc.request.inflight_bytes`<br/>`IGAUGE`, bytes | The size of requests that a database is simultaneously handling in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | | `api.grpc.response.bytes`<br/>`RATE`, bytes | The size of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | -| `api.grpc.response.count`<br/>`RATE`, pieces | The number of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`.<br/>- _status_: The query execution status. To learn more about statuses, see [Error handling](https://cloud.yandex.ru/docs/ydb/oss/public/reference/ydb-sdk/error_handling). | +| `api.grpc.response.count`<br/>`RATE`, pieces | The number of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`.<br/>- _status_: The query execution status. To learn more about statuses, see [Error handling](https://cloud.yandex.com/en/docs/ydb/oss/public/reference/ydb-sdk/error_handling). | | `api.grpc.response.dropped_count`<br/>`RATE`, pieces | The number of responses dropped at the transport (gRPC) layer due to an error.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | -| `api.grpc.response.issues`<br/>`RATE`, pieces | The number of errors of a certain type, which occurred when executing queries during a certain period of time.<br/>Labels:<br/>- _issue_type_: The type of error, the only value is `optimistic_locks_invalidation`. For more information about locks invalidation, see [Transactions and queries to {{ ydb-short-name }}](https://cloud.yandex.ru/docs/ydb/oss/public/develop/concepts/transactions). | +| `api.grpc.response.issues`<br/>`RATE`, pieces | The number of errors of a certain type, which occurred when executing queries during a certain period of time.<br/>Labels:<br/>- _issue_type_: The type of error, the only value is `optimistic_locks_invalidation`. For more information about locks invalidation, see [Transactions and queries to {{ ydb-short-name }}](https://cloud.yandex.com/en/docs/ydb/oss/public/develop/concepts/transactions). | ### Session metrics {#sessions} @@ -86,3 +86,4 @@ You can analyze a transaction's execution time using a histogram counter. The in | `table.query.compilation.cache_evictions`<br/>`RATE`, pieces | The number of queries evicted from the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries) in a certain period of time. | | `table.query.compilation.cache_size_bytes`<br/>`IGAUGE`, bytes | The size of the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries). | | `table.query.compilation.cached_query_count`<br/>`IGAUGE`, pieces | The size of the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries). | + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md new file mode 100644 index 0000000000..d531f9364c --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md @@ -0,0 +1,103 @@ +## Distributed Storage + +Information about the operation of distributed storage is contained in several interconnected tables, each of which is responsible for describing its entity, such as: + +* PDisk +* VSlot +* Group +* Storage Pool + +In addition, there is a separate table that shows statistics on the use of the number of groups in different storage pools and whether these pools can be increased. + +### ds_pdisks + +| **Field** | **Type** | **Key** | **Value** | +| ----------------------- | ----------- | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| NodeId | Uint32 | 0 | ID of the node where a PDisk is running. | +| PDiskId | Uint32 | 1 | ID of the PDisk (unique within the node). | +| Type | String | | Media type (ROT, SSD, NVME). | +| Kind | Uint64 | | A user-defined numeric ID that is needed to group disks with the same type of media into different subgroups. | +| Path | String | | Path to the block device inside the machine. | +| Guid | Uint64 | | A unique ID that is generated randomly when adding a disk to the system and is designed to prevent data loss in the event of disk swapping. | +| BoxId | Uint64 | | ID of the Box that this PDisk belongs to. | +| SharedWithOs | Bool | | A Boolean attribute that is used to filter disks when creating new groups. | +| ReadCentric | Bool | | A Boolean attribute that is used to filter disks when creating new groups. | +| AvailableSize | Uint64 | | The number of bytes that can be allocated on the PDisk. | +| TotalSize | Uint64 | | The total number of bytes on the PDisk. | +| Status | String | | PDisk operation mode that affects its participation in the allocation of groups (ACTIVE, INACTIVE, BROKEN, FAULT, and TO_BE_REMOVED). | +| StatusChangeTimestamp | Timestamp | | The time when the Status was last changed. NULL indicates that the Status hasn't changed since the creation of PDisk. | +| ExpectedSlotCount | Uint32 | | The maximum number of VSlots that can be created on this PDisk. | +| NumActiveSlots | Uint32 | | The number of slots that are currently active. | + +### ds_vslots + +| **Field** | **Type** | **Key** | **Value** | +| ----------------- | --------- | ---------- | ----------------------------------------------------------------------------------------- | +| NodeId | Uint32 | 0 | ID of the node where a VSlot is running. | +| PDiskId | Uint32 | 1 | ID of the PDisk inside the node where the VSlot is running. | +| VSlotId | Uint32 | 2 | ID of the VSlot inside the PDisk. | +| GroupId | Uint32 | | Number of the storage group that this VSlot belongs to. | +| GroupGeneration | Uint32 | | Generation of the storage group configuration that this VSlot belongs to. | +| FailRealm | Uint32 | | Relative number of the fail realm of the VSlot within the storage group. | +| FailDomain | Uint32 | | Relative number of the fail domain of the VSlot within the fail realm. | +| VDisk | Uint32 | | Relative number of the VSlot inside the fail domain. | +| AllocatedSize | Uint64 | | The number of bytes that the VSlot occupies on the PDisk. | +| AvailableSize | Uint64 | | The number of bytes that can be allocated to this VSlot. | +| Status | String | | Status of the VDisk running in this VSlot (INIT_PENDING, REPLICATING, READY, or ERROR). | +| Kind | String | | Preset VDisk operation mode (Default, Log, ...). | + +Please not that the (NodeId, PDiskId) tuple creates an external key to the `ds_pdisks` table and the (GroupId) to the `ds_groups` table. + +### ds_groups + +| **Field** | **Type** | **Key** | **Value** | +| --------------------- | ---------- | ---------- | ------------------------------------------------------------------------------------------------------------ | +| GroupId | Uint32 | 0 | Number of the storage group in the cluster. | +| Generation | Uint32 | | Storage group configuration generation. | +| ErasureSpecies | String | | Group redundancy coding mode (block-4-2, mirror-3-dc, mirror-3of4, ...). | +| BoxId | Uint64 | | ID of the Box that this group is created in. | +| StoragePoolId | Uint64 | | ID of the storage pool inside the Box that this group operates in. | +| EncryptionMode | Uint32 | | Group data encryption and its algorithm (if enabled). | +| LifeCyclePhase | Uint32 | | Availability of a generated encryption key (if encryption is enabled). | +| AllocatedSize | Uint64 | | The number of allocated bytes of data in the group (reduced to user bytes, that is, to redundancy). | +| AvailableSize | Uint64 | | The number of bytes of user data available for allocation (up to redundancy as well). | +| SeenOperational | Bool | | A Boolean flag that indicates whether the group was operational after its creation. | +| PutTabletLogLatency | Interval | | 90th percentile of the PutTabletLog request execution time. | +| PutUserDataLatency | Interval | | 90th percentile of the PutUserData request execution time. | +| GetFastLatency | Interval | | 90th percentile of the GetFast request execution time. | + +In this table, the (BoxId, StoragePoolId) tuple creates an external key to the `ds_storage_pools` table. + +### ds_storage_pools + +| **Field** | **Type** | **Key** | **Value** | +| ---------------- | --------- | ---------- | ------------------------------------------------------------------------------------------------------------- | +| BoxId | Uint64 | 0 | ID of the Box that this storage pool belongs to. | +| StoragePoolId | Uint64 | 1 | ID of the storage pool inside the Box. | +| Name | String | | User-defined storage pool name (used when linking tablets and storage pools). | +| Generation | Uint64 | | Storage pool configuration generation (number of changes). | +| ErasureSpecies | String | | Redundancy coding mode for all groups within this storage pool. | +| VDiskKind | String | | Preset operation mode for all VDisks in this storage pool. | +| Kind | String | | A user-defined string description of the purpose of the pool, which can also be used for filtering. | +| NumGroups | Uint32 | | Number of groups within this storage pool. | +| EncryptionMode | Uint32 | | Data encryption setting for all groups (similar to ds_groups.EncryptionMode). | +| SchemeshardId | Uint64 | | ID of the SchemeShard object of the schema that this storage pool belongs to (as of now, always NULL). | +| PathId | Uint64 | | ID of the node of the schema object inside the specified SchemeShard that this storage pool belongs to. | + +### ds_storage_stats + +Unlike other tables that show physical entities, the `ds_storage_stats` table shows aggregated storage statistics. + +| **Field** | **Type** | **Key** | **Value** | +| ------------------------- | --------- | ---------- | ------------------------------------------------------------------------------------------------- | +| BoxId | Uint64 | 0 | ID of the Box that statistics are calculated for. | +| PDiskFilter | String | 1 | A string description of filters that select a PDisk to create groups (for example, by media type). | +| ErasureSpecies | String | 2 | Redundancy coding mode that statistics are collected for. | +| CurrentGroupsCreated | Uint32 | | Number of groups created with the specified characteristics. | +| CurrentAllocatedSize | Uint64 | | Total space occupied by all groups from CurrentGroupsCreated. | +| CurrentAvailableSize | Uint64 | | Total space that is available to all groups from CurrentGroupsCreated. | +| AvailableGroupsToCreate | Uint32 | | Number of groups with the specified characteristics that can be created taking into account the need for a reserve. | +| AvailableSizeToCreate | Uint64 | | Number of available bytes that will be obtained when creating all groups from AvailableGroupsToCreate. | + +It should be noted that AvailableGroupsToCreate shows the maximum number of groups that can be created if no other types of groups are created. So when extending a storage pool, the count of AvailableGroupsToCreate in several rows of statistics may change. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md index c36f003299..df3c85f771 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md @@ -3,3 +3,4 @@ Examples: {% include [example_yql](partitions_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md index e651429b28..bdeab3c498 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md @@ -1,4 +1,5 @@ - Top 5 of most loaded partitions among all DB tables +Top 5 of most loaded partitions among all DB tables + ```sql SELECT Path, @@ -10,6 +11,7 @@ ``` List of DB tables with in-flight sizes and loads + ```sql SELECT Path, @@ -19,4 +21,5 @@ SUM(CPUCores) as CPU FROM `/cluster/path/to/database/.sys/partition_stats` GROUP BY Path - ```
\ No newline at end of file + ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md index 4edb8c845c..447571f4e1 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md @@ -35,3 +35,4 @@ Table structure: Restrictions: * Cumulative fields (RowReads, RowUpdates, and so on) store the accumulated values since the last start of the tablet serving the partition + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md index 049830f29b..e2fc18f715 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md @@ -2,4 +2,5 @@ Examples: -{% include [example_yql](query_metrics_example_yql.md) %}
\ No newline at end of file +{% include [example_yql](query_metrics_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md index 2e622063c3..f98134848a 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md @@ -1,4 +1,4 @@ - Top 10 queries for the last 6 hours by the total number of rows updated per minute +Top 10 queries for the last 6 hours by the total number of rows updated per minute ```sql SELECT @@ -24,4 +24,5 @@ WHERE SumReadBytes > 0 ORDER BY IntervalEnd DESC, SumReadBytes DESC LIMIT 100 - ```
\ No newline at end of file + ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md index b74f360d20..69ee2da5f6 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md @@ -40,3 +40,4 @@ Restrictions: * The table contains the history for the last 6 hours. * Within the interval, information is provided for no more than 256 different queries. * Statistics may be incomplete if the database is under heavy load. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md index 7c1cb50003..acb4488ba7 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md @@ -3,3 +3,4 @@ Examples: {% include [example_yql.md](tops_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md index b887e8ef37..967f8bded5 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md @@ -1,4 +1,4 @@ - Top queries by execution time for the last minute when queries were made +Top queries by execution time for the last minute when queries were made ```sql $last = ( @@ -26,4 +26,5 @@ Partitions FROM `/cluster/path/to/database/.sys/top_queries_by_read_bytes_one_minute` WHERE Rank = 1 - ```
\ No newline at end of file + ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md index cdff4f948d..2d2249a66c 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md @@ -50,3 +50,4 @@ Restrictions: * Query text limit is 4 KB. * Tables with minute intervals contain the history for the last 6 hours. * Tables with hourly intervals contain the history for the last 2 weeks. + diff --git a/ydb/docs/en/core/troubleshooting/monitoring.md b/ydb/docs/en/core/troubleshooting/monitoring.md index a181d1d03e..598b0fed12 100644 --- a/ydb/docs/en/core/troubleshooting/monitoring.md +++ b/ydb/docs/en/core/troubleshooting/monitoring.md @@ -1,7 +1,7 @@ --- editable: false --- - # Metric reference {% include notitle [ydb_monitoring_sensors.md](_includes/monitoring_sensors.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/system_views.md b/ydb/docs/en/core/troubleshooting/system_views.md index 89e6340905..3aafbefa8c 100644 --- a/ydb/docs/en/core/troubleshooting/system_views.md +++ b/ydb/docs/en/core/troubleshooting/system_views.md @@ -6,4 +6,6 @@ {% include [query_metrics.md](_includes/system_views/query_metrics.md) %} +{% include [distributed_storage.md](_includes/system_views/distributed_storage.md) %} + {% include [notes.md](_includes/system_views/notes.md) %} diff --git a/ydb/docs/en/core/yql/reference/_includes/index/intro.md b/ydb/docs/en/core/yql/reference/_includes/index/intro.md index 7957c0629a..a1cd5dcef1 100644 --- a/ydb/docs/en/core/yql/reference/_includes/index/intro.md +++ b/ydb/docs/en/core/yql/reference/_includes/index/intro.md @@ -1,12 +1,21 @@ ---- -title: Что такое YQL? Обзор Yandex Query Language -description: YQL (Yandex Query Language) — универсальный декларативный язык запросов к системам хранения и обработки данных, диалект SQL. Начать работать с YQL можно в веб-интерфейсе после создания базы данных. -keywords: - - yql - - что такое yql - - Yandex Query Language ---- -# YQL overview - -*YQL* (Yandex Query Language) is a universal declarative query language for data storage and processing systems, a dialect of SQL. +# YQL - Overview + +*YQL* (YDB Query Language) is a universal declarative query language for YDB, a dialect of SQL. YQL has been natively designed for large distributed databases, and therefore has a number of differences from the SQL standard. + +YDB tools support interfaces for sending YQL queries and receiving their execution results: + +{% include [yql/ui_prompt.md](yql/ui_prompt.md) %} + +- [YDB CLI](../../../../reference/ydb-cli/index.md) +- [YDB SDK](../../../../reference/ydb-sdk/index.md) + +This documentation section contains the YQL reference that includes the sections: + +- [Data types](../../types/index.md) with a description of data types used in YQL +- [Syntax](../../syntax/index.md) with a full list of YQL commands +- [Built-in functions](../../builtins/index.md) with a description of the available built-in functions + +You can also take a tutorial to get familiar with the basic YQL commands, in the section + +- [YQL tutorial](../../../tutorial/index.md) diff --git a/ydb/docs/en/core/yql/reference/_includes/index/start.md b/ydb/docs/en/core/yql/reference/_includes/index/start.md index a25d4be33e..e69de29bb2 100644 --- a/ydb/docs/en/core/yql/reference/_includes/index/start.md +++ b/ydb/docs/en/core/yql/reference/_includes/index/start.md @@ -1,6 +0,0 @@ -You can get started with YQL after you create a database. - -The [YQL tutorial](../../../tutorial/) provides examples of YQL instructions you can use to get familiar with the basic syntax and features of YQL. - -See [{#T}](../../../../operations/crud.md) to learn about the basic operations with data. - diff --git a/ydb/docs/en/core/yql/reference/_includes/index/yql/ui_prompt.md b/ydb/docs/en/core/yql/reference/_includes/index/yql/ui_prompt.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/_includes/index/yql/ui_prompt.md diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md index 688e7712eb..467901871f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md @@ -6,9 +6,9 @@ SELECT CAST("xyz" AS Uint64) IS NULL, -- true, because it failed CAST(-1 AS Uint16) IS NULL, -- true, a negative integer cast to an unsigned integer CAST([-1, 0, 1] AS List<Uint8?>), -- [null, 0, 1] - --The element type is optional: the failed element is cast to null. + --The item type is optional: the failed item is cast to null. CAST(["3.14", "bad", "42"] AS List<Float>), -- [3.14, 42] - --The element type is not optional: the failed element has been deleted. + --The item type is not optional: the failed item has been deleted. CAST(255 AS Uint8), -- 255 CAST(256 AS Uint8) IS NULL -- true, out of range diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md index 717dcd8311..74d3ef437c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md @@ -6,7 +6,7 @@ If you know already that you have few distinct values, use the `AGGREGATE_LIST_D The order of elements in the result list depends on the implementation and can't be set externally. To return an ordered list, sort the result, for example, with [ListSort](../../list.md#listsort). -To return a list of multiple values from one line, *DO NOT* use the `AGGREGATE_LIST` function several times, but add all the needed values to a container, for example, via [AsList](../../basic.md#aslist) or [AsTuple](../../basic.md#astuple), then pass this container to a single `AGGREGATE_LIST` call. +To return a list of multiple values from one line, *DO NOT* use the `AGGREGATE_LIST` function several times, but add all the needed values to a container, for example, via [AsList](../../basic.md#aslist) or [AsTuple](../../basic.md#astuple), then pass this container to a single `AGGREGATE_LIST` call. For example, you can combine it with `DISTINCT` and the function [String::JoinFromList](../../../udf/list/string.md) (it's an equivalent of `','.join(list)` in Python) to output to a string all the values found in the column after [GROUP BY](../../../syntax/group_by.md). @@ -33,7 +33,7 @@ These functions also have a short notation: `AGG_LIST` and `AGG_LIST_DISTINCT`. {% note alert %} -Execution is **NOT** lazy, so when you use it, be sure that the list has a reasonable size (about a thousand elements or less). To stay on the safe side, better use a second optional numeric argument that limits the number of items in the list. +Execution is **NOT** lazy, so when you use it, be sure that the list has a reasonable size (about a thousand items or less). To stay on the safe side, better use a second optional numeric argument that limits the number of items in the list. {% endnote %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md index a2ea97b8dc..4b3dea633c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md @@ -95,7 +95,7 @@ SELECT FROM my_table; ``` -LinearHistogram, LogarithmicHistogram and LogHistogram {#linearhistogram} +## LinearHistogram, LogarithmicHistogram, and LogHistogram {#linearhistogram} Plotting a histogram based on an explicitly specified fixed bucket scale. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md index f098eb6cc1..5bac2b8f60 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md @@ -1,6 +1,6 @@ ## PERCENTILE and MEDIAN {#percentile-median} -Calculating percentiles using the amortized version of the [TDigest](https://github.com/tdunning/t-digest) algorithm. `MEDIAN`: An alias for `PERCENTILE(N, 0.5)`. +Calculating percentiles using the amortized version of the [TDigest](https://github.com/tdunning/t-digest) algorithm. `MEDIAN`: An alias for `PERCENTILE(N, 0.5)`. {% note info "Restriction" %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/session_start.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/session_start.md index 1832102f6f..62e1db4f2a 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/session_start.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/session_start.md @@ -1,8 +1,6 @@ ## SessionStart {#session-start} -No arguments. It's allowed only if there is [SessionWindow](../../../syntax/group_by.md#session-window) in -[GROUP BY](../../../syntax/group_by.md) / [PARTITION BY](../../../syntax/window.md#partition). +No arguments. It's allowed only if there is [SessionWindow](../../../syntax/group_by.md#session-window) in [GROUP BY](../../../syntax/group_by.md) / [PARTITION BY](../../../syntax/window.md#partition). Returns the value of the `SessionWindow` key column. If `SessionWindow` has two arguments, it returns the minimum value of the first argument within the group/section. - In the case of the expanded version `SessionWindow`, it returns the value of the second element from the tuple returned by `<calculate_lambda>`, for which the first tuple element is `True`. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md index 75f6ac83cd..8a80dc8ac7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md @@ -1,6 +1,6 @@ ## TOP and BOTTOM {#top-bottom} -Return a list of the maximum/minimum values of an expression. The first argument is an expression, the second argument limits the number of elements. +Return a list of the maximum/minimum values of an expression. The first argument is an expression, the second argument limits the number of items. **Examples** diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md index 39842a9ea7..1c563cba95 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md @@ -3,6 +3,8 @@ The absolute value of the number. **Examples** -``` yql + +```yql SELECT Abs(-123); -- 123 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md index b861aaecd3..0302b69fc3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md @@ -11,7 +11,8 @@ The resulting factory can be used as the second parameter of the function [AGGRE If the aggregate function is applied to two columns instead of one, as, for example, [MIN_BY](../../aggregation.md#minby), then in [AGGREGATE_BY](../../aggregation.md#aggregateby), the first argument passes a `Tuple` of two values. See more details in the description of the applicable aggregate function. **Examples:** -``` yql + +```yql $factory = AggregationFactory("MIN"); SELECT AGGREGATE_BY (value, $factory) AS min_value -- apply the MIN aggregation to the "value" column @@ -20,15 +21,16 @@ FROM my_table; ## AggregateTransform... {#aggregatetransform} -`AggregateTransformInput()` converts an [aggregation factory](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function, to other factory, in which the specified transformation of input elements is performed before starting aggregation. +`AggregateTransformInput()` converts an [aggregation factory](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function, to other factory, in which the specified transformation of input items is performed before starting aggregation. Arguments: 1. Aggregation factory. -2. A lambda function with one argument that converts an input element. +2. A lambda function with one argument that converts an input item. **Examples:** -``` yql + +```yql $f = AggregationFactory("sum"); $g = AggregateTransformInput($f, ($x) -> (cast($x as Int32))); $h = AggregateTransformInput($f, ($x) -> ($x * 2)); @@ -45,7 +47,8 @@ Arguments: 2. A lambda function with one argument that converts the result. **Examples:** -``` yql + +```yql $f = AggregationFactory("sum"); $g = AggregateTransformOutput($f, ($x) -> ($x * 2)); select ListAggregate([1,2,3], $f); -- 6 @@ -54,14 +57,15 @@ select ListAggregate([1,2,3], $g); -- 12 ## AggregateFlatten {#aggregateflatten} -Adapts a factory for [aggregation functions](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function in a way that allows aggregation of list input elements. This operation is similar to [FLATTEN LIST BY](../../../syntax/flatten.md): Each list element is aggregated. +Adapts a factory for [aggregation functions](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function in a way that allows aggregation of list input items. This operation is similar to [FLATTEN LIST BY](../../../syntax/flatten.md): Each list item is aggregated. Arguments: 1. Aggregation factory. **Examples:** -``` yql + +```yql $i = AggregationFactory("AGGREGATE_LIST_DISTINCT"); $j = AggregateFlatten($i); select AggregateBy(x, $j) from ( @@ -69,5 +73,5 @@ select AggregateBy(x, $j) from ( union all select [2,3] as x ); -- [1, 2, 3] +``` -```
\ No newline at end of file diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md index 7c2126c4df..15adedeeda 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md @@ -8,7 +8,7 @@ Specifics: * `AsTuple` and `AsStruct` can be called without arguments, and also the arguments can have different types. * The field names in `AsStruct` are set using `AsStruct(field_value AS field_name)`. * Creating a list requires at least one argument if you need to output the element types. To create an empty list with the given type of elements, use the function [ListCreate](../../list.md#listcreate). You can create an empty list as an `AsList()` call without arguments. In this case, this expression will have the `EmptyList` type. -* Creating a dictionary requires at least one argument if you need to output element types. To create an empty dictionary with the given type of elements, use the function [DictCreate](../../dict.md#dictcreate). You can create an empty dictionary as an `AsDict()` call without arguments, in this case, this expression will have the `EmptyDict` type. +* Creating a dictionary requires at least one argument if you need to output the element types. To create an empty dictionary with the given type of elements, use the function [DictCreate](../../dict.md#dictcreate). You can create an empty dictionary as an `AsDict()` call without arguments, in this case, this expression will have the `EmptyDict` type. * Creating a set requires at least one argument if you need to output element types. To create an empty set with the given type of elements, use the function [SetCreate](../../dict.md#setcreate). You can create an empty set as an `AsSet()` call without arguments, in this case, this expression will have the `EmptySet` type. * `AsList` outputs the common type of elements in the list. A type error is raised in the case of incompatible types. * `AsDict` separately outputs the common types for keys and values. A type error is raised in the case of incompatible types. @@ -18,7 +18,8 @@ Specifics: * `AsSet` and `AsSetStrict` expect keys as arguments. **Examples** -``` yql + +```yql SELECT AsTuple(1, 2, "3") AS `tuple`, AsStruct( @@ -34,3 +35,4 @@ SELECT ) AS `dict`, AsSet(1, 2, 3) AS `set` ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md index 981395a06c..326cdb9812 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md @@ -1,6 +1,6 @@ ## AsTagged, Untag {#as-tagged} -Wraps the value in the [Tagged data type](../../../types/special.md) with the specified tag, preserving the physical data type. `Untag`: the reverse operation. +Wraps the value in the [Tagged data type](../../../types/special.md) with the specified tag, preserving the physical data type. `Untag`: The reverse operation. Required arguments: @@ -14,3 +14,4 @@ Examples of use cases: * Returns to the client's web interface the media files from BASE64-encoded strings{% if feature_webui %}. Tag support in the YQL Web UI [is described here](../../../interfaces/web_tagged.md){% endif %}. {% if feature_mapreduce %}* Prevent passing of invalid values at the boundaries of UDF calls.{% endif %} * Additional refinements at the level of returned columns types. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md index e2c5b33134..2ecf0818d4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md @@ -10,8 +10,10 @@ Arguments: TestBit returns `true/false`. The other functions return a copy of their first argument with the corresponding conversion. **Examples:** -``` yql + +```yql SELECT TestBit(1u, 0), -- true SetBit(8u, 0); -- 9 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md index 1964874931..ef9c9d0bde 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md @@ -8,9 +8,11 @@ Arguments: 2. Index: `Uint32`. **Examples** -``` yql + +```yql SELECT ByteAt("foo", 0), -- 102 ByteAt("foo", 1), -- 111 ByteAt("foo", 9); -- NULL ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md index 25df4c1be5..30cd21dffb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md @@ -8,7 +8,8 @@ Arguments: 2. Lambda function. **Examples:** -``` yql + +```yql $lambda = ($x) -> { RETURN CAST($x as String) }; @@ -20,3 +21,4 @@ $callables = AsTuple( SELECT $callables.0(10), $callables.1(true); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md index 7da96d2271..09fe299a60 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md @@ -7,7 +7,8 @@ Lets you pass potentially empty values to functions that can't handle them by th A short format using the low-priority `??` operator is available (lower than the Boolean operations). You can use the `NVL` alias. **Examples** -``` yql + +```yql SELECT COALESCE( maybe_empty_column, "it's empty!" @@ -27,4 +28,5 @@ SELECT NVL( ) FROM my_table; ``` -<span style="color: gray;">(all three examples above are equivalent)</span>
\ No newline at end of file +<span style="color: gray;">(all three examples above are equivalent)</span> + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md index 2a90f1650d..9ea09391d6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md @@ -14,7 +14,8 @@ For field names in the structure literal, you can use an expression that can be For nested lists, use [AsList](#aslist), for nested dictionaries, use [AsDict](#asdict), for nested sets, use [AsSet](#asset), for nested tuples, use [AsTuple](#astuple), for nested structures, use [AsStruct](#asstruct). **Examples** -``` yql + +```yql $name = "computed " || "member name"; SELECT (1, 2, "3") AS `tuple`, @@ -32,3 +33,4 @@ SELECT } AS `dict`, {1, 2, 3} AS `set` ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md index 75c934c7b6..c7748e2ff9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md @@ -5,10 +5,12 @@ The arguments that follow are optional and work same as [RANDOM](#random). **Examples** -``` yql + +```yql SELECT CurrentTzDate("Europe/Moscow"); ``` -``` yql + +```yql SELECT CurrentTzTimestamp("Europe/Moscow", TableRow()) FROM my_table; ``` @@ -24,7 +26,8 @@ Arguments: Result type: `TzDate`/`TzDatetime`/`TzTimestamp`, depending on the input data type. **Examples** -``` yql + +```yql SELECT AddTimezone(Datetime("2018-02-01T12:00:00Z"), "Europe/Moscow"); ``` @@ -34,11 +37,13 @@ Removing the time zone data and converting the value to date/time in UTC. Arguments: -1. Date: the type is `TzDate`/`TzDatetime`/`TzTimestamp`. +1. Date: the type is `TzDate`/`TzDatetime`/`TzTimestamp`. Result type: `Date`/`Datetime`/`Timestamp`, depending on the input data type. **Examples** -``` yql + +```yql SELECT RemoveTimezone(TzDatetime("2018-02-01T12:00:00,Europe/Moscow")); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md index 8eff367299..36bb5eeb02 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md @@ -1,13 +1,16 @@ ## CurrentUtc... {#current-utc} -`CurrentUtcDate()`, `CurrentUtcDatetime()` and `CurrentUtcTimestamp()`: getting the current date and/or time in UTC. The result data type is specified at the end of the function name. +`CurrentUtcDate()`, `CurrentUtcDatetime()` and `CurrentUtcTimestamp()`: Getting the current date and/or time in UTC. The result data type is specified at the end of the function name. The arguments are optional and work same as [RANDOM](#random). **Examples** -``` yql + +```yql SELECT CurrentUtcDate(); ``` -``` yql + +```yql SELECT CurrentUtcTimestamp(TableRow()) FROM my_table; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md index f123dd7c15..1e201128e2 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md @@ -23,7 +23,8 @@ For the data types `TzDate`, `TzDatetime`, `TzTimestamp`, literals are also set {% include [decimal args](../../../_includes/decimal_args.md) %} **Examples** -``` yql + +```yql SELECT Bool("true"), Uint8("0"), @@ -46,4 +47,5 @@ SELECT TzDatetime("2017-11-27T13:24:00,America/Los_Angeles"), TzTimestamp("2017-11-27T13:24:00.123456,GMT"), Uuid("f9d5cc3f-f1dc-4d9c-b97e-766e57ca4ccb"); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md index 9a47d5b546..d2509df357 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md @@ -17,7 +17,8 @@ Arguments: To check the conditions based on the final calculation result, it's convenient to combine Ensure with [DISCARD SELECT](../../../syntax/discard.md). **Examples** -``` yql + +```yql SELECT Ensure( value, value < 100, @@ -25,7 +26,7 @@ SELECT Ensure( ) AS value FROM my_table; ``` -``` yql +```yql SELECT EnsureType( value, TypeOf(other_value), @@ -33,10 +34,11 @@ SELECT EnsureType( ) AS value FROM my_table; ``` -``` yql +```yql SELECT EnsureConvertibleTo( value, Double?, "expected value to be numeric" ) AS value FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md index f30225d2e9..28729c4441 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md @@ -8,7 +8,8 @@ Arguments: * Enumeration type **Example** -``` yql + +```yql $enum_type = Enum<Foo, Bar>; SELECT Enum("Foo", $enum_type) as Enum1Value, @@ -19,10 +20,12 @@ SELECT Arguments: -* A string with the name of an enumeration element +* A string with the name of an enumeration item **Example** -``` yql + +```yql SELECT AsEnum("Foo"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md index 22a5bd1c78..096709330d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md @@ -17,10 +17,12 @@ Restrictions: * This functionality is fully locked in YQL over YDB. **Examples:** -``` yql + +```yql $now = CurrentUtcDate(); SELECT EvaluateExpr( DateTime::MakeDate(DateTime::StartOfWeek($now) ) ); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md index e802a21663..16e63c45e0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md @@ -5,12 +5,14 @@ Both the [console](../../../interfaces/cli.md) and [web](../../../interfaces/web The `FileContent` and `FilePath` argument is a string with an alias. **Examples** -``` yql + +```yql SELECT "Content of " || FilePath("my_file.txt") || ":\n" || FileContent("my_file.txt"); ``` + ## FolderPath {#folderpath} Getting the path to the root of a directory with several "attached" files with the common prefix specified. @@ -20,7 +22,8 @@ The argument is a string with a prefix among aliases. See also [PRAGMA File](../../../syntax/pragma.md#file) and [PRAGMA Folder](../../../syntax/pragma.md#folder). **Examples** -``` yql + +```yql PRAGMA File("foo/1.txt", "http://url/to/somewhere"); PRAGMA File("foo/2.txt", "http://url/to/somewhere/else"); PRAGMA File("bar/3.txt", "http://url/to/some/other/place"); @@ -47,10 +50,13 @@ The return value is a lazy list. For repeat use, wrap it in the function [ListCo {% endnote %} **Examples:** -``` yql + +```yql SELECT ListLength(ParseFile("String", "my_file.txt")); ``` -``` yql + +```yql SELECT * FROM my_table WHERE int_column IN ParseFile("Int64", "my_file.txt")); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md index 436cd710f9..80fecee7ab 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md @@ -14,13 +14,16 @@ Optional arguments: Returns the first substring position found or `NULL` (meaning that the desired substring hasn't been found starting from the specified position). **Examples** -``` yql + +```yql SELECT FIND("abcdefg_abcdefg", "abc"); -- 0 ``` -``` yql + +```yql SELECT FIND("abcdefg_abcdefg", "abc", 1); -- 8 ``` -``` yql + +```yql SELECT FIND("abcdefg_abcdefg", "abc", 9); -- null ``` @@ -40,12 +43,16 @@ Optional arguments: Returns the first substring position found or `NULL` (meaning that the desired substring hasn't been found starting from the specified position). **Examples** -``` yql + +```yql SELECT RFIND("abcdefg_abcdefg", "bcd"); -- 9 ``` -``` yql + +```yql SELECT RFIND("abcdefg_abcdefg", "bcd", 8); -- 1 ``` -``` yql + +```yql SELECT RFIND("abcdefg_abcdefg", "bcd", 0); -- null -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md index 3bc844b1be..dcd486afac 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md @@ -7,9 +7,11 @@ It's a simplified alternative for [CASE WHEN ... THEN ... ELSE ... END](../../.. You may omit the `else_expression` argument. In this case, if the condition is false (`condition_expression` returned `false`), an empty value is returned with the type corresponding to `then_expression` and allowing for `NULL`. Hence, the result will have an [optional data type](../../../types/optional.md). **Examples** -``` yql + +```yql SELECT IF(foo > 0, bar, baz) AS bar_or_baz, IF(foo > 0, foo) AS only_positive_foo FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md index 27b4b0e8c6..d8da1aff4f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md @@ -1,3 +1,4 @@ # Basic built-in functions -Below are the general-purpose functions. For specialized functions, see separate articles: [aggregate functions](../../aggregation.md){% if feature_window_functions %}, [window functions](../../window.md){% endif %}, and functions for [lists](../../list.md), [dictionaries](../../dict.md), [structures](../../struct.md), [data types](../../types.md){% if feature_codegen %}, and [code generation](../../codegen.md){% endif %}. +Below are the general-purpose functions. For specialized functions, there are separate articles: [aggregate functions](../../aggregation.md){% if feature_window_functions %}, [window functions](../../window.md){% endif %}, and functions for [lists](../../list.md), [dictionaries](../../dict.md), [structures](../../struct.md), [data types](../../types.md){% if feature_codegen %}, and [code generation](../../codegen.md){% endif %}. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md index ac3cb2cf7a..4dc75910f9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md @@ -3,10 +3,12 @@ Returns the length of the string in bytes. This function is also available under the `LEN` name . **Examples** -``` yql + +```yql SELECT LENGTH("foo"); ``` -``` yql + +```yql SELECT LEN("bar"); ``` @@ -15,3 +17,4 @@ SELECT LEN("bar"); To calculate the length of a string in Unicode characters, you can use the function [Unicode::GetLength](../../../udf/list/unicode.md).<br><br>To get the number of elements in the list, use the function [ListLength](../../list.md#listlength). {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md index 22a5c00e70..d4f4afa916 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md @@ -7,6 +7,8 @@ The argument types must be mutually castable and accept `NULL`. `GREATEST` is a synonym for `MAX_OF` and `LEAST` is a synonym for `MIN_OF`. **Examples** -``` yql + +```yql SELECT MIN_OF(1, 2, 3); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md index 1bfc9ae23b..610f9bb0f2 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md @@ -11,9 +11,11 @@ No arguments. If this data is missing, for example, when you run operations in the embedded mode, the functions return an empty string. **Examples** -``` yql + +```yql SELECT CurrentOperationId(), CurrentOperationSharedId(), CurrentAuthenticatedUser(); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md index 6bdffcb6b8..dc9b16d1b3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md @@ -10,8 +10,10 @@ Arguments: If one of the arguments is `Double`, the result is`Double`, otherwise, it's `Float`. If one of the arguments is `Optional`, then the result is `Optional`. **Examples** -``` yql + +```yql SELECT NANVL(double_column, 0.0) FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md index 6e8526f422..d0bc08c13e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md @@ -5,7 +5,8 @@ The reverse operation is [Unwrap](#unwrap). **Examples** -``` yql + +```yql SELECT Just("my_string"); -- String? ``` @@ -22,7 +23,8 @@ Arguments: Reverse operation is [Just](#just). **Examples** -``` yql + +```yql $value = Just("value"); SELECT Unwrap($value, "Unexpected NULL for $value"); @@ -31,9 +33,11 @@ SELECT Unwrap($value, "Unexpected NULL for $value"); `Nothing()`: Create an empty value for the specified [Optional](../../../types/optional.md) data type. **Examples** -``` yql + +```yql SELECT Nothing(String?); -- an empty (NULL) value with the String? type ``` -[Learn more about ParseType and other functions for data types](../../types.md).
\ No newline at end of file +[Learn more about ParseType and other functions for data types](../../types.md). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md index 50432dd22c..1a852f530a 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md @@ -5,7 +5,8 @@ `Unpickle()` is the inverse operation (deserialization), where with the first argument being the data type of the result and the second argument is the string with the result of `Pickle()` or `StablePickle()`. Examples: -``` yql + +```yql SELECT * FROM my_table WHERE Digest::MurMurHash32( @@ -15,3 +16,4 @@ WHERE Digest::MurMurHash32( $buf = Pickle(123); SELECT Unpickle(Int32, $buf); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md index b3e2d6e40c..1722cc6c9c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md @@ -21,25 +21,26 @@ Use cases: * `SELECT RANDOM(some_column), RANDOM(some_column + 1) FROM table;` or `SELECT RANDOM(some_column), RANDOM(other_column) FROM table;`: Two columns, with different numbers in both. **Examples** -``` yql + +```yql SELECT Random(key) -- [0, 1) FROM my_table; ``` -``` yql +```yql SELECT RandomNumber(key) -- [0, Max<Uint64>) FROM my_table; ``` -``` yql +```yql SELECT RandomUuid(key) -- Uuid version 4 FROM my_table; ``` -``` yql +```yql SELECT RANDOM(column) AS rand1, RANDOM(column) AS rand2, -- same as rand1 @@ -47,3 +48,4 @@ SELECT RANDOM(column, 2) AS randAnd2 -- different from randAnd1 FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/s_expressions.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/s_expressions.md index 664bf56eb0..969b4531e6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/s_expressions.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/s_expressions.md @@ -1,6 +1,6 @@ ## YQL::, s-expressions {#s-expressions} -For the full list of internal YQL functions, see the [documentation for s-expressions](/docs/s_expressions/functions), an alternative low-level YQL syntax. Any of the functions listed there can also be called from the SQL syntax by adding the `YQL::` prefix to its name. However, we don't recommend doing this, because this mechanism is primarily intended to temporarily bypass possible issues and for internal testing purposes. +For the full list of internal YQL functions, see the [documentation for s-expressions](/docs/s_expressions/functions), an alternative low-level YQL syntax. Any of the functions listed there can also be called from the SQL syntax by adding the `YQL::` prefix to its name. However, we don't recommend doing this, because this mechanism is primarily intended to temporarily bypass possible issues and for internal testing purposes. If the function is available in SQL syntax without the `YQL::` prefix, then its behavior may differ from the same-name function from the s-expressions documentation, if any. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md index 483ffa429c..5a0de652b4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md @@ -10,15 +10,20 @@ Required arguments: The arguments can be of the `String` or `Utf8` type and can be optional. **Examples** -``` yql + +```yql SELECT StartsWith("abc_efg", "abc") AND EndsWith("abc_efg", "efg"); -- true ``` -``` yql + +```yql SELECT StartsWith("abc_efg", "efg") OR EndsWith("abc_efg", "abc"); -- false ``` -``` yql + +```yql SELECT StartsWith("abcd", NULL); -- null ``` -``` yql + +```yql SELECT EndsWith(NULL, Utf8("")); -- null ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md index 347db0f105..6bb7b716ee 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md @@ -1,16 +1,17 @@ ## StaticMap -Transforms a structure or tuple by applying a lambda function to each element. +Transforms a structure or tuple by applying a lambda function to each item. Arguments: * Structure or tuple. -* Lambda for processing elements. +* Lambda for processing items. -Result: a structure or tuple with the same number and naming of elements as in the first argument, and with element data types determined by lambda results. +Result: a structure or tuple with the same number and naming of items as in the first argument, and with item data types determined by lambda results. **Examples:** -``` yql + +```yql SELECT * FROM ( SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md new file mode 100644 index 0000000000..3643e19691 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md @@ -0,0 +1,16 @@ +## StaticZip + +Merges structures or tuples element-by-element. All arguments (one or more) must be either structures with the same set of fields or tuples of the same length. +The result will be a structure or tuple, respectively. +Each item of the result is a tuple comprised of items taken from arguments. + +**Examples:** + +```yql +$one = <|k1:1, k2:2.0|>; +$two = <|k1:3.0, k2:4|>; + +-- Adding two structures item-by-item +SELECT StaticMap(StaticZip($one, $two), ($tuple)->($tuple.0 + $tuple.1)) AS sum; +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md index c07157c0af..78f533ed6f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md @@ -11,16 +11,20 @@ Optional arguments: * Substring length: The number of bytes starting from the specified position (an integer, or the default `NULL` meaning "up to the end of the source string"). -Indexing starts from zero. If the specified position and length are beyond the string, it returns an empty string. +Indexing starts from zero. If the specified position and length are beyond the string, returns an empty string. If the input string is optional, the result is also optional. **Examples** -``` yql + +```yql SELECT SUBSTRING("abcdefg", 3, 1); -- d ``` -``` yql + +```yql SELECT SUBSTRING("abcdefg", 3); -- defg ``` -``` yql + +```yql SELECT SUBSTRING("abcdefg", NULL, 3); -- abc ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md index d7f71ddfe3..24dbcfe745 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md @@ -6,18 +6,19 @@ No arguments. Returns a string with the full path or an empty string and warning {% note info %} -The functions [TablePath](#tablepath), [TableName](#tablename), and [TableRecordIndex](#tablerecordindex) don't support temporary and anonymous tables (they return an empty row or 0 for [TableRecordIndex](#tablerecordindex)). +The [TablePath](#tablepath), [TableName](#tablename), and [TableRecordIndex](#tablerecordindex) functions don't support temporary and anonymous tables (they return an empty string or 0 for [TableRecordIndex](#tablerecordindex)). These functions are calculated when [executing](../../../syntax/select.md#selectexec) projections in `SELECT`, and by that time the current table may already be temporary. To avoid such a situation, create a subquery for calculating these functions, as shown in the second example below. {% endnote %} **Examples** -``` yql + +```yql SELECT TablePath() FROM CONCAT(table_a, table_b); ``` -``` yql +```yql SELECT key, tpath_ AS path FROM (SELECT a.*, TablePath() AS tpath_ FROM RANGE(`my_folder`) AS a) WHERE key IN $subquery; ``` @@ -32,12 +33,13 @@ Optional arguments: * Specifying the system ("yt") whose rules are used to determine the table name. You need to specify the system only if [USE](../../../syntax/select.md#use) doesn't specify the current cluster. **Examples** -``` yql + +```yql USE hahn; SELECT TableName() FROM CONCAT(table_a, table_b); ``` -``` yql +```yql SELECT TableName(Path, "yt") FROM hahn.FOLDER(folder_name); ``` @@ -48,6 +50,8 @@ Access to the current sequence number of a row in the physical source table, **s No arguments. When used in combination with [CONCAT](../../../syntax/select.md#concat), [RANGE](../../../syntax/select.md#range) and other similar mechanisms, numbering restarts for each input table. If used in an incorrect context, it returns 0. **Example** -``` yql + +```yql SELECT TableRecordIndex() FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md index 9abb22feaf..f6cedf2bf8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md @@ -3,6 +3,8 @@ Getting the entire table row as a structure. No arguments{% if feature_join %}. `JoinTableRow` in case of `JOIN` always returns a structure with table prefixes{% endif %}. **Example** -``` yql + +```yql SELECT TableRow() FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md index 2992859f12..75105178eb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md @@ -3,7 +3,8 @@ Conversion of [primitive data types](../../../types/primitive.md) to a string with their binary representation and back. Numbers are represented in the [little endian](https://en.wikipedia.org/wiki/Endianness#Little-endian) format. **Examples** -``` yql + +```yql SELECT ToBytes(123), -- "\u0001\u0000\u0000\u0000" FromBytes( @@ -11,3 +12,4 @@ SELECT Uint64 ); -- 1234567890ul ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md index 84fa060875..7854ae7db0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md @@ -9,7 +9,8 @@ Arguments: * Variant type **Example** -``` yql + +```yql $var_type = Variant<foo: Int32, bar: Bool>; SELECT @@ -25,7 +26,8 @@ Arguments: * A string with the field name **Example** -``` yql + +```yql SELECT AsVariant(6, "foo") as VariantValue ``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md index 499e422e37..e8a60bd796 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md @@ -7,9 +7,11 @@ Syntax: `WeakField([<table>.]<field>, <type>[, <default_value>])`. The default value is used only if the column is missing in the data schema. To use the default value in any case, use [COALESCE](#coalesce). **Examples:** -``` yql + +```yql SELECT WeakField(my_column, String, "no value"), WeakField(my_table.other_column, Int64) FROM my_table; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md index e77c620ef4..afd230466c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md @@ -17,7 +17,7 @@ In the text representation, S-expressions have the following format: * Atom: ```'"foo"```. The apostrophe character (') denotes quoting of the next line that is usually enclosed in quotation marks. * List: ```'("foo" "bar")```. The apostrophe character (') denotes that there will be no function call in parentheses. -* Calling the built-in function: ```(foo "bar")```. The first element inside the brackets is the mandatory name of the function followed by the function arguments. +* Calling the built-in function: ```(foo "bar")```. The first item inside the brackets is the mandatory name of the function followed by the function arguments. * Declaring a lambda function: ```(lambda '(x y) (+ x y))```. The `lambda` keyword is followed by a list of argument names and then by the body of the lambda function. * The lambda function argument is ```x```. Unlike an atom, a string without an apostrophe character (') references a name in the current scope. When declaring a lambda function, the names of arguments are added to the body's visibility scope, and, if needed, the name is hidden from the global scope. * The ```world```. @@ -33,7 +33,8 @@ To learn more about code generation, see a [separate introductory article](../.. Serializing the code as [S-expressions](/docs/s_expressions). The code must not contain free arguments of functions, hence, to serialize the lambda function code, you must pass it completely, avoiding passing individual expressions that might contain lambda function arguments. **Examples:** -``` yql + +```yql SELECT FormatCode(AtomCode("foo")); -- ( -- (return '"foo") @@ -45,7 +46,8 @@ SELECT FormatCode(AtomCode("foo")); Build a code node with the `world` type. **Examples:** -``` yql + +```yql SELECT FormatCode(WorldCode()); -- ( -- (return world) @@ -57,7 +59,8 @@ SELECT FormatCode(WorldCode()); Build a code node with the `atom` type from a string passed to the argument. **Examples:** -``` yql + +```yql SELECT FormatCode(AtomCode("foo")); -- ( -- (return '"foo") @@ -69,7 +72,8 @@ SELECT FormatCode(AtomCode("foo")); Build a code node with the `list` type from a set of nodes or lists of code nodes passed to arguments. In this case, lists of arguments are built in as separately listed code nodes. **Examples:** -``` yql + +```yql SELECT FormatCode(ListCode( AtomCode("foo"), AtomCode("bar"))); @@ -90,7 +94,8 @@ SELECT FormatCode(ListCode(AsList( Build a code node with the `built-in function call` from a string with the function name and a set of nodes or lists of code nodes passed to arguments. In this case, lists of arguments are built in as separately listed code nodes. **Examples:** -``` yql + +```yql SELECT FormatCode(FuncCode( "Baz", AtomCode("foo"), @@ -117,7 +122,8 @@ You can build a code node with the `lambda function declaration` type from: * The number of arguments and a [lambda function](../../syntax/expressions.md#lambda) with one argument. In this case, a list of nodes of the `argument`type will be passed as an argument to this lambda function. **Examples:** -``` yql + +```yql SELECT FormatCode(LambdaCode(($x, $y) -> { RETURN FuncCode("+", $x, $y); })); @@ -138,7 +144,8 @@ SELECT FormatCode(LambdaCode(2, ($args) -> { Substituting the code node passed in the argument, into the main program code. **Examples:** -``` yql + +```yql SELECT EvaluateCode(FuncCode("Int32", AtomCode("1"))); -- 1 $lambda = EvaluateCode(LambdaCode(($x, $y) -> { @@ -152,7 +159,8 @@ SELECT $lambda(1, 2); -- 3 Substituting the code node representing the result of evaluating an expression passed in the argument, into the main program. **Examples:** -``` yql + +```yql $add3 = EvaluateCode(LambdaCode(($x) -> { RETURN FuncCode("+", $x, ReprCode(1 + 2)); })); @@ -164,7 +172,8 @@ SELECT $add3(1); -- 4 Substituting into the main program the code node that represents an expression or a [lambda function](../../syntax/expressions.md#lambda) passed in the argument. If free arguments of lambda functions were found during the substitution, they are calculated and substituted into the code as in the [ReprCode](#reprcode) function. **Examples:** -``` yql + +```yql $lambda = ($x, $y) -> { RETURN $x + $y }; $makeClosure = ($y) -> { RETURN EvaluateCode(LambdaCode(($x) -> { @@ -175,3 +184,4 @@ $makeClosure = ($y) -> { $closure = $makeClosure(2); SELECT $closure(1); -- 3 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md index f8a4c42172..568a78ff53 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md @@ -7,7 +7,8 @@ Construct an empty dictionary. Two arguments are passed: for a key and a value. [Documentation for the type definition format](../../types/type_string.md). **Examples** -``` yql + +```yql SELECT DictCreate(String, Tuple<String,Double?>); ``` @@ -22,7 +23,8 @@ Construct an empty set. An argument is passed: the key type that can be built by [Documentation for the type definition format](../../types/type_string.md). **Examples** -``` yql + +```yql SELECT SetCreate(String); ``` @@ -32,29 +34,38 @@ SELECT SetCreate(Tuple<Int32?,String>); ## DictLength {#dictlength} -The count of elements in the dictionary. +The count of items in the dictionary. **Examples** -``` yql + +```yql SELECT DictLength(AsDict(AsTuple(1, AsList("foo", "bar")))); ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictLength(dict_column) FROM my_table; ``` + {% endif %} + ## DictHasItems {#dicthasitems} -Check that the dictionary contains at least one element. +Check that the dictionary contains at least one item. **Examples** -``` yql + +```yql SELECT DictHasItems(AsDict(AsTuple(1, AsList("foo", "bar")))) FROM my_table; ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictHasItems(dict_column) FROM my_table; ``` + {% endif %} ## DictItems {#dictitems} @@ -63,47 +74,58 @@ Get dictionary contents as a list of tuples including key-value pairs (`List<Tup **Examples** -``` yql +```yql SELECT DictItems(AsDict(AsTuple(1, AsList("foo", "bar")))); -- [ ( 1, [ "foo", "bar" ] ) ] ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictItems(dict_column) FROM my_table; ``` + {% endif %} + ## DictKeys {#dictkeys} Get a list of dictionary keys. **Examples** -``` yql +```yql SELECT DictKeys(AsDict(AsTuple(1, AsList("foo", "bar")))); -- [ 1 ] ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictKeys(dict_column) FROM my_table; ``` + {% endif %} + ## DictPayloads {#dictpayloads} Get a list of dictionary values. **Examples** -``` yql +```yql SELECT DictPayloads(AsDict(AsTuple(1, AsList("foo", "bar")))); -- [ [ "foo", "bar" ] ] ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictPayloads(dict_column) FROM my_table; ``` + {% endif %} ## DictLookup {#dictlookup} @@ -112,18 +134,21 @@ Get a dictionary element by its key. **Examples** -``` yql +```yql SELECT DictLookup(AsDict( AsTuple(1, AsList("foo", "bar")), AsTuple(2, AsList("bar", "baz")) ), 1); -- [ "foo", "bar" ] ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictLookup(dict_column, "foo") FROM my_table; ``` + {% endif %} ## DictContains {#dictcontains} @@ -132,25 +157,28 @@ Checking if an element in the dictionary using its key. Returns true or false. **Examples** -``` yql +```yql SELECT DictContains(AsDict( AsTuple(1, AsList("foo", "bar")), AsTuple(2, AsList("bar", "baz")) ), 42); -- false ``` + {% if feature_column_container_type %} -``` yql + +```yql SELECT DictContains(dict_column, "foo") FROM my_table; ``` + {% endif %} ## DictAggregate {#dictaggregate} -Apply [aggregation factory](../basic.md#aggregationfactory) to the passed dictionary where each value is a list. The factory is applied separately inside each key. -If the list is empty, the aggregation result is the same as for an empty table: 0 for the `COUNT` function and `NULL` for other functions. -If the list under a certain key is empty in the passed dictionary, such a key is removed from the result. +Apply [aggregation factory](../basic.md#aggregationfactory) to the passed dictionary where each value is a list. The factory is applied separately inside each key.
+If the list is empty, the aggregation result is the same as for an empty table: 0 for the `COUNT` function and `NULL` for other functions.
+If the list under a certain key is empty in the passed dictionary, such a key is removed from the result.
If the passed dictionary is optional and contains `NULL`, the result is also `NULL`. Arguments: @@ -166,7 +194,6 @@ SELECT DictAggregate(AsDict( AsTuple(2, AsList("baz", "qwe"))), AggregationFactory("Max")); -- {1 : "foo", 2 : "qwe" } - ``` ## SetIsDisjoint {#setisjoint} @@ -195,7 +222,8 @@ Arguments: * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1,V2) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. **Examples** -``` yql + +```yql SELECT SetIntersection(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 3 } SELECT SetIntersection( AsDict(AsTuple(1, "foo"), AsTuple(3, "bar")), @@ -214,7 +242,8 @@ So there are two options to make a call: * With the `Dict<K,V1>` and `Dict<K,V2>` arguments. **Examples** -``` yql + +```yql SELECT SetIncludes(ToSet(AsList(1, 2, 3)), AsList(3, 4)); -- false SELECT SetIncludes(ToSet(AsList(1, 2, 3)), ToSet(AsList(2, 3))); -- true ``` @@ -229,7 +258,8 @@ Arguments: * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1?,V2?) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. **Examples** -``` yql + +```yql SELECT SetUnion(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2, 3, 4 } SELECT SetUnion( AsDict(AsTuple(1, "foo"), AsTuple(3, "bar")), @@ -243,7 +273,8 @@ SELECT SetUnion( Construct a dictionary containing all the keys with their values in the first dictionary with no matching key in the second dictionary. **Examples** -``` yql + +```yql SELECT SetDifference(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2 } SELECT SetDifference( AsDict(AsTuple(1, "foo"), AsTuple(2, "bar")), @@ -261,7 +292,8 @@ Arguments: * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1?,V2?) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. **Examples** -``` yql + +```yql SELECT SetSymmetricDifference(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2, 4 } SELECT SetSymmetricDifference( AsDict(AsTuple(1, "foo"), AsTuple(3, "bar")), @@ -269,3 +301,4 @@ SELECT SetSymmetricDifference( ($k, $a, $b) -> { RETURN AsTuple($a, $b) }); -- { 2 : (null, "qwe"), 3 : ("bar", null) } ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md new file mode 100644 index 0000000000..702cb639fd --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md @@ -0,0 +1,17 @@ +# Built-in YQL functions + +- [Basic](../basic.md) +- [Aggregate](../aggregation.md) +{% if feature_window_functions %} +- [Window](../window.md) +{% endif %} +- [For lists](../list.md) +- [For dictionaries](../dict.md) +- [For structures](../struct.md) +- [For types](../types.md) +{% if feature_codegen %} +- [For code generation](../codegen.md) +{% endif %} +- [For JSON](../json.md) +- [C++ libraries](../../udf/list/index.md) + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md index c199107b6e..5d24591e4d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md @@ -1,1129 +1,1150 @@ -# Functions for JSON
-
-**JSON** is a lightweight [data-interchange format](https://www.json.org). In YQL, it's represented by the `Json` type. Unlike relational tables, JSON can store data with no schema defined. Here is an example of a valid JSON object:
-```json
-[
- {
- "name": "Jim Holden",
- "age": 30
- },
- {
- "name": "Naomi Nagata",
- "age": "twenty years old"
- }
-]
-```
-Despite the fact that the `age` field in the first object is of the `Number` type (`"age": 21`) and in the second object its type is `String` (`"age": "twenty years old"`), this is a fully valid JSON object.
-
-To work with JSON, YQL implements a subset of the [SQL support for JavaScript Object Notation (JSON)](https://www.iso.org/standard/67367.html) standard, which is part of the common ANSI SQL standard.
-
-## JsonPath
-
-Values inside JSON objects are accessed using a query language called JsonPath. All functions for JSON accept a JsonPath query as an argument.
-
-Let's look at an example. Suppose we have a JSON object like:
-```json
-{
- "comments": [
- {
- "id": 123,
- "text": "A whisper will do, if it's all that you can manage."
- },
- {
- "id": 456,
- "text": "My life has become a single, ongoing revelation that I haven’t been cynical enough."
- }
- ]
-}
-```
-
-Then, to get the text of the second comment, we can write the following JsonPath query:
-```
-$.comments[1].text
-```
-
-In this query:
-
-1. `$` is a way to access the entire JSON object.
-2. `$.comments` accesses the `comments` key of the JSON object.
-3. `$.comments[1]` accesses the second element of the JSON array (element numbering starts from 0).
-4. `$.comments[1].text` accesses the `text` key of the JSON object.
-5. Query execution result: `"My life has become a single, ongoing revelation that I haven’t been cynical enough."`
-
-### Quick reference
-
-| Operation | Example |
-| --------------------------------------- | ------------------------------------------------- |
-| Retrieving a JSON object key | `$.key` |
-| Retrieving all JSON object keys | `$.*` |
-| Accessing an array element | `$[25]` |
-| Retrieving an array subsegment | `$[2 to 5]` |
-| Accessing the last array element | `$[last]` |
-| Accessing all array elements | `$[*]` |
-| Unary operations | `- 1` |
-| Binary operations | `(12 * 3) % 4 + 8` |
-| Accessing a variable | `$variable` |
-| Logical operations | `(1 > 2) || (3 <= 4) && ("string" == "another")` |
-| Matching a regular expression | `$.name like_regex "^[A-Za-z]+$"` |
-| Checking the string prefix | `$.name starts with "Bobbie"` |
-| Checking if a path exists | `exists ($.profile.name)` |
-| Checking a Boolean expression for null | `($.age > 20) is unknown` |
-| Filtering values | `$.friends ? (@.age >= 18 && @.gender == "male")` |
-| Getting the value type | `$.name.type()` |
-| Getting the array size | `$.friends.size()` |
-| Converting a string to a number | `$.number.double()` |
-| Rounding up a number | `$.number.ceiling()` |
-| Rounding down a number | `$.number.floor()` |
-| Returning the absolute value | `$.number.abs()` |
-| Getting key-value pairs from an object | `$.profile.keyvalue()` |
-
-### Data model
-
-The result of executing all JsonPath expressions is a sequence of JSON values. For example:
-
-* The result of executing the `"Bobbie"` expression is a sequence with the only element `"Bobbie"`. Its length is 1.
-* The result of executing the `$` expression (that takes the entire JSON object) in JSON `[1, 2, 3]` is `[1, 2, 3]`. A sequence of 1 element of the array `[1, 2, 3]`
-* The result of executing the `$[*]` expression (retrieving all array elements) in JSON `[1, 2, 3]` is `1, 2, 3`. A sequence of three elements:`1`, `2`, and `3`
-
-If the input sequence consists of multiple values, some operations are performed for each element (for example, accessing a JSON object key). However, other operations require a sequence of one element as input (for example, binary arithmetic operations).
-
-The behavior of a specific operation is described in the corresponding section of the documentation.
-
-### Execution mode
-
-JsonPath supports two execution modes, `lax` and `strict`. Setting the mode is optional. By default, `lax`. The mode is specified at the beginning of a query. For example, `strict $.key`.
-
-The behavior for each mode is described in the corresponding sections with JsonPath operations.
-
-#### Auto unpacking of arrays
-
-When accessing a JSON object key in `lax` mode, arrays are automatically unpacked.
-
-**Example:**
-```json
-[
- {
- "key": 123
- },
- {
- "key": 456
- }
-]
-```
-
-The `lax $.key` query is successful and returns `123, 456`. As `$` is an array, it's automatically unpacked and accessing the key of the `$.key` JSON object is executed for each element in the array.
-
-The `strict $.key` query returns an error. In `strict` mode, there is no support for auto unpacking of arrays. Since `$` is an array and not an object, accessing the `$.key` object key is impossible. You can fix this by writing `strict $[*].key`.
-
-Unpacking is only 1 level deep. In the event of nested arrays, only the outermost one is unpacked.
-
-#### Wrapping values in arrays
-
-When accessing an array element in `lax` mode, JSON values are automatically wrapped in an array.
-
-**Example:**
-```json
-{
- "name": "Avasarala"
-}
-```
-
-The `lax $[0].name` query is successful and returns `"Avasarala"`. As `$` isn't an array, it's automatically wrapped in an array of length 1. Accessing the first element `$[0]` returns the source JSON object where the `name` key is taken.
-
-The `strict $[0].name` query returns an error. In `strict` mode, values aren't wrapped in an array automatically. Since `$` is an object and not an array, accessing the `$[0]` element is impossible. You can fix this by writing `strict $.name`.
-
-#### Handling errors
-
-Some errors are converted to an empty result when a query is executed in `lax` mode.
-
-### Literals
-
-Values of some types can be specified in a JsonPath query using literals:
-
-| Type | Example |
-| ------------------ | ---------------- |
-| Numbers | `42`, `-1.23e-5` |
-| Boolean values | `false`, `true` |
-| Null | `Null` |
-| Stings | `"Belt"` |
-
-### Accessing JSON object keys
-
-JsonPath supports accessing JSON object keys, such as `$.session.user.name`.
-
-{% note info %}
-
-Accessing keys without quotes is only supported for keys that start with an English letter or underscore and only contain English letters, underscores, numbers, and a dollar sign. Use quotes for all other keys. For example, `$.profile."this string has spaces"` or `$.user."42 is the answer"`
-
-{% endnote %}
-
-For each value from the input sequence:
-
-1. If the value is an array, it's automatically unpacked in `lax` mode.
-2. If the value isn't a JSON object or if it is and the specified key is missing from this JSON object, a query executed in `strict` mode fails. In `lax` mode, an empty result is returned for this value.
-
-The expression execution result is the concatenation of the results for each value from the input sequence.
-
-**Example:**
-```json
-{
- "name": "Amos",
- "friends": [
- {
- "name": "Jim"
- },
- {
- "name": "Alex"
- }
- ]
-}
-```
-
-| | `lax` | `strict` |
-| ---------------- | ---------------- | -------- |
-| `$.name` | `"Amos"` | `"Amos"` |
-| `$.surname` | Empty result | Error |
-| `$.friends.name` | `"Jim", "Alex"` | Error |
-
-### Accessing all JSON object keys
-
-JsonPath supports accessing all JSON object keys at once: `$.*`.
-
-For each value from the input sequence:
-
-1. If the value is an array, it's automatically unpacked in `lax` mode.
-2. If the value isn't a JSON object, a query executed in `strict` mode fails. In `lax` mode, an empty result is returned for this value.
-
-The expression execution result is the concatenation of the results for each value from the input sequence.
-
-**Example:**
-```json
-{
- "profile": {
- "id": 123,
- "name": "Amos"
- },
- "friends": [
- {
- "name": "Jim"
- },
- {
- "name": "Alex"
- }
- ]
-}
-```
-
-| | `lax` | `strict` |
-| ------------- | --------------- | ------------- |
-| `$.profile.*` | `123, "Amos"` | `123, "Amos"` |
-| `$.friends.*` | `"Jim", "Alex"` | Error |
-
-### Accessing an array element
-
-JsonPath supports accessing array elements: `$.friends[1, 3 to last - 1]`.
-
-For each value from the input sequence:
-
-1. If the value isn't an array, a query executed in `strict` mode fails. In `lax` mode, values are automatically wrapped in an array.
-2. The `last` keyword is replaced with the array's last index. Using `last` outside of accessing the array is an error in both modes.
-3. The specified indexes are calculated. Each of them must be a single number, otherwise the query fails in both modes.
-4. If the index is a fractional number, it's rounded down.
-5. If the index goes beyond the array boundaries, the query executed in `strict` mode fails. In `lax` mode, this index is ignored.
-6. If a segment is specified and its start index is greater than the end index (for example, `$[20 to 1]`), the query fails in `strict` mode. In `lax` mode, this segment is ignored.
-7. All elements by the specified indexes are added to the result. Segments include **both ends**.
-
-**Examples**:
-```json
-[
- {
- "name": "Camina",
- "surname": "Drummer"
- },
- {
- "name": "Josephus",
- "surname": "Miller"
- },
- {
- "name": "Bobbie",
- "surname": "Draper"
- },
- {
- "name": "Julie",
- "surname": "Mao"
- }
-]
-```
-
-| | `lax` | `strict` |
-| ----------------------------- | ------------------------------- | ------------------------------- |
-| `$[0].name` | `"Camina"` | `"Camina"` |
-| `$[1, 2 to 3].name` | `"Josephus", "Bobbie", "Julie"` | `"Josephus", "Bobbie", "Julie"` |
-| `$[last - 2].name` | `"Josephus"` | `"Josephus"` |
-| `$[2, last + 200 to 50].name` | `"Bobbie"` | Error |
-| `$[50].name` | Empty result | Error |
-
-### Accessing all array elements
-
-JsonPath supports accessing all array elements at once: `$[*]`.
-
-For each value from the input sequence:
-
-1. If the value isn't an array, a query executed in `strict` mode fails. In `lax` mode, values are automatically wrapped in an array.
-2. All elements of the current array are added to the result.
-
-**Examples:**
-```json
-[
- {
- "class": "Station",
- "title": "Medina"
- },
- {
- "class": "Corvette",
- "title": "Rocinante"
- }
-]
-```
-
-| | `lax` | `strict` |
-| ------------------- | ------------------------- | ----------------------- |
-| `$[*].title` | `"Medina", "Rocinante"` | `"Medina", "Rocinante"` |
-| `lax $[0][*].class` | `"Station"` | Error |
-
-Let's analyze the last example step by step:
-
-1. `$[0]` returns the first element of the array, that is `{"class": "Station", "title": "Medina"}`
-2. `$[0][*]` expects an array for input, but an object was input instead. It's automatically wrapped in an array as `[ {"class": "Station", "title": "Medina"} ]`
-3. Now, `$[0][*]` can be executed and returns all elements of the array, that is `{"class": "Station", "title": "Medina"}`
-4. `$[0][*].class` returns the `class` field value: `"Station"`.
-
-### Arithmetic operations
-
-{% note info %}
-
-All arithmetic operations work with numbers as with Double. Calculations are made with potential [loss of accuracy](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html).
-
-{% endnote %}
-
-#### Unary operations
-
-JsonPath supports unary `+` and `-`.
-
-A unary operation applies to all values from the input sequence. If a unary operation's input is a value that isn't a number, a query fails in both modes.
-
-**Example:**
-```json
-[1, 2, 3, 4]
-```
-
-The `strict -$[*]` query is successful and returns `-1, -2, -3, -4`.
-
-The `lax -$` query fails as `$` is an array and not a number.
-
-#### Binary operations
-
-JsonPath supports binary arithmetic operations (in descending order of priority):
-
-1. Multiplication `*`, dividing floating-point numbers `/`, and taking the remainder `%` (works as the `MOD` function in `SQL`).
-2. Addition `+`, subtraction `-`.
-
-You can change the order of operations using parentheses.
-
-If each argument of a binary operation is not a single number or a number is divided by 0, the query fails in both modes.
-
-**Examples:**
-
-- `(1 + 2) * 3` returns `9`
-- `1 / 2` returns `0.5`
-- `5 % 2` returns `1`
-- `1 / 0` fails
-- If JSON is `[-32.4, 5.2]`, the `$[0] % $[1]` query returns `-1.2`
-- If JSON is `[1, 2, 3, 4]`, the `lax $[*] + $[*]` query fails as the `$[*]` expression execution result is `1, 2, 3, 4`, that is multiple numbers. A binary operation only requires one number for each of its arguments.
-
-### Boolean values
-
-Unlike some other programming languages, Boolean values in JsonPath are not only `true` and `false`, but also `null` (uncertainty).
-
-JsonPath considers any values received from a JSON document to be non-Boolean. For example, a query like `! $.is_valid_user` (a logical negation applied to the `is_valid_user`) field is syntactically invalid because the `is_valid_user` field value is not Boolean (even when it actually stores `true` or `false`). The correct way to write this kind of query is to explicitly use a comparison with a Boolean value, such as `$.is_valid_user == false`.
-
-### Logical operations
-
-JsonPath supports some logical operations for Boolean values.
-
-The arguments of any logical operation must be a single Boolean value. -All logical operations return a Boolean value.
-
-**Logical negation,`!`**
-
-Truth table:
-
-| `x` | `!x` |
-| ------- | ------- |
-| `true` | `false` |
-| `false` | `true` |
-| `Null` | `Null` |
-
-**Logical AND, `&&`**
-
-In the truth table, the first column is the left argument, the first row is the right argument, and each cell is the result of using the Logical AND both with the left and right arguments:
-
-| `&&` | `true` | `false` | `Null` |
-| ------- | ------- | ------- | ------- |
-| `true` | `true` | `false` | `Null` |
-| `false` | `false` | `false` | `false` |
-| `Null` | `Null` | `false` | `Null` |
-
-**Logical OR, `||`**
-
-In the truth table, the first column is the left argument, the first row is the right argument, and each cell is the result of using the logical OR with both the left and right arguments:
-
-| `||` | `true` | `false` | `Null` |
-| ------- | ------- | ------- | ------- |
-| `true` | `true` | `true` | `true` |
-| `false` | `true` | `false` | `Null` |
-| `Null` | `true` | `Null` | `Null` |
-
-**Examples:**
-
-- `! (true == true)`, the result is `false`
-- `(true == true) && (true == false)`, the result is `false`
-- `(true == true) || (true == false)`, the result is `true`
-
-### Comparison operators
-
-JsonPath implements comparison operators for values:
-
-- Equality, `==`
-- Inequality, `!=`and `<>`
-- Less than and less than or equal to, `<` and `=`
-- Greater than and greater than or equal to, `>` and `>=`
-
-All comparison operators return a Boolean value. Both operator arguments support multiple values.
-
-If an error occurs when calculating the operator arguments, it returns `null`. In this case, the JsonPath query execution continues.
-
-The arrays of each of the arguments are automatically unpacked. After that, for each pair where the first element is taken from the sequence of the left argument and the second one from the sequence of the right argument:
-
-1. The elements of the pair are compared
-2. If an error occurs during the comparison, the `ERROR` flag is set.
-3. If the comparison result is true, the flag set is `FOUND`
-4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed.
-
-If the pair analysis results in:
-
-1. The `ERROR` flag is set, the operator returns `null`
-2. The `FOUND` flag is set, the operator returns `true`
-3. Otherwise, it returns `false`
-
-We can say that this algorithm considers all pairs from the Cartesian product of the left and right arguments, trying to find the pair whose comparison returns true.
-
-Elements in a pair are compared according to the following rules:
-
-1. If the left or right argument is an array or object, the comparison fails.
-2. `null == null` returns true
-3. In all other cases, if one of the arguments is `null`, false is returned.
-4. If the left and right arguments are of different types, the comparison fails.
-5. Strings are compared byte by byte.
-6. `true` is considered greater than `false`
-7. Numbers are compared with the accuracy of `1e-20`
-
-**Example:**
-
-Let's take a JSON document as an example
-
-```json
-{
- "left": [1, 2],
- "right": [4, "Inaros"]
-}
-```
-
-and analyze the steps for executing the `lax $.left < $.right` query:
-
-1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"`
-2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND`
-3. Since the query is executed in `lax` mode and the `FOUND` flag is set, we aren't analyzing any more pairs.
-4. Since we have the `FOUND` flag set, the operator returns true.
-
-Let's take the same query in a different execution mode: `strict $.left < $.right`:
-
-1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"`
-2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND`
-3. Let's take the pair `(2, 4)`. The comparison is successful as `2 < 4` is true. Set the flag `FOUND`
-4. Let's take the pair `(1, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR`
-5. Let's take the pair `(2, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR`
-6. Since we have the `ERROR` flag set, the operator returns `null`
-
-### Predicates
-
-JsonPath supports predicates which are expressions that return a Boolean value and check a certain condition. You can use them, for example, in filters.
-#### `like_regex`
-
-The `like_regex` predicate lets you check if a string matches a regular expression. The syntax of regular expressions is the same as in [Hyperscan UDF](../../udf/list/hyperscan.md) and [REGEXP](../../syntax/expressions.md#regexp).
-
-**Syntax**
-
-```
-<expression> like_regex <regexp string> [flag <flag string>]
-```
-
-Where:
-
-1. `<expression>` is a JsonPath expression with strings to be checked for matching the regular expression.
-2. `<regexp string>` is a string with the regular expression.
-3. `flag <flag string>` is an optional section where `<flag string>` is a string with regular expression execution flags.
-
-Supported flags:
-
-- `i`: Disable the case sensitivity.
-
-**Execution**
-
-Before the check, the input sequence arrays are automatically unpacked.
-
-After that, for each element of the input sequence:
-
-1. A check is made to find out if a string matches a regular expression.
-2. If the element isn't a string, the `ERROR` flag is set.
-3. If the check result is true, the `FOUND` flag is set.
-4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed.
-
-If the pair analysis results in:
-
-1. Setting the `ERROR` flag, the predicate returns `null`
-2. Setting the `FOUND` flag, the predicate returns `true`
-3. Otherwise, the predicate returns `false`
-
-**Examples**
-
-1. `"123456" like_regex "^[0-9]+$"` returns `true`
-2. `"123abcd456" like_regex "^[0-9]+$"` returns `false`
-3. `"Naomi Nagata" like_regex "nag"` returns `false`
-4. `"Naomi Nagata" like_regex "nag" flag "i"` returns `true`
-
-#### `starts with`
-
-The `starts with` predicate lets you check if one string is a prefix of another.
-
-**Syntax**
-
-```
-<string expression> starts with <prefix expression>
-```
-
-Where:
-
-1. `<string expression>` is a JsonPath expression with the string to check.
-2. `<prefix expression>` is a JsonPath expression with a prefix string.
-
-This means that the predicate will check that the `<string expression>` starts with the `<prefix expression>` string.
-
-**Execution**
-
-The first argument of the predicate must be a single string.
-
-The second argument of the predicate must be a sequence of (possibly, multiple) strings.
-
-For each element in a sequence of prefix strings:
-
-1. A check is made for whether "an element is a prefix of an input string"
-2. If the element isn't a string, the `ERROR` flag is set.
-3. If the check result is true, the `FOUND` flag is set.
-4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed.
-
-If the pair analysis results in:
-
-1. Setting the `ERROR` flag, the predicate returns `null`
-2. Setting the `FOUND` flag, the predicate returns `true`
-3. Otherwise, the predicate returns `false`
-
-**Examples**
-
-1. `"James Holden" starts with "James"` returns `true`
-2. `"James Holden" starts with "Amos"` returns `false`
-
-#### `exists`
-
-The `exists` predicate lets you check whether a JsonPath expression returns at least one element.
-
-**Syntax**
-
-```
-exists (<expression>)
-```
-
-Where `<expression>` is the JsonPath expression to be checked. Parentheses around the expression are required.
-
-**Execution**
-
-1. The passed JsonPath expression is executed
-2. If an error occurs, the predicate returns `null`
-3. If an empty sequence is obtained as a result of the execution, the predicate returns `false`
-4. Otherwise, the predicate returns `true`
-
-**Examples**
-
-Let's take a JSON document:
-
-```json
-{
- "profile": {
- "name": "Josephus",
- "surname": "Miller"
- }
-}
-```
-
-1. `exists ($.profile.name)` returns `true`
-2. `exists ($.friends.profile.name)` returns `false`
-3. `strict exists ($.friends.profile.name)` returns `null`, because accessing non-existent object keys in `strict` mode is an error.
-
-#### `is unknown`
-
-The `is unknown` predicate lets you check if a Boolean value is `null`.
-
-**Syntax**
-
-```
-(<expression>) is unknown
-```
-
-Where `<expression>` is the JsonPath expression to be checked. Only expressions that return a Boolean value are allowed. Parentheses around the expression are required.
-
-**Execution**
-
-1. If the passed expression returns `null`, the predicate returns `true`
-2. Otherwise, the predicate returns `false`
-
-**Examples**
-
-1. `(1 == 2) is unknown` returns `false`. The `1 == 2` expression returned `false`, which is not `null`
-2. `(1 == "string") is unknown` returns `true`. The `1 == "string"` expression returned `null`, because strings and numbers can't be compared in JsonPath.
-
-### Filters
-
-JsonPath lets you filter values obtained during query execution.
-
-An expression in a filter must return a Boolean value. -Before filtering, the input sequence arrays are automatically unpacked.
-
-For each element of the input sequence:
-
-1. The value of the current filtered `@` object becomes equal to the current element of the input sequence.
-2. Executing the expression in the filter
-3. If an error occurs during the expression execution, the current element of the input sequence is skipped.
-4. If the expression execution result is the only `true` value, the current element is added to the filter result.
-
-**Example:**
-
-Suppose we have a JSON document describing the user's friends
-
-```json
-{
- "friends": [
- {
- "name": "James Holden",
- "age": 35,
- "money": 500
- },
- {
- "name": "Naomi Nagata",
- "age": 30,
- "money": 345
- }
- ]
-}
-```
-
-and we want to get a list of friends who are over 32 years old using a JsonPath query. To do this, you can write the following query:
-
-```
-$.friends ? (@.age > 32)
-```
-
-Let's analyze the query in parts:
-
-- `$.friends` accesses the `friends` array in the JSON document.
-- `? ( ... )` is the filter syntax. An expression inside the parentheses is called a predicate.
-- `` accesses the currently filtered object. In our example, it's the object describing a friend of the user.
-- `.age` accesses the `age` field of the currently filtered object.
-- `.age > 32` compares the `age` field with the value 32. As a result of the query, only the values for which this predicate returned true remain.
-
-The query only returns the first friend from the array of user's friends.
-
-Like many other JsonPath operators, filters can be arranged in chains. Let's take a more complex query that selects the names of friends who are older than 20 and have less than 400 currency units:
-
-```
-$.friends ? (@.age > 20) ? (@.money < 400) . name
-```
-
-Let's analyze the query in parts:
-
-- `$.friends` accesses the `friends` array in the JSON document.
-- `? (@.age > 20)` is the first filter. Since all friends are over 20, it just returns all the elements of the `friends` array.
-- `? (@.money < 400)` is the second filter. It only returns the second element of the `friends` array, since only its `money` field value is less than 400.
-- `.name` accesses the `name` field of filtered objects.
-
-The query returns a sequence of a single element: `"Naomi Nagata"`.
-
-In practice, it's recommended to combine multiple filters into one if possible. The above query is equivalent to `$.friends ? (@.age > 20 && @.money < 400) . name`.
-
-### Methods
-
-JsonPath supports methods that are functions converting one sequence of values to another. The syntax for calling a method is similar to accessing the object key:
-
-```
-$.friends.size()
-```
-
-Just like in the case of accessing object keys, method calls can be arranged in chains:
-
-```
-$.numbers.double().floor()
-```
-
-#### `type`
-
-The `type` method returns a string with the type of the passed value.
-
-For each element of the input sequence, the method adds this string to the output sequence according to the table below:
-
-| Value type | String with type |
-| ------------------ | ------------------------ |
-| Null | `"null"` |
-| Boolean value | `"boolean"` |
-| Number | `"number"` |
-| String | `"string"` |
-| Array | `"array"` |
-| Objects | `"object"` |
-
-**Examples**
-
-1. `"Naomi".type()` returns `"string"`
-2. `false.type()` returns `"boolean"`
-
-#### `size`
-
-The `size` method returns the size of the array.
-
-For each element of the input sequence, the method adds the following to the output sequence:
-
-1. The size of the array if the element type is an array.
-2. For all other types (including objects), it adds `1`
-
-**Examples**
-
-Let's take a JSON document:
-
-```json
-{
- "array": [1, 2, 3],
- "object": {
- "a": 1,
- "b": 2
- },
- "scalar": "string"
-}
-```
-
-And queries to it:
-
-1. `$.array.size()` returns `3`
-2. `$.object.size()` returns `1`
-3. `$.scalar.size()` returns `1`
-
-#### `Double`
-
-The `double` method converts strings to numbers.
-
-Before its execution, the input sequence arrays are automatically unpacked.
-
-All elements in the input sequence must be strings that contain decimal numbers. It's allowed to specify the fractional part and exponent.
-
-**Examples**
-
-1. `"125".double()` returns `125`
-2. `"125.456".double()` returns `125.456`
-3. `"125.456e-3".double()` returns `0.125456`
-
-#### `ceiling`
-
-The `ceiling` method rounds up a number.
-
-Before its execution, the input sequence arrays are automatically unpacked.
-
-All elements in the input sequence must be numbers.
-
-**Examples**
-
-1. `(1.3).ceiling()` returns `2`
-2. `(1.8).ceiling()` returns `2`
-3. `(1.5).ceiling()` returns `2`
-4. `(1.0).ceiling()` returns `1`
-
-#### `floor`
-
-The `floor` method rounds down a number.
-
-Before its execution, the input sequence arrays are automatically unpacked.
-
-All elements in the input sequence must be numbers.
-
-**Examples**
-
-1. `(1.3).floor()` returns `1`
-2. `(1.8).floor()` returns `1`
-3. `(1.5).floor()` returns `1`
-4. `(1.0).floor()` returns `1`
-
-#### `abs`
-
-The `abs` method calculates the absolute value of a number (removes the sign).
-
-Before its execution, the input sequence arrays are automatically unpacked.
-
-All elements in the input sequence must be numbers.
-
-**Examples**
-
-1. `(0.0).abs()` returns `0`
-2. `(1.0).abs()` returns `1`
-3. `(-1.0).abs()` returns `1`
-
-#### `keyvalue`
-
-The `keyvalue` method converts an object to a sequence of key-value pairs.
-
-Before its execution, the input sequence arrays are automatically unpacked.
-
-All elements in the input sequence must be objects.
-
-For each element of the input sequence:
-
-1. Each key-value pair in the element is analyzed.
-2. For each key-value pair, an object is generated with the keys `name` and `value`.
-3. `name` stores a string with the name of the key from the pair.
-4. `value` stores the value from the pair.
-5. All objects for this element are added to the output sequence.
-
-**Examples**
-
-Let's take a JSON document:
-
-```json
-{
- "name": "Chrisjen",
- "surname": "Avasarala",
- "age": 70
-}
-```
-
-The `$.keyvalue()` query returns the following sequence for it:
-
-```json
-{
- "name": "age",
- "value": 70
-},
-{
- "name": "name",
- "value": "Chrisjen"
-},
-{
- "name": "surname",
- "value": "Avasarala"
-}
-```
-
-### Variables
-
-Functions using JsonPath can pass values into a query. They are called variables. To access a variable, write the `$` character and the variable name: `$variable`.
-
-**Example:**
-
-Let the `planet` variable be equal to
-```json
-{
- "name": "Mars",
- "gravity": 0.376
-}
-```
-
-Then the `strict $planet.name` query returns `"Mars"`.
-
-Unlike many programming languages, JsonPath doesn't support creating new variables or modifying existing ones.
-
-## Common arguments
-
-All functions for JSON accept:
-
-1. A JSON value (can be an arbitrary `Json` or `Json?` expression)
-2. A JsonPath query (must be explicitly specified with a string literal)
-3. **(Optional)** `PASSING` section
-
-### PASSING section
-
-Lets you pass values to a JsonPath query as variables.
-
-**Syntax:**
-```yql
-PASSING
- <expression 1> AS <variable name 1>,
- <expression 2> AS <variable name 2>,
- ...
-```
-
-`<expression>` can have the following types:
-
-- Numbers, `Date`, `DateTime`, and `Timestamp` (a `CAST` into `Double` will be made before passing a value to JsonPath)
-- `Utf8`, `Bool`, and `Json`
-
-You can set a `<variable name>` in several ways:
-
-- As an SQL name like `variable`
-- In quotes, for example, `"variable"`
-
-**Example:**
-```yql
-JSON_VALUE(
- $json,
- "$.timestamp - $Now + $Hour"
- PASSING
- 24 * 60 as Hour,
- CurrentUtcTimestamp() as "Now"
-)
-```
-
-## JSON_EXISTS
-
-The `JSON_EXISTS` function checks if a JSON value meets the specified JsonPath.
-
-**Syntax:**
-```yql
-JSON_EXISTS(
- <JSON expression>,
- <JsonPath query>,
- [<PASSING clause>]
- [{TRUE | FALSE | UNKNOWN | ERROR} ON ERROR]
-)
-```
-
-**Return value:** `Bool?`
-
-**Default value:** If the `ON ERROR` section isn't specified, the used section is `FALSE ON ERROR`
-
-**Behavior:**
-
-1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Bool?`
-2. If an error occurs during JsonPath execution, the returned value depends on the `ON ERROR` section:
- - `TRUE`: Return `True`.
- - `FALSE`: Return `False`.
- - `UNKNOWN`: Return an empty `Bool?`.
- - `ERROR`: Abort the entire query.
-3. If the result of JsonPath execution is one or more values, the return value is `True`.
-4. Otherwise, `False` is returned.
-
-**Examples:**
-
-```yql
-$json = CAST(@@{
- "title": "Rocinante",
- "crew": [
- "James Holden",
- "Naomi Nagata",
- "Alex Kamai",
- "Amos Burton"
- ]
-}@@ as Json);
-
-SELECT
- JSON_EXISTS($json, "$.title"), -- True
- JSON_EXISTS($json, "$.crew[*]"), -- True
- JSON_EXISTS($json, "$.nonexistent"); -- False, as JsonPath returns an empty result
-
-SELECT
- -- JsonPath error, False is returned because the default section used is FALSE ON ERROR
- JSON_EXISTS($json, "strict $.nonexistent");
-
-SELECT
- -- JsonPath error, the entire YQL query fails.
- JSON_EXISTS($json, "strict $.nonexistent" ERROR ON ERROR);
-```
-
-## JSON_VALUE
-
-The `JSON_VALUE` function retrieves a scalar value from JSON (anything that isn't an array or object).
-
-**Syntax:**
-``` yql
-JSON_VALUE(
- <JSON expression>,
- <JsonPath query>,
- [<PASSING clause>]
- [RETURNING <type>]
- [{ERROR | NULL | DEFAULT <expr>} ON EMPTY]
- [{ERROR | NULL | DEFAULT <expr>} ON ERROR]
-)
-```
-
-**Return value:** `<type>?`
-
-**Default values:**
-
-1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY`
-2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR`
-3. If the `RETURNING` section isn't specified, then for `<type>`, the type used is `Utf8`
-
-**Behavior:**
-
-1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `<type>?`
-2. If an error occurs, the returned value depends on the `ON ERROR` section:
- - `NULL`: Return an empty `<type>?`
- - `ERROR`: Abort the entire query
- - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the entire query fails, too.
-3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section:
- - `NULL`: Return an empty `<type>?`
- - `ERROR`: Abort the entire query
- - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the behavior matches the `ON ERROR` section.
-4. If the result of JsonPath execution is a single value, then:
- - If the `RETURNING` section isn't specified, the value is converted to `Utf8`.
- - Otherwise, the `CAST` function is run to convert the value to `<type>`. If the `CAST` fails, the behavior matches the `ON ERROR` section. In this case, the value from JSON must match the `<type>` type.
-5. Return the result
-
-Correlation between JSON and YQL types:
-- JSON Number: Numeric types, `Date`, `DateTime`, and `Timestamp`
-- JSON Bool: `Bool`
-- JSON String: `Utf8` and `String`
-
-Errors executing `JSON_VALUE` are as follows:
-
-- Errors evaluating JsonPath
-- The result of JsonPath execution is a number of values or a non-scalar value.
-- The type of value returned by JSON doesn't match the expected one.
-
-`The RETURNING` section supports such types as numbers, `Date`, `DateTime`, `Timestamp`, `Utf8`, `String`, and `Bool`.
-
-**Examples:**
-``` yql
-$json = CAST(@@{
- "friends": [
- {
- "name": "James Holden",
- "age": 35
- },
- {
- "name": "Naomi Nagata",
- "age": 30
- }
- ]
-}@@ as Json);
-
-SELECT
- JSON_VALUE($json, "$.friends[0].age"), -- "35" (type Utf8?)
- JSON_VALUE($json, "$.friends[0].age" RETURNING Uint64), -- 35 (type Uint64?)
- JSON_VALUE($json, "$.friends[0].age" RETURNING Utf8); -- an empty Utf8? due to an error. The JSON's Number type doesn't match the string Utf8 type.
-
-SELECT
- -- "empty" (type String?)
- JSON_VALUE(
- $json,
- "$.friends[50].name"
- RETURNING String
- DEFAULT "empty" ON EMPTY
- );
-
-SELECT
- -- 20 (type Uint64?). The result of JsonPath execution is empty, but the
- -- default value from the ON EMPTY section can't be cast to Uint64.
- -- That's why the value from ON ERROR is used.
- JSON_VALUE(
- $json,
- "$.friends[50].age"
- RETURNING Uint64
- DEFAULT -1 ON EMPTY
- DEFAULT 20 ON ERROR
- );
-
-```
-
-## JSON_QUERY
-
-The `JSON_QUERY` function lets you retrieve arrays and objects from JSON.
-
-**Syntax:**
-``` yql
-JSON_QUERY(
- <JSON expression>,
- <JsonPath query>,
- [<PASSING clause>]
- [WITHOUT [ARRAY] | WITH [CONDITIONAL | UNCONDITIONAL] [ARRAY] WRAPPER]
- [{ERROR | NULL | EMPTY ARRAY | EMPTY OBJECT} ON EMPTY]
- [{ERROR | NULL | EMPTY ARRAY | EMPTY OBJECT} ON ERROR]
-)
-```
-
-**Return value:** `Json?`
-
-**Default values:**
-
-1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY`
-2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR`
-3. If the `WRAPPER` section isn't specified, the section used is `WITHOUT WRAPPER`
-4. If the `WITH WRAPPER` section is specified but `CONDITIONAL` or `UNCONDITIONAL` is omitted, then the section used is `UNCONDITIONAL`
-
-**Behavior:**
-
-{% note info %}
-
-You can't specify the `WITH ... WRAPPER` and `ON EMPTY` sections at the same time.
-
-{% endnote %}
-
-1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Json?`
-2. If the `WRAPPER` section is specified, then:
- - `WITHOUT WRAPPER` or `WITHOUT ARRAY WRAPPER`: Don't convert the result of JsonPath execution in any way.
- - `WITH UNCONDITIONAL WRAPPER` or `WITH UNCONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array.
- - `WITH CONDITIONAL WRAPPER` or `WITH CONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array if it isn't the only array or object.
-3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section:
- - `NULL`: Return an empty `Json?`
- - `ERROR`: Abort the entire query
- - `EMPTY ARRAY`: Return an empty JSON array, `[]`
- - `EMPTY OBJECT`: Return an empty JSON object, `{}`
-4. If an error occurs, the returned value depends on the `ON ERROR` section:
- - `NULL`: Return an empty `Json?`
- - `ERROR`: Abort the entire query
- - `EMPTY ARRAY`: Return an empty JSON array, `[]`
- - `EMPTY OBJECT`: Return an empty JSON object, `{}`
-5. Return the result
-
-Errors running a `JSON_QUERY`:
-
-- Errors evaluating JsonPath
-- The result of JsonPath execution is a number of values (even after using the `WRAPPER` section) or a scalar value.
-
-**Examples:**
-``` yql
-$json = CAST(@@{
- "friends": [
- {
- "name": "James Holden",
- "age": 35
- },
- {
- "name": "Naomi Nagata",
- "age": 30
- }
- ]
-}@@ as Json);
-
-SELECT
- JSON_QUERY($json, "$.friends[0]"); -- {"name": "James Holden", "age": 35}
-
-SELECT
- JSON_QUERY($json, "$.friends.name" WITH UNCONDITIONAL WRAPPER); -- ["James Holden", "Naomi Nagata"]
-
-SELECT
- JSON_QUERY($json, "$.friends[0]" WITH CONDITIONAL WRAPPER), -- {"name": "James Holden", "age": 35}
- JSON_QUERY($json, "$.friends.name" WITH CONDITIONAL WRAPPER); -- ["James Holden", "Naomi Nagata"]
-```
+# Functions for JSON + +**JSON** is a lightweight [data-interchange format](https://www.json.org). In YQL, it's represented by the `Json` type. Unlike relational tables, JSON can store data with no schema defined. Here is an example of a valid JSON object: + +```json +[ + { + "name": "Jim Holden", + "age": 30 + }, + { + "name": "Naomi Nagata", + "age": "twenty years old" + } +] +``` + +Despite the fact that the `age` field in the first object is of the `Number` type (`"age": 21`) and in the second object its type is `String` (`"age": "twenty years old"`), this is a fully valid JSON object. + +To work with JSON, YQL implements a subset of the [SQL support for JavaScript Object Notation (JSON)](https://www.iso.org/standard/67367.html) standard, which is part of the common ANSI SQL standard. + +## JsonPath + +Values inside JSON objects are accessed using a query language called JsonPath. All functions for JSON accept a JsonPath query as an argument. + +Let's look at an example. Suppose we have a JSON object like: + +```json +{ + "comments": [ + { + "id": 123, + "text": "A whisper will do, if it's all that you can manage." + }, + { + "id": 456, + "text": "My life has become a single, ongoing revelation that I haven’t been cynical enough." + } + ] +} +``` + +Then, to get the text of the second comment, we can write the following JsonPath query: + +``` +$.comments[1].text +``` + +In this query: + +1. `$` is a way to access the entire JSON object. +2. `$.comments` accesses the `comments` key of the JSON object. +3. `$.comments[1]` accesses the second element of the JSON array (element numbering starts from 0). +4. `$.comments[1].text` accesses the `text` key of the JSON object. +5. Query execution result: `"My life has become a single, ongoing revelation that I haven’t been cynical enough."` + +### Quick reference + +| Operation | Example | +| --------------------------------------- | ------------------------------------------------- | +| Retrieving a JSON object key | `$.key` | +| Retrieving all JSON object keys | `$.*` | +| Accessing an array element | `$[25]` | +| Retrieving an array subsegment | `$[2 to 5]` | +| Accessing the last array element | `$[last]` | +| Accessing all array elements | `$[*]` | +| Unary operations | `- 1` | +| Binary operations | `(12 * 3) % 4 + 8` | +| Accessing a variable | `$variable` | +| Logical operations | `(1 > 2) || (3 <= 4) && ("string" == "another")` | +| Matching a regular expression | `$.name like_regex "^[A-Za-z]+$"` | +| Checking the string prefix | `$.name starts with "Bobbie"` | +| Checking if a path exists | `exists ($.profile.name)` | +| Checking a Boolean expression for null | `($.age > 20) is unknown` | +| Filtering values | `$.friends ? (@.age >= 18 && @.gender == "male")` | +| Getting the value type | `$.name.type()` | +| Getting the array size | `$.friends.size()` | +| Converting a string to a number | `$.number.double()` | +| Rounding up a number | `$.number.ceiling()` | +| Rounding down a number | `$.number.floor()` | +| Returning the absolute value | `$.number.abs()` | +| Getting key-value pairs from an object | `$.profile.keyvalue()` | + +### Data model + +The result of executing all JsonPath expressions is a sequence of JSON values. For example: + +- The result of executing the `"Bobbie"` expression is a sequence with the only element `"Bobbie"`. Its length is 1. +- The result of executing the `$` expression (that takes the entire JSON object) in JSON `[1, 2, 3]` is `[1, 2, 3]`. A sequence of 1 element of the array `[1, 2, 3]` +- The result of executing the `$[*]` expression (retrieving all array elements) in JSON `[1, 2, 3]` is `1, 2, 3`. A sequence of three items:`1`, `2`, and `3` + +If the input sequence consists of multiple values, some operations are performed for each element (for example, accessing a JSON object key). However, other operations require a sequence of one element as input (for example, binary arithmetic operations). + +The behavior of a specific operation is described in the corresponding section of the documentation. + +### Execution mode + +JsonPath supports two execution modes, `lax` and `strict`. Setting the mode is optional. By default, `lax`. The mode is specified at the beginning of a query. For example, `strict $.key`. + +The behavior for each mode is described in the corresponding sections with JsonPath operations. + +#### Auto unpacking of arrays + +When accessing a JSON object key in `lax` mode, arrays are automatically unpacked. + +**Example:** + +```json +[ + { + "key": 123 + }, + { + "key": 456 + } +] +``` + +The `lax $.key` query is successful and returns `123, 456`. As `$` is an array, it's automatically unpacked and accessing the key of the `$.key` JSON object is executed for each element in the array. + +The `strict $.key` query returns an error. In `strict` mode, there is no support for auto unpacking of arrays. Since `$` is an array and not an object, accessing the `$.key` object key is impossible. You can fix this by writing `strict $[*].key`. + +Unpacking is only 1 level deep. In the event of nested arrays, only the outermost one is unpacked. + +#### Wrapping values in arrays + +When accessing an array element in `lax` mode, JSON values are automatically wrapped in an array. + +**Example:** + +```json +{ + "name": "Avasarala" +} +``` + +The `lax $[0].name` query is successful and returns `"Avasarala"`. As `$` isn't an array, it's automatically wrapped in an array of length 1. Accessing the first element `$[0]` returns the source JSON object where the `name` key is taken. + +The `strict $[0].name` query returns an error. In `strict` mode, values aren't wrapped in an array automatically. Since `$` is an object and not an array, accessing the `$[0]` element is impossible. You can fix this by writing `strict $.name`. + +#### Handling errors + +Some errors are converted to an empty result when a query is executed in `lax` mode. + +### Literals + +Values of some types can be specified in a JsonPath query using literals: + +| Type | Example | +| ------------------ | ---------------- | +| Numbers | `42`, `-1.23e-5` | +| Boolean values | `false`, `true` | +| Null | `Null` | +| Stings | `"Belt"` | + +### Accessing JSON object keys + +JsonPath supports accessing JSON object keys, such as `$.session.user.name`. + +{% note info %} + +Accessing keys without quotes is only supported for keys that start with an English letter or underscore and only contain English letters, underscores, numbers, and a dollar sign. Use quotes for all other keys. For example, `$.profile."this string has spaces"` or `$.user."42 is the answer"` + +{% endnote %} + +For each value from the input sequence: + +1. If the value is an array, it's automatically unpacked in `lax` mode. +2. If the value isn't a JSON object or if it is and the specified key is missing from this JSON object, a query executed in `strict` mode fails. In `lax` mode, an empty result is returned for this value. + +The expression execution result is the concatenation of the results for each value from the input sequence. + +**Example:** + +```json +{ + "name": "Amos", + "friends": [ + { + "name": "Jim" + }, + { + "name": "Alex" + } + ] +} +``` + +| | `lax` | `strict` | +| ---------------- | ---------------- | -------- | +| `$.name` | `"Amos"` | `"Amos"` | +| `$.surname` | Empty result | Error | +| `$.friends.name` | `"Jim", "Alex"` | Error | + +### Accessing all JSON object keys + +JsonPath supports accessing all JSON object keys at once: `$.*`. + +For each value from the input sequence: + +1. If the value is an array, it's automatically unpacked in `lax` mode. +2. If the value isn't a JSON object, a query executed in `strict` mode fails. In `lax` mode, an empty result is returned for this value. + +The expression execution result is the concatenation of the results for each value from the input sequence. + +**Example:** + +```json +{ + "profile": { + "id": 123, + "name": "Amos" + }, + "friends": [ + { + "name": "Jim" + }, + { + "name": "Alex" + } + ] +} +``` + +| | `lax` | `strict` | +| ------------- | --------------- | ------------- | +| `$.profile.*` | `123, "Amos"` | `123, "Amos"` | +| `$.friends.*` | `"Jim", "Alex"` | Error | + +### Accessing an array element + +JsonPath supports accessing array elements: `$.friends[1, 3 to last - 1]`. + +For each value from the input sequence: + +1. If the value isn't an array, a query executed in `strict` mode fails. In `lax` mode, values are automatically wrapped in an array. +2. The `last` keyword is replaced with the array's last index. Using `last` outside of accessing the array is an error in both modes. +3. The specified indexes are calculated. Each of them must be a single number, otherwise the query fails in both modes. +4. If the index is a fractional number, it's rounded down. +5. If the index goes beyond the array boundaries, the query executed in `strict` mode fails. In `lax` mode, this index is ignored. +6. If a segment is specified and its start index is greater than the end index (for example, `$[20 to 1]`), the query fails in `strict` mode. In `lax` mode, this segment is ignored. +7. All elements by the specified indexes are added to the result. Segments include **both ends**. + +**Examples**: + +```json +[ + { + "name": "Camina", + "surname": "Drummer" + }, + { + "name": "Josephus", + "surname": "Miller" + }, + { + "name": "Bobbie", + "surname": "Draper" + }, + { + "name": "Julie", + "surname": "Mao" + } +] +``` + +| | `lax` | `strict` | +| ----------------------------- | ------------------------------- | ------------------------------- | +| `$[0].name` | `"Camina"` | `"Camina"` | +| `$[1, 2 to 3].name` | `"Josephus", "Bobbie", "Julie"` | `"Josephus", "Bobbie", "Julie"` | +| `$[last - 2].name` | `"Josephus"` | `"Josephus"` | +| `$[2, last + 200 to 50].name` | `"Bobbie"` | Error | +| `$[50].name` | Empty result | Error | + +### Accessing all array elements + +JsonPath supports accessing all array elements at once: `$[*]`. + +For each value from the input sequence: + +1. If the value isn't an array, a query executed in `strict` mode fails. In `lax` mode, values are automatically wrapped in an array. +2. All elements of the current array are added to the result. + +**Examples:** + +```json +[ + { + "class": "Station", + "title": "Medina" + }, + { + "class": "Corvette", + "title": "Rocinante" + } +] +``` + +| | `lax` | `strict` | +| ------------------- | ------------------------- | ----------------------- | +| `$[*].title` | `"Medina", "Rocinante"` | `"Medina", "Rocinante"` | +| `lax $[0][*].class` | `"Station"` | Error | + +Let's analyze the last example step by step: + +1. `$[0]` returns the first element of the array, that is `{"class": "Station", "title": "Medina"}` +2. `$[0][*]` expects an array for input, but an object was input instead. It's automatically wrapped in an array as `[ {"class": "Station", "title": "Medina"} ]` +3. Now, `$[0][*]` can be executed and returns all elements of the array, that is `{"class": "Station", "title": "Medina"}` +4. `$[0][*].class` returns the `class` field value: `"Station"`. + +### Arithmetic operations + +{% note info %} + +All arithmetic operations work with numbers as with Double. Calculations are made with potential [loss of accuracy](https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html). + +{% endnote %} + +#### Unary operations + +JsonPath supports unary `+` and `-`. + +A unary operation applies to all values from the input sequence. If a unary operation's input is a value that isn't a number, a query fails in both modes. + +**Example:** + +```json +[1, 2, 3, 4] +``` + +The `strict -$[*]` query is successful and returns `-1, -2, -3, -4`. + +The `lax -$` query fails as `$` is an array and not a number. + +#### Binary operations + +JsonPath supports binary arithmetic operations (in descending order of priority): + +1. Multiplication `*`, dividing floating-point numbers `/`, and taking the remainder `%` (works as the `MOD` function in `SQL`). +2. Addition `+`, subtraction `-`. + +You can change the order of operations using parentheses. + +If each argument of a binary operation is not a single number or a number is divided by 0, the query fails in both modes. + +**Examples:** + +- `(1 + 2) * 3` returns `9` +- `1 / 2` returns `0.5` +- `5 % 2` returns `1` +- `1 / 0` fails +- If JSON is `[-32.4, 5.2]`, the `$[0] % $[1]` query returns `-1.2` +- If JSON is `[1, 2, 3, 4]`, the `lax $[*] + $[*]` query fails as the `$[*]` expression execution result is `1, 2, 3, 4`, that is multiple numbers. A binary operation only requires one number for each of its arguments. + +### Boolean values + +Unlike some other programming languages, Boolean values in JsonPath are not only `true` and `false`, but also `null` (uncertainty). + +JsonPath considers any values received from a JSON document to be non-Boolean. For example, a query like `! $.is_valid_user` (a logical negation applied to the `is_valid_user`) field is syntactically invalid because the `is_valid_user` field value is not Boolean (even when it actually stores `true` or `false`). The correct way to write this kind of query is to explicitly use a comparison with a Boolean value, such as `$.is_valid_user == false`. + +### Logical operations + +JsonPath supports some logical operations for Boolean values. + +The arguments of any logical operation must be a single Boolean value.
+All logical operations return a Boolean value. + +**Logical negation,`!`** + +Truth table: + +| `x` | `!x` | +| ------- | ------- | +| `true` | `false` | +| `false` | `true` | +| `Null` | `Null` | + +**Logical AND, `&&`** + +In the truth table, the first column is the left argument, the first row is the right argument, and each cell is the result of using the Logical AND both with the left and right arguments: + +| `&&` | `true` | `false` | `Null` | +| ------- | ------- | ------- | ------- | +| `true` | `true` | `false` | `Null` | +| `false` | `false` | `false` | `false` | +| `Null` | `Null` | `false` | `Null` | + +**Logical OR, `||`** + +In the truth table, the first column is the left argument, the first row is the right argument, and each cell is the result of using the logical OR with both the left and right arguments: + +| `||` | `true` | `false` | `Null` | +| ------- | ------- | ------- | ------- | +| `true` | `true` | `true` | `true` | +| `false` | `true` | `false` | `Null` | +| `Null` | `true` | `Null` | `Null` | + +**Examples:** + +- `! (true == true)`, the result is `false` +- `(true == true) && (true == false)`, the result is `false` +- `(true == true) || (true == false)`, the result is `true` + +### Comparison operators + +JsonPath implements comparison operators for values: + +- Equality, `==` +- Inequality, `!=`and `<>` +- Less than and less than or equal to, `<` and `=` +- Greater than and greater than or equal to, `>` and `>=` + +All comparison operators return a Boolean value. Both operator arguments support multiple values. + +If an error occurs when calculating the operator arguments, it returns `null`. In this case, the JsonPath query execution continues. + +The arrays of each of the arguments are automatically unpacked. After that, for each pair where the first element is taken from the sequence of the left argument and the second one from the sequence of the right argument: + +1. The elements of the pair are compared +2. If an error occurs during the comparison, the `ERROR` flag is set. +3. If the comparison result is true, the flag set is `FOUND` +4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed. + +If the pair analysis results in: + +1. The `ERROR` flag is set, the operator returns `null` +2. The `FOUND` flag is set, the operator returns `true` +3. Otherwise, it returns `false` + +We can say that this algorithm considers all pairs from the Cartesian product of the left and right arguments, trying to find the pair whose comparison returns true. + +Elements in a pair are compared according to the following rules: + +1. If the left or right argument is an array or object, the comparison fails. +2. `null == null` returns true +3. In all other cases, if one of the arguments is `null`, false is returned. +4. If the left and right arguments are of different types, the comparison fails. +5. Strings are compared byte by byte. +6. `true` is considered greater than `false` +7. Numbers are compared with the accuracy of `1e-20` + +**Example:** + +Let's take a JSON document as an example + +```json +{ + "left": [1, 2], + "right": [4, "Inaros"] +} +``` + +and analyze the steps for executing the `lax $.left < $.right` query: + +1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"` +2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND` +3. Since the query is executed in `lax` mode and the `FOUND` flag is set, we aren't analyzing any more pairs. +4. Since we have the `FOUND` flag set, the operator returns true. + +Let's take the same query in a different execution mode: `strict $.left < $.right`: + +1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"` +2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND` +3. Let's take the pair `(2, 4)`. The comparison is successful as `2 < 4` is true. Set the flag `FOUND` +4. Let's take the pair `(1, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR` +5. Let's take the pair `(2, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR` +6. Since we have the `ERROR` flag set, the operator returns `null` + +### Predicates + +JsonPath supports predicates which are expressions that return a Boolean value and check a certain condition. You can use them, for example, in filters.
+ +#### `like_regex` + +The `like_regex` predicate lets you check if a string matches a regular expression. The syntax of regular expressions is the same as in [Hyperscan UDF](../../udf/list/hyperscan.md) and [REGEXP](../../syntax/expressions.md#regexp). + +**Syntax** + +``` +<expression> like_regex <regexp string> [flag <flag string>] +``` + +Where: + +1. `<expression>` is a JsonPath expression with strings to be checked for matching the regular expression. +2. `<regexp string>` is a string with the regular expression. +3. `flag <flag string>` is an optional section where `<flag string>` is a string with regular expression execution flags. + +Supported flags: + +- `i`: Disable the case sensitivity. + +**Execution** + +Before the check, the input sequence arrays are automatically unpacked. + +After that, for each element of the input sequence: + +1. A check is made to find out if a string matches a regular expression. +2. If the element isn't a string, the `ERROR` flag is set. +3. If the check result is true, the `FOUND` flag is set. +4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed. + +If the pair analysis results in: + +1. Setting the `ERROR` flag, the predicate returns `null` +2. Setting the `FOUND` flag, the predicate returns `true` +3. Otherwise, the predicate returns `false` + +**Examples** + +1. `"123456" like_regex "^[0-9]+$"` returns `true` +2. `"123abcd456" like_regex "^[0-9]+$"` returns `false` +3. `"Naomi Nagata" like_regex "nag"` returns `false` +4. `"Naomi Nagata" like_regex "nag" flag "i"` returns `true` + +#### `starts with` + +The `starts with` predicate lets you check if one string is a prefix of another. + +**Syntax** + +``` +<string expression> starts with <prefix expression> +``` + +Where: + +1. `<string expression>` is a JsonPath expression with the string to check. +2. `<prefix expression>` is a JsonPath expression with a prefix string. + +This means that the predicate will check that the `<string expression>` starts with the `<prefix expression>` string. + +**Execution** + +The first argument of the predicate must be a single string. + +The second argument of the predicate must be a sequence of (possibly, multiple) strings. + +For each element in a sequence of prefix strings: + +1. A check is made for whether "an element is a prefix of an input string" +2. If the element isn't a string, the `ERROR` flag is set. +3. If the check result is true, the `FOUND` flag is set. +4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed. + +If the pair analysis results in: + +1. Setting the `ERROR` flag, the predicate returns `null` +2. Setting the `FOUND` flag, the predicate returns `true` +3. Otherwise, the predicate returns `false` + +**Examples** + +1. `"James Holden" starts with "James"` returns `true` +2. `"James Holden" starts with "Amos"` returns `false` + +#### `exists` + +The `exists` predicate lets you check whether a JsonPath expression returns at least one element. + +**Syntax** + +``` +exists (<expression>) +``` + +Where `<expression>` is the JsonPath expression to be checked. Parentheses around the expression are required. + +**Execution** + +1. The passed JsonPath expression is executed. +2. If an error occurs, the predicate returns `null` +3. If an empty sequence is obtained as a result of the execution, the predicate returns `false` +4. Otherwise, the predicate returns `true` + +**Examples** + +Let's take a JSON document: + +```json +{ + "profile": { + "name": "Josephus", + "surname": "Miller" + } +} +``` + +1. `exists ($.profile.name)` returns `true` +2. `exists ($.friends.profile.name)` returns `false` +3. `strict exists ($.friends.profile.name)` returns `null`, because accessing non-existent object keys in `strict` mode is an error. + +#### `is unknown` + +The `is unknown` predicate lets you check if a Boolean value is `null`. + +**Syntax** + +``` +(<expression>) is unknown +``` + +Where `<expression>` is the JsonPath expression to be checked. Only expressions that return a Boolean value are allowed. Parentheses around the expression are required. + +**Execution** + +1. If the passed expression returns `null`, the predicate returns `true` +2. Otherwise, the predicate returns `false` + +**Examples** + +1. `(1 == 2) is unknown` returns `false`. The `1 == 2` expression returned `false`, which is not `null` +2. `(1 == "string") is unknown` returns `true`. The `1 == "string"` expression returned `null`, because strings and numbers can't be compared in JsonPath. + +### Filters + +JsonPath lets you filter values obtained during query execution. + +An expression in a filter must return a Boolean value.
+Before filtering, the input sequence arrays are automatically unpacked. + +For each element of the input sequence: + +1. The value of the current filtered `@` object becomes equal to the current element of the input sequence. +2. Executing the expression in the filter +3. If an error occurs during the expression execution, the current element of the input sequence is skipped. +4. If the expression execution result is the only `true` value, the current element is added to the filter result. + +**Example:** + +Suppose we have a JSON document describing the user's friends + +```json +{ + "friends": [ + { + "name": "James Holden", + "age": 35, + "money": 500 + }, + { + "name": "Naomi Nagata", + "age": 30, + "money": 345 + } + ] +} +``` + +and we want to get a list of friends who are over 32 years old using a JsonPath query. To do this, you can write the following query: + +``` +$.friends ? (@.age > 32) +``` + +Let's analyze the query in parts: + +- `$.friends` accesses the `friends` array in the JSON document. +- `? ( ... )` is the filter syntax. An expression inside the parentheses is called a predicate. +- `` accesses the currently filtered object. In our example, it's the object describing a friend of the user. +- `.age` accesses the `age` field of the currently filtered object. +- `.age > 32` compares the `age` field with the value 32. As a result of the query, only the values for which this predicate returned true remain. + +The query only returns the first friend from the array of user's friends. + +Like many other JsonPath operators, filters can be arranged in chains. Let's take a more complex query that selects the names of friends who are older than 20 and have less than 400 currency units: + +``` +$.friends ? (@.age > 20) ? (@.money < 400) . name +``` + +Let's analyze the query in parts: + +- `$.friends` accesses the `friends` array in the JSON document. +- `? (@.age > 20)` is the first filter. Since all friends are over 20, it just returns all the elements of the `friends` array. +- `? (@.money < 400)` is the second filter. It only returns the second element of the `friends` array, since only its `money` field value is less than 400. +- `.name` accesses the `name` field of filtered objects. + +The query returns a sequence of a single element: `"Naomi Nagata"`. + +In practice, it's recommended to combine multiple filters into one if possible. The above query is equivalent to `$.friends ? (@.age > 20 && @.money < 400) . name`. + +### Methods + +JsonPath supports methods that are functions converting one sequence of values to another. The syntax for calling a method is similar to accessing the object key: + +``` +$.friends.size() +``` + +Just like in the case of accessing object keys, method calls can be arranged in chains: + +``` +$.numbers.double().floor() +``` + +#### `type` + +The `type` method returns a string with the type of the passed value. + +For each element of the input sequence, the method adds this string to the output sequence according to the table below: + +| Value type | String with type | +| ------------------ | ------------------------ | +| Null | `"null"` | +| Boolean value | `"boolean"` | +| Number | `"number"` | +| String | `"string"` | +| Array | `"array"` | +| Object | `"object"` | + +**Examples** + +1. `"Naomi".type()` returns `"string"` +2. `false.type()` returns `"boolean"` + +#### `size` + +The `size` method returns the size of the array. + +For each element of the input sequence, the method adds the following to the output sequence: + +1. The size of the array if the element type is an array. +2. For all other types (including objects), it adds `1` + +**Examples** + +Let's take a JSON document: + +```json +{ + "array": [1, 2, 3], + "object": { + "a": 1, + "b": 2 + }, + "scalar": "string" +} +``` + +And queries to it: + +1. `$.array.size()` returns `3` +2. `$.object.size()` returns `1` +3. `$.scalar.size()` returns `1` + +#### `Double` + +The `double` method converts strings to numbers. + +Before its execution, the input sequence arrays are automatically unpacked. + +All elements in the input sequence must be strings that contain decimal numbers. It's allowed to specify the fractional part and exponent. + +**Examples** + +1. `"125".double()` returns `125` +2. `"125.456".double()` returns `125.456` +3. `"125.456e-3".double()` returns `0.125456` + +#### `ceiling` + +The `ceiling` method rounds up a number. + +Before its execution, the input sequence arrays are automatically unpacked. + +All elements in the input sequence must be numbers. + +**Examples** + +1. `(1.3).ceiling()` returns `2` +2. `(1.8).ceiling()` returns `2` +3. `(1.5).ceiling()` returns `2` +4. `(1.0).ceiling()` returns `1` + +#### `floor` + +The `floor` method rounds down a number. + +Before its execution, the input sequence arrays are automatically unpacked. + +All elements in the input sequence must be numbers. + +**Examples** + +1. `(1.3).floor()` returns `1` +2. `(1.8).floor()` returns `1` +3. `(1.5).floor()` returns `1` +4. `(1.0).floor()` returns `1` + +#### `abs` + +The `abs` method calculates the absolute value of a number (removes the sign). + +Before its execution, the input sequence arrays are automatically unpacked. + +All elements in the input sequence must be numbers. + +**Examples** + +1. `(0.0).abs()` returns `0` +2. `(1.0).abs()` returns `1` +3. `(-1.0).abs()` returns `1` + +#### `keyvalue` + +The `keyvalue` method converts an object to a sequence of key-value pairs. + +Before its execution, the input sequence arrays are automatically unpacked. + +All elements in the input sequence must be objects. + +For each element of the input sequence: + +1. Each key-value pair in the element is analyzed. +2. For each key-value pair, an object is generated with the keys `name` and `value`. +3. `name` stores a string with the name of the key from the pair. +4. `value` stores the value from the pair. +5. All objects for this element are added to the output sequence. + +**Examples** + +Let's take a JSON document: + +```json +{ + "name": "Chrisjen", + "surname": "Avasarala", + "age": 70 +} +``` + +The `$.keyvalue()` query returns the following sequence for it: + +```json +{ + "name": "age", + "value": 70 +}, +{ + "name": "name", + "value": "Chrisjen" +}, +{ + "name": "surname", + "value": "Avasarala" +} +``` + +### Variables + +Functions using JsonPath can pass values into a query. They are called variables. To access a variable, write the `$` character and the variable name: `$variable`. + +**Example:** + +Let the `planet` variable be equal to + +```json +{ + "name": "Mars", + "gravity": 0.376 +} +``` + +Then the `strict $planet.name` query returns `"Mars"`. + +Unlike many programming languages, JsonPath doesn't support creating new variables or modifying existing ones. + +## Common arguments + +All functions for JSON accept: + +1. A JSON value (can be an arbitrary `Json` or `Json?` expression) +2. A JsonPath query (must be explicitly specified with a string literal) +3. **(Optional)** `PASSING` section + +### PASSING section + +Lets you pass values to a JsonPath query as variables. + +**Syntax:** + +```yql +PASSING + <expression 1> AS <variable name 1>, + <expression 2> AS <variable name 2>, + ... +``` + +`<expression>` can have the following types: + +- Numbers, `Date`, `DateTime`, and `Timestamp` (a `CAST` into `Double` will be made before passing a value to JsonPath) +- `Utf8`, `Bool`, and `Json` + +You can set a `<variable name>` in several ways: + +- As an SQL name like `variable` +- In quotes, for example, `"variable"` + +**Example:** + +```yql +JSON_VALUE( + $json, + "$.timestamp - $Now + $Hour" + PASSING + 24 * 60 as Hour, + CurrentUtcTimestamp() as "Now" +) +``` + +## JSON_EXISTS + +The `JSON_EXISTS` function checks if a JSON value meets the specified JsonPath. + +**Syntax:** + +```yql +JSON_EXISTS( + <JSON expression>, + <JsonPath query>, + [<PASSING clause>] + [{TRUE | FALSE | UNKNOWN | ERROR} ON ERROR] +) +``` + +**Return value:** `Bool?` + +**Default value:** If the `ON ERROR` section isn't specified, the used section is `FALSE ON ERROR` + +**Behavior:** + +1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Bool?` +2. If an error occurs during JsonPath execution, the returned value depends on the `ON ERROR` section: + - `TRUE`: Return `True` + - `FALSE`: Return `False` + - `UNKNOWN`: Return an empty `Bool?` + - `ERROR`: Abort the entire query +3. If the result of JsonPath execution is one or more values, the return value is `True`. +4. Otherwise, `False` is returned. + +**Examples:** + +```yql +$json = CAST(@@{ + "title": "Rocinante", + "crew": [ + "James Holden", + "Naomi Nagata", + "Alex Kamai", + "Amos Burton" + ] +}@@ as Json); + +SELECT + JSON_EXISTS($json, "$.title"), -- True + JSON_EXISTS($json, "$.crew[*]"), -- True + JSON_EXISTS($json, "$.nonexistent"); -- False, as JsonPath returns an empty result. + +SELECT + -- JsonPath error, False is returned because the default section used is FALSE ON ERROR. + JSON_EXISTS($json, "strict $.nonexistent"); + +SELECT + -- JsonPath error, the entire YQL query fails. + JSON_EXISTS($json, "strict $.nonexistent" ERROR ON ERROR); +``` + +## JSON_VALUE + +The `JSON_VALUE` function retrieves a scalar value from JSON (anything that isn't an array or object). + +**Syntax:** + +```yql +JSON_VALUE( + <JSON expression>, + <JsonPath query>, + [<PASSING clause>] + [RETURNING <type>] + [{ERROR | NULL | DEFAULT <expr>} ON EMPTY] + [{ERROR | NULL | DEFAULT <expr>} ON ERROR] +) +``` + +**Return value:** `<type>?` + +**Default values:** + +1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY` +2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR` +3. If the `RETURNING` section isn't specified, then for `<type>`, the type used is `Utf8` + +**Behavior:** + +1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `<type>?` +2. If an error occurs, the returned value depends on the `ON ERROR` section: + - `NULL`: Return an empty `<type>?` + - `ERROR`: Abort the entire query + - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the entire query fails, too. +3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section: + - `NULL`: Return an empty `<type>?` + - `ERROR`: Abort the entire query + - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the behavior matches the `ON ERROR` section. +4. If the result of JsonPath execution is a single value, then: + - If the `RETURNING` section isn't specified, the value is converted to `Utf8`. + - Otherwise, the `CAST` function is run to convert the value to `<type>`. If the `CAST` fails, the behavior matches the `ON ERROR` section. In this case, the value from JSON must match the `<type>` type. +5. Return the result + +Correlation between JSON and YQL types: + +- JSON Number: Numeric types, `Date`, `DateTime`, and `Timestamp` +- JSON Bool: `Bool` +- JSON String: `Utf8` and `String` + +Errors executing `JSON_VALUE` are as follows: + +- Errors evaluating JsonPath +- The result of JsonPath execution is a number of values or a non-scalar value. +- The type of value returned by JSON doesn't match the expected one. + +`The RETURNING` section supports such types as numbers, `Date`, `DateTime`, `Timestamp`, `Utf8`, `String`, and `Bool`. + +**Examples:** + +```yql +$json = CAST(@@{ + "friends": [ + { + "name": "James Holden", + "age": 35 + }, + { + "name": "Naomi Nagata", + "age": 30 + } + ] +}@@ as Json); + +SELECT + JSON_VALUE($json, "$.friends[0].age"), -- "35" (type Utf8?) + JSON_VALUE($json, "$.friends[0].age" RETURNING Uint64), -- 35 (type Uint64?) + JSON_VALUE($json, "$.friends[0].age" RETURNING Utf8); -- an empty Utf8? due to an error. The JSON's Number type doesn't match the string Utf8 type. + +SELECT + -- "empty" (type String?) + JSON_VALUE( + $json, + "$.friends[50].name" + RETURNING String + DEFAULT "empty" ON EMPTY + ); + +SELECT + -- 20 (type Uint64?). The result of JsonPath execution is empty, but the + -- default value from the ON EMPTY section can't be cast to Uint64. + -- That's why the value from ON ERROR is used. + JSON_VALUE( + $json, + "$.friends[50].age" + RETURNING Uint64 + DEFAULT -1 ON EMPTY + DEFAULT 20 ON ERROR + ); +``` + +## JSON_QUERY + +The `JSON_QUERY` function lets you retrieve arrays and objects from JSON. + +**Syntax:** + +```yql +JSON_QUERY( + <JSON expression>, + <JsonPath query>, + [<PASSING clause>] + [WITHOUT [ARRAY] | WITH [CONDITIONAL | UNCONDITIONAL] [ARRAY] WRAPPER] + [{ERROR | NULL | EMPTY ARRAY | EMPTY OBJECT} ON EMPTY] + [{ERROR | NULL | EMPTY ARRAY | EMPTY OBJECT} ON ERROR] +) +``` + +**Return value:** `Json?` + +**Default values:** + +1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY` +2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR` +3. If the `WRAPPER` section isn't specified, the section used is `WITHOUT WRAPPER` +4. If the `WITH WRAPPER` section is specified but `CONDITIONAL` or `UNCONDITIONAL` is omitted, then the section used is `UNCONDITIONAL` + +**Behavior:** + +{% note info %} + +You can't specify the `WITH ... WRAPPER` and `ON EMPTY` sections at the same time. + +{% endnote %} + +1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Json?` +2. If the `WRAPPER` section is specified, then: + - `WITHOUT WRAPPER` or `WITHOUT ARRAY WRAPPER`: Don't convert the result of JsonPath execution in any way. + - `WITH UNCONDITIONAL WRAPPER` or `WITH UNCONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array. + - `WITH CONDITIONAL WRAPPER` or `WITH CONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array if it isn't the only array or object. +3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section: + - `NULL`: Return an empty `Json?` + - `ERROR`: Abort the entire query + - `EMPTY ARRAY`: Return an empty JSON array, `[]` + - `EMPTY OBJECT`: Return an empty JSON object, `{}` +4. If an error occurs, the returned value depends on the `ON ERROR` section: + - `NULL`: Return an empty `Json?` + - `ERROR`: Abort the entire query + - `EMPTY ARRAY`: Return an empty JSON array, `[]` + - `EMPTY OBJECT`: Return an empty JSON object, `{}` +5. Return the result + +Errors running a `JSON_QUERY`: + +- Errors evaluating JsonPath +- The result of JsonPath execution is a number of values (even after using the `WRAPPER` section) or a scalar value. + +**Examples:** + +```yql +$json = CAST(@@{ + "friends": [ + { + "name": "James Holden", + "age": 35 + }, + { + "name": "Naomi Nagata", + "age": 30 + } + ] +}@@ as Json); + +SELECT + JSON_QUERY($json, "$.friends[0]"); -- {"name": "James Holden", "age": 35} + +SELECT + JSON_QUERY($json, "$.friends.name" WITH UNCONDITIONAL WRAPPER); -- ["James Holden", "Naomi Nagata"] + +SELECT + JSON_QUERY($json, "$.friends[0]" WITH CONDITIONAL WRAPPER), -- {"name": "James Holden", "age": 35} + JSON_QUERY($json, "$.friends.name" WITH CONDITIONAL WRAPPER); -- ["James Holden", "Naomi Nagata"] +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md index b3e2b6982c..555302c562 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md @@ -2,25 +2,27 @@ ## ListCreate {#list-create} -Construct an empty list. The only argument specifies a string describing the data type of the list cell or the type itself obtained using [relevant functions](../types.md). YQL doesn't support lists with an unknown cell type. +Construct an empty list. The only argument specifies a string describing the data type of the list cell, or the type itself obtained using [relevant functions](../types.md). YQL doesn't support lists with an unknown cell type. [Documentation for the type definition format](../../types/type_string.md). **Examples** -``` yql + +```yql SELECT ListCreate(Tuple<String,Double?>); ``` -``` yql +```yql SELECT ListCreate(OptionalType(DataType("String"))); ``` -## AsList and AsListStrict {#aslist} +## asList and AsListStrict {#aslist} Construct a list based on one or more arguments. The argument types must be compatible in the case of `AsList` and strictly match in the case of `AsListStrict`. **Examples** -``` yql + +```yql SELECT AsList(1, 2, 3, 4, 5); ``` @@ -30,48 +32,59 @@ The count of items in the list. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListLength(list_column) FROM my_table; ``` + {% endif %} + ## ListHasItems Check that the list contains at least one item. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListHasItems(list_column) FROM my_table; ``` + {% endif %} ## ListCollect {#listcollect} -Convert a lazy list (which can be built by such functions as [ListFilter](#listfilter), [ListMap](#listmap), and [ListFlatMap](#listflatmap)) to an eager list. In contrast to a lazy list, where each new pass re-calculates the list contents, in an eager list the content is built at once by consuming more memory. +Convert a lazy list (it can be built by such functions as [ListFilter](#listfilter), [ListMap](#listmap), [ListFlatMap](#listflatmap)) to an eager list. In contrast to a lazy list, where each new pass re-calculates the list contents, in an eager list the content is built at once by consuming more memory. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListCollect(list_column) FROM my_table; ``` + {% endif %} -## ListSort, ListSortAsc and ListSortDesc {#listsort} + +## ListSort, ListSortAsc, and ListSortDesc {#listsort} Sort the list. By default, the ascending sorting order is applied (`ListSort` is an alias for `ListSortAsc`). Arguments: 1. List. -2. An optional expression to get the sort key from a list element (the element itself by default). +2. An optional expression to get the sort key from a list element (it's the element itself by default). **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListSortDesc(list_column) FROM my_table; ``` + {% endif %} -``` yql + +```yql $list = AsList( AsTuple("x", 3), AsTuple("xx", 1), @@ -92,19 +105,21 @@ The example used a [lambda function](../../syntax/expressions.md#lambda). ## ListExtend and ListExtendStrict {#listextend} Sequentially join lists (concatenation of lists). The arguments can be lists, optional lists, and `NULL`. -The types of list elements must be compatible in the case of `ListExtend` and strictly match in the case of `ListExtendStrict`. +The types of list items must be compatible in the case of `ListExtend` and strictly match in the case of `ListExtendStrict`. If at least one of the lists is optional, then the result is also optional. If at least one argument is `NULL`, then the result type is `NULL`. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListExtend( list_column_1, list_column_2, list_column_3 ) FROM my_table; ``` + {% endif %} ## ListUnionAll {#listunionall} @@ -115,30 +130,34 @@ If at least one of the lists is optional, then the result is also optional. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListUnionAll( list_column_1, list_column_2, list_column_3 ) FROM my_table; ``` + {% endif %} ## ListZip and ListZipAll {#listzip} -Based on the input lists, build a list of pairs containing the list elements with matching indexes (`List<Tuplefirst_list_element_type,second_list_element_type>`). +Based on the input lists, build a list of pairs containing the list items with matching indexes (`List<Tuplefirst_list_element_type,second_list_element_type>`). -The length of the returned list is determined by the shortest list for ListZip and the longest list for ListZipAll. +The length of the returned list is determined by the shortest list for ListZip and the longest list for ListZipAll. When the shorter list is exhausted, a `NULL` value of a relevant [optional type](../../types/optional.md) is paired with the elements of the longer list. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListZip(list_column_1, list_column_2, list_column_3), ListZipAll(list_column_1, list_column_2) FROM my_table; ``` + {% endif %} ## ListEnumerate {#listenumerate} @@ -147,9 +166,11 @@ Build a list of pairs (Tuple) containing the element number and the element itse **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListEnumerate(list_column) FROM my_table; ``` + {% endif %} ## ListReverse {#listreverse} @@ -158,9 +179,11 @@ Reverse the list. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListReverse(list_column) FROM my_table; ``` + {% endif %} ## ListSkip {#listskip} @@ -171,11 +194,13 @@ The first argument specifies the source list and the second argument specifies h **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListSkip(list_column, 3) FROM my_table; ``` + {% endif %} ## ListTake {#listtake} @@ -186,9 +211,11 @@ The first argument specifies the source list and the second argument specifies t **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListTake(list_column, 3) FROM my_table; ``` + {% endif %} ## ListIndexOf {#listindexof} @@ -197,14 +224,16 @@ Searches the list for an element with the specified value and returns its index **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListIndexOf(list_column, 123) FROM my_table; ``` + {% endif %} -## ListMap, ListFlatMap and ListFilter {#listmap} +## ListMap, ListFilter, and ListFlatMap {#listmap} Apply the function specified as the second argument to each list element. The functions differ in their returned result: @@ -223,22 +252,24 @@ Arguments: 1. Source list. 2. Functions for processing list elements, such as: * [Lambda function](../../syntax/expressions.md#lambda). - * `Module::Function` - C++ UDF. -{% if feature_udf_noncpp %} - * [Python UDF](../../udf/python.md), [JavaScript UDF](../../udf/javascript.md) or any other callable value. + * `Module::Function` - C++ UDF; +{% if feature_udf_noncpp %} + * [Python UDF](../../udf/python.md), [JavaScript UDF](../../udf/javascript.md) or any other called value. If the source list is optional, then the output list is also optional. **Examples** {% if feature_column_container_type %} -``` yql -$callable = Python::test(Callable<(Int64)->Bool>, "def test(i): return i % 2"); + +```yql +$callable = Python::test(Callable<(Int64)->Bool>, "defMyFavouriteCrutchtest(i): return i % 2"); SELECT ListMap(list_column, ($x) -> { RETURN $x > 2; }), ListFlatMap(list_column, My::Udf), ListFilter(list_column, $callable) FROM my_table; ``` + {% endif %} {% endif %} @@ -249,19 +280,21 @@ Applies transformation to the source list, skipping empty optional items and str If the source list is optional, then the output list is also optional. **Examples** -``` yql + +```yql SELECT ListNotNull([1,2]), -- [1,2] ListNotNull([3,null,4]); -- [3,4] ``` ## ListFlatten {#listflatten} -Expands the list of lists into a flat list, preserving the order of items. As the top-level list item, you can use an optional list that is interpreted as an empty list in the case of `NULL`. +Expands the list of lists into a flat list, preserving the order of items. As the top-level list item you can use an optional list that is interpreted as an empty list in the case of `NULL`. If the source list is optional, then the output list is also optional. **Examples** -``` yql + +```yql SELECT ListFlatten([[1,2],[3,4]]), -- [1,2,3,4] ListFlatten([null,[3,4],[5,6]]); -- [3,4,5,6] ``` @@ -272,16 +305,18 @@ Returns a copy of the list containing only distinct elements. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListUniq(list_column) FROM my_table; ``` + {% endif %} ## ListAny and ListAll {#listany} -Returns `true` for a list of Boolean values if: +Returns `true` for a list of Boolean values, if: * `ListAny`: At least one element is `true`. * `ListAll`: All elements are `true`. @@ -290,12 +325,14 @@ Otherwise, it returns false. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListAll(bool_column), ListAny(bool_column) FROM my_table; ``` + {% endif %} ## ListHas {#listhas} @@ -304,11 +341,13 @@ Show whether the list contains the specified element. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListHas(list_column, "my_needle") FROM my_table; ``` + {% endif %} ## ListHead, ListLast {#listheadlast} @@ -317,12 +356,14 @@ Returns the first and last item of the list. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListHead(numeric_list_column) AS head, ListLast(numeric_list_column) AS last FROM my_table; ``` + {% endif %} ## ListMin, ListMax, ListSum and ListAvg {#listminy} @@ -331,7 +372,8 @@ Apply the appropriate aggregate function to all elements of the numeric list. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListMax(numeric_list_column) AS max, ListMin(numeric_list_column) AS min, @@ -339,11 +381,12 @@ SELECT ListAvg(numeric_list_column) AS avg FROM my_table; ``` + {% endif %} ## ListFromRange {#listfromrange} -Generate a sequence of numbers with the specified step. It's similar to `xrange` in Python 2, but additionally supports floating points. +Generate a sequence of numbers with the specified step. It's similar to `xrange` in Python 2, but additionally supports floats. Arguments: @@ -355,13 +398,14 @@ Specifics: * The end is not included, i.e. `ListFromRange(1,3) == AsList(1,2)`. * The type for the resulting elements is selected as the broadest from the argument types. For example, `ListFromRange(1, 2, 0.5)` results in a `Double` list. -* The list is "lazy", but if used improperly, it can still consume lots of RAM. +* The list is "lazy", but if it's used incorrectly, it can still consume a lot of RAM. * If the step is positive and the end is less than or equal to the start, the result list is empty. * If the step is negative and the end is greater than or equal to the start, the result list is empty. * If the step is neither positive nor negative (0 or NaN), the result list is empty. **Examples** -``` yql + +```yql SELECT ListFromRange(-2, 2), -- [-2, -1, 0, 1] ListFromRange(2, 1, -0.5); -- [2.0, 1.5] @@ -377,7 +421,8 @@ Required arguments: 2. Number of copies. **Examples** -``` yql + +```yql SELECT ListReplicate(true, 3); -- [true, true, true] ``` @@ -388,12 +433,14 @@ You can set a separator as the second parameter. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListConcat(string_list_column), ListConcat(string_list_column, "; ") FROM my_table; ``` + {% endif %} ## ListExtract {#listextract} @@ -402,19 +449,21 @@ For a list of structures, it returns a list of contained fields having the speci **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ListExtract(struct_list_column, "MyMember") FROM my_table; ``` + {% endif %} ## ListTakeWhile, ListSkipWhile {#listtakewhile} `ListTakeWhile` returns a list from the beginning while the predicate is true, then the list ends. -`ListSkipWhile` skips the list segment from the beginning while the predicate is true, then returns the rest of the list disregarding the predicate. -`ListTakeWhileInclusive` returns a list from the beginning while the predicate is true. Then the list ends, but it also includes the element that triggered the stopping predicate. +`ListSkipWhile` skips the list segment from the beginning while the predicate is true, then returns the rest of the list ignoring the predicate. +`ListTakeWhileInclusive` returns a list from the beginning while the predicate is true. Then the list ends, but it also includes the item on which the stopping predicate triggered. `ListSkipWhileInclusive` skips a list segment from the beginning while the predicate is true, then returns the rest of the list disregarding the predicate, but excluding the element that matched the predicate and starting with the next element after it. Required arguments: @@ -425,7 +474,8 @@ Required arguments: If the input list is optional, then the result is also optional. **Examples** -``` yql + +```yql $data = AsList(1, 2, 5, 1, 2, 7); SELECT @@ -447,41 +497,48 @@ Arguments: 2. [Aggregation factory](../basic.md#aggregationfactory). **Examples** -``` yql + +```yql SELECT ListAggregate(AsList(1, 2, 3), AggregationFactory("Sum")); -- 6 ``` ## ToDict and ToMultiDict {#todict} -Convert a list of tuples containing key-value pairs to a dictionary. If there are conflicting keys in the input list, `ToDict` leaves the first value and `ToMultiDict` builds a list of all the values. +Convert a list of tuples containing key-value pairs to a dictionary. In case of conflicting keys in the input list, `ToDict` leaves the first value and `ToMultiDict` builds a list of all the values. It means that: -* `ToDict` converts `List<TupleK, V>` to `Dict<K, V>` -* `ToMultiDict` converts `List<TupleK, V>` to `Dict<K, ListV>` +* `ToDict` converts `List<TupleK, V="">` to `Dict<K, V="">` +* `ToMultiDict` converts `List<TupleK, V>` to `Dict<K, List<V>>` Optional lists are also supported, resulting in an optional dictionary. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ToDict(tuple_list_column) FROM my_table; ``` + {% endif %} ## ToSet {#toset} -Converts a list to a dictionary where the keys are unique elements of this list, and values are omitted and have the type `Void`. For the `List<T>` list, the result type is `Dict<T, Void>`. + +Converts a list to a dictionary where the keys are unique elements of this list, and values are omitted and have the type `Void`. For the `List<T>` list, the result type is `Dict<T, Void="">`. An optional list is also supported, resulting in an optional dictionary. Inverse function: get a list of keys for the [DictKeys](../dict.md#dictkeys) dictionary. **Examples** {% if feature_column_container_type %} -``` yql + +```yql SELECT ToSet(list_column) FROM my_table; ``` + {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md index 0314eb64f2..1b824272b7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md @@ -10,7 +10,7 @@ Arguments: 2. Field name. 3. Default value. -``` yql +```yql $struct = <|a:1|>; SELECT TryMember( @@ -37,7 +37,8 @@ Arguments: * All the other arguments must be named, each argument adds a new field and the argument's name is used as the field's name (as in [AsStruct](../basic.md#asstruct)). **Examples** -``` yql + +```yql $struct = <|a:1|>; SELECT ExpandStruct( @@ -60,7 +61,8 @@ Arguments: 3. Value of the new field. **Examples** -``` yql + +```yql $struct = <|a:1|>; SELECT AddMember( @@ -82,7 +84,8 @@ Arguments: 2. Field name. **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT RemoveMember( @@ -103,7 +106,8 @@ Arguments: 2. Field name. **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT ForceRemoveMember( @@ -124,7 +128,8 @@ Arguments: 2. List of field names. **Examples** -``` yql + +```yql $struct = <|a:1, b:2, c:3|>; SELECT ChooseMembers( @@ -145,7 +150,8 @@ Arguments: 2. List of field names. **Examples** -``` yql + +```yql $struct = <|a:1, b:2, c:3|>; SELECT RemoveMembers( @@ -166,7 +172,8 @@ Arguments: 2. List of field names. **Examples** -``` yql + +```yql $struct = <|a:1, b:2, c:3|>; SELECT ForceRemoveMembers( @@ -184,7 +191,8 @@ If the resulting field set contains duplicate values, an error is returned. Arguments: two or more structures. **Examples** -``` yql + +```yql $struct1 = <|a:1, b:2|>; $struct2 = <|c:3|>; SELECT @@ -200,10 +208,11 @@ Combining the fields from multiple new structures into another new structure wit If the resulting field set contains duplicate values, an error is returned. -Arguments: two or more tuples of two elements: prefix and structure. +Arguments: two or more tuples of two items: prefix and structure. **Examples** -``` yql + +```yql $struct1 = <|a:1, b:2|>; $struct2 = <|c:3|>; SELECT @@ -220,7 +229,8 @@ Returns an unordered list of field names (possibly removing one Optional level) Argument: structure **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT StructMembers($struct); -- ['a', 'b'] @@ -236,7 +246,8 @@ Arguments: 2. A tuple of field names: the original name, the new name. **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT RenameMembers($struct, [('a', 'c'), ('a', 'e')]); -- (b:2, c:1, e:1) @@ -252,7 +263,8 @@ Arguments: 2. A tuple of field names: the original name, the new name. **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT ForceRenameMembers($struct, [('a', 'c'), ('d', 'e')]); -- (b:2, c:1) @@ -260,12 +272,13 @@ SELECT ## GatherMembers {#gathermembers} -Returns an unordered list of tuples including the field name and value. For the `NULL` argument, `EmptyList` is returned. It can be used only in the cases when the types of elements in the structure are the same or compatible. Returns an optional list for an optional structure. +Returns an unordered list of tuples including the field name and value. For the `NULL` argument, `EmptyList` is returned. It can be used only in the cases when the types of items in the structure are the same or compatible. Returns an optional list for an optional structure. Argument: structure **Examples** -``` yql + +```yql $struct = <|a:1, b:2|>; SELECT GatherMembers($struct); -- [('a', 1), ('b', 2)] @@ -273,7 +286,7 @@ SELECT ## SpreadMembers {#spreadmembers} -Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of edits includes a field that is not in the list of expected fields, an error is returned. +Creates a structure with a specified list of fields and applies a specified list of edits to it in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of edits includes a field that is not in the list of expected fields, an error is returned. Arguments: @@ -281,7 +294,8 @@ Arguments: 2. A list of all possible field names in the structure. **Examples** -``` yql + +```yql SELECT SpreadMembers([('a',1),('a',2)],['a','b']); -- (a: 2, b: null) @@ -289,7 +303,7 @@ SELECT ## ForceSpreadMembers {#forcespreadmembers} -Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of updates includes a field that is not in the list of expected fields, this edit is ignored. +Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is optional or equal to `NULL`, the result has the same type. If the list of updates includes a field that is not in the list of expected fields, this edit is ignored. Arguments: @@ -297,8 +311,10 @@ Arguments: 2. A list of all possible field names in the structure. **Examples** -``` yql + +```yql SELECT ForceSpreadMembers([('a',1),('a',2),('c',100)],['a','b']); -- (a: 2, b: null) ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md index a5659b7963..28c7b6f56d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md @@ -9,7 +9,8 @@ Serializing a type {% if feature_codegen %} or a handle type{% endif %} to a hum Building a type from a string with description. [Documentation for its format](../../types/type_string.md). **Examples** -``` yql + +```yql SELECT FormatType(ParseType("List<Int32>")); -- List<int32> ``` @@ -18,10 +19,12 @@ SELECT FormatType(ParseType("List<Int32>")); -- List<int32> Getting the type of value passed to the argument. **Examples** -``` yql + +```yql SELECT FormatType(TypeOf("foo")); -- String ``` -``` yql + +```yql SELECT FormatType(TypeOf(AsTuple(1, 1u))); -- Tuple<Int32,Uint32> ``` @@ -32,7 +35,8 @@ Returns an instance of the specified type that can only be used to get the type If this instance remains in the computation graph by the end of optimization, the operation fails. **Examples** -``` yql + +```yql SELECT FormatType(TypeOf( InstanceOf(ParseType("Int32")) + InstanceOf(ParseType("Double")) @@ -44,7 +48,8 @@ SELECT FormatType(TypeOf( Returns a type for [primitive data types](../../types/primitive.md) based on type name. **Examples** -``` yql + +```yql SELECT FormatType(DataType("Bool")); -- Bool SELECT FormatType(DataType("Decimal","5","1")); -- Decimal(5,1) ``` @@ -54,7 +59,8 @@ SELECT FormatType(DataType("Decimal","5","1")); -- Decimal(5,1) Adds the option to assign `NULL` to the passed type. **Examples** -``` yql + +```yql SELECT FormatType(OptionalType(DataType("Bool"))); -- Bool? ``` @@ -63,7 +69,8 @@ SELECT FormatType(OptionalType(DataType("Bool"))); -- Bool? Builds a list type or stream type based on the passed element type. **Examples** -``` yql + +```yql SELECT FormatType(ListType(DataType("Bool"))); -- List<Bool> ``` @@ -72,7 +79,8 @@ SELECT FormatType(ListType(DataType("Bool"))); -- List<Bool> Builds a dictionary type based on the passed key types (first argument) and value types (second argument). **Examples** -``` yql + +```yql SELECT FormatType(DictType( DataType("String"), DataType("Double") @@ -84,7 +92,8 @@ SELECT FormatType(DictType( Builds the tuple type from the passed element types. **Examples** -``` yql + +```yql SELECT FormatType(TupleType( DataType("String"), DataType("Double"), @@ -97,7 +106,8 @@ SELECT FormatType(TupleType( Builds the structure type based on the passed element types. The standard syntax of named arguments is used to specify the element names. **Examples** -``` yql + +```yql SELECT FormatType(StructType( DataType("Bool") AS MyBool, ListType(DataType("String")) AS StringList @@ -109,7 +119,8 @@ SELECT FormatType(StructType( Returns the type of a variant based on the underlying type (structure or tuple). **Examples** -``` yql + +```yql SELECT FormatType(VariantType( ParseType("Struct<foo:Int32,bar:Double>") )); -- Variant<'bar':Double,'foo':Int32> @@ -120,7 +131,8 @@ SELECT FormatType(VariantType( Returns the type of the [resource](../../types/special.md) based on the passed string label. **Examples** -``` yql + +```yql SELECT FormatType(ResourceType("Foo")); -- Resource<'Foo'> ``` @@ -133,7 +145,8 @@ Constructs the type of the called value using the following arguments: 3. All the next arguments of CallableType are treated as types of arguments of the callable value, but with a shift for two required arguments (for example, the third argument of the CallableType describes the type of the first argument in the callable value). **Examples** -``` yql + +```yql SELECT FormatType(CallableType( 1, -- optional args count DataType("Double"), -- result type @@ -142,34 +155,38 @@ SELECT FormatType(CallableType( )); -- Callable<(String,[Int64?])->Double> ``` -## GenericType, UnitType and VoidType {#generictype} +## GenericType, UnitType, and VoidType {#generictype} Return the same-name [special data types](../../types/special.md). They have no arguments because they are not parameterized. **Examples** -``` yql + +```yql SELECT FormatType(VoidType()); -- Void ``` ## OptionalItemType, ListItemType and StreamItemType {#optionalitemtype} -{% if feature_codegen %} If a type is passed to these functions, then they {% else %}Do{% endif %} the action reverse to [OptionalType](#optionaltype), [ListType](#listtype), and [StreamType](#streamtype): return the element type based on its container type. +{% if feature_codegen %} If a type is passed to these functions, then they perform {% else %}Perform{% endif %} the action reverse to [OptionalType](#optionaltype), [ListType](#listtype), and [StreamType](#streamtype): return the item type based on its container type. {% if feature_codegen %} -If a type handle is passed to these functions, then they do the action reverse to [OptionalTypeHandle](#optionaltypehandle), [ListTypeHandle](#listtypehandle), and [StreamTypeHandle](#streamtypehandle): they return the handle of the element type based on the type handle of its container. -{% endif %} +If a type handle is passed to these functions, then they perform the action reverse to [OptionalTypeHandle](#optionaltypehandle), [ListTypeHandle](#listtypehandle), and [StreamTypeHandle](#streamtypehandle): they return the handle of the element type based on the type handle of its container. {% endif %} **Examples** -``` yql + +```yql SELECT FormatType(ListItemType( ParseType("List<Int32>") )); -- Int32 ``` + {% if feature_codegen %} -``` yql + +```yql SELECT FormatType(ListItemType( ParseTypeHandle("List<Int32>") )); -- Int32 ``` + {% endif %} ## DictKeyType and DictPayloadType {#dictkeytype} @@ -177,7 +194,8 @@ SELECT FormatType(ListItemType( Returns the type of the key or value based on the dictionary type. **Examples** -``` yql + +```yql SELECT FormatType(DictKeyType( ParseType("Dict<Int32,String>") )); -- Int32 @@ -188,7 +206,8 @@ SELECT FormatType(DictKeyType( Returns the tuple's element type based on the tuple type and the element index (index starts from zero). **Examples** -``` yql + +```yql SELECT FormatType(TupleElementType( ParseType("Tuple<Int32,Double>"), "1" )); -- Double @@ -199,7 +218,8 @@ SELECT FormatType(TupleElementType( Returns the type of the structure element based on the structure type and element name. **Examples** -``` yql + +```yql SELECT FormatType(StructMemberType( ParseType("Struct<foo:Int32,bar:Double>"), "foo" )); -- Int32 @@ -210,7 +230,8 @@ SELECT FormatType(StructMemberType( `CallableResultType` returns the result type based on the type of the called value. `CallableArgumentType` returns the argument type based on the called value type and its index (index starts from zero). **Examples** -``` yql + +```yql $callable_type = ParseType("(String,Bool)->Double"); SELECT FormatType(CallableResultType( @@ -223,13 +244,13 @@ FormatType(CallableArgumentType( ## VariantUnderlyingType {#variantunderlyingtype} -{% if feature_codegen %}If a type is passed to this function, then it {% else %}Does{% endif %} an action reverse to [VariantType](#varianttype): it returns the underlying type based on the variant type. +{% if feature_codegen %}If a type is passed to this function, then it {% else %}Performs{% endif %} an action reverse to [VariantType](#varianttype): it returns the underlying type based on the variant type. {% if feature_codegen %} -If a type handle is passed to this function, it does the action reverse to[VariantTypeHandle](#varianttypehandle): returns the handle of the underlying type based on the handle of the variant type. -{% endif %} +If a type handle is passed to this function, it performs the action reverse to [VariantTypeHandle](#varianttypehandle): returns the handle of the underlying type based on the handle of the variant type. {% endif %} **Examples** -``` yql + +```yql SELECT FormatType(VariantUnderlyingType( ParseType("Variant<foo:Int32,bar:Double>") )), -- Struct<'bar':Double,'foo':Int32> @@ -237,8 +258,10 @@ FormatType(VariantUnderlyingType( ParseType("Variant<Int32,Double>") )); -- Tuple<Int32,Double> ``` + {% if feature_codegen %} -``` yql + +```yql SELECT FormatType(VariantUnderlyingType( ParseTypeHandle("Variant<foo:Int32,bar:Double>") )), -- Struct<'bar':Double,'foo':Int32> @@ -246,9 +269,11 @@ FormatType(VariantUnderlyingType( ParseTypeHandle("Variant<Int32,Double>") )); -- Tuple<Int32,Double> ``` + {% endif %} {% if feature_codegen %} + # Functions for data types during calculations To work with data types during calculations, use handle types: these are [resources](../../types/special.md) that contain an opaque type definition. After constructing the type handle, you can revert to the regular type using the [EvaluateType](#evaluatetype) function. For debug purposes, you can convert a handle type to a string using the [FormatType](#formattype) function. @@ -258,15 +283,18 @@ To work with data types during calculations, use handle types: these are [resour Getting a type handle from the type passed to the argument. **Examples:** -``` yql + +```yql SELECT FormatType(TypeHandle(TypeOf("foo"))); -- String ``` + ## EvaluateType Getting the type from the type handle passed to the argument. The function is evaluated before the start of the main calculation, as well as [EvaluateExpr](../basic.md#evaluate_expr_atom). **Examples:** -``` yql + +```yql SELECT FormatType(EvaluateType(TypeHandle(TypeOf("foo")))); -- String ``` @@ -275,7 +303,8 @@ SELECT FormatType(EvaluateType(TypeHandle(TypeOf("foo")))); -- String Building a type handle from a string with description. [Documentation for its format](../../types/type_string.md). **Examples:** -``` yql + +```yql SELECT FormatType(ParseTypeHandle("List<Int32>")); -- List<int32> ``` @@ -284,7 +313,8 @@ SELECT FormatType(ParseTypeHandle("List<Int32>")); -- List<int32> Getting the top-level type name from the type handle passed to the argument. **Examples:** -``` yql + +```yql SELECT TypeKind(TypeHandle(TypeOf("foo"))); -- Data SELECT TypeKind(ParseTypeHandle("List<Int32>")); -- List ``` @@ -294,7 +324,8 @@ SELECT TypeKind(ParseTypeHandle("List<Int32>")); -- List Getting the name and parameters for a [primitive data type](../../types/primitive.md) from the primitive type handle passed to the argument. Reverse function: [DataTypeHandle](#datatypehandle). **Examples:** -``` yql + +```yql SELECT DataTypeComponents(TypeHandle(TypeOf("foo"))); -- ["String"] SELECT DataTypeComponents(ParseTypeHandle("Decimal(4,1)")); -- ["Decimal", "4", "1"] ``` @@ -304,7 +335,8 @@ SELECT DataTypeComponents(ParseTypeHandle("Decimal(4,1)")); -- ["Decimal", "4", Constructing a handle for a [primitive data type](../../types/primitive.md) from its name and parameters passed to the argument as a list. Reverse function: [DataTypeComponents](#datatypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType(DataTypeHandle( AsList("String") )); -- String @@ -319,7 +351,8 @@ SELECT FormatType(DataTypeHandle( Adds the option to assign `NULL` to the passed type handle. **Examples:** -``` yql + +```yql SELECT FormatType(OptionalTypeHandle( TypeHandle(DataType("Bool")) )); -- Bool? @@ -330,7 +363,8 @@ SELECT FormatType(OptionalTypeHandle( Builds a list type handle or stream type handle based on the passed element type handle. **Examples:** -``` yql + +```yql SELECT FormatType(ListTypeHandle( TypeHandle(DataType("Bool")) )); -- List<Bool> @@ -341,7 +375,8 @@ SELECT FormatType(ListTypeHandle( Constructs a handle for an empty list or dictionary. **Examples:** -``` yql + +```yql SELECT FormatType(EmptyListTypeHandle()); -- EmptyList ``` @@ -350,7 +385,8 @@ SELECT FormatType(EmptyListTypeHandle()); -- EmptyList Getting a list of element type handles from the tuple type handle passed to the argument. Inverse function: [TupleTypeHandle](#tupletypehandle). **Examples:** -``` yql + +```yql SELECT ListMap( TupleTypeComponents( ParseTypeHandle("Tuple<Int32, String>") @@ -366,7 +402,8 @@ SELECT ListMap( Building a tuple type handle from handles of element types passed as a list to the argument. Inverse function: [TupleTypeComponents](#tupletypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType( TupleTypeHandle( AsList( @@ -382,7 +419,8 @@ SELECT FormatType( Getting a list of element type handles and their names from the structure type handle passed to the argument. Inverse function: [StructTypeHandle](#structtypehandle). **Examples:** -``` yql + +```yql SELECT ListMap( StructTypeComponents( ParseTypeHandle("Struct<a:Int32, b:String>") @@ -401,7 +439,8 @@ SELECT ListMap( Building a structure type handle from handles of element types and names passed as a list to the argument. Inverse function: [StructTypeComponents](#structtypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType( StructTypeHandle( AsList( @@ -417,7 +456,8 @@ SELECT FormatType( Getting a key-type handle and a value-type handle from the dictionary-type handle passed to the argument. Inverse function: [DictTypeHandle](#dicttypehandle). **Examples:** -``` yql + +```yql $d = DictTypeComponents(ParseTypeHandle("Dict<Int32,String>")); SELECT @@ -430,7 +470,8 @@ SELECT Building a dictionary-type handle from a key-type handle and a value-type handle passed to arguments. Inverse function: [DictTypeComponents](#dicttypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType( DictTypeHandle( ParseTypeHandle("Int32"), @@ -444,7 +485,8 @@ SELECT FormatType( Getting the tag from the resource type handle passed to the argument. Inverse function: [ResourceTypeHandle](#resourcetypehandle). **Examples:** -``` yql + +```yql SELECT ResourceTypeTag(ParseTypeHandle("Resource<foo>")); -- foo ``` @@ -453,7 +495,8 @@ SELECT ResourceTypeTag(ParseTypeHandle("Resource<foo>")); -- foo Building a resource-type handle from the tag value passed to the argument. Inverse function: [ResourceTypeTag](#resourcetypetag). **Examples:** -``` yql + +```yql SELECT FormatType(ResourceTypeHandle("foo")); -- Resource<'foo'> ``` @@ -462,7 +505,8 @@ SELECT FormatType(ResourceTypeHandle("foo")); -- Resource<'foo'> Getting the tag and the basic type from the decorated type handle passed to the argument. Inverse function: [TaggedTypeHandle](#taggedtypehandle). **Examples:** -``` yql + +```yql $t = TaggedTypeComponents(ParseTypeHandle("Tagged<Int32,foo>")); SELECT FormatType($t.Base), $t.Tag; -- Int32, foo @@ -473,7 +517,8 @@ SELECT FormatType($t.Base), $t.Tag; -- Int32, foo Constructing a decorated type handle based on the base type handle and the tag name passed in arguments. Inverse function: [TaggedTypeComponents](#taggedtypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType(TaggedTypeHandle( ParseTypeHandle("Int32"), "foo" )); -- Tagged<Int32, 'foo'> @@ -484,7 +529,8 @@ SELECT FormatType(TaggedTypeHandle( Building a variant-type handle from the handle of the underlying type passed to the argument. Inverse function: [VariantUnderlyingType](#variantunderlyingtype). **Examples:** -``` yql + +```yql SELECT FormatType(VariantTypeHandle( ParseTypeHandle("Tuple<Int32, String>") )); -- Variant<Int32, String> @@ -495,7 +541,8 @@ SELECT FormatType(VariantTypeHandle( Constructing a handle for Void and Null types, respectively. **Examples:** -``` yql + +```yql SELECT FormatType(VoidTypeHandle()); -- Void SELECT FormatType(NullTypeHandle()); -- Null ``` @@ -505,7 +552,8 @@ SELECT FormatType(NullTypeHandle()); -- Null Getting the handle description for the type of callable value passed to the argument. Inverse function: [CallableTypeHandle](#callabletypehandle). **Examples:** -``` yql + +```yql $formatArgument = ($x) -> { return AsStruct( FormatType($x.Type) as Type, @@ -553,7 +601,8 @@ Constructing the type handle of the called value using the following arguments: Inverse function: [CallableTypeComponents](#callabletypecomponents). **Examples:** -``` yql + +```yql SELECT FormatType( CallableTypeHandle( ParseTypeHandle("String"), @@ -571,9 +620,11 @@ SELECT FormatType( Getting the number of arguments in a lambda function. **Examples:** -``` yql + +```yql SELECT LambdaArgumentsCount(($x, $y)->($x+$y)) ; -- 2 ``` {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md index 60741f1cec..6f5f78cdfb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md @@ -4,7 +4,8 @@ All [aggregate functions](../../aggregation.md) can also be used as window funct In this case, each row includes an aggregation result obtained on a set of rows from the [window frame](../../../syntax/window.md#frame). **Examples:** -``` yql + +```yql SELECT SUM(int_column) OVER w1 AS running_total, SUM(int_column) OVER w2 AS total, @@ -13,3 +14,4 @@ WINDOW w1 AS (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), w2 AS (); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md index f4214c8559..a3dc0c052b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md @@ -5,16 +5,18 @@ Access values from the first and last rows of the [window frame](../../../syntax Optionally, `OVER` can be preceded by the additional modifier `IGNORE NULLS`. It changes the behavior of functions to the first or last __non-empty__ (i.e., non-`NULL`) value among the window frame rows. The antonym of this modifier is `RESPECT NULLS`: it's the default behavior that can be omitted. **Examples** -``` yql + +```yql SELECT FIRST_VALUE(my_column) OVER w FROM my_table WINDOW w AS (ORDER BY key); ``` -``` yql +```yql SELECT LAST_VALUE(my_column) IGNORE NULLS OVER w FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/intro.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/intro.md index d07d7ca595..f1f627c3e9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/intro.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/intro.md @@ -1,2 +1,4 @@ # List of window functions in YQL + The syntax for calling window functions is detailed in a [separate article](../../../syntax/window.md). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md index 1bc1cf901f..334d448fed 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md @@ -3,9 +3,11 @@ Accessing a value from a row in the [section](../../../syntax/window.md#partition) that lags behind (`LAG`) or leads (`LEAD`) the current row by a fixed number. The first argument specifies the expression to be accessed, and the second argument specifies the offset in rows. You may omit the offset. By default, the neighbor row is used: the previous or next, respectively (hence, 1 is assumed by default). For the rows having no neighbors at a given distance (for example `LAG(expr, 3)` `NULL` is returned in the first and second rows of the section. **Examples** -``` yql + +```yql SELECT int_value - LAG(int_value) OVER w AS int_value_diff FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md index 9dab3a24a1..88e66e32d0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md @@ -2,7 +2,7 @@ Number the groups of neighboring rows in the [section](../../../syntax/window.md#partition) that have the same expression value in the argument. `DENSE_RANK` numbers the groups one-by-one, and `RANK` skips `(N - 1)` values, with `N` being the number of rows in the previous group. -If there is no argument, it uses the order specified in the ORDER BY `section` in the window definition. +If there is no argument, it uses the order specified in the `ORDER BY` section in the window definition. If the argument is omitted and `ORDER BY` is not specified, then all rows are considered equal to each other. {% note info %} @@ -12,14 +12,18 @@ Passing an argument to `RANK`/`DENSE_RANK` is a non-standard extension in YQL. {% endnote %} **Examples** -``` yql + +```yql SELECT RANK(my_column) OVER w FROM my_table WINDOW w AS (ORDER BY key); ``` -``` yql + +```yql SELECT RANK() OVER w FROM my_table WINDOW w AS (ORDER BY my_column); + + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md index 33ba59c6b1..49c93db7fc 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md @@ -3,9 +3,11 @@ Row number within a [partition](../../../syntax/window.md#partition). No arguments. **Examples** -``` yql + +```yql SELECT ROW_NUMBER() OVER w AS row_num FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md index 6993f977d4..241d78b744 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md @@ -2,3 +2,4 @@ A non-standard window function `SessionState()` (without arguments) lets you get the session calculation status from [SessionWindow](../../../syntax/group_by.md#session-window) for the current row. It's allowed only if `SessionWindow()` is present in the `PARTITION BY` section in the window definition. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/aggregation.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/aggregation.md index fd2c0ea7c5..2d6f0e1f60 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/aggregation.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/aggregation.md @@ -1,29 +1,29 @@ -{% include [x](_includes/aggregation/simple.md) %} +{% include [simple.md](_includes/aggregation/simple.md) %} -{% include [x](_includes/aggregation/count_distinct_estimate.md) %} +{% include [count_distinct_estimate.md](_includes/aggregation/count_distinct_estimate.md) %} -{% include [x](_includes/aggregation/agg_list.md) %} +{% include [agg_list.md](_includes/aggregation/agg_list.md) %} -{% include [x](_includes/aggregation/max_min_by.md) %} +{% include [max_min_by.md](_includes/aggregation/max_min_by.md) %} -{% include [x](_includes/aggregation/top_bottom.md) %} +{% include [top_bottom.md](_includes/aggregation/top_bottom.md) %} -{% include [x](_includes/aggregation/topfreq_mode.md) %} +{% include [topfreq_mode.md](_includes/aggregation/topfreq_mode.md) %} -{% include [x](_includes/aggregation/stddev_variance.md) %} +{% include [stddev_variance.md](_includes/aggregation/stddev_variance.md) %} -{% include [x](_includes/aggregation/corr_covar.md) %} +{% include [corr_covar.md](_includes/aggregation/corr_covar.md) %} -{% include [x](_includes/aggregation/percentile_median.md) %} +{% include [percentile_median.md](_includes/aggregation/percentile_median.md) %} -{% include [x](_includes/aggregation/histogram.md) %} +{% include [histogram.md](_includes/aggregation/histogram.md) %} -{% include [x](_includes/aggregation/bool_bit.md) %} +{% include [bool_bit.md](_includes/aggregation/bool_bit.md) %} {% if feature_window_functions %} - {% include [x](_includes/aggregation/session_start.md) %} + {% include [session_start.md](_includes/aggregation/session_start.md) %} {% endif %} -{% include [x](_includes/aggregation/aggregate_by.md) %} +{% include [aggregate_by.md](_includes/aggregation/aggregate_by.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/basic.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/basic.md index 212b8194d0..b4262430aa 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/basic.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/basic.md @@ -81,10 +81,12 @@ {% include [x](_includes/basic/staticmap.md) %} +{% include [x](_includes/basic/staticzip.md) %} + {% include [x](_includes/basic/aggr_factory.md) %} {% if tech %} {% include [x](_includes/basic/s_expressions.md) %} -{% endif %}
\ No newline at end of file +{% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/codegen.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/codegen.md index 3fb45da716..bc3bca2b11 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/codegen.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/codegen.md @@ -1,2 +1,2 @@ -{% include [x](_includes/codegen.md) %} +{% include [codegen.md](_includes/codegen.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/dict.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/dict.md index 1d060a286a..24267d59a4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/dict.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/dict.md @@ -1,2 +1,2 @@ -{% include [x](_includes/dict.md) %} +{% include [dict.md](_includes/dict.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/index.md new file mode 100644 index 0000000000..b3cf50bcae --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/index.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/json.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/json.md index 2a96f5886e..4b454b1b63 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/json.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/json.md @@ -1,2 +1,2 @@ -{% include [x](_includes/json.md) %} +{% include [json.md](_includes/json.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml index 8def30316b..9481cd1ccc 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml @@ -1,4 +1,5 @@ items: +- { name: Overview, href: index.md } - { name: Basic, href: basic.md } - { name: Aggregate, href: aggregation.md } - { name: Window, href: window.md, when: feature_window_functions } diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md index ce9c0f9a0a..0ee45fa2f3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md @@ -4,7 +4,7 @@ Performing an action without declaring it (anonymous action). **Syntax** -1. `BEGIN`. +1. `BEGIN`; 1. List of top-level expressions. 1. `END DO`. @@ -17,4 +17,5 @@ DO BEGIN SELECT 1; SELECT 2 -- here and in the previous example, you might omit ';' before END END DO -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md index c32db440e2..97ef941ee7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md @@ -4,7 +4,7 @@ Specifies a named action that is a parameterizable block of multiple top-level e **Syntax** -1. `DEFINE ACTION`: Action definition. +1. `DEFINE ACTION`: action definition. 1. [Action name](../../expressions.md#named-nodes) that will be used to access the defined action further in the query. 1. The values of parameter names are listed in parentheses. 1. `AS` keyword. @@ -47,4 +47,5 @@ DO EMPTY_ACTION(); DO $hello_world(NULL); DO $hello_world("John"); DO $hello_world(NULL, "Earth"); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md index 471fcb10e2..1a69668908 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md @@ -3,18 +3,18 @@ `EVALUATE IF`: Executing an action depending on the condition. It's followed by: 1. Condition. -1. [DO](#do) with the name and parameters of the action or an anonymous action. -1. An optional `ELSE` followed by the second `DO` for a situation where the condition is not met. +2. [DO](#do) with the name and parameters of the action or an anonymous action. +3. An optional `ELSE` followed by the second `DO` for a situation where the condition is not met. ## EVALUATE FOR {#evaluate-for} -`EVALUATE FOR`: Executing an action for each element in the list. It's followed by: +`EVALUATE FOR`: Executing an action for each item in the list. It's followed by: 1. [A named expression](../../expressions.md#named-nodes) applied to each next element in the list. -1. `IN` keyword. -1. The above-declared named expression applied to the list the action is executed on. -1. [DO](#do) with the name and parameters of an action or an anonymous action. In the parameters, you can use both the current element from the first paragraph and any named expressions declared above, including the list itself. -1. An optional `ELSE` followed by the second `DO` for the situation when the list is empty. +2. `IN` keyword. +3. The above-declared named expression applied to the list the action is executed on. +4. [DO](#do) with the name and parameters of an action or an anonymous action. In the parameters, you can use both the current element from the first paragraph and any named expressions declared above, including the list itself. +5. An optional `ELSE` followed by the second `DO` for the situation when the list is empty. **Examples** @@ -72,4 +72,5 @@ ELSE Note that `EVALUATE` is run before the operation starts. Please also note that in `EVALUATE` you can't use [anonymous tables](../../select.md#temporary-tables). -{% endnote %}
\ No newline at end of file +{% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md index 06d47a4380..692ed59a08 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md @@ -45,7 +45,23 @@ ALTER TABLE `series` DROP INDEX `title_index`; ``` {% endif %} -{% if concept_table %} +{% if feature_map_tables %} + +## Renaming a table {#rename} + +```sql +ALTER TABLE old_table_name RENAME TO new_table_name; +``` + +If a table with a new name exists, an error is returned. The possibility of transactional table substitution under load is supported by ad-hoc CLI and SDK methods. + +If a YQL query contains multiple `ALTER TABLE ... RENAME TO ...` commands, each of them will be executed in autocommit mode as a separate transaction. From the external process viewpoint, the tables will be renamed sequentially one by one. To rename multiple tables within a single transaction, use ad-hoc methods available in the CLI and SDK. + +Renaming can be used to move a table from one directory inside the database to another, for example: + +```sql +ALTER TABLE `table1` RENAME TO `/backup/table1`; +``` ## Changing column groups {#column-family} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md index 07e843fb8b..003af35c54 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md @@ -15,7 +15,7 @@ DECLARE $named-node AS data_type; 1. `DECLARE` keyword. 1. `$named-node`: The name by which you can access the passed value. It must start with `$`. 1. `AS` keyword. -1. `data_type`: The data type [represented as a string in the accepted format](../../../types/type_string.md). +1. `data_type` is the data type [represented as a string in the accepted format](../../../types/type_string.md). Only serializable data types are allowed: @@ -32,4 +32,5 @@ DECLARE $y AS String?; DECLARE $z AS List<String>; SELECT $x, $y, $z; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/as.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/as.md index 6ed3562f3f..4da92fdd1f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/as.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/as.md @@ -9,32 +9,36 @@ Can be used in the following scenarios: {% if select_command != "SELECT STREAM" %} **Examples:** -``` yql +```yql SELECT key AS k FROM my_table; ``` -``` yql +```yql SELECT t.key FROM my_table AS t; ``` -``` yql +```yql SELECT MyFunction(key, 123 AS my_optional_arg) FROM my_table; ``` + {% else %} **Examples:** -``` yql + +```yql SELECT STREAM key AS k FROM my_stream; ``` -``` yql +```yql SELECT STREAM s.key FROM my_stream AS s; ``` -``` yql +```yql SELECT STREAM MyFunction(key, 123 AS my_optional_arg) FROM my_stream; ``` + {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md index 482b9722f0..e1f3edfeb5 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md @@ -4,7 +4,8 @@ Checking whether a value is in a range. It's equivalent to two conditions with ` **Examples** -``` yql +```yql SELECT * FROM my_table WHERE key BETWEEN 10 AND 20; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md index da08059551..064bcb9a32 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md @@ -3,7 +3,8 @@ Performs a bitwise conversion of an integer value to the specified integer type. The conversion is always successful, but may lose precision or high-order bits. **Examples** -``` yql + +```yql SELECT BITCAST(100000ul AS Uint32), -- 100000 BITCAST(100000ul AS Int16), -- -31072 @@ -11,3 +12,4 @@ SELECT BITCAST(-1 AS Int16), -- -1 BITCAST(-1 AS Uint16); -- 65535 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md index f94fb18a08..50ee8fd8cd 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md @@ -1,16 +1,14 @@ ## CASE{#case}. Conditional expressions and branching. It's similar to `if`, `switch` and ternary operators in the imperative programming languages. -If the result of the `WHEN` expression is `true`, the value of the `CASE` expression becomes the result -following the condition, and the rest of the `CASE` expression isn't calculated. If the condition is not met, -all the `WHEN` clauses that follow are checked. If -none of the `WHEN` clauses are met, the `CASE` value is assigned the result from the `ELSE` clause. +If the result of the `WHEN` expression is `true`, the value of the `CASE` expression becomes the result following the condition, and the rest of the `CASE` expression isn't calculated. If the condition is not met, all the `WHEN` clauses that follow are checked. If none of the `WHEN` clauses are met, the `CASE` value is assigned the result from the `ELSE` clause. The `ELSE` branch is mandatory in the `CASE` expression. Expressions in `WHEN` are checked sequentially, from top to bottom. Since its syntax is quite sophisticated, it's often more convenient to use the built-in function [IF](../../../builtins/basic.md#if). **Examples** -``` yql + +```yql SELECT CASE WHEN value > 0 @@ -29,3 +27,4 @@ SELECT END FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md index 803a84b851..9820614f91 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md @@ -7,8 +7,9 @@ For structures and tuples, it deletes elements that are omitted in the target ty For more information about casting rules, see [here](../../../types/cast.md). {% endif %} -{% include notitle [Decimal types](../../../_includes/decimal_args.md) %} +{% include [decimal_args](../../../_includes/decimal_args.md) %} **Examples** {% include [cast_examples](../../../_includes/cast_examples.md) %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md index 2db930bc12..f540daa4a9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md @@ -1,12 +1,12 @@ ## Matching a string by pattern {#check-match} -`REGEXP` and `RLIKE` are aliases used to call [Hyperscan::Grep](../../../udf/list/hyperscan.md#grep). `MATCH`: same for [Hyperscan::Match](../../../udf/list/hyperscan.md#match). +`REGEXP` and `RLIKE` are aliases used to call [Hyperscan::Grep](../../../udf/list/hyperscan.md#grep). `MATCH`: Same for [Hyperscan::Match](../../../udf/list/hyperscan.md#match). `LIKE` works as follows: * Patterns can include two special characters: * `%`: Zero or more of any characters. - * `_`: Exactly one of any character. + * `_`: Exactly one of any character.
All other characters are literals that represent themselves. * As opposed to `REGEXP`, `LIKE` must be matched exactly. For example, to search a substring, add `%` at the beginning and end of the pattern. * `ILIKE` is a case-insensitive version of `LIKE`. @@ -16,7 +16,8 @@ All other characters are literals that represent themselves. The most popular way to use the `LIKE` and `REGEXP` keywords is to filter a table using the statements with the `WHERE` clause. However, there are no restrictions on using templates in this context: you can use them in most of contexts involving strings, for example, with concatenation by using `||`. **Examples** -``` yql + +```yql SELECT * FROM my_table WHERE string_column REGEXP '\\d+'; -- the second slash is required because @@ -24,7 +25,7 @@ WHERE string_column REGEXP '\\d+'; -- can accept C-escaped strings ``` -``` yql +```yql SELECT string_column LIKE '___!_!_!_!!!!!!' ESCAPE '!' -- searches for a string of exactly 9 characters: @@ -34,10 +35,11 @@ SELECT FROM my_table; ``` -``` yql +```yql SELECT * FROM my_table WHERE key LIKE 'foo%bar'; -- would probably only physically scan the keys -- starting with "foo", and then, among them, -- will leave only those that end in "bar" ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md index 877294c1e5..bf77137627 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md @@ -8,6 +8,7 @@ Don't confuse this operator with a logical "or": in SQL, it's denoted by the `OR **Examples** -``` sql +```sql SELECT "fo" || "o"; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md index fc2795712d..c0bd34f103 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md @@ -4,38 +4,37 @@ Checking whether a value is inside of a set of values. It's logically equivalent {% note warning "Warning" %} -Unlike a similar keyword in Python, in YQL `IN ` **DOES NOT** searches for a substring inside a string. To search for a substring, use the function [String::Contains](../../../udf/list/string.md) or [LIKE / REGEXP](#like) mentioned above. +Unlike a similar keyword in Python, in YQL `IN ` **DOES NOT** searches for a substring inside a string. To search for a substring, use the function [String::Contains](../../../udf/list/string.md) or [LIKE/REGEXP](#like) mentioned above. {% endnote %} -Immediately after `IN`, you can specify the `COMPACT` hint. -If `COMPACT` is not specified, then `IN` with a subquery is executed as a relevant `JOIN` (`LEFT SEMI` for `IN` and `LEFT ONLY` for `NOT IN`), if possible. +Immediately after `IN`, you can specify the `COMPACT` hint.
+If `COMPACT` is not specified, then `IN` with a subquery is executed as a relevant `JOIN` (`LEFT SEMI` for `IN` and `LEFT ONLY` for `NOT IN`), if possible.
Using the `COMPACT` hint forces the in-memory execution strategy: a hash table is immediately built from the contents of the right `IN` part in-memory, and then the left part is filtered. The `COMPACT` hint must be used with care. Since the hash table is built in-memory, the query may fail if the right part of `IN` contains many large or different elements. -{% if feature_mapreduce %} -Since YQL imposes a limit on the query size in bytes (it's about 1Mb), add large lists of values to your query by URLs and use the [ParseFile]( function./../../builtins/basic.md#parsefile). +{% if feature_mapreduce %} Since YQL imposes a limit on the query size in bytes (it's about 1Mb), add large lists of values to your query by URLs and use the [ParseFile](../../../builtins/basic.md#parsefile) function. {% endif %} **Examples** -``` yql +```yql SELECT column IN (1, 2, 3) FROM my_table; ``` -``` yql +```yql SELECT * FROM my_table WHERE string_column IN ("a", "b", "c"); ``` -``` yql +```yql $foo = AsList(1, 2, 3); SELECT 1 IN $foo; ``` -``` yql +```yql $values = (SELECT column + 1 FROM table); SELECT * FROM my_table WHERE -- filtering by an in-memory hash table for one_table @@ -43,3 +42,4 @@ SELECT * FROM my_table WHERE -- followed by LEFT ONLY JOIN with other_table column2 NOT IN (SELECT other_column FROM other_table); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-distinct-from.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-distinct-from.md index 511623af52..c54862cef2 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-distinct-from.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-distinct-from.md @@ -2,6 +2,7 @@ Comparing of two values. Unlike the regular [comparison operators](#comparison-operators), NULLs are treated as equal to each other. More precisely, the comparison is carried out according to the following rules: + 1) The operators `IS DISTINCT FROM`/`IS NOT DISTINCT FROM` are defined for those and only for those arguments for which the operators `!=` and `=` are defined. 2) The result of `IS NOT DISTINCT FROM` is equal to the logical negation of the `IS DISTINCT FROM` result for these arguments. 3) If the result of the `==` operator is not equal to zero for some arguments, then it is equal to the result of the `IS NOT DISTINCT FROM` operator for the same arguments. @@ -9,3 +10,4 @@ More precisely, the comparison is carried out according to the following rules: 5) The result of `IS NOT DISTINCT FROM` for an empty `Optional` or `NULL` and filled-in `Optional` or non-`Optional` value is `False`. For values of composite types, these rules are used recursively. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md index a503359e01..f8242bf2f4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md @@ -4,7 +4,8 @@ Matching an empty value (`NULL`). Since `NULL` is a special value [equal to noth **Examples** -``` yql +```yql SELECT key FROM my_table WHERE value IS NOT NULL; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md index 1bacb876ee..6f4c1d2ad3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md @@ -11,7 +11,7 @@ When using this syntax to access containers within table columns, be sure to spe **Examples** -``` yql +```yql SELECT t.struct.member, t.tuple.7, @@ -20,7 +20,8 @@ SELECT FROM my_table AS t; ``` -``` yql +```yql SELECT Sample::ReturnsStruct().member; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md index 08ffbaf363..bb4d836725 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md @@ -12,7 +12,7 @@ One or more of the last lambda parameters can be marked with a question mark as **Examples** -``` yql +```yql $f = ($y) -> { $prefix = "x"; RETURN $prefix || $y; @@ -25,8 +25,9 @@ $h = ($x, $y?) -> ($x + ($y ?? 0)); SELECT $f("y"), $g("z"), $h(1), $h(2, 3); -- "xy", "xz", 1, 5 ``` -``` yql +```yql -- if the lambda result is calculated by a single expression, then you can use a more compact syntax: $f = ($x, $_) -> ($x || "suffix"); -- the second argument is not used SELECT $f("prefix_", "whatever"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md index 0bd8b5f036..523d28251e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md @@ -3,41 +3,48 @@ Complex queries may be sophisticated, containing lots of nested levels and/or repeating parts. In YQL, you can use named expressions to assign a name to an arbitrary expression or subquery. Named expressions can be referenced in other expressions or subqueries. In this case, the original expression/subquery is actually substituted at point of use. A named expression is defined as follows: + ``` <named-expr> = <expression> | <subquery>; ``` + Here `<named-expr>` consists of a $ character and an arbitrary non-empty identifier (for example, `$foo`). If the expression on the right is a tuple, you can automatically unpack it by specifying several named expressions separated by commas on the left: + ``` <named-expr1>, <named-expr2>, <named-expr3> ... = <expression-returning-tuple>; ``` + In this case, the number of expressions must match the tuple size. Each named expression has a scope. It starts immediately after the definition of a named expression and ends at the end of the nearest enclosed namescope (for example, at the end of the query or at the end of the body of the [lambda function](#lambda), [ACTION](../../action.md#define-action){% if feature_subquery %}, [SUBQUERY](../../subquery.md#define-subquery){% endif %}{% if feature_mapreduce %}, or the cycle [EVALUATE FOR](../../action.md#evaluate-for){% endif %}). Redefining a named expression with the same name hides the previous expression from the current scope. -If the named expression has never been used, a warning is issued. To avoid such a warning, use the underscore as the first character in the ID (for example `$_foo`). +If the named expression has never been used, a warning is issued. To avoid such a warning, use the underscore as the first character in the ID (for example, `$_foo`). The named expression `$_` is called an anonymous named expression and is processed in a special way: it works as if `$_` would be automatically replaced by `$_<some_uniq_name>`. Anonymous named expressions are convenient when you don't need the expression value. For example, to fetch the second element from a tuple of three elements, you can write: -``` yql + +```yql $_, $second, $_ = AsTuple(1, 2, 3); select $second; ``` An attempt to reference an anonymous named expression results in an error: -``` yql + +```yql $_ = 1; select $_; --- error: Unable to reference anonymous name $_ export $_; --- An error: Can not export anonymous name $_ ``` -{% if feature_mapreduce %} -Moreover, you can't import a named expression with an anonymous alias: -``` yql + +{% if feature_mapreduce %} Moreover, you can't import a named expression with an anonymous alias: + +```yql import utils symbols $sqrt as $_; --- error: Can not import anonymous name $_ ``` -{% endif %} -Anonymous argument names are also supported for [lambda functions](#lambda), [ACTION](../../action.md#define-action){% if feature_subquery %}, [SUBQUERY](../../subquery.md#define-subquery){% endif %}{% if feature_mapreduce %}, and in [EVALUATE FOR](../../action.md#evaluate-for){% endif %}. + +{% endif %} Anonymous argument names are also supported for [lambda functions](#lambda), [ACTION](../../action.md#define-action){% if feature_subquery %}, [SUBQUERY](../../subquery.md#define-subquery){% endif %}{% if feature_mapreduce %}, and in [EVALUATE FOR](../../action.md#evaluate-for){% endif %}. {% note info %} @@ -47,7 +54,7 @@ If named expression substitution results in completely identical subgraphs in th **Examples** -``` yql +```yql $multiplier = 712; SELECT a * $multiplier, -- $multiplier is 712 @@ -60,7 +67,7 @@ SELECT FROM abc_table; ``` -``` yql +```yql $intermediate = ( SELECT value * value AS square, @@ -73,11 +80,12 @@ INNER JOIN $intermediate AS b ON a.value == b.square; ``` -``` yql +```yql $a, $_, $c = AsTuple(1, 5u, "test"); -- unpack a tuple SELECT $a, $c; ``` -``` yql +```yql $x, $y = AsTuple($y, $x); -- swap expression values ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md index 879df86b4f..36f0685055 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md @@ -8,11 +8,11 @@ For the Decimal data type, bankers rounding is used (to the nearest even integer **Examples** -``` yql +```yql SELECT 2 + 2; ``` -``` yql +```yql SELECT 0.0 / 0.0; ``` @@ -25,7 +25,7 @@ The operators `=`, `==`, `!=`, `<>`, `>`, `<` are defined for: **Examples** -``` yql +```yql SELECT 2 > 1; ``` @@ -35,7 +35,7 @@ Use the operators `AND`, `OR`, `XOR` for logical operations on Boolean values (` **Examples** -``` yql +```yql SELECT 3 > 0 AND false; ``` @@ -45,12 +45,12 @@ Bitwise operations on numbers: * `&`, `|`, `^`: AND, OR, and XOR, respectively. Don't confuse bitwise operations with the related keywords. The keywords `AND`, `OR`, and `XOR` are used * for Boolean values only*, but not for numbers. * ` ~ `: A negation. -* `<<`, `>>`: Left or right shifts. -* `|<<`, `>>|`: Circular left or right shifts. +* `<`, `>`: Left or right shifts. +* `|<`, `>|`: Circular left or right shifts. **Examples** -``` yql +```yql SELECT key << 10 AS key, ~value AS value @@ -60,11 +60,10 @@ FROM my_table; ### Precedence and associativity of operators {#operator-priority} Operator precedence determines the order of evaluation of an expression that contains different operators. -For example, the expression `1 + 2 * 3` is evaluated as `1 + (2 * 3)` -because the multiplication operator has a higher precedence than the addition operator. +For example, the expression `1 + 2 * 3` is evaluated as `1 + (2 * 3)` because the multiplication operator has a higher precedence than the addition operator. Associativity determines the order of evaluating expressions containing operators of the same type. -For example, the expression `1 + 2 + 3` is evaluated as `(1 + 2) + 3`, because the addition operator is left-associative. +For example, the expression `1 + 2 + 3` is evaluated as `(1 + 2) + 3` because the addition operator is left-associative. On the other hand, the expression `a ?? b ?? c` is evaluated as `a ?? (b ?? c)` because the `??` operator is right-associative The table below shows precedence and associativity of YQL operators. @@ -72,7 +71,7 @@ The operators in the table are listed in descending order of precedence. | Priority | Operator | Description | Associativity | | --- | --- | --- | --- | -| 1 | <code>a[], a.foo, a()</code> | Accessing a container element, calling a function | Left | +| 1 | <code>a[], a.foo, a()</code> | Accessing a container item, calling a function | Left | | 2 | <code>+a, -a, ~a, NOT a</code> | Unary operators: plus, minus, bitwise and logical negation | Right | | 3 | <code>a || b</code> | [String concatenation](#concatenation) | Left | | 4 | <code>a*b, a/b, a%b</code> | Multiplication, division, remainder of division | Left | @@ -85,3 +84,4 @@ The operators in the table are listed in descending order of precedence. | 10 | <code>a XOR b</code> | Logical XOR | Left | | 11 | <code>a AND b</code> | Logical AND | Left | | 12 | <code>a OR b</code> | Logical OR | Left | + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/tables.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/tables.md index bfff9de481..cb6e91aa27 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/tables.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/tables.md @@ -1,6 +1,7 @@ ## Table expressions {#table-contexts} A table expression is an expression that returns a table. Table expressions in YQL are as follows: + * Subqueries: `(SELECT key, subkey FROM T)` * [Named subqueries](#named-nodes): `$foo = SELECT * FROM T;` (in this case, `$foo` is also a table expression) {% if feature_subquery %} @@ -8,14 +9,13 @@ A table expression is an expression that returns a table. Table expressions in Y {% endif %} Semantics of a table expression depends on the context where it is used. In YQL, table expressions can be used in the following contexts: + * Table context: after [FROM](../../select.md#from). In this case, table expressions work as expected: for example, `$input = SELECT a, b, c FROM T; SELECT * FROM $input` returns a table with three columns. The table context also occurs after [UNION ALL](../../select.md#unionall){% if feature_join %}, [JOIN](../../join.md#join){% endif %}{% if feature_mapreduce and process_command == "PROCESS" %}, [PROCESS](../../process.md#process), [REDUCE](../../reduce.md#reduce){% endif %}; * Vector context: after [IN](#in). In this context, the table expression must contain exactly one column (the name of this column doesn't affect the expression result in any way). A table expression in a vector context is typed as a list (the type of the list element is the same as the column type in this case). Example: `SELECT * FROM T WHERE key IN (SELECT k FROM T1)`; -* A scalar context arises _in all the other cases_. As in a vector context, -a table expression must contain exactly one column, but the value of the table expression is a scalar, -that is, an arbitrarily selected value of this column (if no rows are returned, the result is `NULL`). Example: `$count = SELECT COUNT(*) FROM T; SELECT * FROM T ORDER BY key LIMIT $count / 2`; +* A scalar context arises _in all the other cases_. As in a vector context, a table expression must contain exactly one column, but the value of the table expression is a scalar, that is, an arbitrarily selected value of this column (if no rows are returned, the result is `NULL`). Example: `$count = SELECT COUNT(*) FROM T; SELECT * FROM T ORDER BY key LIMIT $count / 2`; The order of rows in a table context, the order of elements in a vector context, and the rule for selecting a value from a scalar context (if multiple values are returned), aren't defined. This order also cannot be affected by `ORDER BY`: `ORDER BY` without `LIMIT` is ignored in table expressions with a warning, and `ORDER BY` with `LIMIT` defines a set of elements rather than the order within that set. @@ -23,7 +23,7 @@ The order of rows in a table context, the order of elements in a vector context, There is an exception to this rule. Named expression with [PROCESS](../../process.md#process), if used in a scalar context, behaves as in a table context: -``` yql +```yql $input = SELECT 1 AS key, 2 AS value; $process = PROCESS $input; @@ -34,11 +34,12 @@ SELECT $process[0].key; -- that returns 1 SELECT FormatType(TypeOf($input)); -- throws an error: $input in a scalar context must contain one column ``` + {% note warning "Warning" %} A common error is to use an expression in a scalar context rather than a table context or vector context. For example: -``` yql +```yql $dict = SELECT key, value FROM T1; DEFINE SUBQUERY $merge_dict($table, $dict) AS @@ -47,12 +48,11 @@ END DEFINE; SELECT * FROM $merge_dict("Input", $dict); -- $dict is used in a scalar context in this case. -- an error: exactly one column is expected in a scalar context - ``` A correct notation in this case is: -``` yql +```yql DEFINE SUBQUERY $dict() AS SELECT key, value FROM T1; END DEFINE; @@ -68,3 +68,4 @@ SELECT * FROM $merge_dict("Input", $dict); -- $dict - is a subquery template (ra {% endnote %} {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md index a6d1159bea..f4d50b811c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md @@ -65,6 +65,7 @@ This conversion can be convenient in the following cases: `FLATTEN BY` interprets [optional data types](../../../types/optional.md) as lists of length 0 or 1. The table rows with `NULL` are skipped, and the column type changes to a similar non-optional type. -`FLATTEN BY` makes only one conversion at a time, so use `FLATTEN LIST BY` or `FLATTEN OPTIONAL BY` on optional containers, for example,`Optional<List<String>>`. +`FLATTEN BY` makes only one conversion at a time, so use `FLATTEN LIST BY` or `FLATTEN OPTIONAL BY` on optional containers, for example,`Optional<List<String>>`. + +{% endnote %} -{% endnote %}
\ No newline at end of file diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_columns.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_columns.md index 1f9ac2f0f5..b9ce21c63d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_columns.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_columns.md @@ -16,4 +16,5 @@ FROM ( AsStruct( false AS z) ) FLATTEN COLUMNS; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md index 4d2741ce55..3692fe89ef 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md @@ -1,7 +1,8 @@ ### Analogs of FLATTEN BY in other DBMS {#flatten-other-dmb} -* PostgreSQL: `unnest`. -* Hive: `LATERAL VIEW`. -* MongoDB: `unwind`. -* Google BigQuery: `FLATTEN`. -* ClickHouse: `ARRAY JOIN / arrayJoin`.
\ No newline at end of file +* PostgreSQL: `unnest` +* Hive: `LATERAL VIEW` +* MongoDB: `unwind` +* Google BigQuery: `FLATTEN` +* ClickHouse: `ARRAY JOIN / arrayJoin` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_type_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_type_by.md index 4fda3a309b..0f4c5e5bea 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_type_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_type_by.md @@ -40,4 +40,5 @@ SELECT * FROM ( "1;2;3" AS a, AsList("x", "y", "z") AS b ) FLATTEN LIST BY (String::SplitToList(a, ";") as a, b); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/compact.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/compact.md index 949fc67fa8..99bad56ad9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/compact.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/compact.md @@ -5,7 +5,8 @@ Improves aggregation efficiency if the query author knows in advance that none o Unlike the usual GROUP BY, the Map-side combiner stage and additional Reduce are disabled for each field with [DISTINCT](#distinct) aggregation. **Example:** -``` yql + +```yql SELECT key, COUNT (DISTINCT value) AS count -- top 3 keys by the number of unique values @@ -14,3 +15,4 @@ GROUP COMPACT BY key ORDER BY count DESC LIMIT 3; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md index b9391c9013..9070a7cb22 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md @@ -10,7 +10,7 @@ Applying `DISTINCT` to calculated values is not currently implemented. For this **Example** -``` sql +```sql SELECT key, COUNT (DISTINCT value) AS count -- top 3 keys by the number of unique values diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md index df88958c53..bb3e7fa1ac 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md @@ -3,6 +3,7 @@ Group the `SELECT` results by the values of the specified columns or expressions. `GROUP BY` is often combined with [aggregate functions](../../../builtins/aggregation.md) (`COUNT`, `MAX`, `MIN`, `SUM`, `AVG`) to perform calculations in each group. **Syntax** + ```sql SELECT -- In SELECT, you can use: column1, -- key columns specified in GROUP BY @@ -14,8 +15,8 @@ SELECT -- In SELECT, you can use: FROM table GROUP BY column1, column2, ..., - <expr> AS key_n -- When grouping by expression, you can set a name for it using AS, - -- and use it in SELECT + <expr> AS key_n -- When grouping by expression, you can set a name for it using AS, + -- and use it in SELECT ``` The query in the format `SELECT * FROM table GROUP BY k1, k2, ...` returns all columns listed in GROUP BY, i.e., is equivalent to `SELECT DISTINCT k1, k2, ... FROM table`. @@ -57,8 +58,9 @@ GROUP BY subkey as sk, ``` -{% note warning %} +{% note warning "Attention" %} Specifying a name for a column or expression in `GROUP BY .. AS foo` it is an extension on top of YQL. Such a name becomes visible in `WHERE` despite the fact that filtering by `WHERE` is executed [before](../../select.md#selectexec) the grouping. For example, if the `T` table includes two columns, `foo` and `bar`, then the query `SELECT foo FROM T WHERE foo > 0 GROUP BY bar AS foo` would actually filter data by the `bar` column from the source table. {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general_stream.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general_stream.md index a93f68bfdd..9667097c43 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general_stream.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general_stream.md @@ -14,9 +14,11 @@ You can group by the result of calculating an arbitrary expression from the sour Aggregate functions automatically skip `NULL` in their arguments. Among the columns by which grouping is performed, make sure to use the `HOP` construct to define the time window for grouping. -``` yql + +```yql HOP(time_extractor, hop, interval, delay) ``` + The implemented version of the time window is called the **hopping window**. This is a window that moves forward in discrete intervals (the `hop` parameter). The total duration of the window is set by the `interval` parameter. To determine the time of each input event, the `time_extractor` parameter is used. This expression depends only on the input values of the stream's columns and must have the `Timestamp` type. It indicates where exactly to get the time value from input events. In each stream defined by the values of all the grouping columns, the window moves forward independently of other streams. Advancement of the window is totally dependent on the latest event of the stream. Since records in streams get somewhat mixed in time, the `delay` parameter has been added so you can delay the closing of the window by a specified period. Events arriving before the current window are ignored. @@ -30,7 +32,8 @@ Functions with omitted `HOP_START` and `HOP_END` parameters, return a value of t The **tumbling window** known in other systems is a special case of a **hopping window** when `interval` == `hop`. **Examples:** -``` yql + +```yql SELECT STREAM key, COUNT(*) @@ -43,7 +46,7 @@ GROUP BY -- delay = 30 seconds ``` -``` yql +```yql SELECT STREAM double_key, HOP_END() as time, @@ -51,5 +54,6 @@ SELECT STREAM FROM my_stream GROUP BY key + key AS double_key, - HOP(ts, "PT1M", "PT1M", "PT1M"); + HOP(ts, "PT1М", "PT1M", "PT1M"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md index 62a9c816a2..5eca8c3171 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md @@ -4,10 +4,11 @@ Filtering a `SELECT` based on the calculation results of [aggregate functions](. **Example** -``` yql +```yql SELECT key FROM my_table GROUP BY key HAVING COUNT(value) > 100; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having_stream.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having_stream.md index ff098a05db..a73539ba74 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having_stream.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having_stream.md @@ -3,10 +3,12 @@ Filtering a `SELECT STREAM` based on the calculation results of [aggregate functions](../../../builtins/aggregation.md). The syntax is similar to [WHERE](../../select_stream.md#where). **Examples:** -``` yql + +```yql SELECT STREAM key FROM my_table GROUP BY key, HOP(ts, "PT1M", "PT1M", "PT1M") HAVING COUNT(value) > 100; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md index 4c063db4a5..d2c73fa0db 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md @@ -27,6 +27,7 @@ You can combine `ROLLUP`, `CUBE` and `GROUPING SETS`, separating them by commas. The values of columns not used in calculations are replaced with `NULL` in the subtotal. In the overall total, the values of all columns are replaced by `NULL`. `GROUPING`: A function that allows you to distinguish the source `NULL` values from the `NULL` values added while calculating subtotals and overall totals. `GROUPING` returns a bit mask: + * `0`: If `NULL` is used for the original empty value. * `1`: If `NULL` is added for a subtotal or overall total. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md index 3cc4195485..28efac179d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md @@ -19,21 +19,47 @@ The following happens in this case: If `GROUP BY` includes nothing more than SessionWindow, then the input table gets into one partition. 2) Each partition is split into disjoint subsets of rows (sessions). For this, the partition is sorted in the ascending order of the `time_expr` expression. - The session limits are drawn between neighboring elements of the partition, that differ in their `time_expr` values by more than `timeout_expr`. + The session limits are drawn between neighboring items of the partition, that differ in their `time_expr` values by more than `timeout_expr`. 3) The sessions obtained in this way are the final partitions on which aggregate functions are calculated. The SessionWindow() key column (in the example, it's `session_start`) has the value "the minimum `time_expr` in the session". -If `GROUP BY` includes SessionWindow(), you can use a special aggregate function -[SessionStart](../../../builtins/aggregation.md#session-start). +If `GROUP BY` includes SessionWindow(), you can use a special aggregate function [SessionStart](../../../builtins/aggregation.md#session-start). An extended version of SessionWindow with four arguments is also supported: `SessionWindow(<order_expr>, <init_lambda>, <update_lambda>, <calculate_lambda>)` Where: -* `<order_expr>`: An expression used to sort the source partition. -* `<init_lambda>`: A lambda function to initialize the state of session calculation. It has the signature `(TableRow())->State`. It's called once for the first (following the sorting order) element of the source partition. -* `<update_lambda>`: A lambda function to update the status of session calculation and define the session limits. It has the signature `(TableRow(), State)->Tuple<Bool, State>`. It's called for every element of the source partition, except the first one. The new value of state is calculated based on the current row of the table and the previous state. If the first element in the return tuple is `True`, then a new session starts from the _current_ row. The key of the new session is obtained by applying `<calculate_lambda>` to the second element in the tuple. -* `<calculate_lambda>`: A lambda function for calculating the session key (the "value" of SessionWindow() that is also accessible via SessionStart()). The function has the signature `(TableRow(), State)->SessionKey`. It's called for the first element in the partition (after `<init_lambda>`) and those elements for which `<update_lambda>` has returned `True` in the first element in the tuple. Please note that to start a new session, you should make sure that `<calculate_lambda>` has returned a value different from the previous session key. Sessions having the same keys are not merged. For example, if `<calculate_lambda>` returns the sequence `0, 1, 0, 1`, then there will be four different sessions. + +* `<order_expr>`: An expression used to sort the source partition +* `<init_lambda>`: A lambda function to initialize the state of session calculation. It has the signature `(TableRow())->State`. It's called once for the first (following the sorting order) element of the source partition +* `<update_lambda>`: A lambda function to update the status of session calculation and define the session limits. It has the signature `(TableRow(), State)->Tuple<Bool, State>`. It's called for every item of the source partition, except the first one. The new value of state is calculated based on the current row of the table and the previous state. If the first item in the return tuple is `True`, then a new session starts from the _current_ row. The key of the new session is obtained by applying `<calculate_lambda>` to the second item in the tuple. +* `<calculate_lambda>`: A lambda function for calculating the session key (the "value" of SessionWindow() that is also accessible via SessionStart()). The function has the signature `(TableRow(), State)->SessionKey`. It's called for the first item in the partition (after `<init_lambda>`) and those items for which `<update_lambda>` has returned `True` in the first item in the tuple. Please note that to start a new session, you should make sure that `<calculate_lambda>` has returned a value different from the previous session key. Sessions having the same keys are not merged. For example, if `<calculate_lambda>` returns the sequence `0, 1, 0, 1`, then there will be four different sessions. + +Using the extended version of SessionWindow, you can, for example, do the following: divide a partition into sessions, as in the SessionWindow use case with two arguments, but with the maximum session length limited by a certain constant: + +**Example** + +```sql +$max_len = 1000; is the maximum session length. +$timeout = 100; is the timeout (timeout_expr in a simplified version of SessionWindow). + +$init = ($row) -> (AsTuple($row.ts, $row.ts)); is the session status: tuple from 1) value of the temporary column ts in the session's first line and 2) in the current line +$update = ($row, $state) -> { + $is_end_session = $row.ts - $state.0 > $max_len OR $row.ts - $state.1 > $timeout; + $new_state = AsTuple(IF($is_end_session, $row.ts, $state.0), $row.ts); + return AsTuple($is_end_session, $new_state); +}; +$calculate = ($row, $state) -> ($row.ts); +SELECT + user, + session_start, + SessionStart() AS same_session_start, -- It's same as session_start + COUNT(*) AS session_size, + SUM(value) AS sum_over_session, +FROM my_table +GROUP BY user, SessionWindow(ts, $init, $update, $calculate) AS session_start +``` You can use SessionWindow in GROUP BY only once. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md index c16aa22f1a..8b7cb79647 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md @@ -3,37 +3,39 @@ ## Debugging and auxiliary settings {#debug} {% if feature_webui %} + ### `DirectRead` | Value type | Default | -| ---------- | ------- | -| Flag | false | +| --- | --- | +| Flag | false | An auxiliary setting for previewing tables in the [HTTP API](../../../interfaces/http.md) (both for the web interface and console client). {% endif %} ### `config.flags("ValidateUdf", "Lazy")` -| Value type | Default | -| ------------------------ | ------- | -| String: None/Lazy/Greedy | None | +| Value type | Default | +| --- | --- | +| String: None/Lazy/Greedy | None | Validating whether UDF results match the declared signature. The Greedy mode enforces materialization of lazy containers, although the Lazy mode doesn't. ### `{{ backend_name_lower }}.DefaultCluster` -| Value type | Default | -| ------------------------------ | ------- | -| A string with the cluster name | hahn | +| Value type | Default | +| --- | --- | +| A string with the cluster name | hahn | Selecting a cluster for calculations that don't access tables. ### `config.flags("Diagnostics")` | Value type | Default | -| ---------- | ------- | -| Flag | false | +| --- | --- | +| Flag | false | Getting diagnostic information from YQL as an additional result of a query. -{% endif %}
\ No newline at end of file +{% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md index 6b1810b588..dcc3cbe888 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md @@ -14,15 +14,15 @@ Redefinition of settings. **Examples** -``` yql +```yql PRAGMA AutoCommit; ``` -``` yql +```yql PRAGMA TablePathPrefix = "home/yql"; ``` -``` yql +```yql PRAGMA Warning("disable", "1101"); ``` @@ -35,4 +35,5 @@ For the full list of available settings, [see the table below](#pragmas). Unless otherwise specified, a pragma affects all the subsequent expressions up to the end of the module where it's used. If necessary and logically possible, you can change the value of this setting several times within a given query to make it different at different execution steps. There are also special scoped pragmas with the scope defined by the same rules as the scope of [named expressions](../../expressions.md#named-nodes). -Unlike scoped pragmas, regular pragmas can only be used in the global scope (not inside lambda functions, ACTION{% if feature_subquery %}, SUBQUERY{% endif %}, etc.).
\ No newline at end of file +Unlike scoped pragmas, regular pragmas can only be used in the global scope (not inside lambda functions, ACTION{% if feature_subquery %}, SUBQUERY{% endif %}, etc.). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md index ed9d9e2004..835e96ce40 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md @@ -27,12 +27,13 @@ Attach a set of files to the query by URL. Works similarly to adding multiple fi | One or two arguments: the file name and an optional URL | — | Static | Treat the specified attached file as a library from which you can do [IMPORT](../../export_import.md). The syntax type for the library is determined from the file extension: + * `.sql`: For the YQL dialect of SQL <span style="color: green;">(recommended)</span>. * `.yql`: For [s-expressions](/docs/s_expressions). Example with a file attached to the query: -``` yql +```yql PRAGMA library("a.sql"); IMPORT a SYMBOLS $x; SELECT $x; @@ -40,10 +41,20 @@ SELECT $x; If the URL is specified, the library is downloaded from the URL rather than from the pre-attached file as in the following example: -``` yql -PRAGMA library("a.sql","https://paste.company-team.ru/5618566/text"); +```yql +PRAGMA library("a.sql","https://paste.yandex-team.ru/5618566/text"); +IMPORT a SYMBOLS $x; +SELECT $x; +``` + +In this case, you can use text parameter value substitution in the URL: + +```yql +DECLARE $_ver AS STRING; -- "5618566" +PRAGMA library("a.sql","https://paste.yandex-team.ru/{$_ver}/text"); IMPORT a SYMBOLS $x; SELECT $x; ``` {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md index f1d38beb4c..d4f32ba914 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md @@ -1,6 +1,6 @@ ## Global {#pragmas} -### AutoCommit +### AutoCommit {#autocommit} | Value type | Default | | --- | --- | @@ -8,7 +8,7 @@ Automatically run [COMMIT](../../select.md#commit) after every statement. -### TablePathPrefix +### TablePathPrefix {#table-path-prefix} | Value type | Default | | --- | --- | @@ -21,13 +21,14 @@ SELECT * FROM test;` The prefix is not added if the table name is an absolute path (starts with /). -### Warning +### Warning {#warning} | Value type | Default | | --- | --- | | 1. Action<br>2. Warning code or "*" | — | Action: + * `disable`: Disable. * `error`: Treat as an error. * `default`: Revert to the default behavior. @@ -39,21 +40,23 @@ Example: `PRAGMA Warning("disable", "1101");` `PRAGMA Warning("default", "4503");` -In this case, all the warnings are treated as errors, except for the warning 1101(that will be disabled) and 4503(that will be processed by default, that is, remain a warning). Since warnings may be added in new YQL releases, use `PRAGMA Warning("error", "*");` with caution (at least cover such queries with autotests). +In this case, all the warnings are treated as errors, except for the warning `1101` (that will be disabled) and `4503` (that will be processed by default, that is, remain a warning). Since warnings may be added in new YQL releases, use `PRAGMA Warning("error", "*");` with caution (at least cover such queries with autotests). -List of codes for warnings and errors: https://a.yandex-team.ru/arc/trunk/arcadia/yql/core/issue/protos/issue_id.proto +{% include [issue_protos.md](issue_protos.md) %} {% if feature_mapreduce %} -### DqEngine + +### DqEngine {#dqengine} | Value type | Default | | --- | --- | -| String | "disable" | +| disable/auto/force string | "auto" | -When set to "auto", it enables a new compute engine. Computing is made, whenever possible, without creating map/reduce operations. +When set to "auto", it enables a new compute engine. Computing is made, whenever possible, without creating map/reduce operations. When the value is "force", computing is made by the new engine unconditionally. {% endif %} {% if feature_join %} + ### SimpleColumns {#simplecolumns} `SimpleColumns` / `DisableSimpleColumns` @@ -62,9 +65,9 @@ When set to "auto", it enables a new compute engine. Computing is made, whenever | --- | --- | | Flag | true | -When you use `SELECT foo.* FROM ... AS foo`, remove the `foo.` prefix from the names of the result columns. +When you use `SELECT foo.* FROM ... AS foo`, remove the `foo.` prefix from the names of the result columns. -It can be also used with [JOIN](../../join.md), but in this case it may fail in the case of a name conflict (that can be resolved by using [WITHOUT](../../select.md#without) and renaming columns). For JOIN in SimpleColumns mode, an implicit Coalesce is made for key columns: the query `SELECT * FROM T1 AS a JOIN T2 AS b USING(key)` in the SimpleColumns mode works same as `SELECT a.key ?? b.key AS key, ... FROM T1 AS a JOIN T2 AS b USING(key)` +It can be also used with a [JOIN](../../join.md), but in this case it may fail in the case of a name conflict (that can be resolved by using [WITHOUT](../../select.md#without) and renaming columns). For JOIN in SimpleColumns mode, an implicit Coalesce is made for key columns: the query `SELECT * FROM T1 AS a JOIN T2 AS b USING(key)` in the SimpleColumns mode works same as `SELECT a.key ?? b.key AS key, ... FROM T1 AS a JOIN T2 AS b USING(key)` ### CoalesceJoinKeysOnQualifiedAll @@ -77,6 +80,7 @@ It can be also used with [JOIN](../../join.md), but in this case it may fail in Controls implicit Coalesce for the key `JOIN` columns in the SimpleColumns mode. If the flag is set, the Coalesce is made for key columns if there is at least one expression in the format `foo.*` or `*` in SELECT: for example, `SELECT a.* FROM T1 AS a JOIN T2 AS b USING(key)`. If the flag is not set, then Coalesce for JOIN keys is made only if there is an asterisk '*' after `SELECT` ### StrictJoinKeyTypes + `StrictJoinKeyTypes` / `DisableStrictJoinKeyTypes` | Value type | Default | @@ -111,7 +115,8 @@ For more information about the `IN` behavior when operands include `NULL`s, see | Flag | false | Aligns the RANK/DENSE_RANK behavior with the standard if there are optional types in the window sort keys or in the argument of such window functions. It means that: -* The result type is always Uint64 rather than Uint64?. + +* The result type is always Uint64 rather than Uint64?; * NULLs in keys are treated as equal to each other (the current implementation returns NULL). You can explicitly select the old behavior by using the `DisableAnsiRankForNullableKeys` pragma. If no pragma is set, then a warning is issued and the old version works. @@ -122,6 +127,7 @@ You can explicitly select the old behavior by using the `DisableAnsiRankForNulla | Flag | false | Aligns the UNION ALL behavior with the standard if there is `ORDER BY/LIMIT/DISCARD/INSERT INTO` in the combined subqueries. It means that: + * `ORDER BY/LIMIT/INSERT INTO` are allowed only after the last subquery. * `DISCARD` is allowed only before the first subquery. * The specified operators apply to the `UNION ALL` result (unlike the current behavior when they apply only to the subquery). @@ -130,14 +136,15 @@ Aligns the UNION ALL behavior with the standard if there is `ORDER BY/LIMIT/DISC You can explicitly select the old behavior by using the `DisableAnsiOrderByLimitInUnionAll` pragma. If no pragma is set, then a warning is issued and the old version works. ### OrderedColumns {#orderedcolumns} + `OrderedColumns`/`DisableOrderedColumns` Output the [column order](../../select.md#orderedcolumns) in SELECT/JOIN/UNION ALL and preserve it when writing the results. The order of columns is undefined by default. ### PositionalUnionAll {#positionalunionall} -Enable the standard column-by-column execution for [UNION ALL](../../select.md#unionall). This automatically enables - [ordered columns](#orderedcolumns). +Enable the standard column-by-column execution for [UNION ALL](../../select.md#unionall). This automatically enables +[ordered columns](#orderedcolumns). ### RegexUseRe2 @@ -193,4 +200,5 @@ Increasing the limit on the number of dimensions in [GROUP BY](../../group_by.md Use this option with care, because the computational complexity of the query grows exponentially with the number of dimensions. -{% endif %}
\ No newline at end of file +{% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/issue_protos.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/issue_protos.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/issue_protos.md diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md index 94569ebdf9..1cca5678d8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md @@ -10,4 +10,5 @@ An experimental pragma that allows you to reduce the isolation level of the current YDB transaction. -{% endif %}
\ No newline at end of file +{% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md index 1a6ae1786a..50058c132f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md @@ -25,3 +25,4 @@ Strict mode control in all Yson UDF calls, including implicit calls. If the valu | Flag | false | An inverted version of `yson.Strict`. If the value is omitted or is `"true"`, it disables the strict mode. If the value is `"false"`, it enables the strict mode. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md index df86cbf2ae..4a4cf40498 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md @@ -6,8 +6,9 @@ As in case of `ORDER BY`, it supports setting the sort order using the keywords **Examples:** -``` yql +```yql SELECT key || "suffix" as key, -CAST(subkey as Int32) as subkey FROM my_table ASSUME ORDER BY key, subkey DESC; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md index 073718ef2b..09bc26c7f5 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md @@ -6,10 +6,11 @@ It can be used in combination with other operations to obtain other effect. **Examples:** -``` yql +```yql SELECT "Hello, world!"; ``` ```yql SELECT 2 + 2; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md index c6c4a6ecff..d6768bea13 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md @@ -4,19 +4,21 @@ The standard SQL is sensitive to the order of columns in projections (that is, i This applies, for example, to [UNION ALL](#unionall) and positional [ORDER BY](#orderby) (ORDER BY ordinal). The column order is ignored in YQL by default: + * The order of columns in the output tables and query results is undefined * The data scheme of the `UNION ALL` result is output by column names rather than positions If you enable `PRAGMA OrderedColumns;`, the order of columns is preserved in the query results and is derived from the order of columns in the input tables using the following rules: + * `SELECT`: an explicit column enumeration dictates the result order. -* `SELECT` with an asterisk (`SELECT * FROM ...`) inherits the order from its input; -{% if feature_join %}* The order of columns after [JOIN](../../join.md): First output the left-hand columns, then the right-hand ones. If the column order in any of the sides in the `JOIN` output is undefined, the column order in the result is also undefined.{% endif %} +* `SELECT` with an asterisk (`SELECT * FROM ...`) inherits the order from its input. {% if feature_join %}* The order of columns after [JOIN](../../join.md): First output the left-hand columns, then the right-hand ones. If the column order in any of the sides in the `JOIN` output is undefined, the column order in the result is also undefined.{% endif %} * The order in `UNION ALL` depends on the [UNION ALL](#unionall) execution mode. * The column order for [AS_TABLE](#as_table) is undefined. -{% note warning %} +{% note warning "Attention" %} In the YT table schema, key columns always precede non-key columns. The order of key columns is determined by the order of the composite key. When `PRAGMA OrderedColumns;` is enabled, non-key columns preserve their output order. {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md index fae8b2d43a..0f42606613 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md @@ -7,10 +7,11 @@ To commit in the same way automatically after each expression in the query, you **Examples:** -``` yql +```yql INSERT INTO result1 SELECT * FROM my_table; INSERT INTO result2 SELECT * FROM my_table; COMMIT; -- result2 will already include the SELECT contents from the second line: INSERT INTO result3 SELECT * FROM result2; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md index 79cb83ff04..4c1a31cc9b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md @@ -10,9 +10,10 @@ Applying `DISTINCT` to calculated values is not currently implemented. For this **Example** -``` yql +```yql SELECT DISTINCT value -- only unique values from the table FROM my_table; ``` -The `DISTINCT` keyword can also be used to apply [aggregate functions](../../../builtins/aggregation.md) only to distinct values. For more information, see the documentation for [GROUP BY](../../group_by.md).
\ No newline at end of file +The `DISTINCT` keyword can also be used to apply [aggregate functions](../../../builtins/aggregation.md) only to distinct values. For more information, see the documentation for [GROUP BY](../../group_by.md). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md index a3b1bdc45b..f41354bb83 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md @@ -3,17 +3,18 @@ The `SELECT` query result is calculated as follows: * Determine the set of input tables by evaluating the [FROM](#from) clauses. -* Apply [SAMPLE](#sample) / [TABLESAMPLE](#sample) to input tables. -* Execute [FLATTEN COLUMNS](../../flatten.md#flatten-columns) or [FLATTEN BY](../../flatten.md); the aliases set in `FLATTEN BY` become visible after this point. +* Apply [SAMPLE](#sample)/[TABLESAMPLE](#sample) to input tables. +* Execute [FLATTEN COLUMNS](../../flatten.md#flatten-columns) or [FLATTEN BY](../../flatten.md); aliases set in `FLATTEN BY` become visible after this point. {% if feature_join %}* Execute every [JOIN](../../join.md).{% endif %} * Add to (or replace in) the data the columns listed in [GROUP BY ... AS ...](../../group_by.md). * Execute [WHERE](#where) — Discard all the data mismatching the predicate. * Execute [GROUP BY](../../group_by.md), evaluate aggregate functions. * Apply the filter [HAVING](../../group_by.md#having). -{% if feature_window_functions %} * Evaluate [window functions](../../window.md).{% endif %} +{% if feature_window_functions %} * values of [window functions](../../window.md) are calculated.{% endif %} * Evaluate expressions in `SELECT`. * Assign names set by aliases to expressions in `SELECT`. * Apply top-level [DISTINCT](#distinct) to the resulting columns. * Execute similarly every subquery inside [UNION ALL](#unionall), combine them (see [PRAGMA AnsiOrderByLimitInUnionAll](../../pragma.md#pragmas)). * Perform sorting with [ORDER BY](#order-by). -* Apply [OFFSET and LIMIT](#limit-offset) to the result.
\ No newline at end of file +* Apply [OFFSET and LIMIT](#limit-offset) to the result. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md index 9547f62a59..436de60509 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md @@ -27,7 +27,7 @@ Use FOLDER with attributes containing large values with caution (`schema` could **Examples:** -``` yql +```yql USE hahn; $table_paths = ( @@ -40,4 +40,5 @@ $table_paths = ( ); SELECT COUNT(*) FROM EACH($table_paths); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md index ff42de5d73..65b9356c6e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md @@ -6,16 +6,17 @@ Data source for `SELECT`. The argument can accept the table name, the result of **Examples** -``` yql +```yql SELECT key FROM my_table; ``` -``` yql +```yql SELECT * FROM (SELECT value FROM my_table); ``` -``` yql +```yql $table_name = "my_table"; SELECT * FROM $table_name; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md index 9db0c5b1d1..c5fe4b8f2b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md @@ -2,11 +2,11 @@ Accessing named expressions as tables using the `AS_TABLE` function. -`AS_TABLE($variable)` lets you use the value of `$variable` as the data source for the query. In this case, the variable `$variable` must have the type `List<Struct<...>>`. +`AS_TABLE($variable)` lets you use the value of `$variable` as the data source for the query. In this case, the variable `$variable` must have the type `List>>`. **Example** -``` yql +```yql $data = AsList( AsStruct(1u AS Key, "v1" AS Value), AsStruct(2u AS Key, "v2" AS Value), @@ -14,3 +14,4 @@ $data = AsList( SELECT Key, Value FROM AS_TABLE($data); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md index 068df845a1..386e5ca429 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md @@ -4,13 +4,14 @@ An inverted format, first specifying the data source and then the operation. **Examples** -``` yql +```yql FROM my_table SELECT key, value; ``` -``` yql +```yql FROM a_table AS a JOIN b_table AS b USING (key) SELECT *; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md index 3114337889..b084524013 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md @@ -18,7 +18,7 @@ The following functions are defined for these purposes: ```FILTER(`prefix`, `callable`, `suffix`, `view`)```: The `callable` argument must be a callable expression with the `(String)->Bool` signature that will be called for each table/subdirectory in the prefix folder. The query will only include those tables for which the callable value returned `true`. It is most convenient to use, as callable values, [lambda functions](../../expressions.md#lambda){% if yql == true %} or user-defined functions (UDF) written in [Python](../../../udf/python.md) or [JavaScript](../../../udf/javascript.md){% endif %}. -{% note warning %} +{% note warning "Attention" %} All of the above functions don't guarantee the order of the table union. @@ -36,7 +36,7 @@ To get the name of the source table from which you originally obtained each row, **Examples:** -``` yql +```yql USE some_cluster; SELECT * FROM CONCAT( `table1`, @@ -44,7 +44,7 @@ SELECT * FROM CONCAT( `table3`); ``` -``` yql +```yql USE some_cluster; $indices = ListFromRange(1, 4); $tables = ListMap($indices, ($index) -> { @@ -53,19 +53,19 @@ $tables = ListMap($indices, ($index) -> { SELECT * FROM EACH($tables); -- Identical to the previous example ``` -``` yql +```yql USE some_cluster; SELECT * FROM RANGE(`my_folder`); ``` -``` yql +```yql SELECT * FROM some_cluster.RANGE( -- You can specify the cluster before the function name `my_folder`, `from_table`, `to_table`); ``` -``` yql +```yql USE some_cluster; SELECT * FROM RANGE( `my_folder`, @@ -74,7 +74,7 @@ SELECT * FROM RANGE( `my_table`); ``` -``` yql +```yql USE some_cluster; SELECT * FROM RANGE( `my_folder`, @@ -84,7 +84,7 @@ SELECT * FROM RANGE( `my_view`); ``` -``` yql +```yql USE some_cluster; SELECT * FROM LIKE( `my_folder`, @@ -92,7 +92,7 @@ SELECT * FROM LIKE( ); ``` -``` yql +```yql USE some_cluster; SELECT * FROM REGEXP( `my_folder`, @@ -100,7 +100,7 @@ SELECT * FROM REGEXP( ); ``` -``` yql +```yql $callable = ($table_name) -> { return $table_name > "2017-03-13"; }; @@ -110,4 +110,5 @@ SELECT * FROM FILTER( `my_folder`, $callable ); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md index a4433f0429..17de5253c0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md @@ -2,21 +2,22 @@ `LIMIT` limits the output to the specified number of rows. By default, the output is not restricted. -`OFFSET` specifies the offset from the beginning (in rows). By default, it's zero. +`OFFSET`: specifies the offset from the beginning (in rows). By default, it's zero. **Examples** -``` yql +```yql SELECT key FROM my_table LIMIT 7; ``` -``` yql +```yql SELECT key FROM my_table LIMIT 7 OFFSET 3; ``` -``` yql +```yql SELECT key FROM my_table LIMIT 3, 7; -- equivalent to the previous example -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md index 22f7504d4c..2390f02238 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md @@ -3,6 +3,7 @@ Sorting the `SELECT` result using a comma-separated list of sorting criteria. As a criteria, you can use a column value or an expression on columns. Ordering by column sequence number is not supported (`ORDER BY N`where `N` is a number). Each criteria can be followed by the sorting direction: + - `ASC`: Sorting in the ascending order. Applied by default. - `DESC`: Sorting in the descending order. @@ -10,11 +11,13 @@ Multiple sorting criteria will be applied left-to-right. **Example** -``` yql +```yql SELECT key, string_column FROM my_table ORDER BY key DESC, LENGTH(string_column) ASC; ``` + {% if feature_window_functions %} -You can use `ORDER BY` also for [window functions](../../window.md). +You can also use `ORDER BY` for [window functions](../../window.md). {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md index 2281f0148f..dc706d2c75 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md @@ -22,20 +22,21 @@ In the `BERNOULLI` mode, if the `REPEATABLE` keyword is added, the seed is mixe **Examples:** -``` yql +```yql SELECT * FROM my_table TABLESAMPLE BERNOULLI(1.0) REPEATABLE(123); -- one percent of the table ``` -``` yql +```yql SELECT * FROM my_table TABLESAMPLE SYSTEM(1.0); -- about one percent of the table ``` -``` yql +```yql SELECT * FROM my_table SAMPLE 1.0 / 3; -- one-third of the table ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md index 594752820c..30da15826d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md @@ -2,7 +2,7 @@ To make a `SELECT` by secondary index statement, use the following: -``` yql +```yql SELECT * FROM TableName VIEW IndexName WHERE … @@ -12,7 +12,7 @@ SELECT * * Select all the fields from the `series` table using the `views_index` index with the `views >=someValue` criteria: - ``` yql + ```yql SELECT series_id, title, info, release_date, views, uploaded_user_id FROM series VIEW views_index WHERE views >= someValue @@ -20,10 +20,11 @@ SELECT * * [`JOIN`](../../join.md) the `series` and `users` tables on the `userName` field using the `users_index` and `name_index` indexes, respectively: - ``` yql + ```yql SELECT t1.series_id, t1.title FROM series VIEW users_index AS t1 INNER JOIN users VIEW name_index AS t2 ON t1.uploaded_user_id == t2.user_id WHERE t2.name == userName; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md index 4d32c0998f..0117bfd058 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md @@ -8,7 +8,7 @@ This feature lets you ignore conflicts in paths to temporary tables between para **Examples:** -``` yql +```yql INSERT INTO @my_temp_table SELECT * FROM my_input_table ORDER BY value; @@ -21,7 +21,7 @@ SELECT * FROM @my_temp_table WHERE value = "456"; Temporary table names can use [named expressions](../../expressions.md#named-nodes): -``` yql +```yql $tmp_name = "my_temp_table"; INSERT INTO @$tmp_name @@ -30,4 +30,5 @@ SELECT 1 AS one, 2 AS two, 3 AS three; COMMIT; SELECT * FROM @$tmp_name; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md index 4f384b8151..505dc1f182 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md @@ -12,6 +12,7 @@ The order of output columns in this mode is equal to the largest common prefix o If the largest common prefix is empty (for example, if the order isn't specified for one of the inputs), then the output order is undefined. In the "by position" mode, the output of the resulting data schema uses the following rules: + * All inputs must have equal number of columns * The order of columns must be defined for all inputs * The names of the resulting columns must match the names of columns in the first table @@ -21,7 +22,7 @@ The order of the output columns in this mode is the same as the order of columns **Examples** -``` yql +```yql SELECT 1 AS x UNION ALL SELECT 2 AS y @@ -29,12 +30,13 @@ UNION ALL SELECT 3 AS z; ``` -In the default mode, this query returns a selection with three columns x, y, and z. When `PRAGMA PositionalUnionAll;` is enabled, the selection includes only the x column. +In the default mode, this query returns a selection with three columns x, y, and z. When `PRAGMA PositionalUnionAll;` is enabled, the selection only includes the x column. -``` yql +```yql PRAGMA PositionalUnionAll; SELECT 1 AS x, 2 as y UNION ALL SELECT * FROM AS_TABLE([<|x:3, y:4|>]); -- error: the order of columns in AS_TABLE is undefined ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md index acd5bdb0ce..7c41e08bf8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md @@ -2,3 +2,4 @@ * If a column wasn't present in all the input tables, then it's automatically assigned the [optional data type](../../../types/optional.md) (that can accept `NULL`). * If a column in different input tables had different types, then the shared type (the broadest one) is output. * If a column in different input tables had a heterogeneous type, for example, string and numeric, an error is raised. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md index 36364425aa..193e6d526b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md @@ -15,8 +15,9 @@ If the meta attributes of the table specify an automatic UDF call to convert raw **Examples:** -``` yql +```yql USE some_cluster; SELECT * FROM my_table VIEW my_view; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md index 53c899dd7c..3c5e5bf18b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md @@ -4,7 +4,8 @@ Filtering rows in the `SELECT` result based on a condition. **Example** -``` yql +```yql SELECT key FROM my_table WHERE value > 0; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md index 1ca172d97b..b99e65fd32 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md @@ -18,25 +18,26 @@ If you use the `SCHEMA` hint, then with the table functions [EACH](#each), [RANG **Examples:** -``` yql +```yql SELECT key FROM my_table WITH INFER_SCHEMA; ``` -``` yql +```yql $s = (SELECT COUNT(*) FROM my_table WITH XLOCK); INSERT INTO my_table WITH TRUNCATE SELECT EvaluateExpr($s) AS a; ``` -``` yql +```yql SELECT key, value FROM my_table WITH SCHEMA Struct<key:String, value:Int32>; ``` -``` yql +```yql SELECT key, value FROM my_table WITH COLUMNS Struct<value:Int32?>; ``` -``` yql +```yql SELECT key, value FROM EACH($my_tables) WITH SCHEMA Struct<key:String, value:List<Int32>>; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md index 791aac43b3..f61e5e9625 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md @@ -4,12 +4,13 @@ Excluding columns from the result of `SELECT *`. **Examples** -``` yql +```yql SELECT * WITHOUT foo, bar FROM my_table; ``` -``` yql +```yql PRAGMA simplecolumns; SELECT * WITHOUT t.foo FROM my_table AS t CROSS JOIN (SELECT 1 AS foo) AS v; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md index d9f77985c4..f9e2f51a7a 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md @@ -29,7 +29,7 @@ Even if the list of parameters in the subquery template definition is empty, whe {% if feature_mapreduce %} In some cases, instead of `DEFINE SUBQUERY` it's more convenient to use an equivalent [lambda function](../expressions.md#lambda). -In this case, the lambda function must accept, as the first argument, the special object called `world` that passes dependencies to make certain PRAGMA or COMMIT statements visible at the query template's point of use. Make also sure to pass this object as the first argument along with the other arguments (if any) to other query templates, if you use them in your lambda function. +In this case, the lambda function must accept, as the first argument, the special object called `world` that passes dependencies to make certain PRAGMA or COMMIT statements visible at the query template's point of use. Also, make sure to pass this object as the first argument along with the other arguments (if any) to other query templates, if you use them in your lambda function. The return value of the lambda function must have the structure list type (output table) or a list of variants over a tuple of structures (multiple output tables). In the latter case, the following unpacking is usually used at the query template's point of use: ```yql @@ -179,3 +179,24 @@ $s = SubqueryExtendFor([1,2,3],$sub); PROCESS $s(); ``` +## Adding sorting to the SubqueryOrderBy template or indicating the presence of this SubqueryAssumeOrderBy + +The functions take the following arguments: + +* A subquery template without parameters. +* A list of pairs (string indicating the column name and Boolean value: True for sorting in ascending order or False for sorting in descending order). + +And they build a new query template without parameters where sorting is performed or a comment on the use of sorting is added to the result. To use the resulting query template, call the `PROCESS` function, since, when using a `SELECT`, sorting is ignored. + +**Examples:** + +```yql +DEFINE SUBQUERY $sub() as + SELECT * FROM (VALUES (1,'c'), (1,'a'), (3,'b')) AS a(x,y); +end define; + +$sub2 = SubqueryOrderBy($sub, [('x',false), ('y',true)]); + +PROCESS $sub2(); +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md index 775f009a08..53b3c767f1 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md @@ -5,7 +5,7 @@ Adds or updates multiple rows in a table based on primary key matching. Missing {% if feature_mapreduce %} The table is searched by name in the database specified by the [USE](../use.md) operator.{% endif %} {% if feature_replace %} -`UPSERT` and [`REPLACE`](../replace_into.md) are data modification operations that don't require a pre-fetch and run faster and cheaper than other operations because of that. +`UPSERT` and [`REPLACE`](../replace_into.md) are data modification operations that don't require a prefetch and run faster and cheaper than other operations because of that. {% else %} `UPSERT` is the only data modification operation that doesn't require prefetching and runs faster and cheaper than other operations because of that. {% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/window.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/window.md index f406bebf29..2176228d10 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/window.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/window.md @@ -83,7 +83,7 @@ The set of rows in the window frame may change depending on which row is the cur Setting `UNBOUNDED PRECEDING` as the frame start means "from the first partition row" and `UNBOUNDED FOLLOWING` as the frame end — "up to the last partition row". Setting `CURRENT ROW` means "from/to the current row". -If no `frame_definition` is specified, a set of rows to be included in the window frame depends on whether there is `ORDER BY` in the `window_definition`. +If no `frame_definition` is specified, a set of rows to be included in the window frame depends on whether there is `ORDER BY` in the `window_definition`.
Namely, if there is `ORDER BY`, then `ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW` is implicitly assumed. If none, then `ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING`. Further, depending on the specific window function, it's calculated either based on the set of rows in the partition or the set of rows in the window frame. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md index a381efbc22..e3bce9d7dd 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md @@ -29,3 +29,4 @@ {% include [x](_includes/expressions/lambda.md) %} {% include [x](_includes/expressions/items-access.md) %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md index af923dcb90..3594e5b469 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md @@ -53,3 +53,4 @@ {% if feature_mapreduce %} * [EXPORT and IMPORT](export_import.md) {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md index 951d674a5c..ed23eefd90 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md @@ -1,4 +1,4 @@ -# Синтаксис SELECT +# SELECT syntax <!-- File split by includable blocks as part of YQL docs preparation for YQL/YDB opensource --> @@ -6,11 +6,7 @@ {% include [x](_includes/select/from.md) %} -{% if feature_secondary_index %} - - {% include [x](_includes/select/secondary_index.md) %} - -{% endif %} +{% include [x](_includes/select/secondary_index.md) %} {% include [x](_includes/select/with.md) %} @@ -51,3 +47,4 @@ {% include [x](_includes/select/temporary_table.md) %} {% include [x](_includes/select/from_as_table.md) %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml index 3f26c627cc..23ec63ac5e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml @@ -1,31 +1,31 @@ items: -- { name: Overview, href: index.md } -- { name: Lexical structure, href: lexer.md } -- { name: Expressions, href: expressions.md } -- { name: USE, href: use.md, when: feature_mapreduce } -- { name: SELECT, href: select.md } -- { name: VALUES, href: values.md } -- { name: SELECT STREAM, href: select_stream.md, when: feature_mapreduce and process_command == "PROCESS STREAM" } -- { name: CREATE TABLE, href: create_table.md } -- { name: ALTER TABLE, href: alter_table.md, when: feature_map_tables } -- { name: DROP TABLE, href: drop_table.md } -- { name: INSERT, href: insert_into.md } -- { name: UPDATE, href: update.md, when: feature_map_tables } -- { name: DELETE, href: delete.md, when: feature_map_tables } -- { name: REPLACE, href: replace_into.md, when: feature_replace } -- { name: UPSERT, href: upsert_into.md, when: feature_upsert } -- { name: GROUP BY, href: group_by.md } -- { name: JOIN, href: join.md, when: feature_join } -- { name: WINDOW, href: window.md, when: feature_window_functions } -- { name: FLATTEN, href: flatten.md } -- { name: ACTION, href: action.md } -- { name: SUBQUERY, href: subquery.md, when: feature_subquery } -- { name: DISCARD, href: discard.md } -- { name: INTO RESULT, href: into_result.md } -- { name: PROCESS STREAM, href: process.md, when: feature_mapreduce and process_command == "PROCESS STREAM" } -- { name: PROCESS, href: process.md, when: feature_mapreduce and process_command == "PROCESS" } -- { name: REDUCE, href: reduce.md, when: feature_mapreduce and reduce_command == "REDUCE" } -- { name: PRAGMA, href: pragma.md } -- { name: DECLARE, href: declare.md } -- { name: EXPORT and IMPORT, href: export_import.md, when: feature_mapreduce } -- { name: Unsupported statements, href: not_yet_supported.md } +- { name: Overview, href: index.md } +- { name: Lexical structure, href: lexer.md } +- { name: Expressions, href: expressions.md } +- { name: ACTION, href: action.md } +- { name: ALTER TABLE, href: alter_table.md, when: feature_map_tables } +- { name: CREATE TABLE, href: create_table.md } +- { name: DECLARE, href: declare.md } +- { name: DELETE, href: delete.md, when: feature_map_tables } +- { name: DISCARD, href: discard.md } +- { name: DROP TABLE, href: drop_table.md } +- { name: GROUP BY, href: group_by.md } +- { name: EXPORT and IMPORT, href: export_import.md, when: feature_mapreduce } +- { name: FLATTEN, href: flatten.md } +- { name: INSERT, href: insert_into.md } +- { name: INTO RESULT, href: into_result.md } +- { name: JOIN, href: join.md, when: feature_join } +- { name: PRAGMA, href: pragma.md } +- { name: PROCESS STREAM, href: process.md, when: feature_mapreduce and process_command == "PROCESS STREAM" } +- { name: PROCESS, href: process.md, when: feature_mapreduce and process_command == "PROCESS" } +- { name: REDUCE, href: reduce.md, when: feature_mapreduce and reduce_command == "REDUCE" } +- { name: REPLACE, href: replace_into.md, when: feature_replace } +- { name: SELECT, href: select.md } +- { name: SELECT STREAM, href: select_stream.md, when: feature_mapreduce and process_command == "PROCESS STREAM" } +- { name: SUBQUERY, href: subquery.md, when: feature_subquery } +- { name: UPDATE, href: update.md, when: feature_map_tables } +- { name: UPSERT, href: upsert_into.md, when: feature_upsert } +- { name: USE, href: use.md, when: feature_mapreduce } +- { name: VALUES, href: values.md } +- { name: WINDOW, href: window.md, when: feature_window_functions } +- { name: Unsupported statements, href: not_yet_supported.md } diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md index cd112ee587..b89466dfc8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md @@ -27,8 +27,8 @@ SELECT ### Rules for List/Dict -* To create a list, `CAST` is applied to each element in the source list to cast it to the target type. -* If the target element type is non-optional and `CAST` on the element might fail, then such casting is discarded. In this case, the resulting list might be shorter or even empty if every casting failed. +* To create a list, `CAST` is applied to each item in the source list to cast it to the target type. +* If the target item type is non-optional and `CAST` on the item might fail, then such casting is discarded. In this case, the resulting list might be shorter or even empty if every casting failed. * For dictionaries, the casting is totally similar to lists, with `CAST` being applied to keys and values. ```yql @@ -42,7 +42,7 @@ SELECT ### Rules for Struct/Tuple -* A structure or tuple is created by applying `CAST` to each element of the source type to cast it to an element with the same name or target type index. +* A structure or tuple is created by applying `CAST` to each item of the source type to cast it to an item with the same name or target type index. * If some field is missing in the target type, it's simply discarded. * If some field is missing in the source value type, then it can be added only if it's optional and accepts the `NULL` value. * If some field is non-optional in the target type, but its casting might fail, then `CAST` adds Optional to the structure or tuple level and might return `NULL` for the entire result. @@ -53,7 +53,7 @@ SELECT CAST((-2, 0) AS Tuple<Uint16, Utf8>), -- null CAST((3, 4) AS Tuple<Uint16, String>), -- (3, "4"): the type is Tuple<Uint16, String>? CAST(("4",) AS Tuple<Uint16, String?>), -- (4, null) - CAST((5, 6, null) AS Tuple<Uint8?>); -- (5,): the elements were removed. + CAST((5, 6, null) AS Tuple<Uint8?>); -- (5,): the items were removed. SELECT -- One field was removed and one field was added: ("three":null, "two": "42") CAST(<|one:"8912", two:42|> AS Struct<two:Utf8, three:Date?>); diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md index 69dc03e12a..b9146a071d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md @@ -4,7 +4,7 @@ | ------------ | ---------------- | ------------- | | List | `List<Type>`,</br>`List<Int32>` | A variable-length sequence consisting of same-type elements. | | Dictionary | `Dict<KeyType, ValueType>`,</br>`Dict<String,Int32>` | Set of key-value pairs with a fixed type of keys and values. | -| Set | `Set<KeyType>`,</br>`Set<String>` | A set of elements with a fixed type is a special case of a dictionary with the `Void` type of values. | +| Set | `Set<KeyType>`,</br>`Set<String>` | A set of elements with a fixed type is a special case of a dictionary with the `Void` value type. | | Tuple | `Tuple<Type1, ..., TypeN>`,</br>`Tuple<Int32,Double>` | Set of unnamed fixed-length elements with types specified for all elements. | | Structure | `Struct<Name1:Type1, ..., NameN:TypeN>`,</br> `Struct<Name:String,Age:Int32>` | A set of named fields with specified value types, fixed at query start (must be data-independent). | | Stream | `Stream<Type>`,</br> `Stream<Int32>` | Single-pass iterator by same-type values, not serializable | diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_datetime.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_datetime.md index 332cc3a303..8f96cc3504 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_datetime.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_datetime.md @@ -1,9 +1,9 @@ | Type | Description | Notes | | ----- | ----- | ----- | -| `Date` | Date, precision to the day | -| `Datetime` | Date/time, precision to the second | -| `Timestamp` | Date/time, precision to the microsecond | -| `Interval` | Time interval, precision to the microsecond, <br/>Valid values: must not exceed 24 hours. | {% if feature_map_tables %}Can't be used in the primary key{% endif %} | +| `Date` | Date, precision to the day | Range of values for all time types except `Interval`: From 00:00 01.01.1970 to 00:00 01.01.2106. Internal `Date` representation: Unsigned 16-bit integer | +| `Datetime` | Date/time, precision to the second | Internal representation: Unsigned 32-bit integer | +| `Timestamp` | Date/time, precision to the microsecond | Internal representation: Unsigned 64-bit integer | +| `Interval` | Time interval (signed), precision to microseconds | Value range: From -136 years to +136 years. Internal representation: Signed 64-bit integer. {% if feature_map_tables %}Can't be used in the primary key{% endif %} | | `TzDate` | Date with time zone label, precision to the day | Not supported in table columns | | `TzDateTime` | Date/time with time zone label, precision to the second | Not supported in table columns | | `TzTimestamp` | Date/time with time zone label, precision to the microsecond | Not supported in table columns | diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md index e5395df48b..43f545cb85 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md @@ -14,6 +14,6 @@ | `Decimal` | A real number with the specified precision, up to 35 decimal digits | {% if feature_map_tables %}When used in table columns, precision is fixed: Decimal (22,9).</br>Can't be used in the primary key{% endif %} | {% if feature_map_tables %} -`DyNumber` | A binary representation of a real number with an accuracy of up to 38 digits.<br/>Acceptable values: positive numbers from 1×10<sup>-130</sup> up to 1×10<sup>126</sup>–1, negative numbers from -1×10<sup>126</sup>–1 up to -1×10<sup>-130</sup>, and 0.<br/>Compatible with the `Number` type in AWS DynamoDB. It's not recommended for {{ backend_name_lower }}-native applications. | +`DyNumber` | A binary representation of a real number with an accuracy of up to 38 digits.<br/>Acceptable values: positive numbers from 1×10<sup>-130</sup> up to 1×10<sup>126</sup>–1, negative numbers from -1×10<sup>126</sup>–1 to -1×10<sup>-130</sup>, and 0.<br/>Compatible with the `Number` type in AWS DynamoDB. It's not recommended for {{ backend_name_lower }}-native applications. | {% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/json.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/json.md index e0be51a469..c449e86bb0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/json.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/json.md @@ -28,7 +28,7 @@ Unsigned integer types. Real 4-byte number. -* Type in JSON — `number`. +* Type in JSON: `number`. * Sample {{ backend_name }} value: `0.12345679`. * Sample JSON value: `0.12345679`. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/optional.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/optional.md index 4cf2d52056..71bb96d346 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/optional.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/optional.md @@ -39,12 +39,12 @@ SELECT True OR NULL, -- Just(True) (works the same way as True OR <unknown value of type Bool>) False AND NULL, -- Just(False) True AND NULL, -- NULL (to be more precise, Nothing<Bool?> – <unknown value of type Bool>) - NULL OR NOT NULL, -- NULL (all NULLs are considered "different") + NULL OR NOT NULL, -- NULL (all NULLs are considered "different") 1 + NULL, -- NULL (Nothing<Int32?>) - the result of adding 1 together with - -- an unknown Int value) - 1 == NULL, -- NULL (the result of comparing 1 with an unknown Int value) - (1, NULL) == (1, 2), -- NULL (composite elements are compared by component comparison - -- using `AND') + -- an unknown Int value) + 1 == NULL, -- NULL (the result of comparing 1 with an unknown Int value) + (1, NULL) == (1, 2), -- NULL (composite elements are compared by component comparison + -- using `AND') (2, NULL) == (1, 3), -- Just(False) (the expression is equivalent to 2 == 1 AND NULL == 3) ``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md index 2b9a8b401b..17791bc2f9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md @@ -42,11 +42,9 @@ Explicit casting using [CAST](../../syntax/expressions.md#cast): | **Timestamp** | No | Yes | Yes | Yes | Yes | No | | **Interval** | No | Yes | Yes | Yes | Yes | No | -<sup>1</sup> `True` is converted to `1`, `False` is converted to `0`. - +<sup>1</sup> `True` is converted to `1` and `False` to `0`. <sup>2</sup> Any value other than `0` is converted to `True`, `0` is converted to `False`. - -<sup>3</sup> his is possible only in the case of a non-negative value. +<sup>3</sup> Possible only in the case of a non-negative value. <sup>4</sup> Using the built-in function [Yson::ConvertTo](../../udf/list/yson.md#ysonconvertto). ### Converting to date and time data types diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/string.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/string.md index d574f82672..8298c9d2f9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/string.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/string.md @@ -68,7 +68,7 @@ Functions for ASCII strings: The third argument includes the following parameters: - DelimeterString:Bool? — treating a delimiter as a string (true, by default) or a set of characters "any of" (false) - SkipEmpty:Bool? — whether to skip empty strings in the result, is false by default - - Limit:Uint64? — Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last element + - Limit:Uint64? — Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last item * ```String::JoinFromList(List<String>{Flags:AutoMap}, String) -> String``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md index 4553f311e9..327cf1443c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md @@ -5,7 +5,7 @@ Functions for Unicode strings. **List of functions** * ```Unicode::IsUtf(String) -> Bool``` -Checks whether a string is a valid UTF-8 sequence. For example, the string ```"\xF0"``` isn't a valid UTF-8 sequence, but the string ```"\xF0\x9F\x90\xB1"``` correctly describes a UTF-8 cat emoji. +Checks whether a string is a valid UTF-8 sequence. For example, the string ```"\xF0"``` isn't a valid UTF-8 sequence, but the ```"\xF0\x9F\x90\xB1"``` string correctly describes a UTF-8 cat emoji. * ```Unicode::GetLength(Utf8{Flags:AutoMap}) -> Uint64``` Returns the length of a utf-8 string in unicode code points. Surrogate pairs are counted as one character. @@ -15,7 +15,7 @@ Returns the length of a utf-8 string in unicode code points. Surrogate pairs are * ```Unicode::RFind(Utf8{Flags:AutoMap}, Utf8, [Uint64?]) -> Uint64?``` * ```Unicode::Substring(Utf8{Flags:AutoMap}, from:Uint64?, len:Uint64?) -> Utf8``` -Returns a substring starting with ```from``` with the length of ```len``` characters. If the ```len``` argument is omitted, the substring is moved to the end of the source string. +Returns a substring starting with ```from``` with the length of ```len``` characters. If the ```len``` argument is omitted, the substring is taken to the end of the source string. * The ```Unicode::Normalize...``` functions convert the passed UTF-8 string to a [normalization form](https://unicode.org/reports/tr15/#Norm_Forms): * ```Unicode::Normalize(Utf8{Flags:AutoMap}) -> Utf8``` -- NFC @@ -37,7 +37,7 @@ Returns a substring starting with ```from``` with the length of ```len``` charac Arguments: ```input```, ```find```, ```replacement```. Replaces all occurrences of the ```find``` string in the ```input``` with ```replacement```. * ```Unicode::ReplaceFirst(Utf8{Flags:AutoMap}, Utf8, Utf8) -> Utf8``` - Arguments: ```input```, ```findSymbol```, ```replacementSymbol```. Replaces the first occurrence of the ```findSymbol``` character in the ```input``` with ```replacementSymbol```. The character can't be a surrogate pair. + Arguments: ```input```, ```findSymbol```, ```replacementSymbol```. Replaces the first occurrence of the ```findSymbol``` character in the ```input``` with ```replacementSymbol```. The character can't be a surrogate pair. * ```Unicode::ReplaceLast(Utf8{Flags:AutoMap}, Utf8, Utf8) -> Utf8``` Arguments: ```input```, ```findSymbol```, ```replacementSymbol```. Replaces the last occurrence of the ```findSymbol``` character in the ```input``` with ```replacementSymbol```. The character can't be a surrogate pair. @@ -69,7 +69,7 @@ Returns a substring starting with ```from``` with the length of ```len``` charac The third argument includes the following parameters: - DelimeterString:Bool? — treating a delimiter as a string (true, by default) or a set of characters "any of" (false) - SkipEmpty:Bool? - whether to skip empty strings in the result, is false by default - - Limit:Uint64? - Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last element + - Limit:Uint64? - Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last item * ```Unicode::JoinFromList(List<Utf8>{Flags:AutoMap}, Utf8) -> Utf8``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md index c9e9ed9b34..b78e8348f0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md @@ -4,8 +4,8 @@ * ```Url::Normalize(String) -> String?``` -Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, etc. -The normalization result depends only on the URL itself. The normalization **DOES NOT** include operations depending on the external data: transformation based on duplicates, mirrors, etc. +Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, and so on. +The normalization result only depends on the URL itself. The normalization **DOES NOT** include operations depending on the external data: transformation based on duplicates, mirrors, etc. Returned value: @@ -121,7 +121,7 @@ Get a component of the URL. Returns a second-level domain in most cases and a third-level domain for the hostnames like: ***.XXX.YY, where XXX is com, net, org, co, gov, or edu. You can redefine this list using an optional second argument * ```Url::GetOwner(String{Flags:AutoMap}) -> String``` - Returns the domain that's most likely owned by an individual or organization. Unlike Url::GetSignificantDomain, it uses a special whitelist. Besides the ***.co.uk domains, it can return a third-level domain used by free hosting sites and blogs (for example: something.livejournal.com)s + Returns the domain that's most likely owned by an individual or organization. Unlike Url::GetSignificantDomain, it uses a special whitelist. Besides the ***.co.uk domains, it can return a third-level domain used by free hosting sites and blogs (for example: something.livejournal.com) **Examples** @@ -169,3 +169,43 @@ SELECT Url::CutWWW("www.yandex.ru"); -- "yandex.ru" SELECT Url::PunycodeToHostName("xn--d1acpjx3f.xn--p1ai"); -- "яндекс.рф" ``` +## ...Query... {#query} + +[Query](https://docs.python.org/3/library/urllib.parse.html) transformations. + +**List of functions** + +```sql +Url::QueryStringToList(String{Flag:AutoMap}, [ + KeepBlankValues:Bool?, -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false. + Strict:Bool?, -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true. + MaxFields:Uint32?, -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>. + Separator:String? -- A key-value pair separator, defaults to '&'. +]) -> List<Tuple<String, String>> +Url::QueryStringToDict(String{Flag:AutoMap}, [ + KeepBlankValues:Bool?, -- Empty values in percent-encoded queries are interpreted as empty strings, defaults to false. + Strict:Bool?, -- If false, parsing errors are ignored and incorrect fields are skipped, defaults to true. + MaxFields:Uint32?, -- The maximum number of fields. If exceeded, an exception is thrown. Defaults to Max<Uint32>. + Separator:String? -- A key-value pair separator, defaults to '&'. +]) -> Dict<String, List<String>> +Url::BuildQueryString(Dict<String, List<String>>{Flag:AutoMap}, [ + Separator:String? -- A key-value pair separator, defaults to '&'. +]) -> String +Url::BuildQueryString(Dict<String, String>{Flag:AutoMap}, [ + Separator:String? -- A key-value pair separator, defaults to '&'. +]) -> String +Url::BuildQueryString(List<Tuple<String, String>>{Flag:AutoMap}, [ + Separator:String? -- A key-value pair separator, defaults to '&'. +]) -> String +``` + +**Examples** + +```sql +SELECT Url::QueryStringToList("a=1&b=2&a=3"); -- [("a", "1"), ("b", "2"), ("a", "3")] +SELECT Url::QueryStringToDict("a=1&b=2&a=3"); -- {"b" : ["2"], "a" : ["1", "3"]} +SELECT Url::BuildQueryString([("a", "1"), ("a", "3"), ("b", "2")]); -- "a=1&a=3&b=2" +SELECT Url::BuildQueryString({"a" : "1", "b" : "2"}); -- "b=2&a=1" +SELECT Url::BuildQueryString({"a" : ["1", "3"], "b" : ["2", "4"]}); -- "b=2&b=4&a=1&a=3" +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/datetime.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/datetime.md index 02c7c57cce..17fa7ce025 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/datetime.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/datetime.md @@ -12,6 +12,7 @@ In the DateTime module, the main internal representation format is `Resource<TM> * TimezoneId (16 bits). * DayOfYear (9 bits): Day since the beginning of the year. * WeekOfYear (6 bits): Week since the beginning of the year, January 1 is always in week 1. +* WeekOfYearIso8601 (6 bits): Week of the year according to ISO 8601 (the first week is the one that includes January 4). * DayOfWeek (3 bits): Day of the week. If the timezone is not GMT, the components store the local time for the relevant timezone. @@ -69,6 +70,7 @@ Extracting a component from an internal representation. * ```DateTime::GetMonth(Resource<TM>{Flags:AutoMap}) -> Uint8``` * ```DateTime::GetMonthName(Resource<TM>{Flags:AutoMap}) -> String``` * ```DateTime::GetWeekOfYear(Resource<TM>{Flags:AutoMap}) -> Uint8``` +* ```DateTime::GetWeekOfYearIso8601(Resource<TM>{Flags:AutoMap}) -> Uint8``` * ```DateTime::GetDayOfMonth(Resource<TM>{Flags:AutoMap}) -> Uint8``` * ```DateTime::GetDayOfWeek(Resource<TM>{Flags:AutoMap}) -> Uint8``` * ```DateTime::GetDayOfWeekName(Resource<TM>{Flags:AutoMap}) -> String``` @@ -117,7 +119,7 @@ SELECT DateTime::MakeTzTimestamp(DateTime::Update($tm, "Europe/Moscow" as Timezone)); -- 2019-01-01T01:02:03.456789,Europe/Moscow ``` -## From... {#from}} +## From... {#from} Getting a Timestamp from the number of seconds/milliseconds/microseconds since the UTC epoch. When the Timestamp limits are exceeded, NULL is returned. @@ -226,8 +228,7 @@ SELECT ## Shift... {#shift} -Add/subtract the specified number of units to the component in the internal representation and update the other fields. - +Add/subtract the specified number of units to/from the component in the internal representation and update the other fields. Returns either an updated copy or NULL, if an update produces an invalid date or other inconsistencies. **List of functions** diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/hyperscan.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/hyperscan.md index 0309e79059..d12f1046a8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/hyperscan.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/hyperscan.md @@ -28,7 +28,6 @@ SELECT * FROM table WHERE $re(key); -- use it to filter the table **Please note** escaping of special characters in regular expressions. Be sure to use the second slash, since all the standard string literals in SQL can accept C-escaped strings, and the `\d` sequence is not valid sequence (even if it were, it wouldn't search for numbers as intended). - You can enable the case-insensitive mode by specifying, at the beginning of the regular expression, the flag `(?i)`. ## Grep {#grep} @@ -46,7 +45,6 @@ For example, the following two queries are equivalent (also in terms of computin Matches **the whole string** against the regular expression. - To get a result similar to `Grep` (where substring matching is included), enclose the regular expression in `.*` (`.*foo.*` instead of `foo`). However, in terms of code readability, it's usually better to change the function. ## BacktrackingGrep/BacktrackingMatch {#backtrackinggrep} @@ -59,7 +57,6 @@ Those functions are currently called in the binary operators [REGEXP](../../synt Hyperscan lets you match against multiple regular expressions in a single pass through the text, and get a separate response for each match. - However, if you want to match a string against any of the listed expressions (the results would be joined with "or"), it would be more efficient to combine the query parts in a single regular expression with `|` and match it with regular `Grep` or `Match`. When you call `MultiGrep`/`MultiMatch`, regular expressions are passed one per line using [multiline string literals](../../syntax/expressions.md#named-nodes): diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/math.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/math.md index 2368ed2cdb..6549188265 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/math.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/math.md @@ -94,7 +94,6 @@ SELECT Math::Remainder(2.1, 2); -- 0.1 * ```Math::Ldexp(Double{Flags:AutoMap}, Int32{Flags:AutoMap}) -> Double``` * ```Math::Round(Double{Flags:AutoMap}, [Int32?]) -> Double```: The second argument indicates the power of 10 to which we round (it's negative for decimal digits and positive for rounding to tens, thousands, or millions); the default value is 0 - **Examples** ```sql @@ -125,8 +124,6 @@ These functions behave similarly to the built-in % operator in the case of non-n * Math::Mod preserves the sign of the second argument (the denominator). * Math::Rem preserves the sign of the first argument (the numerator). - - Functions return null if the divisor is zero. **Examples** diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/pire.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/pire.md index be9ca7b48a..e608f6b6c6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/pire.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/pire.md @@ -81,18 +81,15 @@ Matches the regular expression with a **part of the string** (arbitrary substrin ## Match {#match} Matches **the whole string** against the regular expression. - To get a result similar to `Grep` (where substring matching is included), enclose the regular expression in `.*`. For example, use `.*foo.*` instead of `foo`. ## MultiGrep/MultiMatch {#multigrep} Pire lets you match against multiple regular expressions in a single pass through the text and get a separate response for each match. - Use the MultiGrep/MultiMatch functions to optimize the query execution speed. Be sure to do it carefully, since the size of the state machine used for matching grows exponentially with the number of regular expressions: * If you want to match a string against any of the listed expressions (the results are joined with "or"), it would be much more efficient to combine the query parts in a single regular expression with `|` and match it using regular Grep or Match. * Pire has a limit on the size of the state machine (YQL uses the default value set in the library). If you exceed the limit, the error is raised at the start of the query: `Failed to glue up regexes, probably the finite state machine appeared to be too large`. - When you call MultiGrep/MultiMatch, regular expressions are passed one per line using [multiline string literals](../../syntax/expressions.md#multiline-string-literals): **Examples** @@ -115,7 +112,6 @@ SELECT ## Capture {#capture} If a string matches the specified regular expression, it returns a substring that matches the group enclosed in parentheses in the regular expression. - Capture is non-greedy: the shortest possible substring is returned. {% note alert %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md index 3efec1f12f..6c778270d8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md @@ -55,8 +55,7 @@ If you leave out the details of implementation and syntax of regular expressions ## Re2::Capture {#capture} -Unlike [Pire::Capture](pire.md#capture) , `Re2:Capture` supports multiple and named capturing groups. - +Unlike [Pire::Capture](pire.md#capture), `Re2::Capture` supports multiple and named capturing groups. Result type: a structure with the fields of the type `String?`. * Each field corresponds to a capturing group with the applicable name. @@ -80,3 +79,44 @@ Works as follows: Returns the number of non-overlapping substrings of the input string that have matched the regular expression. +## Re2::Options {#options} + +Notes on Re2::Options from the official [repository](https://github.com/google/re2/blob/main/re2/re2.h#L595-L617) + +| Parameter | Default | Comments | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- | ------------------------------------------------------------------------------------- | +| CaseSensitive:Bool? | true | match is case-sensitive (regexp can override with (?i) unless in posix_syntax mode) | +| DotNl:Bool? | false | let `.` match `\n` (default ) | +| Literal:Bool? | false | interpret string as literal, not regexp | +| LogErrors:Bool? | true | log syntax and execution errors to ERROR | +| LongestMatch:Bool? | false | search for longest match, not first match | +| MaxMem:Uint64? | - | (see below) approx. max memory footprint of RE2 | +| NeverCapture:Bool? | false | parse all parents as non-capturing | +| NeverNl:Bool? | false | never match \n, even if it is in regexp | +| PosixSyntax:Bool? | false | restrict regexps to POSIX egrep syntax | +| Utf8:Bool? | true | text and pattern are UTF-8; otherwise Latin-1 | +| The following options are only consulted when PosixSyntax == true. <bt>When PosixSyntax == false, these features are always enabled and cannot be turned off; to perform multi-line matching in that case, begin the regexp with (?m). | +| PerlClasses:Bool? | false | allow Perl's \d \s \w \D \S \W | +| WordBoundary:Bool? | false | allow Perl's \b \B (word boundary and not) | +| OneLine:Bool? | false | ^ and $ only match beginning and end of text | + +It is not recommended to use Re2::Options in the code. Most parameters can be replaced with regular expression flags. + +**Flag usage examples** + +```sql +$value = "Foo bar FOO"u; +-- enable case-insensitive mode +$capture = Re2::Capture(@@(?i)(foo)@@); + +SELECT + $capture($value) AS capture; + +$capture = Re2::Capture(@@(?i)(?P<vasya>FOO).*(?P<banan>bar)@@); + +SELECT + $capture($value) AS capture; +``` + +In both cases, the word VASYA will be found. Using the raw string @@regexp@@ lets you avoid double slashes. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md index 9da69f12c9..fed6a90482 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md @@ -18,7 +18,7 @@ Implementation specifics and functionality of the module: * `Yson::Parse***`: Getting a resource with a DOM object from serialized data, with all further operations performed on the obtained resource. * `Yson::From`: Getting a resource with a DOM object from simple YQL data types or containers (lists or dictionaries). * `Yson::ConvertTo***`: Converting a resource to [primitive data types](../../types/primitive.md) or [containers](../../types/containers.md). - * `Yson::Lookup***`: Getting a single list element or a dictionary with optional conversion to the relevant data type. + * `Yson::Lookup***`: Getting a single list item or a dictionary with optional conversion to the relevant data type. * `Yson::YPath***`: Getting one element from the document tree based on the relative path specified, optionally converting it to the relevant data type. * `Yson::Serialize***`: Getting a copy of data from the resource and serializing the data in one of the formats. * For convenience, when serialized Yson and Json are passed to functions expecting a resource with a DOM object, implicit conversion using `Yson::Parse` or `Yson::ParseJson` is done automatically. In SQL syntax, the dot or square brackets operator automatically adds a `Yson::Lookup` call. To serialize a resource, you still need to call `Yson::ConvertTo***` ([here's the ticket about CAST syntax support](https://st.yandex-team.ru/YQL-2610)) or `Yson::Serialize***`. It means that, for example, to get the "foo" element as a string from the Yson column named mycolumn and serialized as a dictionary, you can write: `SELECT Yson::ConvertToString(mycolumn["foo"]) FROM mytable;` or `SELECT Yson::ConvertToString(mycolumn.foo) FROM mytable;`. In the variant with a dot, special characters can be escaped by [general rules for IDs](../../syntax/expressions.md#escape). @@ -28,7 +28,7 @@ The module's functions must be considered as "building blocks" from which you ca * `Yson::Parse*** -> Yson::Serialize***`: Converting from one format to other. * `Yson::Parse*** -> Yson::Lookup -> Yson::Serialize***`: Extracting the value of the specified subtree in the source YSON tree. -* `Yson::Parse*** -> Yson::ConvertToList -> ListMap -> Yson::Lookup***`: Extracting elements by a key from the YSON list. +* `Yson::Parse*** -> Yson::ConvertToList -> ListMap -> Yson::Lookup***`: Extracting items by a key from the YSON list. See also examples of combining YSON functions in the [tutorial](https://yql.yandex-team.ru/Tutorial/yt_17_Yson_and_Json). @@ -36,7 +36,7 @@ See also examples of combining YSON functions in the [tutorial](https://yql.yand ```yql $node = Json(@@ - {"abc": {"def": 123, "ghi": "привет"}} + {"abc": {"def": 123, "ghi": "hello"}} @@); SELECT Yson::SerializeText($node.abc) AS `yson`; -- {"def"=123;"ghi"="\xD0\xBF\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82"} @@ -176,7 +176,6 @@ Yson::Contains(Resource<'Yson2.Node'>{Flags:AutoMap}, String) -> Bool? ``` Checks for a key in the dictionary. If the object type is a map, then it searches among the keys. - If the object type is a list, then the key must be a decimal number, i.e., an index in the list. ## Yson::Lookup... {#ysonlookup} @@ -245,21 +244,22 @@ Yson::Options([AutoConvert:Bool?, Strict:Bool?]) -> Resource<'Yson2.Options'> It's passed in the last optional argument (omitted for brevity) to the methods `Parse...`, `ConvertTo...`, `Contains`, `Lookup...`, and `YPath...` that accept the result of the `Yson::Options` call. By default, all the `Yson::Options` fields are false and when enabled (true), they modify the behavior as follows: * **AutoConvert**: If the value passed to Yson doesn't match the result data type exactly, the value is converted where possible. For example, `Yson::ConvertToInt64` in this mode will convert even Double numbers to Int64. -* **Strict**: By default, all functions from the Yson library return an error in case of issues during query execution (for example, an attempt to parse a string that is not Yson/Json, or an attempt to search by a key in a scalar type, or when a conversion to an incompatible data type has been requested, and so on). If you disable the strict mode, `NULL` is returned instead of an error. +* **Strict**: By default, all functions from the Yson library return an error in case of issues during query execution (for example, an attempt to parse a string that is not Yson/Json, or an attempt to search by a key in a scalar type, or when a conversion to an incompatible data type has been requested, and so on). If you disable the strict mode, `NULL` is returned instead of an error in most cases. When converting to a dictionary or list (`ConvertTo<Type>Dict` or `ConvertTo<Type>List`), improper items are excluded from the resulting collection. **Example:** ```yql -$x = Yson(@@{y = true}@@); -SELECT Yson::LookupBool($x, "z"); --- Error -SELECT Yson::LookupBool($x, "z", Yson::Options()); --- null (all fields are false by default) -SELECT Yson::LookupBool($x, "z", Yson::Options(false as Strict)); --- null -SELECT Yson::LookupBool($x, "z", Yson::Options(true as Strict)); --- Error - -$x = Yson(@@{y = 5.5}@@); -SELECT Yson::LookupInt64($x, "y") --- Error -SELECT Yson::LookupInt64($x, "y", Yson::Options(false as AutoConvert)); --- null -SELECT Yson::LookupInt64($x, "y", Yson::Options(true as AutoConvert)); --- 5 +$yson = @@{y = true; x = 5.5}@@y; +SELECT Yson::LookupBool($yson, "z"); --- null +SELECT Yson::LookupBool($yson, "y"); --- true + +SELECT Yson::LookupInt64($yson, "x"); --- Error +SELECT Yson::LookupInt64($yson, "x", Yson::Options(false as Strict)); --- null +SELECT Yson::LookupInt64($yson, "x", Yson::Options(true as AutoConvert)); --- 5 + +SELECT Yson::ConvertToBoolDict($yson); --- Error +SELECT Yson::ConvertToBoolDict($yson, Yson::Options(false as Strict)); --- { "y": true } +SELECT Yson::ConvertToDoubleDict($yson, Yson::Options(false as Strict)); --- { "x": 5.5 } ``` If you need to use the same Yson library settings throughout the query, it's more convenient to use [PRAGMA yson.AutoConvert;](../../syntax/pragma.md#yson.autoconvert) and/or [PRAGMA yson.Strict;](../../syntax/pragma.md#yson.strict). Only with these `PRAGMA` you can affect implicit calls to the Yson library occurring when you work with Yson/Json data types. diff --git a/ydb/docs/en/core/yql/toc_i.yaml b/ydb/docs/en/core/yql/toc_i.yaml index 5896ca55ae..d38ee02b48 100644 --- a/ydb/docs/en/core/yql/toc_i.yaml +++ b/ydb/docs/en/core/yql/toc_i.yaml @@ -1,7 +1,6 @@ items: -- name: Overview +- name: Обзор href: reference/index.md -- name: YQL reference guide - include: { mode: link, path: reference/toc_i.yaml } -- name: YQL tutorial +- include: { mode: link, path: reference/toc_i.yaml } +- name: Туториал YQL include: { mode: link, path: tutorial/toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md b/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md index 09014a6711..d6c6b823f1 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md @@ -6,8 +6,7 @@ keywords: - yql tutorial - yql basic operations --- - -# Overview +# YQL Tutorial - Overview From this tutorial, you will learn how to perform basic operations with data in {{ ydb-short-name }} and get familiar with the YQL syntax. diff --git a/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md b/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md index 6d5f5dabac..850bdb1e72 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md @@ -14,4 +14,5 @@ The tutorial consists of 15 steps: 1. [{#T}](../../update.md) 1. [{#T}](../../delete.md) 1. [{#T}](../../alter_table.md) -1. [{#T}](../../delete_table.md)
\ No newline at end of file +1. [{#T}](../../delete_table.md) + diff --git a/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md b/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md index 3848e51da7..d4eda537bc 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md @@ -3,3 +3,4 @@ We assume that you already created tables in step [{#T}](../create_demo_tables.md) and populated them with data in step [{#T}](../fill_tables_with_data.md). {% endnote %} + diff --git a/ydb/docs/en/core/yql/tutorial/alter_table.md b/ydb/docs/en/core/yql/tutorial/alter_table.md index 010eb6657a..fe79e0b8a2 100644 --- a/ydb/docs/en/core/yql/tutorial/alter_table.md +++ b/ydb/docs/en/core/yql/tutorial/alter_table.md @@ -19,3 +19,4 @@ Delete the column you added from the table: ```sql ALTER TABLE episodes DROP COLUMN viewers; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/basic_aggregation.md b/ydb/docs/en/core/yql/tutorial/basic_aggregation.md index 80bf81e70b..415fb866b3 100644 --- a/ydb/docs/en/core/yql/tutorial/basic_aggregation.md +++ b/ydb/docs/en/core/yql/tutorial/basic_aggregation.md @@ -19,7 +19,7 @@ SELECT FROM episodes GROUP BY - series_id, -- The query result will follow the listed order of columns. + series_id, -- The query result will follow the listed order of columns season_id -- Multiple columns are separated by a comma. -- Other columns can be listed after a SELECT only if -- they are passed to an aggregate function. @@ -29,4 +29,5 @@ ORDER BY ; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md b/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md index b962e99dc1..e99e6127f2 100644 --- a/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md +++ b/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md @@ -14,8 +14,8 @@ SELECT FROM episodes WHERE - series_id = 1 -- List of conditions to build the result. - AND season_id > 1 -- Logical AND is used for complex conditions. + series_id = 1 -- List of conditions to build the result + AND season_id > 1 -- Logical AND is used for complex conditions ORDER BY -- Sorting the results. series_id, -- ORDER BY sorts the values by one or multiple @@ -27,4 +27,5 @@ LIMIT 3 -- LIMIT N after ORDER BY means ; -- depending on sort order. COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/conditional_values.md b/ydb/docs/en/core/yql/tutorial/conditional_values.md index ff7bfbf4ce..9587168313 100644 --- a/ydb/docs/en/core/yql/tutorial/conditional_values.md +++ b/ydb/docs/en/core/yql/tutorial/conditional_values.md @@ -34,8 +34,8 @@ WHERE series_id IN (1,2) -- IN defines the set of values in the WHERE cla -- IN or NOT IN may lead to undesirable outcomes. AND season_id = 1 GROUP BY - CASE -- CASE evaluates a list of conditions - -- and returns one of multiple possible resulting + CASE -- CASE evaluates a list of conditions and + -- returns one of multiple possible resulting -- expressions. CASE can be used in any -- statement or with any clause -- that supports a given statement. For example, you can use CASE in @@ -52,3 +52,4 @@ GROUP BY COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/create_demo_tables.md b/ydb/docs/en/core/yql/tutorial/create_demo_tables.md index ffcf6b0b32..9a1c14300f 100644 --- a/ydb/docs/en/core/yql/tutorial/create_demo_tables.md +++ b/ydb/docs/en/core/yql/tutorial/create_demo_tables.md @@ -45,3 +45,4 @@ CREATE TABLE episodes COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/delete.md b/ydb/docs/en/core/yql/tutorial/delete.md index d3bc370225..d9d726a626 100644 --- a/ydb/docs/en/core/yql/tutorial/delete.md +++ b/ydb/docs/en/core/yql/tutorial/delete.md @@ -42,3 +42,4 @@ SELECT * FROM episodes WHERE series_id = 1 AND season_id = 1; COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/delete_table.md b/ydb/docs/en/core/yql/tutorial/delete_table.md index 5d008a40bc..0fa9da6de9 100644 --- a/ydb/docs/en/core/yql/tutorial/delete_table.md +++ b/ydb/docs/en/core/yql/tutorial/delete_table.md @@ -6,4 +6,5 @@ Delete the [created](create_demo_tables.md) tables using the [DROP TABLE](../ref DROP TABLE episodes; DROP TABLE seasons; DROP TABLE series; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md b/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md index 83346d766a..2a5d2b4152 100644 --- a/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md +++ b/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md @@ -7,7 +7,7 @@ REPLACE INTO series (series_id, title, release_date, series_info) VALUES - -- By default, numeric literals have type Int32 + -- By default, numeric literals have type Int32 -- if the value is within the range. -- Otherwise, they automatically expand to Int64. ( @@ -116,3 +116,4 @@ VALUES COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/insert_into.md b/ydb/docs/en/core/yql/tutorial/insert_into.md index d7ba507bd9..ee4c778227 100644 --- a/ydb/docs/en/core/yql/tutorial/insert_into.md +++ b/ydb/docs/en/core/yql/tutorial/insert_into.md @@ -1,6 +1,6 @@ # Inserting data with INSERT -Add data to the table using [INSERT INTO](../reference/syntax/insert_into.md). +Add data to the table using [INSERT INTO](../reference/syntax/insert_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -37,3 +37,4 @@ SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/join_tables.md b/ydb/docs/en/core/yql/tutorial/join_tables.md index d9d9793465..7e5f905d8b 100644 --- a/ydb/docs/en/core/yql/tutorial/join_tables.md +++ b/ydb/docs/en/core/yql/tutorial/join_tables.md @@ -25,3 +25,4 @@ ORDER BY -- Sorting of the results. COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/replace_into.md b/ydb/docs/en/core/yql/tutorial/replace_into.md index c79a16e3ea..2e791d3eb4 100644 --- a/ydb/docs/en/core/yql/tutorial/replace_into.md +++ b/ydb/docs/en/core/yql/tutorial/replace_into.md @@ -1,6 +1,6 @@ # Inserting and updating data with REPLACE -Add data to the table using [REPLACE INTO](../reference/syntax/replace_into.md). +Add data to the table using [REPLACE INTO](../reference/syntax/replace_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -32,3 +32,4 @@ SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/select_all_columns.md b/ydb/docs/en/core/yql/tutorial/select_all_columns.md index d3e665e10a..a0d55bbff6 100644 --- a/ydb/docs/en/core/yql/tutorial/select_all_columns.md +++ b/ydb/docs/en/core/yql/tutorial/select_all_columns.md @@ -1,6 +1,6 @@ # Selecting data from all columns -Select all columns from the table using [SELECT](../reference/syntax/select.md). +Select all columns from the table using [SELECT](../reference/syntax/select.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -13,3 +13,4 @@ FROM episodes; -- The table to select the data from. COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/update.md b/ydb/docs/en/core/yql/tutorial/update.md index f33bd6c61f..134d9d004e 100644 --- a/ydb/docs/en/core/yql/tutorial/update.md +++ b/ydb/docs/en/core/yql/tutorial/update.md @@ -1,6 +1,6 @@ # Updating data with UPDATE -Update data in the table using the [UPDATE](../reference/syntax/update.md) operator. +Update data in the table using the [UPDATE](../reference/syntax/update.md) operator: {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -45,3 +45,4 @@ SELECT * FROM episodes WHERE series_id = 1 AND season_id = 1; COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/upsert_into.md b/ydb/docs/en/core/yql/tutorial/upsert_into.md index f9c37407d2..75f2137f48 100644 --- a/ydb/docs/en/core/yql/tutorial/upsert_into.md +++ b/ydb/docs/en/core/yql/tutorial/upsert_into.md @@ -1,6 +1,6 @@ # Inserting and updating data with UPSERT -Add data to the table using [UPSERT INTO](../reference/syntax/upsert_into.md). +Add data to the table using [UPSERT INTO](../reference/syntax/upsert_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -30,3 +30,4 @@ SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; ``` + |
