diff options
| author | bazeltsev <bazeltsev@yandex-team.ru> | 2022-03-25 11:32:50 +0300 |
|---|---|---|
| committer | bazeltsev <bazeltsev@yandex-team.ru> | 2022-03-25 11:32:50 +0300 |
| commit | 6d68f7eae47385bd2a512252e401399b0b3ad8fa (patch) | |
| tree | f5340089072d5000a02cf9ccfba541d20c4ee04c | |
| parent | 0ae0603c05a3ef41e6adb8e6c57451ca01f2c072 (diff) | |
| download | ydb-6d68f7eae47385bd2a512252e401399b0b3ad8fa.tar.gz | |
Update core/en 20220322
ref:54ad45d653185dbceb29655513867c86bb4830bd
386 files changed, 2307 insertions, 1282 deletions
diff --git a/ydb/docs/en/core/best_practices/_includes/batch_upload.md b/ydb/docs/en/core/best_practices/_includes/batch_upload.md index 43fb693392..e7ab3fd4c3 100644 --- a/ydb/docs/en/core/best_practices/_includes/batch_upload.md +++ b/ydb/docs/en/core/best_practices/_includes/batch_upload.md @@ -34,3 +34,4 @@ We recommend the following algorithm for efficiently uploading data to {{ ydb-sh 3. Partition the resulting data set by the number of shards in the table. Each part will contain a set of consecutive rows. 4. Upload the resulting parts to the table shards concurrently. 5. Make a ```COMMIT``` after every 100,000 rows or 1 MB of data. + diff --git a/ydb/docs/en/core/best_practices/_includes/index.md b/ydb/docs/en/core/best_practices/_includes/index.md new file mode 100644 index 0000000000..7c6b906cc8 --- /dev/null +++ b/ydb/docs/en/core/best_practices/_includes/index.md @@ -0,0 +1,11 @@ +# Recommendations + +This section contains articles about the purpose, specifics, and best practices of using YDB tools for app development. + +- [Designing a data schema](../schema_design.md) +- [Partitioning tables](../table_sharding.md) +- [Secondary indexes](../secondary_indexes.md) +- [Paginated output](../paging.md) +- [Uploading large data volumes](../batch_upload.md) +- [Using timeouts](../timeouts.md) + diff --git a/ydb/docs/en/core/best_practices/_includes/paging.md b/ydb/docs/en/core/best_practices/_includes/paging.md index 73497a2aca..130ae7dba6 100644 --- a/ydb/docs/en/core/best_practices/_includes/paging.md +++ b/ydb/docs/en/core/best_practices/_includes/paging.md @@ -59,6 +59,7 @@ In {{ ydb-short-name }}, all columns, including key ones, may have a NULL value. ## Examples of paginated output implementation {% if oss %} + * [C++](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp/examples/pagination) {% endif %} * [Java](https://github.com/yandex-cloud/ydb-java-sdk/tree/master/examples/src/main/java/com/yandex/ydb/examples/pagination) diff --git a/ydb/docs/en/core/best_practices/_includes/schema_design.md b/ydb/docs/en/core/best_practices/_includes/schema_design.md index b0f2e6662c..45379bb56b 100644 --- a/ydb/docs/en/core/best_practices/_includes/schema_design.md +++ b/ydb/docs/en/core/best_practices/_includes/schema_design.md @@ -47,3 +47,4 @@ In {{ ydb-short-name }}, all columns, including key ones, may contain a NULL val ## Row size limit {#limit-string} To achieve high performance, we don't recommend writing rows larger than 8 MB and key columns larger than 2 KB to the DB. + diff --git a/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md b/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md index cb928162b2..f7f5d4f11f 100644 --- a/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md +++ b/ydb/docs/en/core/best_practices/_includes/secondary_indexes.md @@ -1,110 +1,66 @@ # Secondary indexes -This section describes how to use [secondary indexes](../../concepts/secondary_indexes.md) for selecting data. +[An index](https://en.wikipedia.org/wiki/Database_index) is an auxiliary data structure that lets you find data that meets certain criteria in a database, without having to search every row in a DB table, as well as obtain sorted samples without performing actual sorting that requires processing a complete set of all sorted data. -In general, transactions using a global index are [distributed transactions](../../concepts/transactions.md#distributed-tx). A query can be executed as a single-shard transaction in the following cases: +Data in YDB tables is always indexed by primary key. This means that fetching any record from a table with the specified values of primary key fields will always take the minimum fixed time, regardless of the total number of records in the table. A primary key index also lets you get any consecutive range of records in ascending or descending order of primary key values. The execution time of this operation will only depend on the number of records received, regardless of the total number of records in the table. -* Point read by primary key. -* Point read by index column if requested data is a primary key, part of it, or there is a copy of the data in a [covering index](../../concepts/secondary_indexes.md#covering). -* Point blind write in a table with an [asynchronous index](../../concepts/secondary_indexes.md#async). +To use similar features for any table fields or combinations thereof, additional indexes called **secondary indexes** can be built based on them. -{% note warning %} +Transactional systems use indexes to avoid performance degradation and an increase in the query execution cost if the amount of stored data grows. -The size of a response to a client may not exceed 50 MB. The size of data extracted from a single table shard per YQL query may not exceed 5 GB. For large tables and queries, these limits may make it impossible to fully scan all table rows. +This article describes the basic operations with secondary indexes and provides links to detailed materials for each operation. For information about different types of secondary indexes and their specifics, see the [Secondary indexes](../../concepts/secondary_indexes.md) article in the "Concepts" section. -{% endnote %} +## Creating secondary indexes {#create} -## Creating a table {#create} +A secondary index is a data schema object that can be set when creating a table with the [YQL `CREATE TABLE`](../../yql/reference/syntax/create_table.md) statement or added to it later with the [YQL `ALTER TABLE`](../../yql/reference/syntax/alter_table.md) statement. -To create a table named `series`, run the query: +The [`table index add`](../../reference/ydb-cli/commands/secondary_index.md#add) command for creating an index is supported in the YDB CLI. -```sql -CREATE TABLE series -( - series_id Uint64, - title Utf8, - series_info Utf8, - release_date Uint64, - views Uint64, - PRIMARY KEY (series_id), - INDEX views_index GLOBAL ON (views) -); -``` - -The `series` table has a key column named `series_id`. An index by primary key is created in {{ ydb-short-name }} automatically. Since the stored data is sorted in order of ascending primary key values, selecting data by this key will be efficient. An example of this selection is a search for all broadcasts of a series by its `series_id` from the `series` table. We also create an index named `views_index` to the `views` column. It will let you efficiently execute queries using it in the predicate. - -For a more complex select, you can create multiple secondary indexes. For example, to create a table named `series` with two indexes, run the query: - -```sql -CREATE TABLE series -( - series_id Uint64, - title Utf8, - series_info Utf8, - release_date Uint64, - views Uint64, - PRIMARY KEY (series_id), - INDEX views_index GLOBAL ON (views) COVER (release_date), - INDEX date_index GLOBAL ON (release_date) -); -``` +Since an index contains its own data that is derived from table data, an initial index build operation is performed when creating an index in an existing data table. This may take a long time. This operation is run in the background and is non-blocking for the table. However, you can't use a new index until its build completes. -As a result, two secondary indexes are created: `views_index` to the `views` column and `date_index` to the `release_date` column. In this case, `views_index` contains a copy of the data from the `release_date` column. +Indexes can only be used in the order of their fields. If there are two index fields, `a` and `b`, this index can be effectively used for queries like: -{% note info %} +- `where a = $var1 and b = $var2` +- `where a = $var1` +- `where a > $var1`, and other comparison operators +- `where a = $var1 and b > $var2`, and any other comparison operators, but the first field must be checked for equality -You can add a secondary index to an existing table without stopping the service. For more information about online index creation, see the [instructions](../../concepts/secondary_indexes.md#index-add). +At the same time, you can't use this index for the following queries: -{% endnote %} +- `where b = $var1` +- `where a > $var1 and b > $var2`, to be more precise, this entry will be equivalent to `where a > $var1` in terms of using the index +- `where b > $var1` -## Inserting data {#insert} +Considering the above specifics, it's useless to try to index all possible combinations of table columns in advance to speed up the execution of any query. An index is always a trade-off between the speed of searching and writing data and the storage space this data takes. Indexes are created for specific selects and criteria for a search that an application will make in the database. -Sample YQL queries use [prepared queries](https://en.wikipedia.org/wiki/Prepared_statement). To run them in the YQL editor, define the values of the parameters declared using the `DECLARE` statement. +## Using secondary indexes when making a Select {#use} -To add data to the `series` table, run the query: +To access a table by secondary index, its name must be explicitly specified in the `view` section after the table name as described in the article about the YQL [`SELECT`](../../yql/reference/syntax/select#secondary_index) statement. For example, to make a Select from the `orders` table for the customer with the specified ID (`id_customer`), run a query like this: ```sql -DECLARE $seriesId AS Uint64; -DECLARE $title AS Utf8; -DECLARE $seriesInfo AS Utf8; -DECLARE $releaseDate AS Uint32; -DECLARE $views AS Uint64; - -INSERT INTO series (series_id, title, series_info, release_date, views) -VALUES ($seriesId, $title, $seriesInfo, $releaseDate, $views); +DECLARE $customer_id AS Uint64; +SELECT * +FROM orders as o view idx_customer +WHERE o.id_customer = $customer_id ``` -## Updating data {#upsert} - -You can save a new number of views for a particular series in the database with the `UPSERT` operation. - -To update data in the `series` table, run the query: - -```sql -DECLARE $seriesId AS Uint64; -DECLARE $newViews AS Uint64; +, where`idx_customer` is the name of a secondary index on the `orders` table with the `id_customer` field specified first. -UPSERT INTO series (series_id, views) -VALUES ($seriesId, $newViews); -``` +If the `view` section is omitted, a full scan of the `orders` table is performed for making this query. -## Data selection {#select} +In transactional applications, such information requests are executed using paginated output. This helps avoid an increase in costs and execution time if the number of records that meet the filtering criteria grows. The approach to making [queries with pagination](../paging.md) that is described using a primary key as an example is also applicable to columns included in a secondary index. -Without using secondary indexes, a query to select records that match a certain predicate by number of views won't work effectively since {{ ydb-short-name }} scans all the rows of the `series` table to execute it. The query must explicitly specify which index to use. +## Checking the cost of a query {#cost} -To select the `series` table rows that match the predicate by number of views, run the query: +Any query in a transactional application should be checked in terms of how many I/O operations it performed in the database and how much CPU it used to run. You should also make sure that these metrics do not grow indefinitely with the growth of the DB size. After making every query, YDB returns statistics that you can use to analyze the necessary parameters. -```sql -SELECT series_id, title, series_info, release_date, views -FROM series view views_index -WHERE views >= someValue -``` +If you use the YDB CLI, select the `--stats` option to enable statistics output after executing the `yql` command. All YDB SDKs also contain structures that provide statistics after making queries. If you run queries in the UI, there is also a tab with statistics next to the results tab. ## Updating data using a secondary index {#update} -`UPDATE` statements don't let you indicate that a secondary index should be used to search for data, so an attempt to run `UPDATE ... WHERE indexed_field = $value` will result in a full scan of the table. To avoid this, you can first run `SELECT` by index to get the primary key value and then `UPDATE` by the primary key. You can also use `UPDATE ON`. +YQL statements used for making changes to records ([`UPDATE`](../../yql/reference/syntax/update.md), [`UPSERT`](../../yql/reference/syntax/upsert_into.md), and [`REPLACE`](../../yql/reference/syntax/replace_into.md)) don't let you indicate that a secondary index should be used to search for data, so an attempt to run an `UPDATE ... WHERE indexed_field = $value` will result in a full scan of the table. To avoid this, you can first make a `SELECT` by index to get the primary key value and then an `UPDATE` by primary key. You can also use `UPDATE ON`. -To update data in the `series` table, run the query: +To update data in `table1`, run the query: ```sql $to_update = ( @@ -123,7 +79,16 @@ To delete all data about series with zero views from the `series` table, run the ```sql DELETE FROM series ON -SELECT series_id, +SELECT series_id, FROM series view views_index -WHERE views == 0; +WHERE views = 0; ``` + +## Performance of writing data to tables with secondary indexes {#write_performance} + +Additional data structures are needed for secondary indexes to work. The support of these structures leads to an increase in the cost of table data update operations. + +With synchronous index updates, a transaction is only committed after writing all the necessary data, both in the table and in synchronous indexes. As a result, it takes longer to execute transactions and makes it necessary to use [distributed transactions](../../concepts/transactions#distributed-tx) even if adding and updating records in the only partition. + +Asynchronously updated indexes can still use single-shard transactions. However, they only guarantee eventual consistency and still generate load on the database. + diff --git a/ydb/docs/en/core/best_practices/_includes/table_sharding.md b/ydb/docs/en/core/best_practices/_includes/table_sharding.md index 458248f7ca..1c5b93d086 100644 --- a/ydb/docs/en/core/best_practices/_includes/table_sharding.md +++ b/ydb/docs/en/core/best_practices/_includes/table_sharding.md @@ -1,14 +1,16 @@ -# Table partitioning recommendations +# Partitioning tables {{ ydb-short-name }} tables are sorted by primary key in ascending order. Tables are partitioned by splitting the range of key values into consecutive non-overlapping ranges called partitions. -Thanks to partitioning, table data can be distributed across multiple storage devices and the load from operations on a table can use more processor cores and network bandwidth. However, too many partitions in a table may add overhead on RAM and CPU time usage. So optimal partitioning directly affects the efficiency of query execution. To achieve optimal partitioning, {{ ydb-short-name }} provides tools for the initial partitioning of a table when creating it as well as methods for subsequent automatic partitioning. +Thanks to partitioning, table data can be distributed across multiple storage devices and the load from operations on a table can use more processor cores and network bandwidth. However, too many partitions in a table may add overhead on RAM and CPU time usage. So optimal partitioning directly affects the efficiency of query execution. To achieve optimal partitioning, {{ ydb-short-name }} provides tools for the initial partitioning of a table when creating it and methods for subsequent automatic partitioning. When creating a table, you can set the initial partitioning in one of two ways: + * Set up even partitioning by the first column of the primary key if this column is UInt32 or UInt64. In this case, you need to explicitly specify the number of partitions. * Set the exact partitioning keys that determine the number and boundaries of the original partitions. {{ ydb-short-name }} supports two methods for automatic table partitioning: + * By data size. * By load on a datashard (a system component that serves a table partition). @@ -17,3 +19,6 @@ Automatic partitioning modes can be enabled separately or together. Both modes c Automatic partitioning by size is parameterized by the value of the partition size setting. When that value is reach, the partition is split. The default value is 2 GB. A key for splitting is selected based on the histogram of data size distribution across partitions by key sub-ranges. If the total data size in adjacent partitions becomes less than half of the setting size, these partitions are merged. Automatic partitioning by load is triggered based on CPU usage by the datashard serving the partition. All datashards monitor their CPU usage. If a high (>50%) level of usage is detected at some point in time, the partition is split. A key is selected using statistics on accessing the keys of a datashard's own partition. + +For a full list of table parameters, including partitioning ones, with comments on their usage, see [Partitioning tables](../../concepts/datamodel.md#partitioning) in the article about data schema objects in the "Concepts" section. + diff --git a/ydb/docs/en/core/best_practices/_includes/timeouts.md b/ydb/docs/en/core/best_practices/_includes/timeouts.md index 7e997b5de1..07cdd82350 100644 --- a/ydb/docs/en/core/best_practices/_includes/timeouts.md +++ b/ydb/docs/en/core/best_practices/_includes/timeouts.md @@ -2,30 +2,32 @@ title: Using timeouts in Yandex Database (YDB) description: 'The operation_timeout value determines the time during which the query result is interesting to the user. If the operation has not been performed during this time, the server returns an error with the Timeout code and tries to terminate the execution of the request, but the cancellation of the request is not guaranteed. It is always recommended to set both the operation timeout and the transport timeout.' --- +# Using timeouts This section describes available timeouts and provides examples of their usage in various programming languages. -## Prerequisites for using timeouts +## Prerequisites for using timeouts {#intro} -The timeout mechanism in YDB is designed to: -1) Make sure the query execution time doesn't exceed a certain interval after which its result is not interesting for further use. -2) Detect network connectivity issues. +The timeout mechanism in {{ ydb-short-name }} is designed to: + +* Make sure the query execution time doesn't exceed a certain interval after which its result is not interesting for further use. +* Detect network connectivity issues. Both of these use cases are important for ensuring the fault tolerance of the entire system. Let's take a closer look at timeouts. -## Operation timeout +## Operation timeout {#operational} The ``operation_timeout`` value shows the time during which the query result is interesting to the user. If the operation fails during this time, the server returns an error with the ``Timeout`` code and tries to terminate the query, but its cancellation is not guaranteed. So the query that the user was returned the ``Timeout`` error for can be both successfully executed on the server and canceled. -## Timeout for canceling an operation +## Timeout for canceling an operation {#cancel} The ``cancel_after`` value shows the time after which the server will start canceling the query, if it can be canceled. If canceled, the server returns the ``Cancelled`` error code. -## Transport timeout +## Transport timeout {#transport} -The client can set a transport timeout for each query. This value lets you determine the amount of time that the client is ready to wait for a response from the server. If the server doesn't respond during this time, the client will get a transport error with the ``DeadlineExceeded`` code. Be sure to set such a client timeout value that won't trigger transport timeouts under the normal operation of the application and network. +The client must set a transport timeout for each query. This value lets you determine the amount of time that the client is ready to wait for a response from the server. If the server doesn't respond during this time, the client will get a transport error with the ``DeadlineExceeded`` code. Be sure to set such a client timeout value that won't trigger transport timeouts under the normal operation of the application and network. -## Using timeouts +## Using timeouts {#usage} We recommend that you always set an operation timeout and transport timeout. The value of the transport timeout should be 50-100 milliseconds more than that of the operation timeout, that way there is some time left for the client to get a server error with the ``Timeout`` code. @@ -58,7 +60,7 @@ Timeout usage example: #include <ydb/public/sdk/cpp/client/ydb.h> #include <ydb/public/sdk/cpp/client/ydb_table.h> #include <ydb/public/sdk/cpp/client/ydb_value.h> - + using namespace NYdb; using namespace NYdb::NTable; @@ -95,3 +97,4 @@ Timeout usage example: ``` {% endlist %} + diff --git a/ydb/docs/en/core/best_practices/index.md b/ydb/docs/en/core/best_practices/index.md index a4077dc318..eb2590567d 100644 --- a/ydb/docs/en/core/best_practices/index.md +++ b/ydb/docs/en/core/best_practices/index.md @@ -1 +1 @@ -This section provides recommendations on how to use {{ ydb-short-name }} features. +{% include [_includes/index.md](_includes/index.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/cluster/_includes/addition_overlay.md b/ydb/docs/en/core/cluster/_includes/addition_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/cluster/_includes/addition_overlay.md diff --git a/ydb/docs/en/core/cluster/index.md b/ydb/docs/en/core/cluster/index.md new file mode 100644 index 0000000000..d325a1ee0a --- /dev/null +++ b/ydb/docs/en/core/cluster/index.md @@ -0,0 +1,12 @@ +# Managing a YDB cluster + +This section contains articles intended for system administrators responsible for the performance of YDB clusters that application databases are deployed on. + +{% include [addition_overlay](_includes/addition_overlay.md) %} + +{% note warning %} + +The section is under development. + +{% endnote %} + diff --git a/ydb/docs/en/core/cluster/toc_i.yaml b/ydb/docs/en/core/cluster/toc_i.yaml new file mode 100644 index 0000000000..59aee60e4e --- /dev/null +++ b/ydb/docs/en/core/cluster/toc_i.yaml @@ -0,0 +1,15 @@ +items: +- name: Kubernetes + include: { mode: link, path: ../deploy/orchestrated/toc_p.yaml } +- name: Manual + items: + - include: { mode: link, path: ../deploy/manual/toc_p.yaml } + - include: { mode: link, path: ../deploy/configuration/toc_p.yaml } + - name: Production checklist + href: ../deploy/production_checklist.md + - name: Maintaining a cluster's disk subsystem + include: { mode: link, path: ../maintenance/manual/toc_p.yaml } +- name: Embedded UI + include: { mode: link, path: ../maintenance/embedded_monitoring/toc_p.yaml } +- name: System views + href: ../troubleshooting/system_views_cluster.md
\ No newline at end of file diff --git a/ydb/docs/en/core/cluster/toc_p.yaml b/ydb/docs/en/core/cluster/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/cluster/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_includes/connect.md b/ydb/docs/en/core/concepts/_includes/connect.md index 092abfd6d8..29de524ca8 100644 --- a/ydb/docs/en/core/concepts/_includes/connect.md +++ b/ydb/docs/en/core/concepts/_includes/connect.md @@ -48,8 +48,8 @@ The highest level of security and performance is provided when using the **Metad When choosing the authentication mode among those supported by the server and environment, follow the recommendations below: -* **Anonymous** can be used on self-deployed local {{ ydb-short-name }} clusters that are inaccessible over the network. -* **Access Token** can be used when other modes are not supported server side or for setup/debugging purposes. It does not require that the client access IAM. However, if the IAM system supports an API for token rotation, fixed tokens issued by this IAM usually have a short validity period, which makes it necessary to update them manually in the IAM system on a regular basis. +* **You would normally use Anonymous** on self-deployed local {{ ydb-short-name }} clusters that are inaccessible over the network. +* **You would use Access Token** when other modes are not supported server side or for setup/debugging purposes. It does not require that the client access IAM. However, if the IAM system supports an API for token rotation, fixed tokens issued by this IAM usually have a short validity period, which makes it necessary to update them manually in the IAM system on a regular basis. * **Refresh Token** can be used when performing one-time manual operations under a personal account, for example, related to DB data maintenance, performing ad-hoc operations in the CLI, or running applications from a workstation. You can manually obtain this token from IAM once to have it last a long time and save it in an environment variable on a personal workstation to use automatically and with no additional authentication parameters on CLI launch. * **Service Account Key** is mainly used for applications designed to run in environments where the **Metadata** mode is supported, when testing them outside these environments (for example, on a workstation). It can also be used for applications outside these environments, working as an analog of **Refresh Token** for service accounts. Unlike a personal account, service account access objects and roles can be restricted. * **Metadata** is used when deploying applications in clouds. Currently, this mode is supported on virtual machines and in {{ sf-name }} {{ yandex-cloud }}. @@ -62,7 +62,7 @@ When using modes in which the {{ ydb-short-name }} client accesses the IAM syste ## Database location {#database} -Database location (`database`) is a string that defines where the queried database is located in the {{ ydb-short-name }} cluster. Has the format [path to the file]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Path_(computing)){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Путь_к_файлу){% endif %} and uses the `/` character as separator. It always starts with a `/`. +Database location (`database`) is a string that defines where the queried database is located in the {{ ydb-short-name }} cluster. Has the format [path to file]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Path_(computing)){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Путь_к_файлу){% endif %} and uses the `/` character as separator. It always starts with a `/`. A {{ ydb-short-name }} cluster may have multiple databases deployed, and their location is defined by the cluster configuration. Like the endpoint, `database` for cloud databases is displayed in the management console on the desired database page, and can also be obtained via the CLI of the cloud provider. @@ -103,3 +103,4 @@ The OS that the client runs on already include a set of root certificates from t For token rotation, the {{ ydb-short-name }} SDK and CLI use a gRPC request to the {{ yandex-cloud }} IAM API [IamToken - create]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/iam/api-ref/grpc/iam_token_service#Create){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/iam/api-ref/grpc/iam_token_service#Create){% endif %}. In **Refresh Token** mode, the token specified in the OAuth parameter is passed in the `yandex_passport_oauth_token` attribute. In **Service Account Key** mode, a short-lived JWT token is generated from the specified service account attributes and an encryption key and passed in the `jwt` property. The source code for generating the JWT is available as part of the SDK (for example, the `get_jwt()` method in the [Python code](https://github.com/ydb-platform/ydb-python-sdk/blob/main/ydb/iam/auth.py)). In the {{ ydb-short-name }} SDK and CLI, you can substitute the host used for accessing the API for obtaining tokens. This makes it possible to implement a similar API in corporate contexts. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md b/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md index 55d5f64654..680349bbcf 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md +++ b/ydb/docs/en/core/concepts/_includes/databases/base_hierarchy.md @@ -4,8 +4,8 @@ To host databases in {{ ydb-short-name }}, use the following directory structure * _A domain_ is the first-level element where all cluster resources are located. There can only be one domain per cluster. * _An account_ is a second-level element that defines the owner of a DB group. -* _A directory_ is a third-level element set by the account owner for grouping their databases. -* _A database_ is a fourth-level element that corresponds to the DB. +* _A directory_ is a third-level element defined by an account owner for grouping his or her databases. +* _A database_ is a fourth-level element that corresponds to a DB. At the top hierarchy level, the cluster contains a single domain that hosts one or more accounts, while accounts host one or more directories that host databases. diff --git a/ydb/docs/en/core/concepts/_includes/databases/cluster.md b/ydb/docs/en/core/concepts/_includes/databases/cluster.md index fc4fc5a78b..d707b99108 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/cluster.md +++ b/ydb/docs/en/core/concepts/_includes/databases/cluster.md @@ -1,5 +1,6 @@ ## Cluster {#cluster} -_A {{ ydb-short-name }} cluster_ is a set of {{ ydb-short-name }} nodes that the load is distributed across and that communicate with each other through a network. A single cluster may host multiple databases, and specific cluster nodes are allocated for each database. One database is fully hosted in one cluster. +_A {{ ydb-short-name }} cluster_ is a set of interconnected {{ ydb-short-name }} nodes that distribute the load among themselves. A single cluster may host multiple databases, and specific cluster nodes are allocated for each database. One database is fully hosted in one cluster. There are {{ ydb-short-name }} clusters with a single data center and geo-distributed clusters. The smallest single-datacenter cluster consists of a single node. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/compute.md b/ydb/docs/en/core/concepts/_includes/databases/compute.md index 30b5586e1f..cf4dc20369 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/compute.md +++ b/ydb/docs/en/core/concepts/_includes/databases/compute.md @@ -1,3 +1,4 @@ ## Computing resources {#compute-resources} {{ ydb-short-name }} clusters can be deployed on bare metal, VMs, or in a container virtualization environment. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/database.md b/ydb/docs/en/core/concepts/_includes/databases/database.md index 6bc098e67b..d057ef34f7 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/database.md +++ b/ydb/docs/en/core/concepts/_includes/databases/database.md @@ -13,3 +13,4 @@ _A {{ ydb-short-name }} database ({{ ydb-short-name }} DB)_ is an isolated consi * Generates metrics that can be used to monitor the DB performance. Resources for the {{ ydb-short-name }} database (CPU, RAM, nodes, and disk space) are allocated within a {{ ydb-short-name }} cluster. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/intro.md b/ydb/docs/en/core/concepts/_includes/databases/intro.md index ddf87b8818..c4000d11b7 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/intro.md +++ b/ydb/docs/en/core/concepts/_includes/databases/intro.md @@ -4,3 +4,4 @@ description: "A database (DB) {{ ydb-full-name }}: is an isolated, consistent se --- # Terms and definitions +
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/_includes/databases/regions.md b/ydb/docs/en/core/concepts/_includes/databases/regions.md index 1c84c96c6a..488f05bc94 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/regions.md +++ b/ydb/docs/en/core/concepts/_includes/databases/regions.md @@ -1,8 +1,9 @@ ## Regions and availability zones {#regions-az} -_An availability zone_ is a data processing center or its isolated segment where the minimum physical distance between nodes is ensured and the risk of simultaneous failures with other availability zones is minimized. -_A wide geographic region_ is a territory where the distance between availability zones does not exceed 500 km. +_An availability zone_ is a data processing center or an isolated segment thereof with minimal physical distance between nodes and minimal risk of failure at the same time as other availability zones. +_A large geographic region_ is an area within which the distance between availability zones is 500 km or smaller. A geo-distributed {{ ydb-short-name }} cluster contains nodes located in different availability zones within a wide geographic region. {{ ydb-short-name }} performs synchronous data writes to each of the availability zones, ensuring uninterrupted performance if an availability zone fails. In geographically distributed clusters, you can choose a policy for distributing computing resources across data centers. This lets you find the right balance between minimum execution time and minimum downtime if a data center goes offline. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/slots.md b/ydb/docs/en/core/concepts/_includes/databases/slots.md index 6469a59c63..896f62d159 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/slots.md +++ b/ydb/docs/en/core/concepts/_includes/databases/slots.md @@ -1,5 +1,6 @@ ## Slots {#slots} -_A slot_ is a part of the server resources allocated for running a single {{ ydb-short-name }} cluster node, it has a fixed size of 10CPU/50GB RAM. Slots are used if a {{ ydb-short-name }} cluster is deployed on bare-metal instances whose resources are sufficient to host multiple slots. If you use VMs for cluster deployment, their capacity is selected so that the use of slots is not required: one node serves one database and one database can use multiple nodes allocated to it. +_A slot_ is the portion of a server's resources allocated to running a single {{ ydb-short-name }} cluster node. It is constant and equal to 10 CPUs/50 GB RAM. Slots are used if a {{ ydb-short-name }} cluster is deployed on bare-metal instances whose resources are sufficient to host multiple slots. If you use VMs for cluster deployment, their capacity is selected so that the use of slots is not required: one node serves one database and one database can use multiple nodes allocated to it. You can see a list of DB slots in the output of the `discovery list` command of the {{ ydb-short-name }} console client. + diff --git a/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md b/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md index 8901b7ecab..6f9ebb3cd8 100644 --- a/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md +++ b/ydb/docs/en/core/concepts/_includes/databases/storage_groups.md @@ -2,6 +2,7 @@ A storage group is a redundant array of independent disks that are networked in a single logical unit. Storage groups increase fault tolerance through redundancy and improve performance. -Each storage group corresponds to a specific storage schema that affects the number of disks used, the failure model, and the redundancy factor. The `block4-2` schema is commonly used for single data center clusters, where the storage group is located on 8 disks in 8 racks, can withstand the failure of any two disks, and ensures a redundancy factor of 1.5. In multiple data center clusters, we use the `mirror3dc` schema, where storage groups are made up of 9 disks, 3 in each of the three data centers, which can survive the failure of a data center or disk, and ensures a redundancy factor of 3. +Each storage group corresponds to a specific storage schema that affects the number of disks used, the failure model, and the redundancy factor. The `block4-2` arrangement is commonly used for single-datacenter clusters with storage located on 8 disks in 8 racks and capable of surviving a failure of any two disks, and ensuring a redundancy factor of 1.5. Multiple-datacenter clusters use the `mirror3dc` arrangement with storage comprising 9 disks, 3 in each of the three datacenters, and capable of surviving a failure of a datacenter or disk and ensuring a redundancy factor of 3. {{ ydb-short-name }} lets you allocate additional storage groups for available DBs as your data grows. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md b/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md index 7fae5905ae..bb4dfd5c88 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/blockdevice.md @@ -1,3 +1,4 @@ ## Network block storage volume {#volume} {{ ydb-short-name }} can be used as a platform for creating a wide range of data storage and processing systems, for example, by implementing a [network block device](https://en.wikipedia.org/wiki/Network_block_device) on {{ ydb-short-name }}. Network block devices implement an interface for a local block device, as well as ensure fault-tolerance (through redundancy) and good scalability in terms of volume size and the number of input/output operations per unit of time. The downside of a network block device is that any input/output operation on such device requires network interaction, which might increase the latency of the network device compared to the local device. You can deploy a common file system on a network block device and/or run an application directly on the block device, such as a database management system. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/dir.md b/ydb/docs/en/core/concepts/_includes/datamodel/dir.md index fc682b2f73..05138506ef 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/dir.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/dir.md @@ -1,3 +1,4 @@ ## Directories {#dir} For convenience, the service supports creating directories like in a file system, meaning the entire database consists of a directory tree, while tables and other entities are in the leaves of this tree (similar to files in the file system). A directory can host multiple subdirectories and tables. The names of the entities they contain are unique. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/intro.md b/ydb/docs/en/core/concepts/_includes/datamodel/intro.md index 72be848af7..860fc10178 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/intro.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/intro.md @@ -1,3 +1,4 @@ # Data model and schema This section describes the entities that {{ ydb-short-name }} uses within DBs. The {{ ydb-short-name }} core lets you flexibly implement various storage primitives, so new entities may appear in the future. + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/pq.md b/ydb/docs/en/core/concepts/_includes/datamodel/pq.md index c72feaff34..1e469fbc9f 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/pq.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/pq.md @@ -1,3 +1,4 @@ ## Persistent queue {#persistent-queue} -A persistent queue consists of one or more partitions, where each partition is a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) [message queue](https://en.wikipedia.org/wiki/Message_queue) ensuring reliable delivery between two or more components. Data messages are untyped and constitute a data blob. Partitioning is a parallel processing tool that helps ensure high queue bandwidth. Mechanisms are provided to implement at least once and exactly once persistent queue delivery guarantees. A persistent queue in {{ ydb-short-name }} is similar to a topic in [Apache Kafka](https://en.wikipedia.org/wiki/Apache_Kafka). +A persistent queue consists of one or more partitions, where each partition is a [FIFO](https://en.wikipedia.org/wiki/FIFO_(computing_and_electronics)) [message queue](https://en.wikipedia.org/wiki/Message_queue) ensuring reliable delivery between two or more components. Data messages have no type and are data blobs. Partitioning is a parallel processing tool that helps ensure high queue bandwidth. Mechanisms are provided to implement the "at least once" and the "exactly once" persistent queue delivery guarantees. A persistent queue in {{ ydb-short-name }} is similar to a topic in [Apache Kafka](https://en.wikipedia.org/wiki/Apache_Kafka). + diff --git a/ydb/docs/en/core/concepts/_includes/datamodel/table.md b/ydb/docs/en/core/concepts/_includes/datamodel/table.md index 9175c5d338..c80147a419 100644 --- a/ydb/docs/en/core/concepts/_includes/datamodel/table.md +++ b/ydb/docs/en/core/concepts/_includes/datamodel/table.md @@ -1,6 +1,6 @@ ## Table {#table} -A table in {{ ydb-short-name }} is a [relational table](https://en.wikipedia.org/wiki/Table_(database)) that contains a set of related data consisting of rows and columns. Each row is a set of cells that are used for storing specific types of values according to the data schema. The data schema defines the names and types of table columns. An example of a data schema is shown below. The `Series` table consists of four columns named `SeriesId`, `ReleaseDate`, `SeriesInfo`, and `Title` and holding data of type `Uint64?` for the first two and `String?` for the others. The `SeriesId` column is declared the primary key. +A table in {{ ydb-short-name }} is a [relational table](https://en.wikipedia.org/wiki/Table_(database)) containing a set of related data and made up of rows and columns. Each row is a set of cells that are used for storing specific types of values according to the data schema. The data schema defines the names and types of table columns. An example of a data schema is shown below. The `Series` table consists of four columns named `SeriesId`, `ReleaseDate`, `SeriesInfo`, and `Title` and holding data of type `Uint64?` for the first two and `String?` for the others. The `SeriesId` column is declared the primary key.  @@ -14,7 +14,7 @@ Often, when you design a table schema, you already have a set of fields, which c A database table can be sharded by primary key value ranges. Each shard of the table is responsible for a specific range of primary keys. Key ranges served by different shards do not overlap. Different table shards can be served by different distributed database servers (including ones in different locations). They can also move independently between servers to enable rebalancing or ensure shard operability if servers or network equipment goes offline. -If there is not a lot of data or load, the table may consist of a single shard. As the amount of data served by the shard or the load on the shard grows, {{ ydb-short-name }} automatically splits the shard into two shards. The data is split by the median value of the primary key if the shard size exceeds the threshold. If partitioning by load is used, the shard first collects a sample of the requested keys (that can be read, written, and deleted) and, based on this sample, selects a key for partitioning to evenly distribute the load across new shards. So in the case of load-based partitioning, the size of new shards may significantly vary. +If there is not a lot of data or load, the table may consist of a single shard. As the amount of data served by the shard or the load on the shard grows, {{ ydb-short-name }} automatically splits this shard into two shards. The data is split by the median value of the primary key if the shard size exceeds the threshold. If partitioning by load is used, the shard first collects a sample of the requested keys (that can be read, written, and deleted) and, based on this sample, selects a key for partitioning to evenly distribute the load across new shards. So in the case of load-based partitioning, the size of new shards may significantly vary. The size-based shard split threshold and automatic splitting can be configured (enabled/disabled) individually for each database table. @@ -31,7 +31,7 @@ The following table partitioning parameters are defined in the data schema: * Type: `Enum` (`ENABLED`, `DISABLED`). * Default value: `ENABLED`. -Automatic partitioning by partition size. If a partition size exceeds the value specified by the [AUTO_PARTITIONING_PARTITION_SIZE_MB](#auto_partitioning_partition_size_mb) parameter, it is enqueued for splitting. If the total size of two or more adjacent partitions is less than 50% of the [AUTO_PARTITIONING_PARTITION_SIZE_MB](#auto_partitioning_partition_size_mb) value, they are enqueued for merging. +Automatic partitioning by partition size. If a partition size exceeds the value specified by the [`AUTO_PARTITIONING_PARTITION_SIZE_MB`](#auto_partitioning_partition_size_mb) parameter, it is enqueued for splitting. If the total size of two or more adjacent partitions is less than 50% of the [`AUTO_PARTITIONING_PARTITION_SIZE_MB`](#auto_partitioning_partition_size_mb) value, they are enqueued for merging. #### AUTO_PARTITIONING_BY_LOAD @@ -40,7 +40,7 @@ Automatic partitioning by partition size. If a partition size exceeds the value Automatic partitioning by load. If a shard consumes more than 50% of the CPU for a few dozens of seconds, it is enqueued for splitting. If the total load on two or more adjacent shards uses less than 35% of a single CPU core within an hour, they are enqueued for merging. -Performing split or merge operations uses the CPU and takes time. Therefore, when dealing with a variable load, we recommend both enabling this mode and setting [AUTO_PARTITIONING_MIN_PARTITIONS_COUNT](#auto_partitioning_min_partitions_count) to a value other than 1 so that decreased load does not cause the number of partitions to drop below the required value resulting in a need to split them again when load increases. +Performing split or merge operations uses the CPU and takes time. Therefore, when dealing with a variable load, we recommend both enabling this mode and setting [`AUTO_PARTITIONING_MIN_PARTITIONS_COUNT`](#auto_partitioning_min_partitions_count) to a value other than 1 so that decreased load does not cause the number of partitions to drop below the required value resulting in a need to split them again when load increases. When choosing the minimum number of partitions, it makes sense to consider that one table partition can only be hosted on one server and use no more than 1 CPU core for data update operations. Hence, you can set the minimum number of partitions for a table on which a high load is expected to at least the number of nodes (servers) or, preferably, to the number of CPU cores allocated to the database. @@ -49,7 +49,7 @@ When choosing the minimum number of partitions, it makes sense to consider that * Type: `Uint64`. * Default value: `2000 MB` ( `2 GB` ). -Partition size threshold in MB. If exceeded, a shard splits. Takes effect when the [AUTO_PARTITIONING_BY_SIZE](#auto_partitioning_by_size) mode is enabled. +Partition size threshold in MB. If exceeded, a shard splits. Takes effect when the [`AUTO_PARTITIONING_BY_SIZE`](#auto_partitioning_by_size) mode is enabled. #### AUTO_PARTITIONING_MIN_PARTITIONS_COUNT @@ -85,14 +85,14 @@ When automatic partitioning is enabled, make sure to set the correct value for [ ### Reading data from replicas {#read_only_replicas} -When making queries in {{ ydb-short-name }}, the actual execution of a query to each shard is performed at a single point serving the distributed transaction protocol. By storing data in shared storage, you can run one or more shard followers without allocating additional space in the storage: the data is already stored in replicated format and you can serve more than one reader (but the writer is still strictly one at every moment). +When making queries in {{ ydb-short-name }}, the actual execution of a query to each shard is performed at a single point serving the distributed transaction protocol. By storing data in shared storage, you can run one or more shard followers without allocating additional storage space: the data is already stored in replicated format, and you can serve more than one reader (but there is still only one writer at any given moment). Reading data from followers allows you: -* Serving clients demanding minimal delays otherwise unachievable in a multi-DC cluster. This is achieved by bringing the query execution point closer to the query formulation point, which eliminates the delay associated with inter-DC transfers. As a result, you can both preserve all the storage reliability guarantees of the multi-DC cluster and respond to occasional read queries in milliseconds. -* Handling read queries from followers without affecting the modifying queries running on the shard. This can be useful both for isolating different scenarios and for increasing the partition bandwidth. -* Ensuring continued maintenance when moving the partition leader (both in regular cases (balancing) and in an emergency). It lets the processes in the cluster survive without affecting the reading clients. -* Increasing the general limit of the shard data read performance if many read queries access the same keys. +* To serve clients demanding minimal delay, which is otherwise unachievable in a multi-DC cluster. This is accomplished by executing queries soon after they are formulated, which eliminates the delay associated with inter-DC transfers. As a result, you can both preserve all the storage reliability guarantees of a multi-DC cluster and respond to occasional read queries in milliseconds. +* To handle read queries from followers without affecting modifying queries running on a shard. This can be useful both for isolating different scenarios and for increasing the partition bandwidth. +* To ensuring continued service when moving a partition leader (both in a planned manner for load balancing and in an emergency). It lets the processes in the cluster survive without affecting the reading clients. +* To increasing the overall shard read performance if many read queries access the same keys. You can enable running read replicas for each shard of the table in the table data schema. The read replicas (followers) are typically accessed without leaving the data center network, which ensures response delays in milliseconds. @@ -104,7 +104,7 @@ The internal state of each of the followers is restored exactly and fully consis Besides the data status in storage, followers also receive a stream of updates from the leader. Updates are sent in real time, immediately after the commit to the log. However, they are sent asynchronously, resulting in some delay (usually no more than dozens of milliseconds, but sometimes longer in the event of cluster connectivity issues) in applying updates to followers relative to their commit on the leader. Therefore, reading data from followers is only supported in the [transaction mode](../transactions.md#modes) `StaleReadOnly()`. -If there are multiple followers, their delay from the leader may vary: although each follower of each of the shards retains internal consistency, artifacts may be observed between different shards. Please allow for this in your application code. For that same reason, it's currently impossible to perform cross-shard transactions from followers. +If there are multiple followers, their delay from the leader may vary: although each follower of each of the shards retains internal consistency, artifacts may be observed from shard to shard. Please provide for this in your application code. For that same reason, it's currently impossible to perform cross-shard transactions from followers. ### Deleting expired data (TTL) {#ttl} diff --git a/ydb/docs/en/core/concepts/_includes/index/intro.md b/ydb/docs/en/core/concepts/_includes/index/intro.md index 6a66cf5a2c..518a0358b5 100644 --- a/ydb/docs/en/core/concepts/_includes/index/intro.md +++ b/ydb/docs/en/core/concepts/_includes/index/intro.md @@ -4,7 +4,7 @@ description: "Yandex Database (YDB): is a horizontally scalable distributed faul --- # {{ ydb-short-name }} overview -*{{ ydb-short-name }}* is a horizontally scalable distributed fault-tolerant DBMS. {{ ydb-short-name }} is designed to meet high performance requirements. For example, a typical server can handle dozens of thousands of queries per second. The system is designed to handle hundreds of petabytes of data. {{ ydb-short-name }} can operate in single data center and geo-distributed (cross data center) modes on a cluster of thousands of servers. +_{{ ydb-short-name }}_ is a horizontally scalable distributed fault tolerant DBMS. {{ ydb-short-name }} is designed for high performance with a typical server being capable of handling tens of thousands of queries per second. The system is designed to handle hundreds of petabytes of data. {{ ydb-short-name }} can operate both in single datacenter and geodistributed (across several datacenters) modes on a cluster made up of thousands of servers. {{ ydb-short-name }} provides: @@ -14,7 +14,7 @@ description: "Yandex Database (YDB): is a horizontally scalable distributed faul * High availability with automatic failover in case a server, rack, or availability zone goes offline. * Automatic data partitioning as data or load grows. -To interact with {{ ydb-short-name }}, you can use the [{{ ydb-short-name }} CLI](../../../reference/ydb-cli/index.md) or [SDK](../../../reference/ydb-sdk/index.md) for {% if oss %}C++,{% endif %} Java, Python, Node.js, PHP, and Go. +To interact with {{ ydb-short-name }}, you can use the [{{ ydb-short-name }} CLI](../../../reference/ydb-cli/index.md) or [SDK](../../../reference/ydb-sdk/index.md) for {% if oss %}C++,{% endif %} Java, Python, Node.js, PHP, and Go. {{ ydb-short-name }} supports a relational [data model](../../../concepts/datamodel.md) and manages tables with a predefined schema. To make it easier to organize tables, directories can be created like in the file system. @@ -24,5 +24,5 @@ Database commands are mainly written in YQL, an SQL dialect. This gives the user {{ ydb-short-name }} natively supports different processing options, such as [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) and [OLAP](https://en.wikipedia.org/wiki/Online_analytical_processing). The current version offers limited analytical query support. This is why we can say that {{ ydb-short-name }} is currently an OLTP database. -{{ ydb-short-name }} is used in Yandex services as a high-performance [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) DBMS. In particular, {{ yandex-cloud }} {{ objstorage-full-name}} use {{ ydb-short-name }} to store data and are based on its components. +{{ ydb-short-name }} is used in Yandex services as a high-performance [OLTP](https://en.wikipedia.org/wiki/Online_transaction_processing) DBMS. In particular, {{ yandex-cloud }} {{ objstorage-full-name }} use {{ ydb-short-name }} to store data and are based on its components. diff --git a/ydb/docs/en/core/concepts/_includes/index/when_use.md b/ydb/docs/en/core/concepts/_includes/index/when_use.md index a1ee985aad..1d9c2734fd 100644 --- a/ydb/docs/en/core/concepts/_includes/index/when_use.md +++ b/ydb/docs/en/core/concepts/_includes/index/when_use.md @@ -2,10 +2,11 @@ {{ ydb-short-name }} can be used as an alternative solution in the following cases: -* When using NoSQL systems if strong data consistency is required. -* When using NoSQL systems if you need to make transactional changes to data stored in different rows of one or more tables. -* In systems that need to process and store large amounts of data and allow for almost infinite horizontal scalability (using industrial clusters of 5000+ nodes, processing millions of RPS and storing petabytes of data). -* In systems with low load, when support for a separate DB instance will be a waste of money (consider using {{ ydb-short-name }} in serverless mode instead). +* When using NoSQL systems, if strong data consistency is required. +* When using NoSQL systems, if you need to make transactional updates to data stored in different rows of one or more tables. +* In systems that need to process and store large amounts of data and allow for virtually unlimited horizontal scalability (using industrial clusters of 5000+ nodes, processing millions of RPS, and storing petabytes of data). +* In low-load systems, when supporting a separate DB instance would be a waste of money (consider using {{ ydb-short-name }} in serverless mode instead). * In systems with unpredictable or seasonally fluctuating load (you can add/reduce computing resources on request and/or in serverless mode). * In high-load systems that shard load across relational DB instances. -* When developing a new product that has no reliable forecast of expected load or is natively designed for an expected high load beyond the capabilities of traditional relational databases. +* When developing a new product with no reliable load forecast or with an expected high load beyond the capabilities of conventional relational databases. + diff --git a/ydb/docs/en/core/concepts/_includes/limits-ydb.md b/ydb/docs/en/core/concepts/_includes/limits-ydb.md index 638b55e6d8..4471c34295 100644 --- a/ydb/docs/en/core/concepts/_includes/limits-ydb.md +++ b/ydb/docs/en/core/concepts/_includes/limits-ydb.md @@ -7,7 +7,7 @@ This section describes the parameters of limits set in {{ ydb-short-name }}. The table below shows the limits that apply to schema objects: tables, databases, and columns. The_Object_ column specifies the type of schema object that the limit applies to. The _Error type_ column shows the status that the query ends with if an error occurs. For more information about statuses, see [Error handling in the API](../../reference/ydb-sdk/error_handling.md). -| Objects | Limit | Value | Explanation | Internal<br>name | Error<br>type | +| Object | Limit | Value | Explanation | Internal<br>name | Error<br>type | | :--- | :--- | :--- | :--- | :---: | :---: | | Database | Maximum path depth | 32 | Maximum number of nested path elements (directories, tables). | MaxDepth | SCHEME_ERROR | | Database | Maximum number of paths (schema objects) | 200,000 | Maximum number of path elements (directories, tables) in a database. | MaxPaths | GENERIC_ERROR | @@ -36,8 +36,9 @@ The table below lists the limits that apply to query execution. The _Call_ colum | Parameter | Value | Call | Explanation | Status<br>in case of<br>a violation<br>of the limit | | :--- | :--- | :--- | :--- | :---: | -| Maximum number of rows in query results | 1000 | ExecuteDataQuery | Complete results of some queries executed using the `ExecuteDataQuery` method may contain more rows than allowed. In this case, the maximum allowed number of rows will be returned in response to the query and the result will have the `truncated` flag. | SUCCESS | +| Maximum number of rows in query results | 1000 | ExecuteDataQuery | Complete results of some queries executed using the `ExecuteDataQuery` method may contain more rows than allowed. In this case, a query will return the maximum number of rows allowed, and the result will have the `truncated` flag set. | SUCCESS | | Maximum query result size | 50 MB | ExecuteDataQuery | Complete results of some queries may exceed the set limit. In this case, a query will fail returning no data. | PRECONDITION_FAILED | | Maximum number of sessions per cluster node | 1000 | CreateSession | Using the library for working with {{ ydb-short-name }}, an application can create sessions within a connection. Sessions are linked to a node. You can create a limited number of sessions with a single node. | OVERLOADED | | Maximum query text length | 10 KB | ExecuteDataQuery | Limit on the length of YQL query text. | BAD_REQUEST | -| Maximum size of parameter values | 50 MB | ExecuteDataQuery | Limit on the total size of parameters passed when executing a previously prepared query. | BAD_REQUEST | +| Maximum size of parameter values | 50 MB | ExecuteDataQuery | Limit on the total size of the parameters passed when executing a previously prepared query. | BAD_REQUEST | + diff --git a/ydb/docs/en/core/concepts/_includes/scan_query.md b/ydb/docs/en/core/concepts/_includes/scan_query.md index a7c187efb5..f283042adb 100644 --- a/ydb/docs/en/core/concepts/_includes/scan_query.md +++ b/ydb/docs/en/core/concepts/_includes/scan_query.md @@ -1,17 +1,17 @@ # Scan queries in {{ ydb-short-name }} -Scan Queries is a separate data access interface designed primarily for performing analytical ad hoc queries on a DB. +Scan Queries is a separate data access interface designed primarily for running analytical ad hoc queries against a DB. This method of executing queries has the following unique features: * Only *Read-Only* queries. * In *SERIALIZABLE_RW* mode, a data snapshot is taken and then used for all subsequent operations. As a result, the impact on OLTP transactions is minimal (only taking a snapshot). -* The result of the query is a data stream ([grpc stream](https://grpc.io/docs/what-is-grpc/core-concepts/)). This means scan queries have no limit on the number of rows in the result. +* The output of a query is a data stream ([grpc stream](https://grpc.io/docs/what-is-grpc/core-concepts/)). This means scan queries have no limit on the number of rows in the result. * Due to the high overhead, it is only suitable for ad hoc queries. {% node info %} -From the *Scan Queries* interface, you can query [system tables](../../troubleshooting/system_views.md). +From the _Scan Queries_ interface, you can query [system tables](../../troubleshooting/system_views_db.md). {% endnote %} @@ -19,7 +19,7 @@ Scan queries cannot currently be considered an effective solution for running OL * The query duration is limited to 5 minutes. * Many operations (including sorting) are performed entirely in memory, which may lead to resource shortage errors when running complex queries. -* Only one strategy is currently used for Joins: *MapJoin* (a.k.a. *Broadcast Join*), where the "right" table is converted to a map, so it should be no more than units of gigabytes. +* A single strategy is currently in use for joins: *MapJoin* (a.k.a. *Broadcast Join*) where the "right" table is converted to a map; and therefore, must be no more than single gigabytes in size. * Prepared form isn't supported, so for each call, a query is compiled. * There is no optimization for point reads or reading small ranges of data. * The SDK doesn't support automatic retry. @@ -53,3 +53,4 @@ class TTableClient { ``` {% endif %} + diff --git a/ydb/docs/en/core/concepts/_includes/secondary_indexes.md b/ydb/docs/en/core/concepts/_includes/secondary_indexes.md index 9e41aec395..673f06bfa7 100644 --- a/ydb/docs/en/core/concepts/_includes/secondary_indexes.md +++ b/ydb/docs/en/core/concepts/_includes/secondary_indexes.md @@ -39,7 +39,7 @@ Online index creation consists of the following steps: The index is ready to use. -Possible effects on user transactions: +Possible impact on user transactions: * There may be an increase in delays because transactions are now distributed (when creating a synchronous index). * There may be an enhanced background of `OVERLOADED` errors because index table automatic shard splitting is actively running during data writes. @@ -48,185 +48,16 @@ The rate of data writes is selected to minimize their impact on user transaction Creating an index is an asynchronous operation. If the client-server connection is interrupted after the operation has started, index building continues. You can manage asynchronous operations using the {{ ydb-short-name }} CLI. -## Examples of working with a secondary index {#example} +## Creating and deleting secondary indexes {#ddl} -### Creating a secondary index {#add} +A secondary index can be: -{% list tabs %} +- Created when creating a table with the YQL [`CREATE TABLE`](../../yql/reference/syntax/create_table.md) statement. +- Added to an existing table with the YQL [`ALTER TABLE`](../../yql/reference/syntax/alter_table.md) statement or the YDB CLI [`table index add`](../../reference/ydb-cli/commands/secondary_index.md#add) command. +- Deleted from an existing table with the YQL [`ALTER TABLE`](../../yql/reference/syntax/alter_table.md) statement or the YDB CLI [`table index drop`](../../reference/ydb-cli/commands/secondary_index.md#drop) command. +- Deleted together with the table using the YQL [`DROP TABLE`](../../yql/reference/syntax/drop_table.md) statement or the YDB CLI `table drop` command. -- CLI +## Purpose and use of secondary indexes {#best_practices} - Run the command: +For information about the purpose and use of secondary indexes for app development, see the [recommendations](../../best_practices/secondary_indexes.md). - ```bash - {{ ydb-cli }} \ - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - table index add global[-sync|-async] \ - --index-name title_index \ - --columns title \ - series - ``` - - * `--endpoint`: DB endpoint. - * `--database`: Full DB path. - * `-sync|-async`: Index type. - * `--index-name`: The name of the index being created. - * `--columns`: A list of columns the index is created by. - - If necessary, you can duplicate data in a covering index by adding the `--cover` option that specifies a list of columns to copy data to this index from. - - The create index command looks like this: - - ```bash - {{ ydb-cli }} \ - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - table index add global[-sync|-async] \ - --index-name title_index \ - --columns title \ - --cover release_date \ - series - ``` - - Result: - - ```text - ┌────────────────────────────────────────┬───────┬────────┐ - | id | ready | status | - ├────────────────────────────────────────┼───────┼────────┤ - | ydb://buildindex/7?id=1407375091598308 | false | | - └────────────────────────────────────────┴───────┴────────┘ - ``` - - The build index operation is started, where `id` is the operation ID. - -- YQL - - Run the query: - - ```sql - ALTER TABLE `series` ADD INDEX `title_index` GLOBAL [SYNC|ASYNC] ON (`title`); - ``` - - If necessary, you can duplicate data in a covering index by using the `COVER` keyword: - - ```sql - ALTER TABLE `series` ADD INDEX `title_index` GLOBAL [SYNC|ASYNC] ON (`title`) COVER(`release_date`); - ``` - - This starts building the index. Wait for the operation to complete. - - {% note warning %} - - You can't cancel index building with YQL. If necessary, use the {{ ydb-short-name }} CLI. - - {% endnote %} - -{% endlist %} - -{% note info %} - -If you don't specify the index type, a synchronous index is created by default. - -{% endnote %} - -### Getting the status of a build secondary index operation {#get} - -{% list tabs %} - -- CLI - - Run the command: - - ```bash - {{ ydb-cli }} \ - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - operation get ydb://buildindex/7?id=1407375091598308 - ``` - * `--endpoint`: DB endpoint. - * `--database`: Full DB path. - - Result: - - ```text - ┌────────────────────────────────────────┬───────┬─────────┬───────┬──────────┬───────────────────────────────────────────────────────────────┬─────────────┐ - | id | ready | status | state | progress | table | index | - ├────────────────────────────────────────┼───────┼─────────┼───────┼──────────┼───────────────────────────────────────────────────────────────┼─────────────┤ - | ydb://buildindex/7?id=1407375091598308 | true | SUCCESS | Done | 100.00% | /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj/series | title_index | - └────────────────────────────────────────┴───────┴─────────┴───────┴──────────┴───────────────────────────────────────────────────────────────┴─────────────┘ - ``` - -{% endlist %} - -### Canceling the build secondary index operation {#cancel} - -{% list tabs %} - -- CLI - - Run the command: - - ```bash - {{ ydb-cli }} \ - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - operation cancel ydb://buildindex/7?id=1407375091598308 - ``` - * `--endpoint`: DB endpoint. - * `--database`: Full DB path. - -{% endlist %} - -### Deleting a completed or canceled build secondary index operation {#forget} - -{% list tabs %} - -- CLI - - Run the command: - - ```bash - {{ ydb-cli }} \ - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - operation forget ydb://buildindex/7?id=1407375091598308 - ``` - * `--endpoint`: DB endpoint. - * `--database`: Full DB path. - -{% endlist %} - -### Deleting a secondary index {#drop} - -{% list tabs %} - -- CLI - - Run the command: - - ```bash - --endpoint ydb.serverless.yandexcloud.net:2135 \ - --database /ru-central1/b1g4ej5ju4rf5kelpk4b/etn01lrprvnlnhv8v5kj \ - table index drop \ - --index-name title_index \ - series - ``` - * `--endpoint`: DB endpoint. - * `--database`: Full DB path. - * `--index-name`: The name of the index to delete. - -- YQL - - Run the query: - - ```sql - ALTER TABLE `series` DROP INDEX `title_index`; - ``` - -{% endlist %} - -#### What's next - -For other examples of working with secondary indexes, see the [recommendations](../../best_practices/secondary_indexes.md). diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md index a64ac12dd0..540c543c9e 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/01_intro.md @@ -7,7 +7,6 @@ keywords: - dedicated editable: false --- - # Serverless and Dedicated modes {{ ydb-short-name }} You can create and use multiple {{ ydb-short-name }} databases. When creating a database, one of two operating modes is selected for each database: Serverless or Dedicated. The mode can't be changed later. @@ -15,3 +14,4 @@ You can create and use multiple {{ ydb-short-name }} databases. When creating a * _Serverless_: A DB that doesn't require you to configure, administer, or monitor load or manage resources. To create a database, you only need to enter a name, and you'll get the URL for the connection. Payment is charged for the execution of queries and the actual amount of stored data. * _Dedicated_: You determine the computing resources that will be reserved for the database: CPU and RAM on the nodes, the number of nodes, and the storage size. You need to make sure there are sufficient resources to handle the load and add more when necessary. Payment is charged for dedicated resources per hour, regardless of their actual use. + diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md index 88ee85494a..33130369fe 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/02_sls_pars.md @@ -20,7 +20,7 @@ The **Throttling limit** can be enabled or disabled. We recommend that you alway If the limit is exceeded, a query is not accepted for execution and the `Throughput limit exceeded` error is returned. This error means that you can safely resend your query later. We recommend that you use the statements supplied as part of the {{ ydb-short-name }} SDK when re-sending. The proposed statements implement repetition algorithms that use different repetition strategies depending on error type and code: zero delay, exponentially increasing delay, fast or slow repetition, and others. -The limit is triggered with some delay, so the `Throughput limit exceeded` error is returned for a subsequent query rather than the specific query that resulted in exceeding the limit. Once the limit is triggered, queries won't be accepted for execution for a period approximately equal to the ratio of the queries exceeding the limit to the limit itself. If you use 1000 RUs, for instance, for the execution of a single query while your limit is 100 RUs, new queries won't be accepted for 10 seconds. +The limit is triggered with some delay, so the `Throughput limit exceeded` error is returned for a subsequent query rather than the specific query that resulted in exceeding the limit. Once the limit is triggered, queries won't be accepted for execution for a period approximately equal to the ratio of the queries exceeding the limit to the limit itself. For example, if you use 1000 RUs for the execution of a single query while your limit is 100 RUs/sec, new queries won't be accepted for 10 seconds. To reduce the risk of rejection under uneven load, {{ ydb-short-name }} flexibly applies restrictions using a bandwidth reservation mechanism (`burst capacity`). As long as you use fewer RU processing requests than specified in the restriction, the unused bandwidth is reserved. During peak usage, more bandwidth than specified in the restriction may be temporarily available from the accumulated reserve. @@ -39,3 +39,4 @@ The **Maximum amount of data** limit for a serverless database limits the amount You can change the **Maximum amount of data** limit interactively at any time, both via the graphical console and the CLI and raise or reduce it without limitations. This allows you to quickly adjust it as needed. We don't recommend setting the **Maximum amount of data** limit below the current actual amount because in this state, all data modification operations, including DELETE, become unavailable. You will only be able to reduce the amount of data with the DROP TABLE or DROP INDEX commands. If the limit is accidentally set below the actual volume, we recommend returning it to the operating value exceeding the actual volume with some redundancy. + diff --git a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md index d26360dad4..d396434cd8 100644 --- a/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md +++ b/ydb/docs/en/core/concepts/_includes/serverless_and_dedicated/10_arch_diff.md @@ -1,4 +1,4 @@ -## {{ ydb-short-name }} architecture in different operating modes {#arch-diff} +## {{ ydb-short-name }} architecture in different modes of operation {#arch-diff} ### Separate compute and storage layers {#separate-layers} @@ -10,11 +10,11 @@ It should be noted that the concept of a database comprises user tables and, acc ### {{ ydb-short-name }} Dedicated mode {#dedicated} -Dedicated mode assumes that resources for tablet instances and for executing YQL queries are selected from the compute resources explicitly allocated to the database. Compute resources are VMs that have a certain amount of vCPUs and memory. The task of selecting the optimal amount of resources for the DB is currently the user's responsibility. If there aren't enough resources to serve the load, the latency of requests increases, which may eventually lead to the denial of service for requests, such as that with the `OVERLOADED` return code. The user can add compute resources (VMs) to the database in the UI or CLI to ensure it has the necessary amount of computing resources. Adding compute resources to the DB is relatively fast and comparable to the time it takes to start a VM. After that, {{ ydb-short-name }} automatically balances tablet instances across the cluster based on the resources added. +Dedicated mode assumes that resources for tablet instances and for executing YQL queries are selected from the compute resources explicitly allocated to the database. Computational resources are VMs that have a certain number of vCPUs and some memory. The task of selecting the optimal amount of resources for the DB is currently the user's responsibility. If there aren't enough resources to serve the load, the latency of requests increases, which may eventually lead to the denial of service for requests, such as that with the `OVERLOADED` return code. The user can add compute resources (VMs) to the database in the UI or CLI to ensure it has the necessary amount of computing resources. Adding compute resources to the DB is relatively fast and comparable to the time it takes to start a VM. Subsequently, {{ ydb-short-name }} automatically balances tablets across a cluster account taken of resources added. ### {{ ydb-short-name }} Serverless mode {#serverless} -In Serverless mode, the {{ ydb-short-name }} infrastructure determines the amount of computing resources to be allocated to serve the user database. The amount of allocated resources can be either very large (an arbitrary number of cores) or very small (significantly less than one core). If a user created a DB that contains a single table with a single entry and makes DB queries very rarely, {{ ydb-short-name }} actually uses a small amount of RAM on tablet instances that are part of the user DB. This is possible due to the fact that the user database components are objects rather than processes. If the load increases, the DB components start using more CPU time and memory. If the load grows to the point where there are not enough VM resources, the {{ ydb-short-name }} infrastructure can balance the system granularly, generating tablet instances on other VMs. +In Serverless mode, the {{ ydb-short-name }} infrastructure determines the amount of computational resources to allocate to support a user database. The amount of allocated resources can be either very large (an arbitrary number of cores) or very small (significantly less than one core). If a user created a DB with a single table with a single entry and only rarely makes DB queries, {{ ydb-short-name }} actually uses a small amount of RAM on tablet instances that are part of the user DB. This is possible due to the fact that the user database components are objects rather than processes. If the load increases, the DB components start using more CPU time and memory. If load grows to the point where there aren't enough VM resources, the {{ ydb-short-name }} infrastructure can balance the system granularly spawning tablet instances on other VMs. This technology lets you package virtual entities (tablet instances) very tightly together into physical resources based on actual consumption. This makes it possible to invoice the user for the operations performed rather than the allocated resources. diff --git a/ydb/docs/en/core/concepts/_includes/transactions.md b/ydb/docs/en/core/concepts/_includes/transactions.md index 79ba0eba7c..b14cf5d58d 100644 --- a/ydb/docs/en/core/concepts/_includes/transactions.md +++ b/ydb/docs/en/core/concepts/_includes/transactions.md @@ -4,13 +4,13 @@ This section describes the specifics of YQL implementation for {{ ydb-short-name ## Query language {#query-language} -The main tool for creating, modifying, and managing data in {{ ydb-short-name }} is a declarative query language called YQL. YQL is an SQL dialect that can be considered a database interaction standard. {{ ydb-short-name }} also supports a set of special RPCs that can be used, for example, to manage a tree schema or cluster. +The main tool for creating, modifying, and managing data in {{ ydb-short-name }} is a declarative query language called YQL. YQL is an SQL dialect that can be considered a database interaction standard. {{ ydb-short-name }} also supports a set of special RPCs useful in managing a tree schema or a cluster, for instance. ## Transaction modes {#modes} By default, {{ ydb-short-name }} transactions are performed in *Serializable* mode. It provides the strictest [isolation level](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Serializable) for custom transactions. This mode guarantees that the result of successful parallel transactions is equal to their specific execution sequence, while there are no [read anomalies](https://en.wikipedia.org/wiki/Isolation_(database_systems)#Read_phenomena) for successful transactions. -If the requirements for the consistency or freshness of data read by a transaction can be lowered, the user can use execution modes with lower guarantees: +If consistency or freshness requirement for data read by a transaction can be relaxed, a user can take advantage of execution modes with lower guarantees: * *Online Read-Only*. Each of the reads in the transaction reads data that is most recent at the time of its execution. The consistency of retrieved data depends on the *allow_inconsistent_reads* setting: * *false* (consistent reads). In this mode, each individual read returns consistent data, but no consistency between different reads is guaranteed. Reading the same table range twice may return different results. @@ -29,8 +29,8 @@ Listed below are the features and limitations of YQL support in {{ ydb-short-nam * Multi-statement transactions (transactions made up of a sequence of YQL statements) are supported. Transactions may interact with client software, or in other words, client interactions with the database might look as follows: `BEGIN; make a SELECT; analyze the SELECT results on the client side; ...; make an UPDATE; COMMIT`. We should note that if the transaction body is fully formed before accessing the database, it will be processed more efficiently. * {{ ydb-short-name }} does not support transactions that combine DDL and DML queries. The conventional notion [ACID]{% if lang == "en" %}(https://en.wikipedia.org/wiki/ACID){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/ACID){% endif %} of a transactions is applicable specifically to DML queries, that is, queries that change data. DDL queries must be idempotent, meaning repeatable if an error occurs. If you need to manipulate a schema, each manipulation is transactional, while a set of manipulations is not. -* The version of YQL in {{ ydb-short-name }} uses the [Optimistic concurrency control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) mechanism. Optimistic locking is applied to entities affected during a transaction. When the transaction is complete, the mechanism verifies that the locks have not been invalidated. For the user, locking optimism means that when transactions are competing with one another, the one that finishes first wins. Competing transactions fail with the `Transaction locks invalidated` error. -* All changes made during the transaction accumulate in the database server memory and are committed when the transaction completes. If the locks are not invalidated, all the changes accumulated are committed atomically, but if at least one lock is invalidated, none of the changes are committed. The above model involves certain restrictions: changes made by a single transaction must fit inside the available memory, and the transaction "doesn't see" its changes. +* The version of YQL in {{ ydb-short-name }} uses the [Optimistic Concurrency Control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) mechanism. Optimistic locking is applied to entities affected during a transaction. When the transaction is complete, the mechanism verifies that the locks have not been invalidated. For the user, locking optimism means that when transactions are competing with one another, the one that finishes first wins. Competing transactions fail with the `Transaction locks invalidated` error. +* All changes made during the transaction accumulate in the database server memory and are committed when the transaction completes. If the locks are not invalidated, all the changes accumulated are committed atomically, but if at least one lock is invalidated, none of the changes are committed. The above model involves certain restrictions: changes made by a single transaction must fit inside available memory, and a transaction "doesn't see" its changes. A transaction should be formed in such a way so that the first part of the transaction only reads data, while the second part of the transaction only changes data. The query structure then looks as follows: diff --git a/ydb/docs/en/core/concepts/_includes/ttl.md b/ydb/docs/en/core/concepts/_includes/ttl.md index a42166901b..c816a05f22 100644 --- a/ydb/docs/en/core/concepts/_includes/ttl.md +++ b/ydb/docs/en/core/concepts/_includes/ttl.md @@ -20,7 +20,7 @@ expiration_time = valueof(ttl_column) + expire_after_seconds {% note info %} -TTL doesn't guarantee that the item will be deleted exactly at `expiration_time`, it might happen later. If it's important to exclude logically obsolete but not yet physically deleted items from the selection, use request-level filtering. +TTL doesn't guarantee that the item will be deleted exactly at `expiration_time`, it might happen later. If it's important to exclude logically obsolete but not yet physically deleted items from the selection, use query-level filtering. {% endnote %} @@ -49,7 +49,7 @@ Data is deleted by the *Background Removal Operation* (*BRO*), consisting of two * `Uint32` * `Uint64` * `DyNumber` -* The value of the TTL column of the numeric type (`Uint32`, `Uint64`, or `DyNumber`) is interpreted as a value from the [Unix era]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Unix_time){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Unix-время ){% endif %} set in: +* The value in the TTL column with a numeric type (`Uint32`, `Uint64`, `DyNumber`) is interpreted as a [Unix-time]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Unix_time){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Unix-время ){% endif %} value specified in: * Seconds * Milliseconds * Microseconds @@ -241,3 +241,4 @@ The current TTL settings can be obtained from the table description: ``` {% endlist %} + diff --git a/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg b/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg index b0ff8b6a77..502ed357b3 100644 --- a/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg +++ b/ydb/docs/en/core/concepts/cluster/_assets/Slide_blob.svg @@ -1,3 +1,3 @@ <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> -<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="691px" height="551px" viewBox="-0.5 -0.5 691 551" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T16:05:40.464Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="HF0g-8VF_2eGPBkH-fRg" version="12.7.0" type="device"><diagram id="LqqcKPgxwdzCRuipa-4Q" name="Page-1">3ZtRc5s4EMc/DTO9h3SQBBg/GidNc3PXyYw7vV7fFKMYWox8shzb/fQnjADDUocktpH74qBFYPhp/9LuKrbIeL65FXQR/c1DlljYDjcWubYwxraH1Z/Mss0tCA2d3DITcahtlWES/2TaaGvrKg7ZstZRcp7IeFE3Tnmasqms2agQfF3v9siT+rcu6IwBw2RKE2j9Jw5llFt9PKjsH1k8i4pvRt4wPzOnRWf9JsuIhny9ZyI3FhkLzmV+NN+MWZLRK7jk1334xdnywQRLZZcLxt6Hb1+Cu/8E+7ZBn/5EH++/Ply5bn6bJ5qs9Bvrp5XbAoHgqzRk2V1siwTrKJZssqDT7OxajbqyRXKeqBZSh488lXoUka/aNIlnqWok7FE9ZQAfWr/HExOSbfZM+iVuGZ8zKbaqiz5LNE/tUaTwlHU1PG7BPNobGmJrI9UuMStvXVFTBxrcSyAOnofIQuVWusmFjPiMpzS5qaxBHXPV5y/OFxrudyblVtOlK8kPo1eIxfZrdr/3btH8V99+17je1Fpb3frlGC35SkzZARCFQKmYMdnB6zIoB4dcsITK+Kmuxbbx05fe81g9c+kq2K77Cm66QP6k+qqGF5SP8XrHQMAvPtOHhMm78G0iO4KKkFdH40MV+S0ick6lIYcYoaHzawZ31Awankkz/us0MxKCbve6LbIOy+7fgwaN5avRH9nuof7qIH+CowoYA6ccR1QFGknv+i2VaIp+OwQSv6V+SUf94jOtecQ7j37B95DD+iUN/Tb6n0a/cFG5ZSkTii9Pe5cwwYZJGMCaSLboHZNjm4UJTnRjzn/ErH9QvlmgPAAqSPjDburuG5VrWPQLE8h7KkxIEzzDwgwfgAKIlhFdZIfTlUi2gaDTH9m6/ByrCmzekvkaQa6vhgfjgxfIs1G4uEIOhNlWuCid9eg0h60KbXE79Yqyzquo60wVECaUIQMRT2ky0ifmcRjmMR1bxj+ztFc7rQ4s1H3dwHKvs3upMG6pCxdHYo0ayYLbUiRqQY1PRbooUu2h9pyLx4z8OmYPUiZnpQwLLv7FQ8YD+znIbbPG6SDDpJjAWfjSKBPHMFeGqctvQNnBhlGGOQ++/GnZtQ2jDMPbbF6232Hvj8uHPXgW9nkjDRgiX75He6atgTBy/pxv7UwkF3TG7tJHDqi/KHlr8pdZBfQoc3DdX8sq1B5L1ArTPxVNbMb+aVU7rlWO3+Oyknz8zR+N1JjqMSYN7xg0Bv3EO6YY5knljktRB+yzbtSg05JWIrtFOyerh2CY8VS8kHG8hr3zgslLxQvOOj3zKktG/fGCaUjF65NpvNz+9Qh3B3RgoLEZHBg0IwOna2SAThYZEDN2ld+wwA87LvDEOfYC/zY/hgGubd1gyx9aI2fGUgt7dJ75XZLlEaj/lRlMnb1PBQTWFm4VlUW2MFflpj73C4lxyMxIBF4vd9I1nieeUXInMOxGh/SO+/desPbbvXsv3PAuBO86BPcfjTcFbwAyuMxcmOC7/ssz8c0SPMwbM1HXFG+awPtfnmBp1ewVvX9kbss/bCgYo91nYO8+0e4z2H26lmLh52dHuk+T6lmL180fVRxjlJpVkZY8a9gySOTlg6Sa1Q+M8hpd9TstcvM/</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="0" y="250" width="520" height="300" fill="#ffffff" stroke="#000000" pointer-events="all"/><path d="M 170 60 L 170 155 L 260 155 L 260 243.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 260 248.88 L 256.5 241.88 L 260 243.63 L 263.5 241.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="130" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 131px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TabletId</div></div></div></foreignObject><text x="170" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TabletId</text></switch></g><path d="M 250 60 L 250 110 L 75 110 L 75 253.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 75 258.88 L 71.5 251.88 L 75 253.63 L 78.5 251.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="210" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 211px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel</div></div></div></foreignObject><text x="250" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel</text></switch></g><path d="M 330 60 L 330 170 L 275 170 L 275 353.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 275 358.88 L 271.5 351.88 L 275 353.63 L 278.5 351.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="290" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 291px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Generation</div></div></div></foreignObject><text x="330" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Generation</text></switch></g><rect x="370" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 371px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Step</div></div></div></foreignObject><text x="410" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Step</text></switch></g><rect x="450" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 451px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Cookie</div></div></div></foreignObject><text x="490" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Cookie</text></switch></g><rect x="530" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 531px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobSize</div></div></div></foreignObject><text x="570" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobSize</text></switch></g><rect x="610" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PartId</div></div></div></foreignObject><text x="650" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PartId</text></switch></g><path d="M 420 -200 L 415 -200 Q 410 -200 410 -190 L 410 70 Q 410 80 405 80 L 402.5 80 Q 400 80 405 80 L 407.5 80 Q 410 80 410 90 L 410 350 Q 410 360 415 360 L 420 360" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(-90,410,80)" pointer-events="all"/><rect x="385" y="90" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 100px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">BlobId</div></div></div></foreignObject><text x="410" y="104" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobId</text></switch></g><rect x="155" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 170px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">64</div></div></div></foreignObject><text x="170" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">64</text></switch></g><rect x="240" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">8</div></div></div></foreignObject><text x="250" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="315" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 330px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="330" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="395" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="410" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="475" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 490px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">24</div></div></div></foreignObject><text x="490" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">24</text></switch></g><rect x="545" y="0" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 570px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">28 (26)</div></div></div></foreignObject><text x="570" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">28 (26)</text></switch></g><rect x="640" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 650px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">4</div></div></div></foreignObject><text x="650" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="15" y="260" width="120" height="280" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 267px; margin-left: 16px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletStorageInfo</div></div></div></foreignObject><text x="75" y="279" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletStorageInfo</text></switch></g><path d="M 125 310 L 170 310 L 170 405 L 208.63 405" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 213.88 405 L 206.88 408.5 L 208.63 405 L 206.88 401.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="25" y="290" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 310px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 0</div></div></div></foreignObject><text x="75" y="314" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 0</text></switch></g><rect x="25" y="330" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 350px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 1</div></div></div></foreignObject><text x="75" y="354" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 1</text></switch></g><rect x="25" y="370" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 390px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 2</div></div></div></foreignObject><text x="75" y="394" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 2</text></switch></g><rect x="25" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel N</div></div></div></foreignObject><text x="75" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel N</text></switch></g><rect x="215" y="360" width="120" height="180" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 367px; margin-left: 216px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletChannelInfo</div></div></div></foreignObject><text x="275" y="379" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletChannelInfo</text></switch></g><path d="M 325 410 L 398.63 410" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 410 L 396.88 413.5 L 398.63 410 L 396.88 406.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0≤gen<10</div></div></div></foreignObject><text x="275" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0≤gen<10</text></switch></g><rect x="405" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><path d="M 325 460 L 398.63 460" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 460 L 396.88 463.5 L 398.63 460 L 396.88 456.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">10≤gen<125</div></div></div></foreignObject><text x="275" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">10≤gen<125</text></switch></g><rect x="405" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 54321</div></div></div></foreignObject><text x="455" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 54321</text></switch></g><path d="M 325 510 L 398.63 510" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 510 L 396.88 513.5 L 398.63 510 L 396.88 506.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">125≤gen</div></div></div></foreignObject><text x="275" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">125≤gen</text></switch></g><rect x="405" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><rect x="425" y="260" width="90" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 275px; margin-left: 470px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 18px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Таблетка</div></div></div></foreignObject><text x="470" y="280" fill="#000000" font-family="Helvetica" font-size="18px" text-anchor="middle">Таблетка</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file +<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="691px" height="551px" viewBox="-0.5 -0.5 691 551" content="<mxfile host="drawio.yandex-team.ru" modified="2021-07-15T16:05:40.464Z" agent="Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36" etag="HF0g-8VF_2eGPBkH-fRg" version="12.7.0" type="device"><diagram id="LqqcKPgxwdzCRuipa-4Q" name="Page-1">3ZtRc5s4EMc/DTO9h3SQBBg/GidNc3PXyYw7vV7fFKMYWox8shzb/fQnjADDUocktpH74qBFYPhp/9LuKrbIeL65FXQR/c1DlljYDjcWubYwxraH1Z/Mss0tCA2d3DITcahtlWES/2TaaGvrKg7ZstZRcp7IeFE3Tnmasqms2agQfF3v9siT+rcu6IwBw2RKE2j9Jw5llFt9PKjsH1k8i4pvRt4wPzOnRWf9JsuIhny9ZyI3FhkLzmV+NN+MWZLRK7jk1334xdnywQRLZZcLxt6Hb1+Cu/8E+7ZBn/5EH++/Ply5bn6bJ5qs9Bvrp5XbAoHgqzRk2V1siwTrKJZssqDT7OxajbqyRXKeqBZSh488lXoUka/aNIlnqWok7FE9ZQAfWr/HExOSbfZM+iVuGZ8zKbaqiz5LNE/tUaTwlHU1PG7BPNobGmJrI9UuMStvXVFTBxrcSyAOnofIQuVWusmFjPiMpzS5qaxBHXPV5y/OFxrudyblVtOlK8kPo1eIxfZrdr/3btH8V99+17je1Fpb3frlGC35SkzZARCFQKmYMdnB6zIoB4dcsITK+Kmuxbbx05fe81g9c+kq2K77Cm66QP6k+qqGF5SP8XrHQMAvPtOHhMm78G0iO4KKkFdH40MV+S0ick6lIYcYoaHzawZ31Awankkz/us0MxKCbve6LbIOy+7fgwaN5avRH9nuof7qIH+CowoYA6ccR1QFGknv+i2VaIp+OwQSv6V+SUf94jOtecQ7j37B95DD+iUN/Tb6n0a/cFG5ZSkTii9Pe5cwwYZJGMCaSLboHZNjm4UJTnRjzn/ErH9QvlmgPAAqSPjDburuG5VrWPQLE8h7KkxIEzzDwgwfgAKIlhFdZIfTlUi2gaDTH9m6/ByrCmzekvkaQa6vhgfjgxfIs1G4uEIOhNlWuCid9eg0h60KbXE79Yqyzquo60wVECaUIQMRT2ky0ifmcRjmMR1bxj+ztFc7rQ4s1H3dwHKvs3upMG6pCxdHYo0ayYLbUiRqQY1PRbooUu2h9pyLx4z8OmYPUiZnpQwLLv7FQ8YD+znIbbPG6SDDpJjAWfjSKBPHMFeGqctvQNnBhlGGOQ++/GnZtQ2jDMPbbF6232Hvj8uHPXgW9nkjDRgiX75He6atgTBy/pxv7UwkF3TG7tJHDqi/KHlr8pdZBfQoc3DdX8sq1B5L1ArTPxVNbMb+aVU7rlWO3+Oyknz8zR+N1JjqMSYN7xg0Bv3EO6YY5knljktRB+yzbtSg05JWIrtFOyerh2CY8VS8kHG8hr3zgslLxQvOOj3zKktG/fGCaUjF65NpvNz+9Qh3B3RgoLEZHBg0IwOna2SAThYZEDN2ld+wwA87LvDEOfYC/zY/hgGubd1gyx9aI2fGUgt7dJ75XZLlEaj/lRlMnb1PBQTWFm4VlUW2MFflpj73C4lxyMxIBF4vd9I1nieeUXInMOxGh/SO+/desPbbvXsv3PAuBO86BPcfjTcFbwAyuMxcmOC7/ssz8c0SPMwbM1HXFG+awPtfnmBp1ewVvX9kbss/bCgYo91nYO8+0e4z2H26lmLh52dHuk+T6lmL180fVRxjlJpVkZY8a9gySOTlg6Sa1Q+M8hpd9TstcvM/</diagram></mxfile>" style="background-color: rgb(255, 255, 255);"><defs/><g><rect x="0" y="250" width="520" height="300" fill="#ffffff" stroke="#000000" pointer-events="all"/><path d="M 170 60 L 170 155 L 260 155 L 260 243.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 260 248.88 L 256.5 241.88 L 260 243.63 L 263.5 241.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="130" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 131px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TabletId</div></div></div></foreignObject><text x="170" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TabletId</text></switch></g><path d="M 250 60 L 250 110 L 75 110 L 75 253.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 75 258.88 L 71.5 251.88 L 75 253.63 L 78.5 251.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="210" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 211px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel</div></div></div></foreignObject><text x="250" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel</text></switch></g><path d="M 330 60 L 330 170 L 275 170 L 275 353.63" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 275 358.88 L 271.5 351.88 L 275 353.63 L 278.5 351.88 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="290" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 291px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Generation</div></div></div></foreignObject><text x="330" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Generation</text></switch></g><rect x="370" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 371px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Step</div></div></div></foreignObject><text x="410" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Step</text></switch></g><rect x="450" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 451px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Cookie</div></div></div></foreignObject><text x="490" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Cookie</text></switch></g><rect x="530" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 531px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">BlobSize</div></div></div></foreignObject><text x="570" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobSize</text></switch></g><rect x="610" y="20" width="80" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 78px; height: 1px; padding-top: 40px; margin-left: 611px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">PartId</div></div></div></foreignObject><text x="650" y="44" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">PartId</text></switch></g><path d="M 420 -200 L 415 -200 Q 410 -200 410 -190 L 410 70 Q 410 80 405 80 L 402.5 80 Q 400 80 405 80 L 407.5 80 Q 410 80 410 90 L 410 350 Q 410 360 415 360 L 420 360" fill="none" stroke="#000000" stroke-miterlimit="10" transform="rotate(-90,410,80)" pointer-events="all"/><rect x="385" y="90" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 100px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">BlobId</div></div></div></foreignObject><text x="410" y="104" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">BlobId</text></switch></g><rect x="155" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 170px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">64</div></div></div></foreignObject><text x="170" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">64</text></switch></g><rect x="240" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 250px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">8</div></div></div></foreignObject><text x="250" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">8</text></switch></g><rect x="315" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 330px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="330" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="395" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 410px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">32</div></div></div></foreignObject><text x="410" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">32</text></switch></g><rect x="475" y="0" width="30" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 490px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">24</div></div></div></foreignObject><text x="490" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">24</text></switch></g><rect x="545" y="0" width="50" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 570px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">28 (26)</div></div></div></foreignObject><text x="570" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">28 (26)</text></switch></g><rect x="640" y="0" width="20" height="20" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 10px; margin-left: 650px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">4</div></div></div></foreignObject><text x="650" y="14" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">4</text></switch></g><rect x="15" y="260" width="120" height="280" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 267px; margin-left: 16px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletStorageInfo</div></div></div></foreignObject><text x="75" y="279" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletStorageInfo</text></switch></g><path d="M 125 310 L 170 310 L 170 405 L 208.63 405" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 213.88 405 L 206.88 408.5 L 208.63 405 L 206.88 401.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="25" y="290" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 310px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 0</div></div></div></foreignObject><text x="75" y="314" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 0</text></switch></g><rect x="25" y="330" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 350px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 1</div></div></div></foreignObject><text x="75" y="354" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 1</text></switch></g><rect x="25" y="370" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 390px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel 2</div></div></div></foreignObject><text x="75" y="394" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel 2</text></switch></g><rect x="25" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 26px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Channel N</div></div></div></foreignObject><text x="75" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Channel N</text></switch></g><rect x="215" y="360" width="120" height="180" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe flex-start; justify-content: unsafe center; width: 118px; height: 1px; padding-top: 367px; margin-left: 216px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">TTabletChannelInfo</div></div></div></foreignObject><text x="275" y="379" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">TTabletChannelInfo</text></switch></g><path d="M 325 410 L 398.63 410" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 410 L 396.88 413.5 L 398.63 410 L 396.88 406.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">0≤gen<10</div></div></div></foreignObject><text x="275" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">0≤gen<10</text></switch></g><rect x="405" y="390" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 410px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="414" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><path d="M 325 460 L 398.63 460" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 460 L 396.88 463.5 L 398.63 460 L 396.88 456.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">10≤gen<125</div></div></div></foreignObject><text x="275" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">10≤gen<125</text></switch></g><rect x="405" y="440" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 460px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 54321</div></div></div></foreignObject><text x="455" y="464" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 54321</text></switch></g><path d="M 325 510 L 398.63 510" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="stroke"/><path d="M 403.88 510 L 396.88 513.5 L 398.63 510 L 396.88 506.5 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="all"/><rect x="225" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 226px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">125≤gen</div></div></div></foreignObject><text x="275" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">125≤gen</text></switch></g><rect x="405" y="490" width="100" height="40" fill="#ffffff" stroke="#000000" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 98px; height: 1px; padding-top: 510px; margin-left: 406px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 12px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: normal; word-wrap: normal; ">Group 12345</div></div></div></foreignObject><text x="455" y="514" fill="#000000" font-family="Helvetica" font-size="12px" text-anchor="middle">Group 12345</text></switch></g><rect x="425" y="260" width="90" height="30" fill="none" stroke="none" pointer-events="all"/><g transform="translate(-0.5 -0.5)"><switch><foreignObject style="overflow: visible; text-align: left;" pointer-events="none" width="100%" height="100%" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: flex; align-items: unsafe center; justify-content: unsafe center; width: 1px; height: 1px; padding-top: 275px; margin-left: 470px;"><div style="box-sizing: border-box; font-size: 0; text-align: center; "><div style="display: inline-block; font-size: 18px; font-family: Helvetica; color: #000000; line-height: 1.2; pointer-events: all; white-space: nowrap; ">Tablet</div></div></div></foreignObject><text x="470" y="280" fill="#000000" font-family="Helvetica" font-size="18px" text-anchor="middle">Tablet</text></switch></g></g><switch><g requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"/><a transform="translate(0,-5)" xlink:href="https://desk.draw.io/support/solutions/articles/16000042487" target="_blank"><text text-anchor="middle" font-size="10px" x="50%" y="100%">Viewer does not support full SVG 1.1</text></a></switch></svg>
\ No newline at end of file diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md index d3d6cd7701..a00da63b70 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/intro.md @@ -3,3 +3,4 @@ An approximate general {{ ydb-short-name }} schema is shown below.  + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md index 3ecbfb22b3..5f38895738 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/nodes.md @@ -9,3 +9,4 @@ Nodes can be static and dynamic. A configuration of static nodes, that is, their complete list with the address for connecting via Interconnect specified, is stored in a configuration file and is read once when the process is started. A set of static nodes changes very rarely. This usually happens when expanding clusters or moving nodes from one physical machine to another. To change a set of static nodes, you must apply the updated configuration to **every** node and then perform a rolling restart of the entire cluster. Dynamic nodes are not known in advance and are added to the system as new processes are started. This may be due, for example, to the creation of new tenants in {{ ydb-short-name }} installations as a database. When a dynamic node is registered, its process first connects to one of the static nodes via gRPC, transmits information about itself through a special service called Node Broker, and receives a NodeID to use to log into the system. The mechanism for assigning nodes is pretty much similar to the DHCP in the context of distributing IP addresses. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md index 1e8e2eaf0c..e61d309020 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/common_scheme_ydb/tablets.md @@ -1,6 +1,6 @@ ## Tablets {#tablets} -Special microservices called *tablets* are run on each node. Each tablet has a specific type and ID and is a singleton meaning that only one tablet with a specific ID can be running in the entire cluster at any given time. A tablet can launch on any suitable node. *Generation* is an important property of a tablet that increases with each subsequent launch. Please note that the distributed nature of the system and various kinds of issues, such as problems with network partitioning, may result in a situation whereby the same tablet will actually be running on two different nodes at the same time. However, BlobStorage guarantees that only one of them will be able successfully to complete operations that change its state and that the generation that each successful operation runs in will not decrease over time. +Special microservices called *tablets* are run on each node. Each tablet has a specific type and ID and is a singleton meaning that only one tablet with a specific ID can be running in the entire cluster at any given time. A tablet can launch on any suitable node. *Generation* is an important property of a tablet that increases with each subsequent launch. Please note that the distributed nature of the system and various kinds of issues, such as problems with network partitioning, may result in a situation whereby the same tablet will actually be running on two different nodes at the same time. However, BlobStorage guarantees that only one of them will be able successfully to complete operations that change its state and that the generation that each successful operation runs in will not decrease over time. You can find out on what node the tablet in the current generation is running through the *StateStorage* service. To send messages to tablets, use a special set of libraries named *tablet pipe*. With it, knowing the ID of the target tablet, you can easily send the desired message to it. @@ -35,3 +35,4 @@ This mechanism works as follows: For each channel, the TTabletStorageInfo structure contains the TTabletChannelInfo substructure with generation ranges and the group number corresponding to each range. The ranges are strictly adjacent to each other, the last range is open. Group numbers may overlap in different ranges and even across different channels: this is legal and quite common. When writing a blob, a tablet selects the most recent range for the corresponding channel since a write is always performed on behalf of a tablet's current generation. When reading a blob, the group number is fetched based on the BlobId.Generation of the blob being read. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md index 5ab8faaddd..64aece9883 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/detailed_distributed_storage.md @@ -354,3 +354,4 @@ VSlot ### Messages via pipe ### Transaction types + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md index 98aadd8cb7..16003775be 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface.md @@ -25,7 +25,7 @@ When performing reads, the blob ID is specified, which can be arbitrary, but pre Blobs are written in a logical entity called *group*. A special actor called DS proxy is created on every node for each group that is written to. This actor is responsible for performing all operations related to the group. The actor is created automatically through the NodeWarden service that will be described below. -Physically, a group is a set of multiple physical devices (OS block devices) that are located on different nodes so that the failure of one device correlates as little as possible with the failure of another device. These devices are usually located in different racks or different datacenters. On each of these devices, some space is allocated for the group, which is managed by a special service called *VDisk*. Each VDisk runs on top of a block storage device from which it is separated by another service called *PDisk*. Blobs are broken into fragments based on *erasure coding* with these fragments written to VDisks. Before splitting into fragments, optional encryption of the data in the group can be performed. +Physically, a group is a set of multiple physical devices (OS block devices) that are located on different nodes so that the failure of one device correlates as little as possible with the failure of another device. These devices are usually located in different racks or different datacenters. On each of these devices, some space is allocated for the group, which is managed by a special service called _VDisk_. Each VDisk runs on top of a block storage device from which it is separated by another service called _PDisk_. Blobs are broken into fragments based on _erasure coding_ with these fragments written to VDisks. Before splitting into fragments, optional encryption of the data in the group can be performed. This scheme is shown in the figure below. @@ -45,29 +45,30 @@ All fail realms have the same number of fail domains, and all fail domains inclu Each PDisk has an ID that consists of the number of the node that it is running on and the internal number of the PDisk inside this node. This ID is usually written as NodeId:PDiskId. For example, 1:1000. If you know the PDisk ID, you can calculate the service ActorId of this disk and send it a message. -Each VDisk runs on top a specific PDisk and has a *slot ID* comprising three fields (NodeID:PDiskId:VSlotId) as well as the above-mentioned VDisk ID. Strictly speaking, there are different concepts: a slot is a reserved location on a PDISK occupied by a VDisk while a VDisk is an element of a group that occupies a certain slot and performs operations with the slot. Similarly to PDisks, if you know the slot ID, you can calculate the service ActorId of the running VDisk and send it a message. To send messages from the DS proxy to the VDisk, an intermediate actor called *BS_QUEUE* is used. +Each VDisk runs on top a specific PDisk and has a _slot ID_ comprising three fields (NodeID:PDiskId:VSlotId) as well as the above-mentioned VDisk ID. Strictly speaking, there are different concepts: a slot is a reserved location on a PDISK occupied by a VDisk while a VDisk is an element of a group that occupies a certain slot and performs operations with the slot. Similarly to PDisks, if you know the slot ID, you can calculate the service ActorId of the running VDisk and send it a message. To send messages from the DS proxy to the VDisk, an intermediate actor called _BS_QUEUE_ is used. -The composition of each group is not constant. It may change while the system is running. Hence the concept of a group generation. Each "GroupId:GroupGeneration" pair corresponds to a fixed set of slots (a vector that consists of N slot IDs, where N is equal to group size) that stores the data of an entire group. *Group generation is not to be confused with tablet generation since they are not in any way related*. +The composition of each group is not constant. It may change while the system is running. Hence the concept of a group generation. Each "GroupId:GroupGeneration" pair corresponds to a fixed set of slots (a vector that consists of N slot IDs, where N is equal to group size) that stores the data of an entire group. _Group generation is not to be confused with tablet generation since they are not in any way related_. As a rule, groups of two adjacent generations differ by no more than one slot. ### Subgroups -A special concept of a *subgroup* is introduced for each blob. It is an ordered subset of group disks with a strictly constant number of elements that will store the blob's data and that depends on the encoding type (the number of elements in a group must be the same or greater). For single-datacenter groups with conventional encoding, this subset is selected as the first N elements of a cyclic disk permutation in the group, where the permutation depends on the BlobId hash. +A special concept of a _subgroup_ is introduced for each blob. It is an ordered subset of group disks with a strictly constant number of elements that will store the blob's data and that depends on the encoding type (the number of elements in a group must be the same or greater). For single-datacenter groups with conventional encoding, this subset is selected as the first N elements of a cyclic disk permutation in the group, where the permutation depends on the BlobId hash. Each disk in the subgroup corresponds to a disk in the group, but is limited by the allowed number of stored blobs. For example, for block-4-2 encoding with four data parts and two parity parts, the functional purpose of the disks in a subgroup is as follows: | Number in the subgroup | Possible PartIds | -| ---------------------- | ---------------- | -| 0 | 1 | -| 1 | 2 | -| 2 | 3 | -| 3 | 4 | -| 4 | 5 | -| 5 | 6 | -| 6 | 1,2,3,4,5,6 | -| 7 | 1,2,3,4,5,6 | - -In this case, PartID=1..4 corresponds to the data parts (which are obtained by splitting the original blob into 4 equal parts), and PartID=5..6 are parity fragments. Disks numbered 6 and 7 in the subgroup are called *handoff disks*. Any part, either one or more, can be written to them. Disks 0..5 can only store the corresponding blob parts. +| ------------------- | ------------------- | +| 0 | 1 | +| 1 | 2 | +| 2 | 3 | +| 3 | 4 | +| 4 | 5 | +| 5 | 6 | +| 6 | 1,2,3,4,5,6 | +| 7 | 1,2,3,4,5,6 | + +In this case, PartID=1..4 corresponds to the data parts (which are obtained by splitting the original blob into 4 equal parts), and PartID=5..6 are parity fragments. Disks numbered 6 and 7 in the subgroup are called _handoff disks_. Any part, either one or more, can be written to them. Disks 0..5 can only store the corresponding blob parts. In practice, when performing writes, the system tries to write 6 parts to the first 6 disks of the subgroup and, in the vast majority of cases, these attempts are successful. However, if any of the disks is not available, a write operation cannot succeed, which is when handoff disks kick in receiving the parts belonging to the disks that did not respond in time. It may turn out that several fragments of the same blob are sent to the same handoff disk as a result of complicated brakes and races. This is acceptable although it makes no sense in terms of storage: each fragment must have its own unique disk. + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md index de01b80777..05d8f54d26 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/distributed_storage_interface_hidden.md @@ -1,3 +1,4 @@ ### Garbage collection ### Tablet suspension + diff --git a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md index 92b360ec02..cb1e4205f7 100644 --- a/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md +++ b/ydb/docs/en/core/concepts/cluster/_includes/distributed_storage/intro.md @@ -3,3 +3,4 @@ {{ ydb-short-name }} BlobStorage is a subsystem {{ ydb-short-name }} that assures reliable data storage. Lets you store *blobs* (binary fragments from 1 byte to 10 megabytes in size) with a unique identifier. + diff --git a/ydb/docs/en/core/concepts/cluster/index.md b/ydb/docs/en/core/concepts/cluster/index.md index 7eb85e1dc3..747731bb96 100644 --- a/ydb/docs/en/core/concepts/cluster/index.md +++ b/ydb/docs/en/core/concepts/cluster/index.md @@ -5,3 +5,4 @@ The information in this section is mainly intended for {{ ydb-short-name }} admi The [General schema {{ ydb-short-name }}](common_scheme_ydb.md) provides general information on nodes and tablets. The [Cluster disk subsystem](distributed_storage.md) section presents more detail on the features of a distributed {{ ydb-short-name }} storage system. + diff --git a/ydb/docs/en/core/concepts/connect.md b/ydb/docs/en/core/concepts/connect.md index c0cae8a17b..baae235208 100644 --- a/ydb/docs/en/core/concepts/connect.md +++ b/ydb/docs/en/core/concepts/connect.md @@ -1,2 +1 @@ - {% include [connect.md](_includes/connect.md) %} diff --git a/ydb/docs/en/core/concepts/datatypes.md b/ydb/docs/en/core/concepts/datatypes.md index 6fd29cf5df..7676dc957a 100644 --- a/ydb/docs/en/core/concepts/datatypes.md +++ b/ydb/docs/en/core/concepts/datatypes.md @@ -1 +1,2 @@ -This page has been deleted, for information about data types, see [YQL Data Types](../yql/reference/types/index.md).
\ No newline at end of file +This page has been deleted, for information about data types, see[YQL Data Types](../yql/reference/types/index.md). + diff --git a/ydb/docs/en/core/concepts/toc_i.yaml b/ydb/docs/en/core/concepts/toc_i.yaml index a925de540d..fa3795c630 100644 --- a/ydb/docs/en/core/concepts/toc_i.yaml +++ b/ydb/docs/en/core/concepts/toc_i.yaml @@ -1,15 +1,15 @@ items: -- { name: Overview, href: index.md } -- { name: Terms and definitions, href: databases.md } -- { name: Connecting to and authenticating with a database, href: connect.md } -- { name: Data model and schema, href: datamodel.md } -- { name: Serverless and Dedicated operation modes, href: serverless_and_dedicated.md } -- { name: Data types, href: datatypes.md, hidden: true } # Deprecated -- { name: Transactions, href: transactions.md } -- { name: Secondary indexes, href: secondary_indexes.md } -- { name: Time to Live (TTL), href: ttl.md } -- { name: Scan queries, href: scan_query.md } -- { name: Database limits, href: limits-ydb.md } +- { name: Overview, href: index.md } +- { name: Terms and definitions, href: databases.md } +- { name: Connecting to and authenticating with a database, href: connect.md } +- { name: Data model and schema, href: datamodel.md } +- { name: Serverless and Dedicated operation modes, href: serverless_and_dedicated.md } +- { name: Data types, href: datatypes.md, hidden: true } # Deprecated +- { name: Transactions, href: transactions.md } +- { name: Secondary indexes, href: secondary_indexes.md } +- { name: Time to Live (TTL), href: ttl.md } +- { name: Scan queries, href: scan_query.md } +- { name: Database limits, href: limits-ydb.md } - name: YDB cluster items: - name: Overview diff --git a/ydb/docs/en/core/db/_includes/cloud_remark.md b/ydb/docs/en/core/db/_includes/cloud_remark.md new file mode 100644 index 0000000000..b56e309914 --- /dev/null +++ b/ydb/docs/en/core/db/_includes/cloud_remark.md @@ -0,0 +1,2 @@ +YDB cloud services provide self-service tools for creating, modifying, and deleting databases. Documentation for these features is available as part of the documentation for the respective cloud services. + diff --git a/ydb/docs/en/core/db/index.md b/ydb/docs/en/core/db/index.md new file mode 100644 index 0000000000..395e716613 --- /dev/null +++ b/ydb/docs/en/core/db/index.md @@ -0,0 +1,8 @@ +# Managing databases + +This section contains articles describing YDB tools for database management and maintenance intended for SREs and application developers and administrators. + +{% include [cloud_remark.md](_includes/cloud_remark.md) %} + +In YDB non-cloud configurations, the operations for creating, modifying, and deleting databases are run by cluster administrators. For operation descriptions, see [Managing a cluster](../cluster/index.md). + diff --git a/ydb/docs/en/core/db/toc_i.yaml b/ydb/docs/en/core/db/toc_i.yaml new file mode 100644 index 0000000000..124ff5cab6 --- /dev/null +++ b/ydb/docs/en/core/db/toc_i.yaml @@ -0,0 +1,9 @@ +items: +# - name: Cloud Yandex.Cloud +# include: { mode: link, path: yandexcloud/toc_i.yaml } +# - name: Internal cloud Yandex-Team +# include: { mode: link, path: yandex-team/toc_i.yaml } +- name: Backup and recovery + href: ../maintenance/backup_and_recovery.md +- name: Diagnostics + include: { mode: link, path: ../troubleshooting/toc_p.yaml } diff --git a/ydb/docs/en/core/db/toc_p.yaml b/ydb/docs/en/core/db/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/db/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/deploy/_includes/index.md b/ydb/docs/en/core/deploy/_includes/index.md index 441adb6a58..1c3bff01ca 100644 --- a/ydb/docs/en/core/deploy/_includes/index.md +++ b/ydb/docs/en/core/deploy/_includes/index.md @@ -1,9 +1,14 @@ -# Administration of YDB clusters +# Deploying databases and clusters -This section provides information on the deployment and configuration of YDB clusters for administrators. Step-by-step tutorials for quick deployment intended for application developers are given in [Getting Started - Self-deployment](../../getting_started/self_hosted/index.md). +This section provides information on the deployment and configuration of YDB databases and clusters. + +Step-by-step tutorials for self-deployment of local DBs intended for application developers are given in [Getting started - Self-deployment](../../getting_started/self_hosted/index.md). + +Information on cluster deployment and configuration is intended for administrators. {% note warning %} The section is under development. -{% endnote %}
\ No newline at end of file +{% endnote %} + diff --git a/ydb/docs/en/core/deploy/configuration/config.md b/ydb/docs/en/core/deploy/configuration/config.md index 9ea38d81fa..e15733e534 100644 --- a/ydb/docs/en/core/deploy/configuration/config.md +++ b/ydb/docs/en/core/deploy/configuration/config.md @@ -207,3 +207,4 @@ For a configuration located in 3 availability zones, specify 3 rings. For a conf ## Sample cluster configurations The [repository](https://github.com/ydb-platform/ydb/tree/main/ydb/deploy/yaml_config_examples/) provides model examples of cluster configurations for self-deployment. Check them out before deploying a cluster. + diff --git a/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md b/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md index 9e3fb6b77d..b11fc4b957 100644 --- a/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md +++ b/ydb/docs/en/core/deploy/manual/_includes/generate-ssl.md @@ -107,4 +107,5 @@ openssl ca -config ca.cnf -keyfile secure/ca.key -cert certs/ca.crt -policy sign -extensions signing_node_req -out certs/node.crt -outdir certs/ -in node.csr -batch ``` -Create similar certificate-key pairs for each node.
\ No newline at end of file +Create similar certificate-key pairs for each node. + diff --git a/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md b/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md index 1246a554e0..e907b0c060 100644 --- a/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md +++ b/ydb/docs/en/core/deploy/manual/_includes/prepare-configs.md @@ -41,4 +41,5 @@ hosts: body: 3 data_center: 'zone-c' rack: '1' -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/deploy/manual/concepts.md b/ydb/docs/en/core/deploy/manual/concepts.md index 6e5d0bd90f..e26c6e4df8 100644 --- a/ydb/docs/en/core/deploy/manual/concepts.md +++ b/ydb/docs/en/core/deploy/manual/concepts.md @@ -6,3 +6,4 @@ YDB consists of two components: * Static nodes that provide the data storage layer. * Dynamic nodes that implement data access and processing. Each database requires one or more separate dynamic nodes. + diff --git a/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md b/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md index 3a024353cb..ba509a6e57 100644 --- a/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md +++ b/ydb/docs/en/core/deploy/manual/deploy-ydb-on-premises.md @@ -29,7 +29,7 @@ sudo groupadd ydb sudo useradd ydb -g ydb ``` -To make sure the {{ ydb-short-name }} server has access to block store disks to run, add the user to start the process under to the disk group: +To make sure the {{ ydb-short-name }} server has access to block store disks to run, add the user to start the process under to the disk group. ```bash sudo usermod -aG disk ydb @@ -43,11 +43,11 @@ We don't recommend using disks that are used by other processes (including the O {% endnote %} -1. Create a partition on the selected disk. +1. Create a partition on the selected disk {% note alert %} -Be careful! The following step will delete all partitions on the specified disks! +Be careful! The following step will delete all partitions on the specified disks. Make sure that you specified the disks that have no other data! {% endnote %} @@ -78,7 +78,7 @@ mkdir /opt/ydb/bin mkdir /opt/ydb/cfg ``` -2. Copy the binary file, libraries, and configuration file to the appropriate directories: +3. Copy the binary file, libraries, and configuration file to the appropriate directories: ```bash sudo cp -i ydbd-main-linux-amd64/bin/ydbd /opt/ydb/bin/ @@ -87,7 +87,7 @@ sudo cp -i ydbd-main-linux-amd64/lib/libiconv.so /opt/ydb/lib/ sudo cp -i ydbd-main-linux-amd64/lib/libidn.so /opt/ydb/lib/ ``` -3. Format the disk with the built-in command: +3. Format the disk with the built-in command ```bash sudo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib /opt/ydb/bin/ydbd admin bs disk obliterate /dev/disk/by-partlabel/ydb_disk_01 @@ -105,7 +105,7 @@ Prepare the configuration files: {% include [prepare-configs.md](_includes/prepare-configs.md) %} - Save the {{ ydb-short-name }} configuration file as `/opt/ydb/cfg/config.yaml`. + Save the {{ ydb-short-name }} configuration file as `/opt/ydb/cfg/config.yaml` - Protected mode @@ -113,21 +113,20 @@ Prepare the configuration files: {% include [generate-ssl.md](_includes/generate-ssl.md) %} - Create directories for certificates on each node: + Create directories for certificates on each node ```bash mkdir /opt/ydb/certs chmod 0750 /opt/ydb/certs ``` - Copy the node certificates and keys: + Copy the node certificates and keys ```bash sudo -u ydb cp certs/ca.crt certs/node.crt certs/node.key /opt/ydb/certs/ ``` {% include [prepare-configs.md](_includes/prepare-configs.md) %} - 3. In the `interconnect_config` and `grpc_config` sections, specify the path to the certificate, key, and CA certificates: ```text @@ -137,7 +136,7 @@ Prepare the configuration files: path_to_certificate_file: "/opt/ydb/certs/node.crt" path_to_private_key_file: "/opt/ydb/certs/node.key" path_to_ca_file: "/opt/ydb/certs/ca.crt" - + grpc_config: cert: "/opt/ydb/certs/node.crt" key: "/opt/ydb/certs/node.key" @@ -175,7 +174,7 @@ Prepare the configuration files: Wants=network-online.target StartLimitInterval=10 StartLimitBurst=15 - + [Service] Restart=always RestartSec=1 @@ -191,7 +190,7 @@ Prepare the configuration files: LimitNOFILE=65536 LimitCORE=0 LimitMEMLOCK=3221225472 - + [Install] WantedBy=multi-user.target ``` @@ -232,7 +231,7 @@ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd admin database sudo su - ydb cd /opt/ydb export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib - /opt/ydbd/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml \ + /opt/ydbd/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml \ --tenant /Root/testdb --node-broker --node-broker --node-broker ``` @@ -260,7 +259,7 @@ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd admin database SyslogFacility=daemon SyslogLevel=err Environment=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib - ExecStart=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml --tenant /Root/testdb --node-broker --node-broker --node-broker + ExecStart=LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd server --grpc-port 2136 --ic-port 19002 --mon-port 8766 --yaml-config /opt/ydb/cfg/config.yaml --tenant /Root/testdb --node-broker --node-broker --node-broker LimitNOFILE=65536 LimitCORE=0 LimitMEMLOCK=32212254720 @@ -279,7 +278,7 @@ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/ydb/lib ; /opt/ydb/bin/ydbd admin database ## Test the created database {# try-first-db} -1. Install the YDB CLI as described in [Installing the YDB CLI](../../reference/ydb-cli/install.md). +1. Install the YDB CLI as described in [Installing the YDB CLI](../../reference/ydb-cli/install.md) 2. Create a `test_table`: ```bash @@ -288,3 +287,4 @@ ydb -e grpc://<node1.domain>:2136 -d /Root/testdb scripting yql \ ``` Where node.domain is the FQDN of the server running the dynamic nodes that support the `/Root/testdb` database. + diff --git a/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md b/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md index be98c709ca..4c7f3a5731 100644 --- a/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md +++ b/ydb/docs/en/core/deploy/orchestrated/yc_managed_kubernetes.md @@ -6,7 +6,7 @@ To use [{{ k8s }}](https://kubernetes.io/) to create a cluster [{{ ydb-short-nam 1. Create a {{ k8s }} cluster. - You can use an existing {{ k8s }} cluster or [create]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create){% endif %} a new one. + You can use an already running {{ k8s }} cluster or [create]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-create){% endif %} a new one. {% note warning %} @@ -16,7 +16,7 @@ To use [{{ k8s }}](https://kubernetes.io/) to create a cluster [{{ ydb-short-nam 1. Install the {{ k8s }} CLI [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl). -1. [Define]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials){% endif %} a kubectl configuration. +1. [Set]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/managed-kubernetes/operations/kubernetes-cluster/kubernetes-cluster-get-credetials){% endif %} the kubectl configuration. 1. Install the {{ k8s }} [Helm 3](https://helm.sh/docs/intro/install/) package manager. diff --git a/ydb/docs/en/core/deploy/production_checklist.md b/ydb/docs/en/core/deploy/production_checklist.md index cfb6b345fa..47b8fa6e2a 100644 --- a/ydb/docs/en/core/deploy/production_checklist.md +++ b/ydb/docs/en/core/deploy/production_checklist.md @@ -12,8 +12,8 @@ When planning a deployment, it's important to choose the right cluster topology TBD. mirror-3dc vs block-4-2: Whether we want the following in the doc: -- Much more network traffic. -- Higher storage overhead. +- Much more network traffic +- Higher storage overhead ## Hardware configuration @@ -22,4 +22,5 @@ For correct operation, we recommend using the x86_64 CPU architecture with suppo - SSE - ??? -## Software configuration
\ No newline at end of file +## Software configuration + diff --git a/ydb/docs/en/core/faq/_includes/common.md b/ydb/docs/en/core/faq/_includes/common.md index df3d0bd04d..44ddf58f73 100644 --- a/ydb/docs/en/core/faq/_includes/common.md +++ b/ydb/docs/en/core/faq/_includes/common.md @@ -6,7 +6,7 @@ description: "What is Yandex Database? For what tasks is it worth using Yandex D #### What is {{ ydb-short-name }}? {#what-is-ydb} -{{ ydb-short-name }} is a distributed fault-tolerant SQL DBMS. {{ ydb-short-name }} provides high availability and scalability while simultaneously ensuring strict consistency and ACID transaction support. Queries are made using an SQL dialect (YQL). +{{ ydb-short-name }} is a distributed fault-tolerant SQL DBMS. {{ ydb-short-name }} provides high availability and scalability while simultaneously ensuring strict consistency and ACID transaction support. Queries are made using an SQL dialect (YQL). {{ ydb-short-name }} is a fully managed database. DB instances are created through the {{ ydb-short-name }} database management service. @@ -20,13 +20,13 @@ To read data, {{ ydb-short-name }} uses a model of strict data consistency. #### How do I design a primary key? {#create-pk} -To design the primary key properly, follow the rules given below. +To design a primary key properly, follow the rules below. * Avoid situations where the main load falls on a single partition of a table. With even load distribution, it's easier to achieve high overall performance. This rule implies that you shouldn't use a monotonically increasing sequence, such as timestamp, as a table's primary key. -* The fewer table partitions used during query execution, the faster it's executed. For greater performance, follow the one query — one partition rule. +* The fewer table partitions a query uses, the faster it runs. For greater performance, follow the one query — one partition rule. * Avoid situations where a small part of the DB is under much heavier load than the rest of the DB. @@ -34,12 +34,11 @@ For more information, see [Schema design](../../best_practices/schema_design.md) #### How do I evenly distribute load across table partitions? {#balance-shard-load} -You can use the following techniques to evenly distribute the load across table partitions and increase overall DB performance. +You can use the following techniques to distribute the load evenly across table partitions and increase overall DB performance. * To avoid using uniformly increasing primary key values, you can: * Change the order of its components. * use a hash of the key column values as the primary key. - * Reduce the number of partitions used in a single query. For more information, see [Schema design](../../best_practices/schema_design.md#balance-shard-load). @@ -48,7 +47,7 @@ For more information, see [Schema design](../../best_practices/schema_design.md# In {{ ydb-short-name }}, all columns, including key ones, may contain a `NULL` value, but we don't recommend using ` NULL` as values in key columns. -According to the SQL standard (ISO/IEC 9075), you can't compare `NULL` with other values. Therefore, the use of concise SQL statements with simple comparison operators may lead to skipping rows containing NULL during filtering, for example. +Per the SQL standard (ISO/IEC 9075), you can't compare `NULL` with other values. Therefore, the use of concise SQL statements with simple comparison operators may result in rows containing NULL being skipped during filtering, for example. #### Is there an optimal size of a database row? {#string-size} @@ -70,14 +69,14 @@ For more information, see [Paginated output](../../best_practices/paging.md). #### How do I effectively upload large amounts of data to {{ ydb-short-name }}? {#batch_upload} -To speed up uploading large amounts of data, follow these recommendations: +To increase upload speed for large amounts of data, follow the recommendations below: * When creating a table, explicitly specify the required number of partitions or their boundaries. This will help you effectively use system bandwidth as soon as you start uploading data by avoiding unnecessary re-partitioning of the table. * Don't insert data in separate transactions for each row. It's more efficient to insert multiple rows at once (batch inserts). This reduces the overhead on the transaction mechanism itself. * In addition to the previous step, within each transaction (batch), insert rows from the primary key-sorted set of data to minimize the number of partitions that the transaction affects. * Avoid writing data sequentially in ascending or descending order of the primary key value to evenly distribute the load across all table partitions. -For more information, see [Uploading large data volumes](../../best_practices/batch_upload.md). +For more detail, see [Uploading large volumes of data](../../best_practices/batch_upload.md). #### How do I delete expired data? {#ttl} diff --git a/ydb/docs/en/core/faq/_includes/errors.md b/ydb/docs/en/core/faq/_includes/errors.md index a9c637963c..a46b79e2db 100644 --- a/ydb/docs/en/core/faq/_includes/errors.md +++ b/ydb/docs/en/core/faq/_includes/errors.md @@ -1,10 +1,10 @@ # Errors -#### What should I do if I frequently get the "Transaction locks invalidated" error? {#locks-invalidated} +#### What do I do if I frequently get the "Transaction locks invalidated" error? {#locks-invalidated} -Typically, if you get this error, repeat a transaction, as {{ ydb-short-name }} uses optimistic locking. If this error occurs frequently, it means that a large number of rows are being read in a transaction or that many transactions are competing for the same "hot" rows. It makes sense to view the queries running in the transaction and check if they're reading unnecessary rows. +Typically, if you get this error, repeat a transaction, as {{ ydb-short-name }} uses optimistic locking. If this error occurs frequently, this is the result of a transaction reading a large number of rows or of many transactions competing for the same "hot" rows. It makes sense to view the queries running in the transaction and check if they're reading unnecessary rows. -#### Why does the error "Exceeded maximum allowed number of active transactions" occur? {#exceed-number-transactions} +#### What causes the "Exceeded maximum allowed number of active transactions" error? {#exceed-number-transactions} The logic on the client side should try to keep transactions as short as possible. diff --git a/ydb/docs/en/core/faq/_includes/sdk.md b/ydb/docs/en/core/faq/_includes/sdk.md index 1b88be1744..ca13aa06e9 100644 --- a/ydb/docs/en/core/faq/_includes/sdk.md +++ b/ydb/docs/en/core/faq/_includes/sdk.md @@ -1,6 +1,6 @@ # SDK -#### What may cause the error "Status: OVERLOADED Error: Pending previous query completion" in the C++ SDK? +#### Possible causes for "Status: OVERLOADED Error: Pending previous query completion" in the C++ SDK Q: When running two queries, I try to get a response from the future method of the second one. It returns: `Status: OVERLOADED Why: <main>: Error: Pending previous query completion`. @@ -14,7 +14,7 @@ Make sure not to wrap SDK components in a singleton, since their lifetime should Using `fork()` in multithreaded applications is an antipattern. Since both the SDK and the gRPC library are multithreaded applications, their stability is not guaranteed. -#### What should I do if I get the error "Active sessions limit exceeded" even though the current number of active sessions is within the limit? {#active-sessions-does-not-exceed-the-limit} +#### What do I do if I get the "Active sessions limit exceeded" error even though the current number of active sessions is within limits? {#active-sessions-does-not-exceed-the-limit} The limit applies to the number of active sessions. An active session is a session passed to the client to be used in its code. A session is returned to the pool in a destructor. In this case, the session itself is a replicated object. You may have saved a copy of the session in the code. @@ -24,5 +24,5 @@ Yes, the C++ SDK lets you override the DB parameters and token when creating a c #### What should I do if a VM has failed and it's impossible to make a query? {#vms-failed-and-you-cant-make-a-request} -To detect the unavailability of a VM, set a client timeout. All queries contain the client timeout parameters. The timeout value should be an order of magnitude greater than the expected query execution time. +To detect that a VM is unavailable, set a client timeout. All queries contain the client timeout parameters. The timeout value should be an order of magnitude greater than the expected query execution time. diff --git a/ydb/docs/en/core/faq/_includes/serverless.md b/ydb/docs/en/core/faq/_includes/serverless.md index c7287c5bab..2922f74331 100644 --- a/ydb/docs/en/core/faq/_includes/serverless.md +++ b/ydb/docs/en/core/faq/_includes/serverless.md @@ -2,13 +2,13 @@ #### How do secondary indexes affect the request cost? -Operations with indexes are estimated according to the same rules as operations with tables. They are reflected in the request statistics and included in the total indicators that are used to calculate the cost in Request Units (RU). For more information, see the [pricing policy for the serverless {{ ydb-short-name }} API]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/ydb/pricing/request_units_yql){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/ydb/pricing/request_units_yql){% endif %}. +Operations with indexes are estimated according to the same rules as operations with tables. They are reflected in the request statistics and included in the total indicators that are used to calculate the cost in Request Units (RU). Read more in [pricing policy for the serverless {{ ydb-short-name }} API]{% if lang == "en" %}(https://cloud.yandex.com/en/docs/ydb/pricing/request_units_yql){% endif %}{% if lang == "ru" %}(https://cloud.yandex.ru/docs/ydb/pricing/request_units_yql){% endif %}. When reading data from a table using an index, the request statistics will show the number of rows read from the index and their volume. When adding a new row to a table, a record is also added to each index that exists in this table, with the number of added records and the volume of written data shown in the statistics. -When changing a table row, the operation of deleting the old record from the index and adding a new one will be reflected in the statistics for all indexes that include updated fields. +Whenever a table row update occurs, the statistics will reflect a deletion operation for the old record and an insert for the new one for all indexes that include the fields being updated. When deleting a table row, the statistics will include the deletion of records from all indexes in this table. diff --git a/ydb/docs/en/core/faq/_includes/yql.md b/ydb/docs/en/core/faq/_includes/yql.md index 4498af3792..b2e5a44c8e 100644 --- a/ydb/docs/en/core/faq/_includes/yql.md +++ b/ydb/docs/en/core/faq/_includes/yql.md @@ -4,7 +4,7 @@ #### How do I select table rows by a list of keys? {#explicit-keys} -You can select table rows from the specified list of values of the table's primary key (or key prefix) using the `IN` operator: +You can select table rows based on a specified list of table primary key (or key prefix) values using the `IN` operator: ```sql DECLARE $keys AS List<UInt64>; @@ -26,7 +26,7 @@ To select rows effectively, make sure that the value types in the parameters mat #### Is search by index performed for conditions containing the LIKE operator? {#like-index} -You can only use the `LIKE` operator to search by table index if it sets the row prefix: +You can only use the `LIKE` operator to search a table index if it specifies a row prefix: ```sql SELECT * FROM string_key_table @@ -37,25 +37,25 @@ WHERE Key LIKE "some_prefix%"; 1000 rows is the response size limit per YQL query. If a response is shortened, it is flagged as `Truncated`. To output more table rows, you can use [paginated output](../../best_practices/paging.md) or the `ReadTable` operation. -#### What should I do if I get the "Datashard: Reply size limit exceeded" error? {#reply-size-exceeded} +#### What do I do if I get the Datashard: Reply size limit exceeded error in response to a query? {#reply-size-exceeded} -This error means that, when executing the query, an attempt was made to return more than 50 MB of data from one of the involved datashards, which exceeds the allowed limit. +This error means that, as a query was running, one of the participating data shards attempted to return over 50 MB of data, which exceeds the allowed limit. Recommendations: * A general recommendation is to reduce the amount of data processed in a transaction. -* If a Join operation is running, make sure that its method is [Index Lookup Join](#index-lookup-join). +* If a query involves a `Join`, it is a good idea to make sure that the method used is [Index lookup Join](#index-lookup-join). * If a simple selection is performed, make sure that it is done by keys, or add `LIMIT` in the query. -#### What should I do if I get the "Datashard program size limit exceeded" error? {#program-size-exceeded} +#### What do I do is I get the "Datashard program size limit exceeded" in response to a query? {#program-size-exceeded} -This error means that the size of the program (including the parameter values) for one of the datashards exceeded the 50 MB limit. In most cases, this indicates an attempt to write more than 50 MB of records to database tables in one transaction. All modifying operations in a transaction such as `UPSERT`, `REPLACE`, `INSERT`, or `UPDATE` count as records. +This error means that the size of a program (including parameter values) exceeded the 50-MB limit for one of the data shards. In most cases, this indicates an attempt to write over 50 MB of data to database tables in a single transaction. All modifying operations in a transaction such as `UPSERT`, `REPLACE`, `INSERT`, or `UPDATE` count as records. You need to reduce the total size of records in one transaction. Normally, we don't recommend combining queries that logically don't require transactionality in a single transaction. When adding/updating data in batches, we recommend reducing the size of one batch to values not exceeding a few megabytes. #### How do I update only those values whose keys are not in the table? {#update-non-existent} -You can use `LEFT JOIN` to mark the keys that are missing from the table and then update their values: +You can use the `LEFT JOIN` operator to identify the keys a table is missing and update their values: ```sql DECLARE $values AS List<Struct<Key: UInt64, Value: String>>; @@ -72,18 +72,18 @@ WHERE t.Key IS NULL; #### Are there any specific features of Join operations? {#join-operations} -Join operations in {{ ydb-short-name }} are performed in one of two ways: +A `Join` in {{ ydb-short-name }} is performed using one of the two methods below: * Common Join. * Index Lookup Join. #### Common Join {#common-join} -The contents of both tables (the left and right parts of a Join) are sent to the node executing the query, where the operation is performed on the entire data. This is a universal way to perform a Join operation, which is used when more optimal methods are not applicable. For large tables, this method is either slow or doesn't work in general due to exceeding the data transfer limits. +The contents of both tables (to the left and to the right of `Join`) are sent to the requesting node which applies the operation to the totality of the data. This is the most generic way of performing a `Join` that is used whenever other optimizations are unavailable. For large tables, this method is either slow or doesn't work in general due to exceeding the data transfer limits. #### Index Lookup Join {#index-lookup-join} -For rows from the left part of a `Join` operation, a lookup is performed for the corresponding values in the right part. This method is used when the right part is a table and the `Join` operation key is the prefix of its primary key or of the secondary index key. In this method, limited selections are made from the right table instead of full reads. This lets you use it when working with large tables. +For rows on the left of `Join`, relevant values are looked up to the right. You use this method whenever the right part is a table and the `Join` key is its primary or secondary index key prefix. In this method, limited selections are made from the right table instead of full reads. This lets you use it when working with large tables. {% note info %} diff --git a/ydb/docs/en/core/getting_started/_includes/auth.md b/ydb/docs/en/core/getting_started/_includes/auth.md index a2e68dab65..4dba70099c 100644 --- a/ydb/docs/en/core/getting_started/_includes/auth.md +++ b/ydb/docs/en/core/getting_started/_includes/auth.md @@ -6,4 +6,5 @@ Full information about all available authentication methods is given in the [DB ## Learn more about YDB {#next} -Go to [YDB CLI: Getting started](../cli.md) to learn more about YDB.
\ No newline at end of file +Go to [YDB CLI: Getting started](../cli.md) to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/cli.md b/ydb/docs/en/core/getting_started/_includes/cli.md index 341da386d9..074186a1b3 100644 --- a/ydb/docs/en/core/getting_started/_includes/cli.md +++ b/ydb/docs/en/core/getting_started/_includes/cli.md @@ -4,8 +4,8 @@ To run commands via the CLI, you will need database connection settings you can retrieve when [creating](../create_db.md) a connection: -* [Endpoint](../../concepts/connect.md#endpoint). -* [Database name](../../concepts/connect.md#database). +* [Endpoint](../../concepts/connect.md#endpoint) +* [Database name](../../concepts/connect.md#database) You may also need a token or login/password if the database requires [authentication](../auth.md). To execute the below scenario, you need to select an option for saving them in an environment variable. @@ -44,7 +44,7 @@ To test connection, you can use the command for [listing objects](../../referenc {{ ydb-cli }} -e <endpoint> -d <database> scheme ls ``` -If the command is successful, a list of objects in the database is shown in response. If you haven't created anything in the database yet, the output will only contain the `.sys` and `.sys_health` system directories with [diagnostic representations of YDB](../../troubleshooting/system_views.md). +If the command is successful, a list of objects in the database is shown in response. If you haven't created anything in the database yet, the output will only contain the `.sys` and `.sys_health` system directories with [diagnostic representations of YDB](../../troubleshooting/system_views_db.md). {% include [cli/ls_examples.md](cli/ls_examples.md) %} @@ -63,7 +63,7 @@ You will be interactively prompted for connection parameters to be linked with t Check that the profile is OK with the `scheme ls` command: ```bash -{{ ydb-cli }} --profile db1 scheme ls +{{ ydb-cli }} -p db1 scheme ls ``` ## Executing an YQL script {#yql} @@ -71,7 +71,7 @@ Check that the profile is OK with the `scheme ls` command: The {{ ydb-short-name }} CLI `scripting yql` command lets you execute any command (both DDL and DML) in [YQL](../../yql/reference/index.md), an SQL dialect supported by {{ ydb-short-name }}: ```bash -{{ ydb-cli }} --profile <profile_name> yql -s <yql_request> +{{ ydb-cli }} -p <profile_name> yql -s <yql_request> ``` For example: @@ -79,19 +79,19 @@ For example: * Creating a table: ```bash - {{ ydb-cli }} --profile db1 yql -s "create table t1( id uint64, primary key(id))" + {{ ydb-cli }} -p db1 yql -s "create table t1( id uint64, primary key(id))" ``` * Adding a record: ```bash - {{ ydb-cli }} --profile db1 yql -s "insert into t1(id) values (1)" + {{ ydb-cli }} -p db1 yql -s "insert into t1(id) values (1)" ``` * Data selects: ```bash - {{ ydb-cli }} --profile db1 yql -s "select * from t1" + {{ ydb-cli }} -p db1 yql -s "select * from t1" ``` If you get the `Profile db1 does not exist` error, that means you neglected to create a profile in the [previous step](#profile). @@ -104,4 +104,5 @@ The YDB CLI supports individual commands with complete sets of options for any e ## Learn more about YDB {#next} -Proceed to the [YQL - Getting started](../yql.md) article to learn more about YDB.
\ No newline at end of file +Proceed to the [YQL - Getting started](../yql.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/create_db.md b/ydb/docs/en/core/getting_started/_includes/create_db.md index a42e08b892..c571284620 100644 --- a/ydb/docs/en/core/getting_started/_includes/create_db.md +++ b/ydb/docs/en/core/getting_started/_includes/create_db.md @@ -15,3 +15,4 @@ There are three methods you can use to deploy {{ ydb-short-name }}: ## Learn more about YDB {#next} After creating your database, proceed to the [Authentication - Getting started](../auth.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/auth.md b/ydb/docs/en/core/getting_started/_includes/index/auth.md index e69cc900cd..37886a7ad5 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/auth.md +++ b/ydb/docs/en/core/getting_started/_includes/index/auth.md @@ -1 +1,2 @@ -* [Authentication ](../../auth.md): Learn what you need to access the database. +* [Authentication](../../auth.md): Learn what you need to access the database. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/cli.md b/ydb/docs/en/core/getting_started/_includes/index/cli.md index e17513d8f4..fd6d6bc205 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/cli.md +++ b/ydb/docs/en/core/getting_started/_includes/index/cli.md @@ -1 +1,2 @@ * [Using the command line interface (CLI)](../../cli.md): Install the {{ ydb-short-name }} CLI, connect to the DB, and view the usage examples. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/create_db.md b/ydb/docs/en/core/getting_started/_includes/index/create_db.md index 81e30281d4..c51b9dec5c 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/create_db.md +++ b/ydb/docs/en/core/getting_started/_includes/index/create_db.md @@ -1 +1,2 @@ -* [Creating a database ](../../create_db.md): Create your own open-source {{ ydb-short-name }} database. +* [Creating a database](../../create_db.md): Create your own open-source {{ ydb-short-name }} database. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/intro.md b/ydb/docs/en/core/getting_started/_includes/index/intro.md index 03cf6cda74..5526958d49 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/intro.md +++ b/ydb/docs/en/core/getting_started/_includes/index/intro.md @@ -3,3 +3,4 @@ The articles in this section will tell you how to quickly get started with {{ ydb-short-name }}. Each article provides one or more simple scenarios of doing an action, and additionally includes a link to other documents describing all the available {{ ydb-short-name }} features. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/sdk.md b/ydb/docs/en/core/getting_started/_includes/index/sdk.md index cd34153a1f..8f551767ed 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/sdk.md +++ b/ydb/docs/en/core/getting_started/_includes/index/sdk.md @@ -1 +1,2 @@ * [{{ ydb-short-name }} SDK](../../sdk.md): Get familiar with {{ ydb-short-name }} SDKs for different programming languages and run simple DB queries from sample apps. + diff --git a/ydb/docs/en/core/getting_started/_includes/index/yql.md b/ydb/docs/en/core/getting_started/_includes/index/yql.md index 5dbab8ad6b..3e6c2991a8 100644 --- a/ydb/docs/en/core/getting_started/_includes/index/yql.md +++ b/ydb/docs/en/core/getting_started/_includes/index/yql.md @@ -1 +1,2 @@ * [YQL](../../yql.md): Get familiar with YQL, the primary query language for {{ ydb-short-name }} and a SQL dialect. + diff --git a/ydb/docs/en/core/getting_started/_includes/sdk.md b/ydb/docs/en/core/getting_started/_includes/sdk.md index 5a53ec55eb..08b92ecad1 100644 --- a/ydb/docs/en/core/getting_started/_includes/sdk.md +++ b/ydb/docs/en/core/getting_started/_includes/sdk.md @@ -11,3 +11,4 @@ Follow these steps to launch a simple app using {{ ydb-short-name }}: * [Go](../../reference/ydb-sdk/example/go/index.md) For more information about the {{ ydb-short-name }} SDK, see [Working with the {{ ydb-short-name }} SDK](../../reference/ydb-sdk/index.md). + diff --git a/ydb/docs/en/core/getting_started/_includes/useful_links.md b/ydb/docs/en/core/getting_started/_includes/useful_links.md index c8203a4e9a..c32f99efb5 100644 --- a/ydb/docs/en/core/getting_started/_includes/useful_links.md +++ b/ydb/docs/en/core/getting_started/_includes/useful_links.md @@ -2,10 +2,11 @@ {% if oss %} -* [{{ ydb-short-name }} OpenSource](https://ydb.tech). +* [{{ ydb-short-name }} OpenSource](https://ydb.tech) {% endif %} -* [{{ yandex-cloud }} management console](https://console.cloud.yandex.com). +* [{{ yandex-cloud }} management console](https://console.cloud.yandex.com) + +* [GitHub account ydb-platform](https://github.com/ydb-platform) -* [GitHub account ydb-platform](https://github.com/ydb-platform). diff --git a/ydb/docs/en/core/getting_started/_includes/yql.md b/ydb/docs/en/core/getting_started/_includes/yql.md index e4ef10340f..dc151d6ce9 100644 --- a/ydb/docs/en/core/getting_started/_includes/yql.md +++ b/ydb/docs/en/core/getting_started/_includes/yql.md @@ -16,9 +16,9 @@ In {{ ydb-short-name }}, you can make YQL queries to a database using: {% include [yql/ui_prompt.md](yql/ui_prompt.md) %} -* [{{ ydb-short-name }} CLI](#cli). +* [{{ ydb-short-name }} CLI](#cli) -* [{{ ydb-short-name }} SDK](../sdk.md). +* [{{ ydb-short-name }} SDK](../sdk.md) {% include [yql/ui_execute.md](yql/ui_execute.md) %} @@ -27,7 +27,7 @@ In {{ ydb-short-name }}, you can make YQL queries to a database using: To execute scripts using the {{ ydb-short-name }} CLI, first do the following: * [Install the CLI](../cli.md#install). -* Define and check [DB connection settings](../cli#scheme-ls). +* Define and check [DB connection settings](../cli#scheme-ls) * [Create a `db1` profile](../cli.md#profile) configured to connect to your database. Save the text of the scripts below to a file. Name it `script.yql` to be able to run the statements given in the examples by simply copying them through the clipboard. Next, run `{{ ydb-cli }} yql` indicating the use of the `db1` profile and reading the script from the `script.yql` file: @@ -38,7 +38,7 @@ Save the text of the scripts below to a file. Name it `script.yql` to be able to ## Working with a data schema {#ddl} -### Creating a table {#create-table} +### Creating tables {#create-table} A table with the specified columns is created [using the YQL `CREATE TABLE`](../../yql/reference/syntax/create_table.md) statement. Make sure the primary key is defined in the table. Column data types are described in [YQL data types](../../yql/reference/types/index.md). @@ -88,7 +88,7 @@ Check that the tables are actually created in the database. {% include [yql/ui_scheme_ls.md](yql/ui_scheme_ls.md) %} -To get a list of existing DB tables via the {{ ydb-short-name }} CLI, make sure that the prerequisites under [Executing YQL scripts in the {{ ydb-short-name }} CLI](#cli) below are complete and run [the `scheme ls` statement](../cli.md#ping): +To get a list of existing DB tables via the {{ ydb-short-name }} CLI, make sure that the prerequisites under [Executing YQL scripts in the {{ ydb-short-name }} CLI](#cli) below are complete and run the [`scheme ls` statement](../cli.md#ping): ```bash {{ ydb-cli }} --profile db1 scheme ls @@ -213,3 +213,4 @@ You can learn more about YQL use cases by completing tasks from the [YQL tutoria ## Learn more about YDB {#next} Proceed to the [YDB SDK - Getting started](../sdk.md) article to learn more about YDB. + diff --git a/ydb/docs/en/core/getting_started/index.md b/ydb/docs/en/core/getting_started/index.md index 28128865c0..12643df073 100644 --- a/ydb/docs/en/core/getting_started/index.md +++ b/ydb/docs/en/core/getting_started/index.md @@ -1,6 +1 @@ ---- -title: Getting started with Yandex Database (YDB). Overview ---- - -# Getting started with {{ ydb-name }} - +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/index.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/index.md index a9a5e6ebc5..709cc7f167 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/index.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/index.md @@ -2,8 +2,9 @@ This section contains articles describing simple YDB deployment scenarios. They can be used by application developers to quickly deploy their own YDB database for development or testing purposes. -* [Running YDB in Docker](../ydb_docker.md). -* [Running YDB from a binary file](../ydb_local.md). -* [Running YDB in Minikube](../ydb_minikube.md). +- [Running YDB in Docker](../ydb_docker.md) +- [Running YDB from a binary file](../ydb_local.md) +- [Running YDB in Minikube](../ydb_minikube.md) + +For information about deploying YDB clusters for administrators, see [Managing a cluster](../../../cluster/index.md). -For information about deploying YDB clusters for administrators, see [Administration](../../../deploy/index.md). diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/01_intro.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/01_intro.md index 41af591d4a..59e9566d9f 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/01_intro.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/01_intro.md @@ -9,13 +9,14 @@ As a result of completing the instructions below, you'll get a local YDB databas {% list tabs %} - gRPC - * [Endpoint ](../../../../concepts/connect.md#endpoint): `grpc://localhost:2136`. - * [Database location](../../../../concepts/connect.md#database): `/local`. - * [Authentication ](../../../../concepts/connect.md#auth-modes): Anonymous (without authentication). + - [Endpoint](../../../../concepts/connect.md#endpoint): `grpc://localhost:2136` + - [Database location](../../../../concepts/connect.md#database): `/local` + - [Authentication ](../../../../concepts/connect.md#auth-modes): Anonymous (without authentication) - gRPCs/TLS - * [Endpoint ](../../../../concepts/connect.md#endpoint): `grpcs://localhost:2135`. - * [Database location](../../../../concepts/connect.md#database): `/local`. - * [Authentication ](../../../../concepts/connect.md#auth-modes): Anonymous (without authentication). + - [Endpoint](../../../../concepts/connect.md#endpoint): `grpcs://localhost:2135` + - [Database location](../../../../concepts/connect.md#database): `/local` + - [Authentication ](../../../../concepts/connect.md#auth-modes): Anonymous (without authentication) {% endlist %} + diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/03_start.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/03_start.md index ab54c8d51f..54225d8460 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/03_start.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/03_start.md @@ -30,14 +30,14 @@ If started successfully, you'll see the ID of the created container. - `/ydb_data` for storing data. If this directory is not mounted, the container will be started without saving data to the host system disk. - `/ydb_certs` for storing certificates to establish a TLS connection. The started container will write to this directory the certificates to be used for a TLS client connection. If this directory is not mounted, you won't be able to connect via TLS due to having no certificate information. -`-e`: means setting environment variables in `<name>=<value>`format. The YDB container uses the following environment variables: +`-e`: means setting environment variables in `<name>=<value>` format. The YDB container uses the following environment variables: -* `YDB_DEFAULT_LOG_LEVEL`: The logging level. Acceptable values: `CRIT`, `ERROR`, `WARN`, `NOTICE`, and `INFO`. Defaults to `NOTICE`. -* `GRPC_PORT`: The port for unencrypted connections. Defaults to 2136. -* `GRPC_TLS_PORT`: The port for TLS connections. Defaults to 2135. -* `MON_PORT`: The port for the built-in web UI with [monitoring and introspection tools](../../../../maintenance/embedded_monitoring/ydb_monitoring.md). Defaults to 8765. -* `YDB_PDISK_SIZE`: The size of the disk for storing data in `<NUM>GB` format (for example, `YDB_PDISK_SIZE=128GB`). Acceptable values: `64GB` and higher. Defaults to 64GB. -* `YDB_USE_IN_MEMORY_PDISKS`: Using disks in memory. Acceptable values are `true` and `false`, defaults to `false`. If enabled, the container's file system is not used for working with data, all data is only stored in the memory of a process and is lost when it's stopped. Currently, you can start the container on Apple M1 in this mode only. +- `YDB_DEFAULT_LOG_LEVEL`: The logging level. Acceptable values: `CRIT`, `ERROR`, `WARN`, `NOTICE`, and `INFO`. Defaults to `NOTICE`. +- `GRPC_PORT`: The port for unencrypted connections. Defaults to 2136. +- `GRPC_TLS_PORT`: The port for TLS connections. Defaults to 2135. +- `MON_PORT`: The port for the built-in web UI with [monitoring and introspection tools](../../../../maintenance/embedded_monitoring/ydb_monitoring.md). Defaults to 8765. +- `YDB_PDISK_SIZE`: The size of the disk for storing data in `<NUM>GB` format (for example, `YDB_PDISK_SIZE=128GB`). Acceptable values: `64GB` and higher. Defaults to 64GB. +- `YDB_USE_IN_MEMORY_PDISKS`: Using disks in memory. Acceptable values are `true` and `false`, defaults to `false`. If enabled, the container's file system is not used for working with data, all data is only stored in the memory of a process and is lost when it's stopped. Currently, you can start the container on Apple M1 in this mode only. `-p` means publishing container ports on the host system. All applicable ports must be explicitly listed even if default values are used. @@ -46,3 +46,4 @@ If started successfully, you'll see the ID of the created container. It may take several minutes to initialize the Docker container, depending on the allocated resources. The database will not be available until the container is initialized. {% endnote %} + diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/04_request.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/04_request.md index 18e232ff80..cb1210bce5 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/04_request.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/04_request.md @@ -18,7 +18,7 @@ A precompiled version of the [YDB CLI](../../../../reference/ydb-cli/index.md) i docker exec <container_id> /ydb -e localhost:2136 -d /local scheme ls ``` -Where: +, where `<container_id>`: The container ID output when you [start](#start) it. diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/06_license.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/06_license.md index 5b30437008..4b6be5ea22 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/06_license.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_docker/06_license.md @@ -13,3 +13,4 @@ View all the included components and their licenses: ```bash docker run --rm -it --entrypoint cat {{ ydb_local_docker_image }} THIRD_PARTY_LICENSES ``` + diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_local.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_local.md index 245b8e908a..af31c91b0f 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_local.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_local.md @@ -6,9 +6,9 @@ This section describes how to deploy a local {{ ydb-short-name }} cluster using As a result of completing the steps below, you'll get a YDB database running on a local VM instance that you can connect to using the following: -* [Endpoint ](../../../concepts/connect.md#endpoint): `grpc://localhost:2136`. -* [Database location](../../../concepts/connect.md#database): `/Root/test`. -* [Authentication ](../../../concepts/connect.md#auth-modes): Anonymous (without authentication). +- [Endpoint](../../../concepts/connect.md#endpoint): `grpc://localhost:2136` +- [Database location](../../../concepts/connect.md#database): `/Root/test` +- [Authentication](../../../concepts/connect.md#auth-modes): Anonymous (without authentication) ## Installation {#install} @@ -28,9 +28,9 @@ The local YDB server can be started in disk or in-memory mode: - Storing data on disk - * To store data on disk, when running the script for the first time, a 64 GB file named `ydb.data` is created in the working directory. Make sure you have enough free space to create it. + - To store data on disk, when running the script for the first time, a 64 GB file named `ydb.data` is created in the working directory. Make sure you have enough free space to create it. - * Run the following command from the working directory: + - Run the following command from the working directory: ```bash ./start.sh disk @@ -38,9 +38,9 @@ The local YDB server can be started in disk or in-memory mode: - Storing data in memory - * Under in-memory data storage, the data is lost when stopping the server. + - Under in-memory data storage, the data is lost when stopping the server. - * Run the following command from the working directory: + - Run the following command from the working directory: ```bash ./start.sh ram @@ -75,3 +75,4 @@ To work with the DB structure and data, you can also use the web interface embed ## Additional information {#advanced} Instructions for administrators on how to deploy clusters from multiple nodes and configure them are given in [Deploy YDB on-premises](../../../deploy/manual/deploy-ydb-on-premises.md) in the "Administration" section. + diff --git a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_minikube.md b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_minikube.md index d7f4753715..522496e9cf 100644 --- a/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_minikube.md +++ b/ydb/docs/en/core/getting_started/self_hosted/_includes/ydb_minikube.md @@ -1,3 +1,4 @@ # Running {{ ydb-short-name }} in Minikube The article is under development. + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md b/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md index 90cd6fa9fa..3b51f1584a 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/build/local_oss.md @@ -2,7 +2,7 @@ The documentation is built using the [YFM-Docs](https://github.com/yandex-cloud/yfm-docs) utility. -The procedure for installing YFM-Docs is described on the [introductory page of the documentation for this utility]{% if lang == "en" %}(https://ydocs.tech/en/tools/docs/){% endif %}{% if lang == "ru" %}(https://ydocs.tech/ru/tools/docs/){% endif %}. +The procedure for installing YFM-Docs is described on [introductory documentation page for this utility]{% if lang == "en" %}(https://ydocs.tech/en/tools/docs/){% endif %}{% if lang == "ru" %}(https://ydocs.tech/ru/tools/docs/){% endif %}. To build the {{ ydb-short-name }} OpenSource documentation, run the command: @@ -12,8 +12,8 @@ $ yfm -i <source_dir> -o <output_dir> --allowHTML Where: -* `source_dir` is the directory where the contents of [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) is cloned. -* `output_dir` is the output directory for HTML files. +- `source_dir` is the directory where the contents of [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) is cloned. +- `output_dir` is the output directory for HTML files. Building the documentation takes a few seconds and there should be no errors logged to the stdout log. @@ -31,6 +31,6 @@ python3 -m http.server 8888 -d ~/docs/ydboss With the server run in this way, the locally built documentation is available at the links: -* [http://localhost:8888/ru](http://localhost:8888/ru) (in Russian) -* [http://localhost:8888/en](http://localhost:8888/en) (in English) +- [http://localhost:8888/ru](http://localhost:8888/ru) (in Russian) +- [http://localhost:8888/en](http://localhost:8888/en) (in English) diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md b/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md index f238f0a24c..f671bc04a3 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/build/tech.md @@ -19,3 +19,4 @@ This overlay allows redefining the content, preserving the placement of articles Since the overlay of the `core` and `overlay` directories is only done when building the documentation, links from the articles in `overlay` to the articles in `core` won't be clickable in an integrated development environment (IDE) that knows nothing about the fact that some directories will be overlaid on top of each other during the build. {% endnote %} + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/content.md b/ydb/docs/en/core/how_to_edit_docs/_includes/content.md index 3626c8a682..ff65803d33 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/content.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/content.md @@ -18,8 +18,8 @@ When adding content, you should first choose where to add it: to the OpenSource In the sections where the requirements differ for basic content and customization, there are two corresponding bookmarks: -* **Core**: The core of any documentation, the basic content. -* **Overlay**: Content that is overlaid on top of the core and adapting it to a custom build. +- **Core**: The core of any documentation, the basic content. +- **Overlay**: Content that is overlaid on top of the core and adapting it to a custom build. All the content of the OpenSource build is part of the core, and zero customization is applied when building it, so when you change it, you'll only need the "Core" bookmark. @@ -31,9 +31,9 @@ The basic structure of the source code directories is based on the subject of th Each subject directory must contain the "Overview" article with the `index.md` filename. The "Overview" article: -* Describes what the articles in this directory tell about. -* Optionally, provides a list of links to all or some of the most important articles. -* Optionally, provides a set of "See also" links to other related articles and sections. +- Describes what the articles in this directory tell about. +- Optionally, provides a list of links to all or some of the most important articles. +- Optionally, provides a set of "See also" links to other related articles and sections. The presence of the "Overview" article lets you refer to the entire directory rather than a specific article and convert articles into directories without losing the referential integrity. @@ -78,7 +78,7 @@ The presence of the "Overview" article lets you refer to the entire directory ra `subject1/article1.md`: ```markdown - + {% include [definition.md](_includes/article1/definition.md) %} {% include [examples.md](_includes/article1/examples.md) %} @@ -106,43 +106,43 @@ The presence of the "Overview" article lets you refer to the entire directory ra When redefining the content, you can: - * Add additional content to the beginning or end of an article. + - Add additional content to the beginning or end of an article. `subject1/article1.md`: ```md - + {% include [article1](_includes/article1.md) %} In addition to the basic authorization methods, our company uses nanotube-based authorization. ``` - * Insert additional content between include directives. + - Insert additional content between include directives. `subject1/article1.md`: ```markdown - + {% include [definition.md](_includes/article1/definition.md) %} - + In our company, the amount of DB data is limited to 150ZB. - + {% include [examples.md](_includes/article1/examples.md) %} - + Example 2: The quick brown fox jumps over the lazy dog. ``` - * Remove some of the content of the original article from the build by removing the corresponding include directive. + - Remove some of the content of the original article from the build by removing the corresponding include directive. `subject1/article1.md`: ```markdown - + {% include [definition.md](_includes/article1/definition.md) %} - + In our company, the amount of DB data is limited to 150ZB. - + Example: The quick brown fox jumps over the lazy dog. ``` @@ -155,10 +155,10 @@ The presence of the "Overview" article lets you refer to the entire directory ra ### Specifics of using the include directive {#include-hints} -* The `_includes` contents are not published, so it is useless and forbidden to refer to them, it can only be included in articles by the `{% include ... %}` directive. Unfortunately, when building documentation locally, the contents of _includes remain available and operational, and the error only occurs when deploying the documentation on the farm. -* Make sure to leave empty lines around the `{% include ... %}` directive, otherwise, it won't be executed during the build. -* Write a file name in square brackets so that there is something to click on when viewing it in the default viewer on github (which does not understand what include is), and so that one include in it visually differs from the other. -* After `{%` at the beginning and before `%}`, a space is required at the end, otherwise the include won't be executed during the build. +- The `_includes` contents are not published, so it is useless and forbidden to refer to them, it can only be included in articles by the `{% include ... %}` directive. Unfortunately, when building documentation locally, the contents of _includes remain available and operational, and the error only occurs when deploying the documentation on the farm. +- Make sure to leave empty lines around the `{% include ... %}` directive, otherwise, it won't be executed during the build. +- Write a file name in square brackets so that there is something to click on when viewing it in the default viewer on github (which does not understand what include is), and so that one include in it visually differs from the other. +- After `{%` at the beginning and before `%}`, a space is required at the end, otherwise the include won't be executed during the build. ### Other use cases for include {#include-reuse} @@ -291,15 +291,15 @@ The text specified in square brackets is displayed in the browser until the imag Desirable image formats: -* Charts: [.SVG]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Scalable_Vector_Graphics){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/SVG){% endif %} -* Screenshots: [.PNG]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Portable_Network_Graphics){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/PNG){% endif %} +- Charts: [.SVG]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Scalable_Vector_Graphics){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/SVG){% endif %} +- Screenshots: [.PNG]{% if lang == "en" %}(https://en.wikipedia.org/wiki/Portable_Network_Graphics){% endif %}{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/PNG){% endif %} Since an image is part of an article, it is impossible to post the image without this article. If there is no text of the article yet, determine the subject directory for its placement and the file name of the future article and only specify a link to the image in the text (keep in mind that the [text of the article is placed in the `_includes` subdirectory](#articles)!), and do not include the article itself in the TOC until it is ready for publication. When inserting images, you can also specify: -* A hint to be shown when hovering over an image: ``. We don't recommend using it, since lots of modern devices have no mouse pointer. -* Image size: ``. We recommend that you use it when specifying the XSIZE for SVG images so that they are expanded properly by the width of the documentation screen, regardless of the resolution they were saved with: ``. If you only specify the X-size in this way, the Y-size is selected automatically with the proportions maintained. +- A hint to be shown when hovering over an image: ``. We don't recommend using it, since lots of modern devices have no mouse pointer. +- Image size: ``. We recommend that you use it when specifying the XSIZE for SVG images so that they are expanded properly by the width of the documentation screen, regardless of the resolution they were saved with: ``. If you only specify the X-size in this way, the Y-size is selected automatically with the proportions maintained. ## Backward compatibility {#compatibility} @@ -320,4 +320,5 @@ If you cannot but move an article, do the following: hidden: true ``` -As a result, this article won't be listed anywhere in the TOC, but will be available when following a direct link provided that you keep it somewhere.
\ No newline at end of file +As a result, this article won't be listed anywhere in the TOC, but will be available when following a direct link provided that you keep it somewhere. + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md b/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md index 75e00dbde3..8a19c28e9f 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/contribute.md @@ -1,7 +1,8 @@ -## How to participate in the development of documentation +# How to participate in the development of documentation The source code of the {{ ydb-short-name }} documentation is open and published at [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs). The {{ ydb-short-name }} team accepts requests for edits and updates to the {{ ydb-short-name }} documentation, which anyone can offer through [Pull Requests on github.com](https://docs.github.com/en/desktop/contributing-and-collaborating-using-github-desktop/working-with-your-remote-repository-on-github-or-github-enterprise/creating-an-issue-or-pull-request). -Before creating a pull request, see the documentation development requirements described in the articles of this section and [check the build locally](../build.md).
\ No newline at end of file +Before creating a pull request, see the documentation development requirements described in the articles of this section and [check the build locally](../build.md). + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md b/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md index c5f09de906..2544111301 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/customize.md @@ -8,9 +8,9 @@ The `overlay` subdirectory is designed for customizing content and is technical A file with an article in `core` never contains content directly, it only contains one or more instructions of include blocks of this content stored in the `_includes` subdirectory. As a result, replacing this file in `overlay` does not lead to the need to copy the content, but lets you make the following types of its adaptation: -* Add additional content to the beginning or end of an article. -* Insert additional content between include directives. -* Remove some of the content of the original article from the build by removing the corresponding include directive. +- Add additional content to the beginning or end of an article. +- Insert additional content between include directives. +- Remove some of the content of the original article from the build by removing the corresponding include directive. Instructions for placing articles in core and overlay are given in the [Content creation guide - Articles](content.md#articles). @@ -21,10 +21,10 @@ The TOC in the {{ ydb-short-name }} documentation is built from a set of files p As with articles, a proxy approach is used for TOC files as follows: 1. A file with menu items for core is named `toc_i.yaml`. It is never redefined in `overlay`. -1. Next to the `toc_i.yaml` file in core, there is a file named `toc_p.yaml`. It contains a link to `toc_i.yaml` and is designed to be redefined in `overlay`. -1. Other sections are included in the TOC via a link to the `toc_p.yaml` file. -1. If there is no need to adjust the contents of the TOC in a certain directory, no toc* files are created in `overlay`. This results in using toc_p --> toc_i from core in the build. -1. If the contents of the TOC need to be adjusted in a certain directory, a file named `toc_p.yaml` is created in `overlay` with the `- include: { mode: link, path: toc_i.yaml }` from core added to its content and additional items for the adapted build added above or below. +2. Next to the `toc_i.yaml` file in core, there is a file named `toc_p.yaml`. It contains a link to `toc_i.yaml` and is designed to be redefined in `overlay`. +3. Other sections are included in the TOC via a link to the `toc_p.yaml` file. +4. If there is no need to adjust the contents of the TOC in a certain directory, no toc* files are created in `overlay`. This results in using toc_p --> toc_i from core in the build. +5. If the contents of the TOC need to be adjusted in a certain directory, a file named `toc_p.yaml` is created in `overlay` with the `- include: { mode: link, path: toc_i.yaml }` from core added to its content and additional items for the adapted build added above or below. It is good practice not to include the "Overview" in toc_i.yaml but rather include it directly in toc_p.yaml instead. This article must be the first in each submenu and always has the same article name (index.md). Including a separate item in toc_p lets you add new articles to the adapted documentation before the articles from core, but keeping the "Overview" in the first place: @@ -55,4 +55,5 @@ A TOC file name may not be `toc.yaml`, since the build tool searches for files w The only exception is the initial `toc.yaml` that is placed in the documentation build root and always contains the command to copy the core content and proceed to TOC handling in core: `include: { mode: merge, path: core/toc_m.yaml }`. -Instructions for making a TOC in core and overlay are given in the [Content creation guide - TOC](content.md#toc).
\ No newline at end of file +Instructions for making a TOC in core and overlay are given in the [Content creation guide - TOC](content.md#toc). + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md index 4251959495..6508653300 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/advanced.md @@ -1,3 +1,4 @@ ## Articles for advanced readers -* [Creating customized documentation](../customize.md) +- [Creating customized documentation](../customize.md) + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md index 5579aae6bd..4db6754f88 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/basic.md @@ -1,5 +1,6 @@ ## Main articles -* [Content creation guide](../../content.md): Describes how to create articles in technical terms. -* [Structure of subject directories](../../subjects.md): Describes where to place certain content. -* [Building documentation](../../build.md): Describes how to build documentation. +- [Content creation guide](../../content.md): Describes how to create articles in technical terms. +- [Structure of subject directories](../../subjects.md): Describes where to place certain content. +- [Building documentation](../../build.md): Describes how to build documentation. + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md b/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md index e9605e71c7..13385e4398 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/index/intro.md @@ -3,3 +3,4 @@ ## Introduction The [https://github.com/ydb-platform/ydb/tree/master/docs](https://github.com/ydb-platform/ydb/tree/master/docs) repository contains the open source code of the {{ ydb-short-name }} documentation in Russian and English. Based on it, an OpenSource version of the {{ ydb-short-name }} documentation is built and published at [http://ydb.tech](http://ydb.tech). It is also used to create customized {{ ydb-short-name }} documentation versions for enterprise environments and cloud services. + diff --git a/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md b/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md index 0106c749bb..5d1daa1266 100644 --- a/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md +++ b/ydb/docs/en/core/how_to_edit_docs/_includes/subjects.md @@ -7,15 +7,15 @@ Documentation articles are arranged in a hierarchical structure of subject direc A topic description includes: 1. The target audience of the article. The audience depends on the role of a potential reader when working with {{ ydb-short-name }}, with the following three basic audiences: - * Developer of applications for {{ ydb-short-name }}. - * {{ ydb-short-name }} administrator. - * {{ ydb-short-name }} developer/contributor. + - Developer of applications for {{ ydb-short-name }} + - {{ ydb-short-name }} administrator + - {{ ydb-short-name }} developer/contributor 2. Purpose: The reader's task or problem the solution of which is described in the content. Basically, the directory structure directly affects the structure of the documentation table of contents. Typical exceptions are: -* Intermediate stages of documentation creation. A new directory is created immediately to preserve the referential integrity in the future, but its content is still insufficient to be designed as a separate submenu in the table of contents. In this case, articles can be temporarily included in existing submenus. -* Historically established directories whose transfer is undesirable due to the loss of referential integrity. +- Intermediate stages of documentation creation. A new directory is created immediately to preserve the referential integrity in the future, but its content is still insufficient to be designed as a separate submenu in the table of contents. In this case, articles can be temporarily included in existing submenus. +- Historically established directories whose transfer is undesirable due to the loss of referential integrity. ## Level 1 structure @@ -25,9 +25,11 @@ The following subject directories are placed on level 1: | --- | ---------- | ---------- | | getting_started | All | Provide quick solutions to standard problems that arise when starting to work with a new database: what is it, how to install it, how to connect, how to perform elementary operations with data, where to go further in the documentation and on what issues. | | concepts | Application developer | Describe the main structural components of {{ ydb-short-name }} that one will deal with when developing applications for {{ ydb-short-name }}. Provide an insight into the role of each of these components in developing applications. Provide detailed information about component configuration options available to application developers. | -| reference, yql | Application developer | A reference guide for daily use on tools for accessing {{ ydb-short-name }} functions: CLI, YQL, and SDK. | -| best_practices | Application developer | Common approaches to addressing the main challenges that arise when developing applications. | -| maintenance | Application developer | How to maintain your {{ ydb-short-name }} databases deployed on a large cluster administered by someone, or when independently deploying small databases for development: backups, monitoring, logs, and more. | -| troubleshooting | ? | Tools for identifying the cause of issues. | -| how_to_edit_docs | Developers {{ ydb-short-name }} | How to update the {{ ydb-short-name }} documentation. | -| faq | All | Questions and answers |
\ No newline at end of file +| reference, yql | Application developer | A reference guide for daily use on tools for accessing {{ ydb-short-name }} functions: CLI, YQL, and SDK | +| best_practices | Application developer | Common approaches to addressing the main challenges that arise when developing applications | +| troubleshooting | ? | Tools for identifying the cause of issues | +| how_to_edit_docs | Developers {{ ydb-short-name }} | How to update the {{ ydb-short-name }} documentation | +| deploy | All | Deploying and configuring {{ ydb-short-name }} databases and clusters. Cloud, orchestrated, and manual deployments. | +| maintenance | All | How to maintain {{ ydb-short-name }} databases and clusters: backups, monitoring, logs, and disk swap. | +| faq | All | Questions and answers | + diff --git a/ydb/docs/en/core/how_to_edit_docs/informal.md b/ydb/docs/en/core/how_to_edit_docs/informal.md index 0adcf1143f..e986f00243 100644 --- a/ydb/docs/en/core/how_to_edit_docs/informal.md +++ b/ydb/docs/en/core/how_to_edit_docs/informal.md @@ -2,3 +2,4 @@ 1. `DRY`. If you have to copy and paste some content, most likely you are doing something wrong. 1. `../../../...` If you have to add relative references that point to many levels up, most likely you are doing something wrong. + diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/cli_overlay.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/cli_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/cli_overlay.md diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/options_overlay.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/options_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/options_overlay.md diff --git a/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/others_overlay.md b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/others_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/maintenance/_includes/backup_and_recovery/others_overlay.md diff --git a/ydb/docs/en/core/maintenance/_includes/index.md b/ydb/docs/en/core/maintenance/_includes/index.md new file mode 100644 index 0000000000..402705b008 --- /dev/null +++ b/ydb/docs/en/core/maintenance/_includes/index.md @@ -0,0 +1,14 @@ +# YDB maintenance + +This section contains articles on YDB database and cluster maintenance. When using a cloud database, only articles on DB maintenance are relevant, since the maintenance of a cluster under these databases is carried out by the cloud provider team. + +## Database maintenance + +- [Backup and recovery](../backup_and_recovery.md) +- [Diagnostics](../../troubleshooting/index.md) + +## Cluster maintenance + +- [Embedded UI](../embedded_monitoring/overview.md) +- [Cluster disk subsystem](../manual/index.md) + diff --git a/ydb/docs/en/core/maintenance/backup_and_recovery.md b/ydb/docs/en/core/maintenance/backup_and_recovery.md index 912a3cdf25..7fb4df420c 100644 --- a/ydb/docs/en/core/maintenance/backup_and_recovery.md +++ b/ydb/docs/en/core/maintenance/backup_and_recovery.md @@ -1,23 +1,29 @@ -{% include [intro.md](_includes/backup_and_recovery/01_intro.md) %} +# Backup and recovery -{% include [prerequisites.md](_includes/backup_and_recovery/02_prerequisites.md) %} +Backup protects against data loss and lets you restore data from a backup copy in the event of a loss. -{% include [limitations.md](_includes/backup_and_recovery/03_limitations.md) %} +YDB provides multiple solutions for backup and recovery: -{% include [fs_backup_header.md](_includes/backup_and_recovery/04_fs_backup_1_header.md) %} +- Backing up data to files and restoring it with a command run by an admin in the YDB CLI +- Backing up data to S3-compatible storage with a command run by an admin in the YDB CLI -{% include [fs_backup_body.md](_includes/backup_and_recovery/04_fs_backup_2_body.md) %} +{% include [_includes/backup_and_recovery/options_overlay.md](_includes/backup_and_recovery/options_overlay.md) %} -{% include [fs_restore.md](_includes/backup_and_recovery/05_fs_restore.md) %} +## YDB CLI {#cli} -{% include [s3_header.md](_includes/backup_and_recovery/06_s3_1_header.md) %} +### Files {#files} -{% include [s3_prerequisites.md](_includes/backup_and_recovery/06_s3_2_prerequisites.md) %} +To back up data to a file, run the [`tools dump`](../reference/ydb-cli/export_import/tools_dump.md) command in the YDB CLI as an admin. To learn more about this command, follow the [link](../reference/ydb-cli/export_import/tools_dump.md) in the YDB CLI reference. -{% include [s3_access_keys.md](_includes/backup_and_recovery/06_s3_3_access_keys.md) %} +To restore data from a file backup created with the `tools dump` command in the YDB CLI, run the [`tools restore`](../reference/ydb-cli/export_import/tools_restore.md) command in the YDB CLI. To learn more about this command, follow the [link](../reference/ydb-cli/export_import/tools_restore.md) in the YDB CLI reference. -{% include [s3_export.md](_includes/backup_and_recovery/06_s3_4_export.md) %} +### S3-compatible storage {#s3} -{% include [s3_restore.md](_includes/backup_and_recovery/06_s3_5_restore.md) %} +To back up data to S3-compatible storage (such as [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html)), run the [`export s3`](../reference/ydb-cli/export_import/s3_export.md) command in the YDB CLI as an admin. To learn more about this command, follow the [link](../reference/ydb-cli/export_import/s3_export.md) in the YDB CLI reference. + +To restore data from a backup created in S3-compatible storage with the`export s3` command in the YDB CLI, run the [`import s3`](../reference/ydb-cli/export_import/s3_import.md) command in the YDB CLI. To learn more about this command, follow the [link](../reference/ydb-cli/export_import/s3_import.md) in the YDB CLI reference. + +{% include [_includes/backup_and_recovery/cli_overlay.md](_includes/backup_and_recovery/cli_overlay.md) %} + +{% include [_includes/backup_and_recovery/others_overlay.md](_includes/backup_and_recovery/others_overlay.md) %} -{% include [s3_forget.md](_includes/backup_and_recovery/06_s3_6_forget.md) %} diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md b/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md index db4985d1dc..238a7cbb5b 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/hive.md @@ -67,3 +67,4 @@ Click **Reassign Groups** to open the window with parameters for balancing: After specifying all the parameters, click "Query" to get the number of channels moved and unlock the "Reassign" button. Clicking this button starts reassigning. + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/index.md b/ydb/docs/en/core/maintenance/embedded_monitoring/index.md new file mode 100644 index 0000000000..05cd6e80fb --- /dev/null +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/index.md @@ -0,0 +1,9 @@ +# Embedded UI + +{{ ydb-short-name }} provides tools for monitoring and determining system health: + +* [{{ ydb-short-name }} Monitoring](ydb_monitoring.md): The main monitor of the cluster. It shows the health of nodes and storage groups. +* [Interconnect overview](interconnect_overview.md): The state of cluster interconnects. +* [Logs](logs.md): Each {{ ydb-short-name }} component writes messages to logs of different levels. They can be used to detect severe issues or identify the root causes of issues. +* [Charts](charts.md): {{ ydb-short-name }} collects a range of metrics that can be used to determine the health of the entire system or of a specific component. + diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md b/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md index 9764f99f49..1852a74e90 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/overview.md @@ -1,9 +1,2 @@ -# Overview of introspection tools - -{{ ydb-short-name }} provides tools for monitoring and determining system health: - -* [{{ ydb-short-name }} Monitoring](ydb_monitoring.md): The main monitor of the cluster. It shows the health of nodes and storage groups. -* [Interconnect overview](interconnect_overview.md): The state of cluster interconnects. -* [Logs](logs.md): Each {{ ydb-short-name }} component writes messages to logs of different levels. They can be used to detect severe issues or identify the root causes of issues. -* [Charts](charts.md): {{ ydb-short-name }} collects a range of metrics that can be used to determine the health of the entire system or a specific component. +The page has been moved to a [new location](index.md). Please update the links. diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/toc_i.yaml b/ydb/docs/en/core/maintenance/embedded_monitoring/toc_i.yaml new file mode 100644 index 0000000000..94d33b8034 --- /dev/null +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/toc_i.yaml @@ -0,0 +1,17 @@ +items: +- name: Overview + href: overview.md + hidden: true +- name: Viewer + href: viewer.md + when: false +- name: YDB Monitoring + href: ydb_monitoring.md +- name: Hive web-viewer + href: hive.md +- name: Connections overview + href: interconnect_overview.md +- name: Logs + href: logs.md +- name: Charts + href: charts.md
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/toc_p.yaml b/ydb/docs/en/core/maintenance/embedded_monitoring/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md b/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md index 9bb9e5d1ec..e00579fc20 100644 --- a/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md +++ b/ydb/docs/en/core/maintenance/embedded_monitoring/ydb_monitoring.md @@ -16,9 +16,9 @@ An example of the page layout is shown in the screenshot below. In the upper-right corner of the page, you can see details about the node that created the current page: -* host. -* node uptime. -* ydb version run by the node. +* host +* node uptime +* ydb version run by the node Below is the cluster summary: @@ -77,7 +77,6 @@ http://<endpoint>:8765/monitoring/node/<node-id>/ Information about the node is presented in the following sections: * **Pools**: CPU utilization broken down by the internal stream pools, with roughly the following pool functions: - * **System**: The tasks of critical system components. * **User**: User tasks, queries executed by tablets. * **Batch**: Long-running background tasks. @@ -87,14 +86,12 @@ Information about the node is presented in the following sections: High pool utilization might degrade performance and increase the system response time. * **Common info**: Basic information about the node: - * **Version**: The {{ ydb-short-name }} version. * **Uptime**: The node uptime. * **DC**: The availability zone where the node resides. * **Rack**: The ID of the rack where the node resides. * **Load average**: Average host CPU utilization for different time intervals: - * 1 minute. * 5 minutes. * 15 minutes. @@ -120,7 +117,7 @@ Each storage group can also be expanded into a list of VDisks, with the followin * The unique (within the node) PDiskID where the VDisk resides. * The ID of the node where the VDisk resides. * Free/available space on the block store volume. -* The path used to access the block store volume. +* Path used to access block storage. This list can be used to identify storage groups that were affected by disk or node failure. @@ -152,7 +149,6 @@ In the `Tenant Info` section, you can see the following information: * **Pools**: The total CPU utilization by the tenant nodes broken down by internal stream pools (for more information about pools, see the [tenant page](#tenant_page)). * **Metrics**: Data about tablet utilization for this tenant: - * **Memory**: The RAM utilized by tablets. * **CPU**: CPU utilized by tablets. * **Storage**: The amount of data stored by tablets. @@ -165,7 +161,7 @@ In the `Tenant Info` section, you can see the following information: The tenant page also includes the following tabs: * **HealthCheck**: The report regarding cluster issues, if any. -* **Storage**: The [list of storage groups](#tenant_storage_page) that includes information about which VDisks reside on which nodes and devices. +* **Storage**: The [list of storage groups](#tenant_storage_page) that includes information about which VDisks reside on which nodes and block store volumes. * **Compute**: The [list of nodes](#tenant_compute_page), which includes the nodes and tablets running on them. * **Schema**: The [tenant's schema](#tenant_scheme) that lets you view tables, execute YQL queries, view a list of the slowest queries and the most loaded shards. * **Network**: The [health of the cluster network](#tenant_network). @@ -227,3 +223,4 @@ The indicator colors have the following meaning: * **Red**: There are critical problems, the component is down (or runs with limitations). If a component includes other components, then in the absence of its own issues, the state is determined by aggregating the states of its parts. + diff --git a/ydb/docs/en/core/maintenance/index.md b/ydb/docs/en/core/maintenance/index.md new file mode 100644 index 0000000000..eb2590567d --- /dev/null +++ b/ydb/docs/en/core/maintenance/index.md @@ -0,0 +1 @@ +{% include [_includes/index.md](_includes/index.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md b/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md index 68a4b5580e..8ab7c7660a 100644 --- a/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md +++ b/ydb/docs/en/core/maintenance/manual/adding_storage_groups.md @@ -31,3 +31,4 @@ Command { ``` kikimr -s <endpoint> admin bs config invoke --proto-file DefineStoragePool.txt ``` + diff --git a/ydb/docs/en/core/maintenance/manual/balancing_load.md b/ydb/docs/en/core/maintenance/manual/balancing_load.md index 86c02b1526..245bbcf236 100644 --- a/ydb/docs/en/core/maintenance/manual/balancing_load.md +++ b/ydb/docs/en/core/maintenance/manual/balancing_load.md @@ -41,3 +41,4 @@ Command { ``` kikimr -s <endpoint> admin bs config invoke --proto-file DefineHostConfig.txt ``` + diff --git a/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md b/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md index f2b1c57cbe..50807b42e7 100644 --- a/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md +++ b/ydb/docs/en/core/maintenance/manual/change_actorsystem_configs.md @@ -38,10 +38,10 @@ Mapping pools to task types is done by setting the pool sequence number in speci List of fields with their respective tasks: -* **SysExecutor**: System. -* **UserExecutor**: User. -* **BatchExecutor**: Batch. -* **IoExecutor**: IO. +* **SysExecutor**: System +* **UserExecutor**: User +* **BatchExecutor**: Batch +* **IoExecutor**: IO Example: @@ -122,7 +122,7 @@ After changing the configuration, restart the node. ## On dynamic nodes -Dynamic nodes take the configuration from the [CMS](cms.md). To change it, you can use the following command. +Dynamic nodes take the configuration from the [CMS](cms.md). To change it, you can use the following command: ```proto ConfigureRequest { @@ -133,7 +133,7 @@ ConfigureRequest { Config { ActorSystemConfig { <actor system config> - } + } } MergeStrategy: 3 } @@ -144,3 +144,4 @@ ConfigureRequest { ```bash kikimr -s <endpoint> admin console execute --domain=<domain> --retry=10 actorsystem.txt ``` + diff --git a/ydb/docs/en/core/maintenance/manual/cluster_expansion.md b/ydb/docs/en/core/maintenance/manual/cluster_expansion.md index 9952b6c538..ec5def9702 100644 --- a/ydb/docs/en/core/maintenance/manual/cluster_expansion.md +++ b/ydb/docs/en/core/maintenance/manual/cluster_expansion.md @@ -18,7 +18,7 @@ Node { NodeId: 2 Port: <ic-port> - Host: "<new-host>" + Host: "<new-host>"| InterconnectHost: "<new-host>" Location { DataCenter: "DC1" @@ -76,3 +76,4 @@ ``` kikimr -s <endpoint> admin bs config invoke --proto-file DefineBox.txt ``` + diff --git a/ydb/docs/en/core/maintenance/manual/cms.md b/ydb/docs/en/core/maintenance/manual/cms.md index ea3e55a02b..625399ae45 100644 --- a/ydb/docs/en/core/maintenance/manual/cms.md +++ b/ydb/docs/en/core/maintenance/manual/cms.md @@ -41,3 +41,4 @@ The UsageScope field is optional and is needed to use settings for a specific te ``` ./kikimr -s <endpoint> admin console configs update <protobuf-file> ``` + diff --git a/ydb/docs/en/core/maintenance/manual/disk_end_space.md b/ydb/docs/en/core/maintenance/manual/disk_end_space.md index 39c63f95c4..2c89942455 100644 --- a/ydb/docs/en/core/maintenance/manual/disk_end_space.md +++ b/ydb/docs/en/core/maintenance/manual/disk_end_space.md @@ -34,23 +34,23 @@ If the block store volume is running out of space, you can apply defragmentation 1. Check the health of the groups in the cluster. There shouldn't be any problem groups on the same node with the problem block store volume. -2. Log in via SSH to the node hosting this block store volume. +2. Log in via SSH to the node hosting this block store volume 3. Check if you can [restart the process](node_restarting.md#restart_process). -4. Stop the process. +4. Stop the process ```bash sudo systemctl stop kikimr ``` -5. Format the block store volume. +5. Format the block store volume ```bash sudo kikimr admin blobstorage disk obliterate <path to the store volume part label> ``` -6. Run the process. +6. Run the process ```bash sudo systemctl start kikimr @@ -59,3 +59,4 @@ If the block store volume is running out of space, you can apply defragmentation ## Moving individual VDisks from full block store volumes If defragmentation can't help freeing up space on the block store volume, you can [move](moving_vdisks.md#moving_disk) individual VDisks. + diff --git a/ydb/docs/en/core/maintenance/manual/failure_model.md b/ydb/docs/en/core/maintenance/manual/failure_model.md index 878465e73a..fd567cd2de 100644 --- a/ydb/docs/en/core/maintenance/manual/failure_model.md +++ b/ydb/docs/en/core/maintenance/manual/failure_model.md @@ -16,3 +16,4 @@ If multiple VDisks have failed in the group, SelfHeal stops moving VDisks. If th ## The number of failed VDisks has exceeded the failure model {#exceeded_the_failure_model} The availability and operability of the system might be lost. Make sure to revive at least one VDisk without losing the data stored on it. + diff --git a/ydb/docs/en/core/maintenance/manual/index.md b/ydb/docs/en/core/maintenance/manual/index.md index c534ac74c3..e35a8c7ded 100644 --- a/ydb/docs/en/core/maintenance/manual/index.md +++ b/ydb/docs/en/core/maintenance/manual/index.md @@ -22,3 +22,4 @@ A {{ ydb-short-name }} cluster lets you: * [Configure](change_actorsystem_configs.md) the actor system on your nodes. * Edit configs via [CMS](cms.md). * [Add](adding_storage_groups.md) new storage groups. + diff --git a/ydb/docs/en/core/maintenance/manual/moving_vdisks.md b/ydb/docs/en/core/maintenance/manual/moving_vdisks.md index 09fa0c8289..a8ff299640 100644 --- a/ydb/docs/en/core/maintenance/manual/moving_vdisks.md +++ b/ydb/docs/en/core/maintenance/manual/moving_vdisks.md @@ -16,11 +16,11 @@ If SelfHeal is disabled or fails to move VDisks, you'll have to run this operati 1. Make sure that the VDisk has actually failed. - Write down the node's FQDN, ic-port, VDisk path, and pdisk-id. + Write down the node's FQDN, ic-port, VDisk path, and pdisk-id -2. Go to any cluster node. +2. Go to any cluster node -3. Move the VDisk. +3. Move the VDisk ```bash kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<host>" IcPort: <ic-port>} Path: "<Path to the storage volume part label>" PDiskId: <pdisk-id> Status: BROKEN } }' @@ -28,14 +28,15 @@ If SelfHeal is disabled or fails to move VDisks, you'll have to run this operati ## Enable the VDisk back after reassignment {#return_a_device_to_work} -1. In Monitoring, make sure that the PDisk is actually operable. +1. In Monitoring, make sure that the PDisk is actually operable - Write down the node's FQDN, ic-port, store path, and pdisk-id. + Write down the node's FQDN, ic-port, store path, and pdisk-id -2. Go to any cluster node. +2. Go to any cluster node -3. Enable the PDisk back. +3. Enable the PDisk back ```bash kikimr admin bs config invoke --proto 'Command { UpdateDriveStatus { HostKey: { Fqdn: "<host>" IcPort: <ic-port>} Path: "<Path to the storage volume part label>" PDiskId: <pdisk-id> Status: ACTIVE } }' ``` + diff --git a/ydb/docs/en/core/maintenance/manual/node_restarting.md b/ydb/docs/en/core/maintenance/manual/node_restarting.md index aa8747f27f..433c336512 100644 --- a/ydb/docs/en/core/maintenance/manual/node_restarting.md +++ b/ydb/docs/en/core/maintenance/manual/node_restarting.md @@ -6,7 +6,7 @@ To make sure that the process is stoppable, follow these steps. 1. Access the node via SSH. -1. Execute the command: +1. Execute the command ```bash kikimr cms request restart host {node_id} --user {user} --duration 60 --dry --reason 'some-reason' @@ -14,13 +14,13 @@ To make sure that the process is stoppable, follow these steps. If the process is stoppable, you'll see `ALLOW`. -1. Stop the process: +1. Stop the process ```bash sudo service kikimr stop ``` -1. Restart the process if needed: +1. Restart the process if needed ```bash sudo service kikimr start @@ -38,3 +38,4 @@ Go to the [Hive web-viewer](../embedded_monitoring/hive.md) page. Click "View Nodes" to see a list of all nodes. Before disabling the node, first disable the transfer of tablets through the Active button, then click Drain, and wait for all the tablets to be moved away. + diff --git a/ydb/docs/en/core/maintenance/manual/scrubbing.md b/ydb/docs/en/core/maintenance/manual/scrubbing.md index 08627ec2b6..4da321e362 100644 --- a/ydb/docs/en/core/maintenance/manual/scrubbing.md +++ b/ydb/docs/en/core/maintenance/manual/scrubbing.md @@ -4,3 +4,4 @@ The Scrub settings let you adjust the interval from the beginning of the previou `$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { ScrubPeriodicitySeconds: 86400 MaxScrubbedDisksAtOnce: 1 } }'` If ScrubPeriodicitySeconds is 0, Scrubbing is disabled. + diff --git a/ydb/docs/en/core/maintenance/manual/selfheal.md b/ydb/docs/en/core/maintenance/manual/selfheal.md index 0f9011c946..012c02908e 100644 --- a/ydb/docs/en/core/maintenance/manual/selfheal.md +++ b/ydb/docs/en/core/maintenance/manual/selfheal.md @@ -7,7 +7,7 @@ SelfHeal includes two parts. Detecting faulty disks and moving them carefully to SelfHeal is enabled by default. Below are instructions how to enable or disable SelfHeal. -1. Enabling detection: +1. Enabling detection Open the page @@ -17,11 +17,11 @@ Below are instructions how to enable or disable SelfHeal. Status field: Enable - Or via the CLI: + Or via the CLI - * Go to any node. + * Go to any node - * Create a file with modified configurations. + * Create a file with modified configurations Sample config.txt file @@ -41,13 +41,13 @@ Below are instructions how to enable or disable SelfHeal. } ``` - * Update the config on the cluster: + * Update the config on the cluster ```bash kikimr admin console configs update config.txt ``` -2. Enable SelfHeal: +2. Enable SelfHeal ```bash kikimr -s <endpoint> admin bs config invoke --proto 'Command{EnableSelfHeal{Enable: true}}' @@ -64,7 +64,7 @@ If there are no settings yet, click Create, if there are, click the "pencil" ico * **Dry run**: Enables/disables the mode in which the CMS doesn't change the BSC setting. * **Config update interval (sec.)**: BSC config update interval. * **Retry interval (sec.)**: Config update retry interval. -* **State update interval (sec.)**: PDisk state update interval, the State is what we're monitoring (through a whiteboard, for example). +* **State update interval (sec.)**: PDisk state update interval, the State is what we're monitoring (through a whiteboard, for example) * **Timeout (sec.)**: PDisk state update timeout * **Change status retries**: The number of retries to change the PDisk Status in the BSC, the Status is what is stored in the BSC (ACTIVE, FAULTY, BROKEN, and so on). * **Change status retry interval (sec.)**: Interval between retries to change the PDisk Status in the BSC. The CMS is monitoring the disk state with the **State update interval**. If the disk remains in the same state for several **Status update interval** cycles, the CMS changes its Status in the BSC. @@ -107,3 +107,4 @@ To enable the donor disks, run the following command: `$ kikimr admin bs config invoke --proto 'Command { UpdateSettings { EnableDonorMode: true } }'` Similarly, when changing the setting to `false`, the command disables the mode. + diff --git a/ydb/docs/en/core/maintenance/manual/toc_i.yaml b/ydb/docs/en/core/maintenance/manual/toc_i.yaml new file mode 100644 index 0000000000..4c7342cc31 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/toc_i.yaml @@ -0,0 +1,23 @@ +items: +- name: How to stay within the failure model + href: failure_model.md +- name: Disk load balancing + href: balancing_load.md +- name: Methods to free up space on physical devices + href: disk_end_space.md +- name: Cluster extension + href: cluster_expansion.md +- name: Adding storage groups + href: adding_storage_groups.md +- name: Safe restart and shutdown of nodes + href: node_restarting.md +- name: Enabling/disabling SelfHeal + href: selfheal.md +- name: Enabling/disabling Scrubbing + href: scrubbing.md +- name: Moving VDisks + href: moving_vdisks.md +- name: Updating configurations via CMS + href: cms.md +- name: Updating configuration of the actor system + href: change_actorsystem_configs.md
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/manual/toc_p.yaml b/ydb/docs/en/core/maintenance/manual/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/maintenance/manual/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/toc_i.yaml b/ydb/docs/en/core/maintenance/toc_i.yaml index 5779af0e59..356f1bae10 100644 --- a/ydb/docs/en/core/maintenance/toc_i.yaml +++ b/ydb/docs/en/core/maintenance/toc_i.yaml @@ -1,47 +1,11 @@ items: - - name: Backups - href: backup_and_recovery.md - - name: Introspection tools for self-deployment - items: - - name: Overview - href: embedded_monitoring/overview.md - - name: Viewer - href: embedded_monitoring/viewer.md - when: false - - name: YDB Monitoring - href: embedded_monitoring/ydb_monitoring.md - - name: Hive web-viewer - href: embedded_monitoring/hive.md - - name: Connections overview - href: embedded_monitoring/interconnect_overview.md - # when: false - - name: Logs - href: embedded_monitoring/logs.md - - name: Charts - href: embedded_monitoring/charts.md +- name: Backups + href: backup_and_recovery.md +- name: Diagnostics + include: { mode: link, path: ../troubleshooting/toc_p.yaml } +- name: Cluster maintenance + items: + - name: Embedded UI + include: { mode: link, path: embedded_monitoring/toc_p.yaml } - name: Maintaining a cluster's disk subsystem - items: - - name: Overview - href: manual/index.md - - name: How to stay within the failure model - href: manual/failure_model.md - - name: Disk load balancing - href: manual/balancing_load.md - - name: Methods to free up space on physical devices - href: manual/disk_end_space.md - - name: Cluster extension - href: manual/cluster_expansion.md - - name: Adding storage groups - href: manual/adding_storage_groups.md - - name: Safe restart and shutdown of nodes - href: manual/node_restarting.md - - name: Enabling/disabling SelfHeal - href: manual/selfheal.md - - name: Enabling/disabling Scrubbing - href: manual/scrubbing.md - - name: Moving VDisks - href: manual/moving_vdisks.md - - name: Updating configurations via CMS - href: manual/cms.md - - name: Updating configuration of the actor system - href: manual/change_actorsystem_configs.md
\ No newline at end of file + include: { mode: link, path: manual/toc_p.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/maintenance/toc_p.yaml b/ydb/docs/en/core/maintenance/toc_p.yaml index 5bfec4365d..94ce110868 100644 --- a/ydb/docs/en/core/maintenance/toc_p.yaml +++ b/ydb/docs/en/core/maintenance/toc_p.yaml @@ -1,2 +1,4 @@ items: +- name: Overview + href: index.md - include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md index ea76108aea..0660f7c088 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_cloud.md @@ -1,4 +1,5 @@ -* If the value of the `IAM_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. -* Otherwise, if the value of the `YC_TOKEN` environment variable is set, the **Refresh Token** authentication mode is used and the token to transfer to the IAM endpoint is taken from this variable value when repeating the request. -* Otherwise, if the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. -* Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **Service Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. +- If the value of the `IAM_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. +- Otherwise, if the value of the `YC_TOKEN` environment variable is set, the **Refresh Token** authentication mode is used and the token to transfer to the IAM endpoint is taken from this variable value when repeating the request. +- Otherwise, if the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. +- Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **Service Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md index 6b2b62cb05..fbed4967db 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/env_static.md @@ -1 +1,2 @@ -* If the value of the `YDB_USER` or `YDB_PASSWORD` environment variable is set, the username and password based authentication mode is used. The username is read from the `YDB_USER` variable. If it is not set, an error saying `User password was provided without user name` is returned. The password is read from the `YDB_PASSWORD` variable. If it is not set, then, depending on whether the `--no-password` command-line option is specified, either an empty password is used or a password is requested interactively. +- If the value of the `YDB_USER` or `YDB_PASSWORD` environment variable is set, the username and password based authentication mode is used. The username is read from the `YDB_USER` variable. If it is not set, an error saying `User password was provided without user name` is returned. The password is read from the `YDB_PASSWORD` variable. If it is not set, then, depending on whether the `--no-password` command-line option is specified, either an empty password is used or a password is requested interactively. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md index 2e4db90e10..837fdae778 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud.md @@ -1,4 +1,5 @@ -* `--iam-token-file <filepath>` : The **Access Token** authentication mode is used based on the contents of the file specified in this option value. -* `--yc-token-file <filepath>` : The **Refresh Token** authentication mode is used based on the contents of the file specified in this option value. -* `--use-metadata-credentials` : The **Metadata** authentication mode is used. -* `--sa-key-file <filepath>` : The **Service Account Key** authentication mode is used with the key and other parameters taken from the JSON file specified in this option value. +- `--iam-token-file <filepath>` : The **Access Token** authentication mode is used based on the contents of the file specified in this option value. +- `--yc-token-file <filepath>` : The **Refresh Token** authentication mode is used based on the contents of the file specified in this option value. +- `--use-metadata-credentials` : The **Metadata** authentication mode is used. +- `--sa-key-file <filepath>` : The **Service Account Key** authentication mode is used with the key and other parameters taken from the JSON file specified in this option value. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md index 59b04333a5..f1937dc14e 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_cloud_additional.md @@ -1,3 +1,4 @@ When using authentication modes that involve token rotation along with regularly re-requesting them from IAM (**Refresh Token** and **Service Account Key**), a special parameter can be set to indicate where the IAM service is located: -* `--iam-endpoint <URL>` : Sets the URL of the IAM service to request new tokens in authentication modes with token rotation. The default value is `"iam.api.cloud.yandex.net"`. +- `--iam-endpoint <URL>` : Sets the URL of the IAM service to request new tokens in authentication modes with token rotation. The default value is `"iam.api.cloud.yandex.net"`. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md index 791d21c038..1d65d46fc7 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_header.md @@ -1 +1,2 @@ The authentication mode and parameters are selected by setting one of the following options: + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md index 35be48b5e6..5a525922d9 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_multiple.md @@ -5,3 +5,4 @@ $ {{ ydb-cli }} --use-metadata-credentials --iam-token-file ~/.ydb/token scheme More than one auth method were provided via options. Choose exactly one of them Try "--help" option for more info. ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md index e2b36ddf53..b830d88295 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/auth/options_static.md @@ -1,3 +1,4 @@ -* `--user <username>` : The username and password based authentication mode is used with the username set in this option value. Additionally, you can specify: - * `--password-file <filename>` : The password is read from the specified file. - * `--no-password` : Defines an empty password. The password will be requested interactively if none of the password identification options listed above are specified in the command line parameters. +- `--user <username>` : The username and password based authentication mode is used with the username set in this option value. Additionally , you can specify: + - `--password-file <filename>` : The password is read from the specified file. + - `--no-password` : Defines an empty password. The password will be requested interactively if none of the password identification options listed above are specified in the command line parameters. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md b/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md index 8677af73fe..674808ddc8 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/commands.md @@ -6,13 +6,13 @@ General syntax for calling {{ ydb-short-name }} CLI commands: {{ ydb-cli }} [global options] <command> [<subcommand> ...] [command options] ``` -Where: +, where: -* `{{ ydb-cli}}` is the command to run the {{ ydb-short-name }} CLI from the OS command line. -* `[global options]` are [global options](../commands/global-options.md) that are common for all {{ ydb-short-name }} CLI commands. -* `<command>` is the command. -* `[<subcomand> ...]` are subcommands specified if the selected command contains subcommands. -* `[command options]` are command options specific to each command and subcommands. +- `{{ ydb-cli}}` is the command to run the {{ ydb-short-name }} CLI from the OS command line. +- `[global options]` are [global options](../commands/global-options.md) that are common for all {{ ydb-short-name }} CLI commands. +- `<command>` is the command. +- `[<subcomand> ...]` are subcommands specified if the selected command contains subcommands. +- `[command options]` are command options specific to each command and subcommands. ## Commands {#list} @@ -30,15 +30,15 @@ Any command can be run from the command line with the `--help` option to get hel | [config profile set](../profile/activate.md) | Activating a [profile](../profile/index.md) | | [discovery list](../commands/discovery-list.md) | List of endpoints | | [discovery whoami](../commands/discovery-whoami.md) | Authentication | -| export s3 | Exporting data to S3 storage | +| [export s3](../export_import/s3_export.md) | Exporting data to S3 storage | | import file csv | Importing data from a CSV file | | import file tsv | Importing data from a TSV file | -| import s3 | Importing data from S3 storage | +| [import s3](../export_import/s3_import.md) | Importing data from S3 storage | | [init](../profile/create.md) | Initializing the CLI, creating a [profile](../profile/index.md) | -| operation cancel | Interrupting a long-running operation | -| operation forget | Deleting a long-running operation from the history | -| operation get | Getting the status of a long-running operation | -| operation list | List of long-running operations | +| operation cancel | Aborting a background operation | +| operation forget | Removing a background operation from history | +| operation get | Background operation status | +| operation list | List of background operations | | [scheme describe](../commands/scheme-describe.md) | Description of a data schema object | | [scheme ls](../commands/scheme-ls.md) | List of data schema objects | | [scheme mkdir](../commands/dir.md#mkdir) | Creating a directory | @@ -54,21 +54,22 @@ Any command can be run from the command line with the `--help` option to get hel | table attribute add | Adding a table attribute | | table attribute drop | Deleting a table attribute | | table drop | Deleting a table | -| [table index add global](../commands/index-ops.md) | Adding a synchronous index | -| [table index add global-async](../commands/index-ops.md) | Adding an asynchronous index | -| [table index add global-sync](../commands/index-ops.md) | Adding a synchronous index | -| [table index drop](../commands/index-ops.md) | Deleting an index | +| [table index add global-async](../commands/secondary_index.md#add) | Adding an asynchronous index | +| [table index add global-sync](../commands/secondary_index.md#add) | Adding a synchronous index | +| [table index drop](../commands/secondary_index.md#drop) | Deleting an index | | [table query execute](../commands/query.md) | Executing a YQL query | | [table query explain](../commands/explain-plan.md) | YQL query execution plan | | [table readtable](../commands/readtable.md) | Streaming table reads | | table ttl drop | Deleting TTL parameters | | table ttl set | Setting TTL parameters | | tools copy | Copying tables | -| tools dump | Dumping a directory or table to the file system | +| [tools dump](../export_import/tools_dump.md) | Dumping a directory or table to the file system | | [tools rename](../commands/tools/rename.md) | Renaming tables | -| tools restore | Restoring data from the file system | +| [tools restore](../export_import/tools_restore.md) | Restoring data from the file system | + {% if ydb-cli == "ydb" %} -|[update](../commands/service.md) | Updating the {{ ydb-short-name }} CLI | -|[version](../commands/service.md) | Displaying the version of the {{ ydb-short-name }} CLI | +[update](../commands/service.md) | Updating the {{ ydb-short-name }} CLI +[version](../commands/service.md) | Displaying the version of the {{ ydb-short-name }} CLI {% endif %} -|[workload](../commands/workload/index.md) | Generating YQL load | Running a YQL script (with streaming support) | +[workload](../commands/workload/index.md) | Generating YQL load | Running a YQL script (with streaming support) + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md b/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md index 6aaa6bef91..456fb2d56f 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/connect.md @@ -23,8 +23,8 @@ DB connection options in the command line are specified before defining the comm {{ ydb-cli }} <connection_options> <command> <command_options> ``` -* `-e, --endpoint <endpoint>` : [Endpoint](../../../concepts/connect.md#endpoint): The main connection parameter that allows finding the {{ ydb-short-name }} server on the network. If no port is specified, port 2135 is used. If no protocol is specified, gRPCs (with encryption) is used in {{ ydb-short-name }} CLI public builds. -* `-d, --database <database>` : [DB location](../../../concepts/connect.md#database). +- `-e, --endpoint <endpoint>` : [Endpoint](../../../concepts/connect.md#endpoint): The main connection parameter that allows finding the {{ ydb-short-name }} server on the network. If no port is specified, port 2135 is used. If no protocol is specified, gRPCs (with encryption) is used in {{ ydb-short-name }} CLI public builds. +- `-d, --database <database>` : [DB location](../../../concepts/connect.md#database). {% include [auth/options.md](auth/options.md) %} @@ -54,16 +54,17 @@ If all the steps described in the beginning of this article are completed, but t If the authentication mode is known, but the necessary additional parameters are not, the command is aborted and an error message describing the issue is returned: -* `(No such file or directory) util/system/file.cpp:857: can't open "<filepath>" with mode RdOnly|Seq (0x00000028)`: Couldn't open and read the `<filepath>` file specified in one of the parameters with the file name and path. +- `(No such file or directory) util/system/file.cpp:857: can't open "<filepath>" with mode RdOnly|Seq (0x00000028)`: Couldn't open and read the `<filepath>` file specified in one of the parameters with the file name and path. ## Additional parameters {#additional} When using gRPCs (with encryption), you may need to [select a root certificate](../../../concepts/connect.md#tls-cert): -* `--ca-file <filepath>` : Root certificate PEM file for a TLS connection. +- `--ca-file <filepath>` : Root certificate PEM file for a TLS connection. Currently, root certificates are not stored in profiles and can only be defined by command line options. ## Verifying authentication {#whoami} The {{ ydb-short-name }} CLI [`discovery whoami`](../commands/discovery-whoami.md) auxiliary command lets you check the account that you actually used to authenticate with the server. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/example_db1.md b/ydb/docs/en/core/reference/ydb-cli/_includes/example_db1.md new file mode 100644 index 0000000000..9253ffe752 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/example_db1.md @@ -0,0 +1,2 @@ +The examples use a profile named `db1`. For information about how to create it, see the [Getting started with the YDB CLI](../../../getting_started/cli.md#profile) article in the "Getting started " section. + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/_includes/index.md index b407502fb7..c49fd1e0ec 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/index.md @@ -15,6 +15,7 @@ For a full description of {{ ydb-short-name }} CLI commands, see the following a * [Working with directories](../commands/dir.md). * [Making a DB query](../commands/query.md). * [Streaming table reads](../commands/readtable.md). -* [Working with secondary indexes](../commands/index-ops.md). +* [Working with secondary indexes](../commands/secondary_index.md). * [Getting a list of DB endpoints](../commands/discovery-list.md). * [Load testing](../commands/workload/index.md). + diff --git a/ydb/docs/en/core/reference/ydb-cli/_includes/install.md b/ydb/docs/en/core/reference/ydb-cli/_includes/install.md index 81e4f76a92..87c5f29882 100644 --- a/ydb/docs/en/core/reference/ydb-cli/_includes/install.md +++ b/ydb/docs/en/core/reference/ydb-cli/_includes/install.md @@ -75,3 +75,4 @@ {% endnote %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md index 697af43f5a..54b58c7c1e 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/conn_options_ref.md @@ -1 +1,2 @@ where [connection options] are [database connection options](../../connect.md#command-line-pars) + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md index 8d4eee7dba..2283df1256 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/dir.md @@ -25,17 +25,17 @@ Issues: pathId: [OwnerId: <some>, LocalPathId: <some>], path type: EPathTypeDir, path state: EPathStateNoChanges ``` -The full path syntax starting with a `/` character is also supported. The full path must begin with the [database location](../../../../concepts/connect.md#database)specified in the connection parameters or with which operations are allowed via the established connection to the cluster. +The full path syntax starting with a `/` character is also supported. The full path must begin with the [database location](../../../../concepts/connect.md#database) specified in the connection parameters or with which operations are allowed via the established connection to the cluster. Examples: -* Creating a directory at the database root: +- Creating a directory at the database root ```bash {{ ydb-cli }} --profile db1 scheme mkdir dir1 ``` -* Creating directories at the specified path from the database root: +- Creating directories at the specified path from the database root ```bash {{ ydb-cli }} --profile db1 scheme mkdir dir1/dir2/dir3 @@ -81,4 +81,4 @@ Names of objects issued in [YQL](../../../../yql/reference/index.md) queries may ## Implicit creation of directories during import {#import} -The data import command creates a directory tree mirroring the original imported catalog. +The data import command creates a directory tree mirroring the original imported catalog.
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md index e28160d6bd..2f1b93166e 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-list.md @@ -10,9 +10,9 @@ The `discovery list` information command lets you get a list of [endpoints](../. The output rows in the response contain the following information: -1. Endpoint, including protocol and port. -2. Availability zone (in square brackets). -3. The `#` character is used for the list of {{ ydb-short-name }} services available on this endpoint. +1. Endpoint, including protocol and port +2. Availability zone (in square brackets) +3. The `#` character is used for the list of {{ ydb-short-name }} services available on this endpoint An endpoint discovery request to the {{ ydb-short-name }} cluster is executed in the {{ ydb-short-name }} SDK at driver initialization so that you can use the `discovery list` CLI command to localize connection issues. @@ -24,3 +24,4 @@ grpcs://vm-etn01q5-ysor.etn01q5k.ydb.mdb.yandexcloud.net:2135 [sas] #table_servi grpcs://vm-etn01q5-arum.etn01ftr.ydb.mdb.yandexcloud.net:2135 [vla] #table_service #scripting #discovery #rate_limiter #locking #kesus grpcs://vm-etn01q5beftr.ydb.mdb.yandexcloud.net:2135 [myt] #table_service #scripting #discovery #rate_limiter #locking #kesus ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md index 7119609561..a3b4ede641 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/discovery-whoami.md @@ -22,3 +22,4 @@ User SID: aje5kkjdgs0puc18976co@as User has no groups ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md index e323ac3c3c..44690bd9b8 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/global-options.md @@ -6,5 +6,6 @@ DB connection options are described in [Connecting to and authenticating with a ## Service options {#service-options} -* `--profile <name>` : Indicates the use of the DB connection profile with the specified name when executing a {{ ydb-short-name }} CLI command. Most connection parameters can be stored in the profile. -* `-v, --verbose` : Outputs detailed information about all operations being executed. Specifying this option is helpful when locating DB connection issues. +- `--profile <name>` : Indicates the use of the DB connection profile with the specified name when executing a {{ ydb-short-name }} CLI command. Most connection parameters can be stored in the profile. +- `-v, --verbose` : Outputs detailed information about all operations being executed. Specifying this option is helpful when locating DB connection issues. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/index.md new file mode 100644 index 0000000000..db8a93d9df --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/index.md @@ -0,0 +1,12 @@ +# Overview + +This section contains descriptions of the {{ ydb-short-name }} CLI commands and examples of how you can use them to perform the following operations in the DB: + +* [Listing objects](../../commands/scheme-ls.md). +* [Getting information about schema objects](../../commands/scheme-describe.md). +* [Getting a list of DB endpoints](../../commands/discovery-list.md). +* [Making a DB query](../../commands/query.md). +* [Streaming table reads](../../commands/readtable.md). +* [Working with secondary indexes](../../commands/operations-index.md). +* [Working with directories](../../commands/scheme-mkdir.md). + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md index bc18e0cf13..1251c2aa4c 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/scheme-ls.md @@ -14,59 +14,62 @@ In the `path` parameter, you can specify the [directory](../dir.md) you want to The following options are available for the command: -* `-l` : Full details about attributes of each object. -* `-R` : Recursive traversal of all subdirectories. +- `-l` : Full details about attributes of each object +- `-R` : Recursive traversal of all subdirectories **Examples** -* Getting objects from the root database directory in a compressed format: +{% include [example_db1.md](../../_includes/example_db1.md) %} + +- Getting objects from the root database directory in a compressed format ```bash {{ ydb-cli }} --profile db1 scheme ls ``` -* Getting objects in all database directories in a compressed format: +- Getting objects in all database directories in a compressed format ```bash {{ ydb-cli }} --profile db1 scheme ls -R ``` -* Getting objects from the given database directory in a compressed format: +- Getting objects from the given database directory in a compressed format ```bash {{ ydb-cli }} --profile db1 scheme ls dir1 {{ ydb-cli }} --profile db1 scheme ls dir1/dir2 ``` -* Getting objects in all subdirectories in the given directory in a compressed format: +- Getting objects in all subdirectories in the given directory in a compressed format ```bash {{ ydb-cli }} --profile db1 scheme ls dir1 -R {{ ydb-cli }} --profile db1 scheme ls dir1/dir2 -R ``` -* Getting complete information on objects in the root database directory: +- Getting complete information on objects in the root database directory ```bash {{ ydb-cli }} --profile db1 scheme ls -l ``` -* Getting complete information about objects in a given database directory: +- Getting complete information about objects in a given database directory ```bash {{ ydb-cli }} --profile db1 scheme ls dir1 -l {{ ydb-cli }} --profile db1 scheme ls dir2/dir3 -l ``` -* Getting complete information about objects in all database directories: +- Getting complete information about objects in all database directories ```bash {{ ydb-cli }} --profile db1 scheme ls -lR ``` -* Getting complete information on objects in all subdirectories of a given database directory: +- Getting complete information on objects in all subdirectories of a given database directory ```bash {{ ydb-cli }} --profile db1 scheme ls dir1 -lR {{ ydb-cli }} --profile db1 scheme ls dir2/dir3 -lR ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/secondary_index.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/secondary_index.md new file mode 100644 index 0000000000..ac12c35d5c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/secondary_index.md @@ -0,0 +1,107 @@ +# Creating and deleting secondary indexes + +The `table index` command lets you create and delete [secondary indexes](../../../../concepts/secondary_indexes.md): + +```bash +{{ ydb-cli }} [connection options] table index [subcommand] [options] +``` + +{% include [conn_options_ref.md](conn_options_ref.md) %} + +For information about the purpose and use of secondary indexes for app development, see [Secondary indexes](../../../../best_practices/schema_design.md) in the "Recommendations" section. + +## Creating a secondary index {#add} + +A secondary index is created with `table index add`: + +```bash +{{ ydb-cli }} [connection options] table index add <sync_async> <table> \ + --index-name STR --columns STR [--cover STR] +``` + +Parameters: + +`<sync_async>`: Secondary index type. Set `global_sync` to build an index with [synchronous updates](../../../../concepts/secondary_indexes.md#sync) or `global_async` for an index with [asynchronous updates](../../../../concepts/secondary_indexes.md#async). + +`<table>`: Path and name of the table that the index is being built for. + +`--index-name STR`: Required parameter that sets the index name. We recommend setting such index names that clearly indicate which columns they include. Index names are unique in the context of a table. + +`--columns STR`: Required parameter that sets the structure and order of columns included in the index key. A list of comma-separated column names, without a space. The index key will consist of these columns with the columns of the table primary key added. + +`--cover STR`: Optional parameter that sets the structure of [cover columns](../../../../concepts/secondary_indexes.md#cover) for the index. Their values won't be included in the index key, but will be copied to the entries in the index to get their values when searching by index without having to access the table. + +If the command is successful, a background build index operation is run and the operation ID is returned in the `id` field with semigraphics formatting to further get information about its status with the `operation get` command. To abort an incomplete build index operation, run the `operation cancel` command. + +Once the index build is either completed or aborted, you can delete the build operation record by running the `operation forget` command. + +To get information about the status of all build index operations, run the `operation list buildindex` command. + +**Examples** + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +Adding a synchronous index by the `air_date` column to the `episodes` table from the [article on YQL](../../../../getting_started/yql.md) in the "Getting started" section: + +```bash +{{ ydb-cli }} -p db1 table index add global-sync episodes \ + --index-name idx_aired --columns air_date +``` + +Adding an asynchronous index by the `release_date` and `title` columns along with copying to the index the `series_info` column value for the `series` table from the [article on YQL](../../../../getting_started/yql.md) in the "Getting started" section: + +```bash +{{ ydb-cli }} -p db1 table index add global-async series \ + --index-name idx_rel_title --columns release_date,title --cover series_info +``` + +Output (the ID of the operation when it's actually run will be different): + +```text +┌──────────────────────────────────┬───────┬────────┐ +| id | ready | status | +├──────────────────────────────────┼───────┼────────┤ +| ydb://buildindex/7?id=2814749869 | false | | +└──────────────────────────────────┴───────┴────────┘ +``` + +Getting the operation status (use the actual operation ID): + +```bash +{{ ydb-cli }} -p db1 operation get ydb://buildindex/7?id=281474976866869 +``` + +Returned value: + +```text +┌──────────────────────────────────┬───────┬─────────┬───────┬──────────┬─────────────────┬───────────┐ +| id | ready | status | state | progress | table | index | +├──────────────────────────────────┼───────┼─────────┼───────┼──────────┼─────────────────┼───────────┤ +| ydb://buildindex/7?id=2814749869 | true | SUCCESS | Done | 100.00% | /local/episodes | idx_aired | +└──────────────────────────────────┴───────┴─────────┴───────┴──────────┴─────────────────┴───────────┘ +``` + +Deleting information about the build index operation (use the actual operation ID): + +```bash +{{ ydb-cli }} -p db1 operation forget ydb://buildindex/7?id=2814749869 +``` + +## Deleting a secondary index {#drop} + +A secondary index is deleted with `table index drop`: + +```bash +{{ ydb-cli }} [connection options] table index drop <table> --index-name STR +``` + +**Example** + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +Deleting the `idx_aired` index built in the above index creation example from the episodes table: + +```bash +{{ ydb-cli }} -p db1 table index drop episodes --index-name idx_aired +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md index 83a7503d98..dcf959c4a4 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/service.md @@ -4,10 +4,11 @@ These commands have to do with the {{ ydb-short-name }} CLI client itself and do | Name | Description | | --- | --- | -| `-?`, `-h`, `--help` | Output the {{ ydb-short-name }} CLI syntax help. | -| `Version` | Output the {{ ydb-short-name }} CLI version (for public builds). | -| `update` | Update the {{ ydb-short-name }} CLI to the latest version (for public builds). | -| `--license` | Show the license (for public builds). | -| `--credits` | Show third-party product licenses (for public builds). | +| `-?`, `-h`, `--help` | Output the {{ ydb-short-name }} CLI syntax help | +| `Version` | Output the {{ ydb-short-name }} CLI version (for public builds) | +| `update` | Update the {{ ydb-short-name }} CLI to the latest version (for public builds) | +| `--license` | Show the license (for public builds) | +| `--credits` | Show third-party product licenses (for public builds) | If it is not known whether the used {{ ydb-short-name }} CLI build is public, you can find out if a particular service command is supported through help output. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md index cff6d86014..ee82fd9b59 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/rename.md @@ -1,6 +1,6 @@ # Renaming a table -Using the `tools rename` subcommand, you can [rename](../../../../../concepts/datamodel.md#rename) one or more tables at the same time, move a table to a different directory within the same database, or replace one table with another within the same transaction. +Using the `tools rename` subcommand, you can [rename](../../../../../concepts/datamodel.md#rename) one or more tables at the same time, move a table to another directory within the same database, replace one table with another one within the same transaction. General command format: @@ -23,20 +23,20 @@ A single run of the `tools rename` command executes a single rename transaction | Parameter name | Parameter description | | --- | --- | -| `--item <value>=<value>,...` | Description of the rename operation. Can be specified multiple times if multiple rename operations need to be executed within a single transaction.</br></br>Required properties:</br><ul><li>`source`, `src`, and `s`: Path to the source table.</li><li>`destination`, `dst`, and `d`: Path to the destination table. If the destination path contains folders, they must be [created in advance](../../dir.md#mkdir).</li></ul>Advanced properties:</br><ul> <li>`replace`, `force`: Overwrite the destination table. If `True`, the destination table is overwritten with its data deleted. `False`: If the destination table exists, an error is returned and the entire rename transaction is canceled. Default value: `False`.</li></ul> | +| `--item <property>=<value>,...` | Description of the rename operation. Can be specified multiple times if multiple rename operations need to be executed within a single transaction.</br></br>Required properties:</br><ul><li>`source`, `src`, and `s`: Path to the source table.</li><li>`destination`, `dst`, and `d`: Path to the destination table. If the destination path contains folders, they must be [created in advance](../../dir.md#mkdir).</li></ul>Advanced properties:</br><ul> <li>`replace`, `force`: Overwrite the destination table. If `True`, the destination table is overwritten with its data deleted. `False`: If the destination table exists, an error is returned and the entire rename transaction is canceled. Default value: `False`.</li></ul> | | `--timeout <value>` | Operation timeout, ms. | When including multiple rename operations in a single `tools rename` call, they're executed in the specified order, but within a single transaction. This lets you rotate the table under load without data loss: the first operation is renaming the working table to the backup one and the second is renaming the new table to the working one. ## Examples {#examples} -* Renaming a single table: +- Renaming a single table: ```bash {{ ydb-cli }} tools rename --item src=old_name,dst=new_name ``` -* Renaming multiple tables within a single transaction: +- Renaming multiple tables within a single transaction: ```bash {{ ydb-cli }} tools rename \ @@ -45,7 +45,7 @@ When including multiple rename operations in a single `tools rename` call, they' --item source=new-project/third_table,destination=new-project/series ``` -* Moving tables to a different directory: +- Moving tables to a different directory: ```bash {{ ydb-cli }} tools rename \ @@ -54,17 +54,18 @@ When including multiple rename operations in a single `tools rename` call, they' --item source=new-project/third_table,destination=cinema/third_table ``` -* Replacing a table +- Replacing a table ```bash {{ ydb-cli }} tools rename \ --item replace=True,source=pre-prod-project/main_table,destination=prod-project/main_table ``` -* Rotating a table +- Rotating a table ```bash {{ ydb-cli }} tools rename \ --item source=prod-project/main_table,destination=prod-project/main_table.backup \ --item source=pre-prod-project/main_table,destination=prod-project/main_table ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md index 42cd5c6aa3..f8d012dd9d 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/_includes/tools/restore.md @@ -25,7 +25,7 @@ View a description of the command to restore data from a dump: | `-o`<br/>`--output` | Required parameter.<br/>The path on the local file system where the dump objects will be placed. The dump folder must not exist or must be empty. | | `--scheme-only` | Make a dump of the DB schema only. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | | `--avoid-copy` | Avoid copying. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | -| `--save-partial-result` | Save the results of an incomplete dump. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | +| `--save-partial-result` | Save the results of an incomplete dump. Possible values:<br/><ul><li>`0`: No.</li><li>`1`: Yes.</li>Default value is `0`. | | `--consistency-level` | Consistency level. Possible values:<br/><ul><li>`database`: Consistency at the DB level.</li><li>`table`: Consistency at the table level.</li>Default value is `database`. | `-p <value>`<br/>`--path <value>` | Required parameter.<br/>The path in the DB by which the folder or table is restored. diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md index e5cacf169f..f9a5dc8b22 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-ls.md @@ -1 +1 @@ -{% include [intro.md](_includes/scheme-ls.md) %} +{% include [_includes/scheme-ls.md](_includes/scheme-ls.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md index 5ca3443385..180e27a7ca 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/scheme-mkdir.md @@ -1 +1,2 @@ This page has been deleted, the content has been moved to a [new section](dir.md). + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/secondary_index.md b/ydb/docs/en/core/reference/ydb-cli/commands/secondary_index.md new file mode 100644 index 0000000000..4c7071ffe7 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/commands/secondary_index.md @@ -0,0 +1 @@ +{% include [_includes/secondary_index.md](_includes/secondary_index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md index 1af0f57f21..e1430ca139 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md @@ -22,3 +22,4 @@ See the description of the command to run the data load: The following types of load tests are supported at the moment: * [Stock](../stock.md): An online store warehouse simulator. + diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md index 37c369878f..895db3ea78 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/stock.md @@ -116,7 +116,7 @@ To run this type of load, execute the command: ``` * `global workload options`: The [global options for all types of load](#global_workload_options). -* `specific workload options`: [Parameters of a specific type of load](#customer_history_options). +* `specific workload options`: [Parameters of a specific type of load](#customer_history_options) ### Parameters for getCustomerHistory {#customer_history_options} @@ -147,7 +147,7 @@ To run this type of load, execute the command: ``` * `global workload options`: The [global options for all types of load](#global_workload_options). -* `specific workload options`: [Parameters of a specific type of load](#random_customer_history_options). +* `specific workload options`: [Parameters of a specific type of load](#random_customer_history_options) ### Parameters for getRandomCustomerHistory {#random_customer_history_options} @@ -180,7 +180,7 @@ To run this type of load, execute the command: ``` * `global workload options`: The [global options for all types of load](#global_workload_options). -* `specific workload options`: [Parameters of a specific type of load](#insert_random_order_options). +* `specific workload options`: [Parameters of a specific type of load](#insert_random_order_options) ### Parameters for insertRandomOrder {#insert_random_order_options} @@ -237,7 +237,7 @@ To run this type of load, execute the command: ``` * `global workload options`: The [global options for all types of load](#global_workload_options). -* `specific workload options`: [Parameters of a specific type of load](#submit_random_order_options). +* `specific workload options`: [Parameters of a specific type of load](#submit_random_order_options) ### Parameters for submitRandomOrder {#submit_random_order_options} @@ -294,13 +294,13 @@ To run this type of load, execute the command: ``` * `global workload options`: The [global options for all types of load](#global_workload_options). -* `specific workload options`: [Parameters of a specific type of load](#submit_same_order_options). +* `specific workload options`: [Parameters of a specific type of load](#submit_same_order_options) ### Parameters for submitSameOrder {#submit_same_order_options} | Parameter name | Short name | Parameter description | | --- | --- | --- | -| `--products <value>` | `-p <value>`| | Number of products per order. The default value is 100. | +| `--products <value>` | `-p <value>` | Number of products per order. The default value is 100. | ## Examples of running the loads @@ -368,3 +368,4 @@ Txs Txs/Sec Retries Errors p50(ms) p95(ms) p99(ms) pMax(ms) * `p99(ms)`: 99th percentile of request latency, in ms. * `pMax(ms)`: 100th percentile of request latency, in ms. * `Timestamp`: Timestamp of the end of the time window. + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/file_structure.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/file_structure.md new file mode 100644 index 0000000000..3cbd66e4fd --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/file_structure.md @@ -0,0 +1,39 @@ +# File structure of data export + +The file structure described below is used for exporting data both to the file system and S3-compatible object storage. + +## Directories {#dir} + +Each directory in a database corresponds to a directory in the file structure. The directory hierarchy in the file structure corresponds the directory hierarchy in the database. If some DB directory contains no objects (neither tables nor subdirectories), this directory's file structure contains a single zero size file named `empty_dir`. + +## Tables {#tables} + +Each DB table also has a corresponding same-name directory in the file structure directory hierarchy, which contains: + +- The `scheme.pb` file with information about the table structure and its parameters in [text protobuf](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.text_format) format. +- One or more `data_XX.csv` files with data in `CSV` format, where `XX` is the file sequence number. Data export starts with the `data_00.csv` file. Each subsequent file is created once the size of the current file exceeds 100 MB. + +## Data files {#datafiles} + +Data is stored in `.csv` files, one file line per table entry, without a row with column headers. URL-encoded format is used for string representation. For example, a file line for a table with uint64 and uft8 columns containing the number 1 and the string "Hello", respectively, looks like this: + +``` +1,"%D0%9F%D1%80%D0%B8%D0%B2%D0%B5%D1%82" +``` + +## Example {#example} + +When exporting tables created within a tutorial when [Getting started with YQL](../../../../getting_started/yql.md#create-table) in the "Getting started" section, the following file structure is created: + +``` +├── episodes +│ ├── data_00.csv +│ └── scheme.pb +├── seasons +│ ├── data_00.csv +│ └── scheme.pb +└── series + ├── data_00.csv + └── scheme.pb +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/index.md new file mode 100644 index 0000000000..65dc716760 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/index.md @@ -0,0 +1,16 @@ +# Exporting and importing data + +The YDB CLI contains a set of commands designed to export and import data and descriptions of data schema objects. Data can be exported to create backups for subsequent recovery and for other purposes. + +- [The export file structure](../file_structure.md) is used for exporting data both to the file system and S3-compatible object storage. + +- [Exporting data to the file system using `tools dump`](../tools_dump.md) + +- [Importing data from the file system using `tools restore`](../tools_restore.md) + +- [Connecting to and authenticating with S3-compatible object storage](../s3_conn.md) + +- [Exporting data to S3-compatible object storage using `export s3`](../s3_export.md) + +- [Data import from S3-compatible object storage using `import s3`](../s3_import.md) + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn.md new file mode 100644 index 0000000000..b5180a8b99 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn.md @@ -0,0 +1,116 @@ +# Connecting to S3-compatible object storage + +The `export s3` and `import s3` commands for exporting data to and importing data from S3-compatible storage, respectively, use the same S3 connection and authentication parameters. For information about how to find out these parameters for some cloud providers, see the [Getting S3 connection parameters](#procure) section below. + +## Connection {#conn} + +To connect to S3, make sure to specify the endpoint and bucket: + +`--s3-endpoint HOST`: S3 endpoint. `HOST`: Valid hostname such as `storage.yandexcloud.net` + +`--bucket STR`: S3 bucket. `STR`: String with the bucket name. + +## Authentication {#auth} + +To establish a connection, except when importing data from a public bucket, you'll need to authenticate under an account with write (for import) or read (for export) permission granted for this bucket. + +To authenticate in S3, the following two parameters are required: + +- Access key ID (access_key_id) +- Secret access key (secret_access_key) + +The YDB CLI takes these parameter values from the following sources (in order of priority): + +1. The command line. +2. Environment variables. +3. The `~/.aws/credentials` file. + +### Command line parameters + +`--access-key STR`: Access key ID `--secret-key STR`: Secret access key + +### Environment variables + +If any authentication parameter is not specified in the command line, the YDB CLI tries to get it from the following environment variables: + +`AWS_ACCESS_KEY_ID`: Access key ID `AWS_SECRET_ACCESS_KEY`: Secret access key + +### AWS authentication file + +If any authentication parameter is not specified in the command line and the YDB CLI couldn't fetch it from the environment variable, it tries to get it from the `~/.aws/credentials` file that is used for authentication in the [AWS CLI](https://aws.amazon.com/ru/cli/). You can create this file with the AWS CLI `aws configure` command. + +## Getting S3 connection parameters {#procure} + +### {{ yandex-cloud }} + +Follow the instructions below to get [{{ yandex-cloud }} Object Storage](https://cloud.yandex.ru/docs/storage/) access keys using the {{ yandex-cloud }} CLI. + +1. [Install and configure the](https://cloud.yandex.ru/docs/cli/quickstart) {{ yandex-cloud }} CLI. + +2. Run the following command to get the ID of your folder in the cloud (you'll need to specify it in the commands below): + + ```bash + yc config list + ``` + + In the command output, the cloud folder ID is in the `folder-id:` line: + + ```yaml + folder-id: b2ge70qdcff4bo9q6t19 + ``` + +3. [Run the following command to create a service account](https://cloud.yandex.ru/docs/iam/operations/sa/create): + + ```bash + yc iam service-account create --name s3account + ``` + + You can specify any account name except `s3account` or use an existing one. In this case, you'll also need to replace it when copying commands below via the clipboard. + +3. [Run the following command to assign roles for the service account](https://cloud.yandex.ru/docs/iam/operations/sa/assign-role-for-sa) according to the required S3 access level: + + {% list tabs %} + + - Read (to import data to the YDB database) + + ```bash + yc resource-manager folder add-access-binding <folder-id> \ + --role storage.viewer --subject serviceAccount:s3account + ``` + + - Write (to export data from the YDB database) + + ```bash + yc resource-manager folder add-access-binding <folder-id> \ + --role storage.editor --subject serviceAccount:s3account + ``` + + {% endlist %} + + where `<folder-id>` is the cloud folder ID obtained in step 2. + + You can also view a [full list](https://cloud.yandex.ru/docs/iam/concepts/access-control/roles#object-storage) of {{ yandex-cloud }} roles. + +4. Run the following command to get [static access keys](https://cloud.yandex.ru/docs/iam/operations/sa/create-access-key): + + ```bash + yc iam access-key create --service-account-name s3account + ``` + + If successful, the command returns access_key attributes and the secret value: + + ```yaml + access_key: + id: aje6t3vsbj8lp9r4vk2u + service_account_id: ajepg0mjt06siuj65usm + created_at: "2018-11-22T14:37:51Z" + key_id: 0n8X6WY6S24N7OjXQ0YQ + secret: JyTRFdqw8t1kh2-OJNz4JX5ZTz9Dj1rI9hxtzMP1 + ``` + + In this output: + - `access_key.key_id` is the access key ID. + - `secret` is the secret access key. + +{% include [s3_conn_procure_overlay.md](s3_conn_procure_overlay.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn_procure_overlay.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn_procure_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_conn_procure_overlay.md diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_export.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_export.md new file mode 100644 index 0000000000..3651e7e986 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_export.md @@ -0,0 +1,158 @@ +# Exporting data to S3-compatible storage + +Running the `export s3` command starts, on the server side, exporting data and information about data schema objects to S3-compatible storage in the format described in the [File structure](../file_structure.md) article: + +```bash +{{ ydb-cli }} [connection options] export s3 [options] +``` + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +## Command line parameters {#pars} + +`[options]`: Command parameters: + +### S3 connection parameters {#s3-conn} + +To run the command to export data to S3, make sure to specify the [S3 connection parameters](../s3_conn.md). Since data export is performed asynchronously by the YDB server, the specified endpoint must be available to establish a server-side connection. + +### List of exported objects {#items} + +`--item STRING`: Description of the object to export. The `--item` parameter can be specified several times if you need to export multiple objects. The `STRING` format is `<property>=<value>,...`, with the following properties required: + +- `source`, `src`, or `s`: Path to the directory or table to be exported, where `.` indicates the DB root directory. In the specified directory, the following are exported: any objects whose names do not begin with a dot and, recursively, any subdirectories whose names do not begin with a dot. +- `destination`, `dst`, or `d`: Path to S3 (key prefix) to store the exported objects to. + +`--exclude STRING`: Pattern ([PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) for excluding paths from the export destination. This parameter can be specified several times for different patterns. + +### Additional parameters {#aux} + +`--description STRING`: Operation text description stored in the history of operations. `--retries NUM`: Number of import retries the server will make. Defaults to 10. +`--format STRING`: Result output format. + +- `pretty`: Human-readable format (default). +- `proto-json-base64`: Protobuf that supports JSON values encoded as binary strings using base64 encoding. + +## Exporting data {#exec} + +### Export result {#result} + +If successful , the `export s3` command outputs summary information about the enqueued operation for exporting data to S3 in the format specified in the `--format` option. The actual export operation is performed by the server asynchronously. The summary displays the operation ID that can be used later to check the status and actions with the operation: + +- In the `pretty` output mode used by default, the operation identifier is output in the id field with semigraphics formatting: + + ``` + ┌───────────────────────────────────────────┬───────┬─────... + | id | ready | stat... + ├───────────────────────────────────────────┼───────┼─────... + | ydb://export/6?id=281474976788395&kind=s3 | true | SUCC... + ├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... + | StorageClass: NOT_SET + | Items: + ... + ``` + +- In proto-json-base64 output mode, the ID is in the "id" attribute: + + ``` + {"id":"ydb://export/6?id=281474976788395&kind=s3","ready":true, ... } + ``` + +### Export status {#status} + +Data is exported in the background. You can get information about the status and progress of the export operation by running the `operation get` command with the **quoted** operation ID passed as the command parameter. For example: + +```bash +{{ ydb-cli }} -p db1 operation get "ydb://export/6?id=281474976788395&kind=s3" +``` + +The format of the `operation get` command output is also specified in the `--format` option. + +Although the operation ID format is URL, there is no guarantee that it's retained later. It should only be interpreted as a string. + +You can track the completion of the export operation by changes in the "progress" attribute: + +- In the `pretty` output mode used by default, a successful operation is indicated by the "Done" value in the `progress` field with semigraphics formatting: + + ``` + ┌───── ... ──┬───────┬─────────┬──────────┬─... + | id | ready | status | progress | ... + ├──────... ──┼───────┼─────────┼──────────┼─... + | ydb:/... | true | SUCCESS | Done | ... + ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴... + ... + ``` + +- In proto-json-base64 output mode, a completed operation is indicated by the `PROGRESS_DONE` value of the `progress` attribute: + + ``` + {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... } + ``` + +### Ending the export operation {#forget} + +When exporting data, a directory named `export_*` is created in the DB root directory, where `*` is the numeric part of the export ID. This directory stores tables containing a consistent snapshot of the exported data as of the start of the export operation. + +Once the data is exported, use the `operation forget` command to make sure the export operation is ended, that is, removed from the list of operations along with deleting all the files created for it: + +```bash +{{ ydb-cli }} -p db1 operation forget "ydb://export/6?id=281474976788395&kind=s3" +``` + +### List of export operations {#list} + +To get a list of export operations, run the `operation list export/s3` command: + +```bash +{{ ydb-cli }} -p db1 operation list export/s3 +``` + +The format of the `operation list` command output is also specified in the `--format` option. + +## Examples {#examples} + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +### Exporting a database {#example-full-db} + +Exporting all DB objects whose names do not begin with a dot and are not placed inside directories whose names begin with a dot to the `export1` directory in the `mybucket` bucket, using S3 authentication parameters from environment variables or the `~/.aws/credentials` file: + +``` +ydb -p db1 export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --item src=.,dst=export1 +``` + +### Exporting multiple directories {#example-specific-dirs} + +Exporting objects from DB directories named dir1 and dir2 to the `export1` directory in the `mybucket` bucket using explicitly specified S3 authentication parameters: + +``` +ydb -p db1 export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key VJGSOScgs-5kDGeo2hO9 --secret-key fZ_VB1Wi5-fdKSqH6074a7w0J4X0 \ + --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2 +``` + +### Getting operation IDs {#example-list-oneline} + +To get a list of export operation IDs in a format that is convenient for processing in bash scripts, use [jq](https://stedolan.github.io/jq/download/): + +```bash +{{ ydb-cli }} -p db1 operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" +``` + +You'll get an output where each new line contains the operation ID. For example: + +``` +ydb://export/6?id=281474976789577&kind=s3 +ydb://export/6?id=281474976789526&kind=s3 +ydb://export/6?id=281474976788779&kind=s3 +``` + +These IDs can be used, for example, to run a loop that will end all current operations: + +```bash +{{ ydb-cli }} -p db1 operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p db1 operation forget $line;done +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_import.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_import.md new file mode 100644 index 0000000000..e928f65cd1 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/s3_import.md @@ -0,0 +1,157 @@ +# Importing data from S3-compatible storage + +Running the `import s3` command starts, on the server side, importing data and information about data schema objects from S3-compatible storage in the format described in the [File structure](../file_structure.md) article: + +```bash +{{ ydb-cli }} [connection options] import s3 [options] +``` + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +Unlike [`tools restore`](../tools_restore.md), the `import s3` command always creates entire objects, meaning that none of the objects being imported (neither directories nor tables) should exist for the command to run successfully. + +If you need to import additional data from S3 to existing tables, you can copy the S3 contents to the file system (for example, using [S3cmd](https://s3tools.org/s3cmd)) and run the [`tools restore`](../tools_restore.md) command. + +## Command line parameters {#pars} + +`[options]`: Command parameters: + +### S3 connection parameters {#s3-conn} + +To run the command to import data from S3, make sure to specify the [S3 connection parameters](../s3_conn.md). Since data import is performed asynchronously by the YDB server, the specified endpoint must be available to establish a server-side connection. + +### List of imported objects {#items} + +`--item STRING`: Description of the object to import. The `--item` parameter can be specified several times if you need to import multiple objects. The `STRING` format is `<property>=<value>,...`, with the following properties required: + +- `source`, `src`, or `s`: Path to S3 (key prefix) specifying the directory or table to import. +- `destination`, `dst`, or`d`: Path to the DB that will store the imported directory or table. The final element of the path must not exist. All directories specified in the path will be created if they don't exist. + +### Additional parameters {#aux} + +`--description STRING`: Operation text description stored in the history of operations. `--retries NUM`: Number of import retries the server will make. Defaults to 10. +`--format STRING`: Result output format. + +- `pretty`: Human-readable format (default). +- `proto-json-base64`: Protobuf that supports JSON values encoded as binary strings using base64 encoding. + +## Importing data {#exec} + +### Import result {#result} + +If successful , the `import s3` command outputs summary information about the enqueued operation for importing data from S3 in the format specified in the `--format` option. The actual import operation is performed by the server asynchronously. The summary displays the operation ID that can be used later to check the status and actions with the operation: + +- In the `pretty` output mode used by default, the operation identifier is output in the id field with semigraphics formatting: + + ``` + ┌───────────────────────────────────────────┬───────┬─────... + | id | ready | stat... + ├───────────────────────────────────────────┼───────┼─────... + | ydb://import/8?id=281474976788395&kind=s3 | true | SUCC... + ├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... + | Items: + ... + ``` + +- In proto-json-base64 output mode, the ID is in the "id" attribute: + + ``` + {"id":"ydb://export/8?id=281474976788395&kind=s3","ready":true, ... } + ``` + +### Import status {#status} + +Data is imported in the background. You can get information about the status and progress of the import operation by running the `operation get` command with the **quoted** operation ID passed as the command parameter. For example: + +```bash +{{ ydb-cli }} -p db1 operation get "ydb://import/8?id=281474976788395&kind=s3" +``` + +The format of the `operation get` command output is also specified in the `--format` option. + +Although the operation ID format is URL, there is no guarantee that it's retained later. It should only be interpreted as a string. + +You can track the completion of the import operation by changes in the "progress" attribute: + +- In the `pretty` output mode used by default, a successful operation is indicated by the "Done" value in the `progress` field with semigraphics formatting: + + ``` + ┌───── ... ──┬───────┬─────────┬──────────┬─... + | id | ready | status | progress | ... + ├──────... ──┼───────┼─────────┼──────────┼─... + | ydb:/... | true | SUCCESS | Done | ... + ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴... + ... + ``` + +- In proto-json-base64 output mode, a completed operation is indicated by the `PROGRESS_DONE` value of the `progress` attribute: + + ``` + {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... } + ``` + +### Ending the import operation {#forget} + +Once the data is imported, use the `operation forget` command to make sure the import operation is removed from the list of operations: + +```bash +{{ ydb-cli }} -p db1 operation forget "ydb://import/8?id=281474976788395&kind=s3" +``` + +### List of import operations {#list} + +To get a list of import operations, run the `operation list import/s3` command: + +```bash +{{ ydb-cli }} -p db1 operation list import/s3 +``` + +The format of the `operation list` command output is also specified in the `--format` option. + +## Examples {#examples} + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +### Importing data to the DB root {#example-full-db} + +Importing the contents of the `export1` directory in the `mybucket` bucket to the root of the database, using S3 authentication parameters from environment variables or the `~/.aws/credentials` file: + +``` +ydb -p db1 import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --item src=export1,dst=. +``` + +### Importing multiple directories {#example-specific-dirs} + +Importing objects from the dir1 and dir2 directories of the `mybucket` S3 bucket to the same-name DB directories using explicitly specified authentication parameters in S3: + +``` +ydb -p db1 import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key VJGSOScgs-5kDGeo2hO9 --secret-key fZ_VB1Wi5-fdKSqH6074a7w0J4X0 \ + --item src=export/dir1,dst=dir1 --item src=export/dir2,dst=dir2 +``` + +### Getting operation IDs {#example-list-oneline} + +To get a list of export operation IDs in a format that is convenient for processing in bash scripts, use [jq](https://stedolan.github.io/jq/download/): + +```bash +{{ ydb-cli }} -p db1 operation list import/s3 --format proto-json-base64 | jq -r ".operations[].id" +``` + +You'll get an output where each new line contains the operation ID. For example: + +``` +ydb://import/8?id=281474976789577&kind=s3 +ydb://import/8?id=281474976789526&kind=s3 +ydb://import/8?id=281474976788779&kind=s3 +``` + +These IDs can be used, for example, to run a loop that will end all current operations: + +```bash +{{ ydb-cli }} -p db1 operation list import/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p db1 operation forget $line;done +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md new file mode 100644 index 0000000000..922115158a --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md @@ -0,0 +1,53 @@ +# Dumping data to the file system + +The `tools dump` command dumps data and information about data schema objects to the client file system in the format described in the [File structure](../file_structure.md) article: + +```bash +{{ ydb-cli }} [connection options] tools dump [options] +``` + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +`[options]`: Command parameters: + +`-p PATH` or `--path PATH`: Path to the DB directory whose objects are to be dumped or path to the table. By default, the DB root directory. The following will be dumped: all subdirectories whose names do not begin with a dot and tables whose names do not begin with a dot inside these subdirectories. To dump such tables or the contents of such directories, you can explicitly specify their names in this parameter. + +`-o PATH` or `--output PATH`: Path to the directory in the client file system that data should be dumped to. If the specified directory doesn't exist, it will be created. Anyway, the entire path to it must exist. If the specified directory does exist, it must be empty. If the parameter is not specified, a directory with a name in `backup_YYYYDDMMTHHMMSS` format will be created in the current directory, where YYYYDDMM indicates the date and HHMMSS the export start time. + +`--exclude STRING`: Pattern ([PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) for excluding paths from the export destination. This parameter can be specified several times for different patterns. + +`--scheme-only`: Only dump information about data schema objects and no data. + +`--consistency-level VAL`: Consistency level. Possible options: + +- `database`: Fully consistent export with a single snapshot taken before starting the export operation. Applied by default. +- `table`: Consistency within each table being dumped with separate independent snapshots taken for each such table. It can run faster and have less impact on handling the current DB load. + +`--avoid-copy`: Do not create a dump snapshot. The snapshot used by default to ensure consistency may not be applicable in some cases (such as for tables with external blobs). + +`--save-partial-result`: Do not delete the result of a partially completed dump. If this option is not enabled, the result will be deleted in case an error occurs when dumping data. + +## Examples + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +### Exporting a database + +With a directory named `backup_...` automatically created in the current directory: + +``` +{{ ydb-cli }} --profile db1 tools dump +``` + +To the specified directory: + +``` +{{ ydb-cli }} --profile db1 tools dump -o ~/backup_db1 +``` + +### Exporting the structure of tables in the specified DB directory and its subdirectories + +``` +{{ ydb-cli }} --profile db1 tools dump -p dir1 --scheme-only +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_restore.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_restore.md new file mode 100644 index 0000000000..47a7e558a7 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_restore.md @@ -0,0 +1,90 @@ +# Importing data from the file system + +The `tools restore` command creates data schema objects in the DB and imports to them the data from the file system that was previously dumped there with the `tools dump` command or prepared manually following the rules described in the [File structure](../file_structure.md) article: + +```bash +{{ ydb-cli }} [connection options] tools restore -p PATH -i PATH [options] +``` + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +If a table already exists in the database, no changes are made to its schema. This may prevent the data import operation from being performed if some columns of the imported files are missing in the DB table or have an incorrect type. + +Data import to DB tables is performed using the [YQL `REPLACE`](../../../../yql/reference/syntax/replace_into.md) statement. If before the start of the import operation, the table contained any entries, those of them whose keys exist in the imported files are replaced with data from the files. The entries with keys that are missing in the imported files remain unchanged. + +## Required parameters {#mandatory} + +`-p PATH` or `--path PATH`: Path to the DB directory that data will be imported to. To import data to the root directory, specify `.`. Any missing directories specified in the path will be created. + +`-i PATH` or `--input PATH`: Path to the directory in the client file system that data will be imported from. + +## Optional parameters {#optional} + +`[options]`: Optional command parameters: + +`--restore-data VAL`: Data import flag, 1 (yes) or 0 (no), defaults to 1. If 0, the import operation will only create schema objects without data being restored to them. If there is no data in the file system (only the data schema is exported), it doesn't matter if you change the flag value. + +`--restore-indexes VAL`: Index import flag, 1 (yes) or 0 (no), defaults to 1. If 0, when running the import operation, secondary indexes will neither be registered in the data schema nor populated with data. + +`--dry-run`: Mode for checking if the data schema in the DB and file system match without making any changes to the DB, 1 (yes) or 0 (no), defaults to 0. If this mode is enabled, it is checked that: + +- All tables in the file system are present in the DB. +- The object data schema in the DB and file system match. + +`--save-partial-result`: Save the result of an incomplete import operation. If this option isn't enabled, in the event of an error during the import operation, the state of the DB is restored to the point before the operation is started. + +### Load limit parameters {#limiters} + +The following parameters let you limit the load on the DB generated by data import processes. + +{% note warning %} + +Some of the parameters listed below have valid default values. This means that even if none of them is specified in the `tools restore` command call, the load will still be limited. + +{% endnote %} + +`--bandwidth VAL`: Limits the amount of data that can be imported per second, defaults to 0 (not set). `VAL` stands for the data volume that is specified as a prefixed number like 2MiB. +`--rps VAL`: Limit on the number of requests per second for importing data packets to the DB, defaults to 30. +`--in-flight VAL`: Limit on the number of concurrently executed requests, defaults to 10. +`--upload-batch-rows VAL`: Limit on the number of rows in the imported batch, defaults to 0 (unlimited). `VAL` stands for the amount of rows, specified as a number with an optional decimal prefix, such as 1K. +`--upload-batch-bytes VAL`: Limit on the size of an imported batch, defaults to 512KB. `VAL` stands for the data volume that is specified as a prefixed number like 1MiB. +`--upload-batch-rus VAL`: Only applies to Serverless databases, limits the use of Request Units (RU) per import of a single batch, defaults 30 RUs. The batch size is selected for the specified value. `VAL` stands for the amount of RUs, specified as a number with an optional decimal prefix, such as 100 or 1K. + +## Examples {#examples} + +{% include [example_db1.md](../../_includes/example_db1.md) %} + +### Importing data to the DB root + +From the current directory of the file system: + +``` +{{ ydb-cli }} -p db1 tools restore -p . -i . +``` + +From the specified directory of the file system: + +``` +{{ ydb-cli }} -p db1 tools restore -p . -i ~/backup_db1 +``` + +### Importing data to the specified DB directory + +From the current directory of the file system: + +``` +{{ ydb-cli }} -p db1 tools restore -p dir1/dir2 -i . +``` + +From the specified directory of the file system: + +``` +{{ ydb-cli }} -p db1 tools restore -p dir1/dir2 -i ~/backup_db1 +``` + +Checking if the data schema in the DB and file system match: + +``` +{{ ydb-cli }} -p db1 tools restore -p dir1/dir2 -i ~/backup_db1 --dry-run +``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/file_structure.md b/ydb/docs/en/core/reference/ydb-cli/export_import/file_structure.md new file mode 100644 index 0000000000..b3f76f809c --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/file_structure.md @@ -0,0 +1,2 @@ + +{% include [file_structure.md](_includes/file_structure.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/index.md b/ydb/docs/en/core/reference/ydb-cli/export_import/index.md new file mode 100644 index 0000000000..b3cf50bcae --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/index.md @@ -0,0 +1,2 @@ + +{% include [index.md](_includes/index.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/s3_conn.md b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_conn.md new file mode 100644 index 0000000000..3dcccf4c5f --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_conn.md @@ -0,0 +1 @@ +{% include [s3_conn.md](_includes/s3_conn.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/s3_export.md b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_export.md new file mode 100644 index 0000000000..f8a7bae0ab --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_export.md @@ -0,0 +1 @@ +{% include [s3_export.md](_includes/s3_export.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/s3_import.md b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_import.md new file mode 100644 index 0000000000..b9b1861f64 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/s3_import.md @@ -0,0 +1 @@ +{% include [s3_import.md](_includes/s3_import.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/export_import/toc_i.yaml new file mode 100644 index 0000000000..b14caf2943 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/toc_i.yaml @@ -0,0 +1,13 @@ +items: +- name: File structure of data export + href: file_structure.md +- name: Exporting data to the file system + href: tools_dump.md +- name: Importing data from the file system + href: tools_restore.md +- name: Connecting to and authenticating with S3 + href: s3_conn.md +- name: Exporting data to S3 + href: s3_export.md +- name: Importing data from S3 + href: s3_import.md
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/toc_p.yaml b/ydb/docs/en/core/reference/ydb-cli/export_import/toc_p.yaml new file mode 100644 index 0000000000..94ce110868 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/toc_p.yaml @@ -0,0 +1,4 @@ +items: +- name: Overview + href: index.md +- include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/tools_dump.md b/ydb/docs/en/core/reference/ydb-cli/export_import/tools_dump.md new file mode 100644 index 0000000000..ce24f8e2f5 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/tools_dump.md @@ -0,0 +1 @@ +{% include [tools_dump.md](_includes/tools_dump.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/tools_restore.md b/ydb/docs/en/core/reference/ydb-cli/export_import/tools_restore.md new file mode 100644 index 0000000000..4856ba9b5b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/tools_restore.md @@ -0,0 +1 @@ +{% include [tools_restore.md](_includes/tools_restore.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md index 09ec2f73d0..45761c8cca 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/activate.md @@ -1,4 +1,4 @@ -# Activating a profile +# Activated profile Executing {{ ydb-short-name }} CLI commands on a database require establishing a connection to the database. If the {{ ydb-short-name }} CLI couldn't identify a certain connection parameter by [command-line parameters and environment variables](../../connect.md), it's taken from the activated profile. @@ -6,9 +6,9 @@ Profile activation is an easy way to get started with the {{ ydb-short-name }} C However, this simplicity may lead to undesirable behavior in further operation, as soon as you need to work with multiple databases: -* The activated profile is applied implicitly, meaning that it can be applied by mistake when a certain connection parameter is missing in the command line. -* The activated profile is applied implicitly, meaning that it can be applied by mistake when a typo is made in the name of an environment variable. -* The activated profile cannot be used in scripts, since it is saved in a file and its change in one terminal window will affect all other windows, possibly leading to an unexpected change of the DB in the middle of the loop being executed in the script. +- The activated profile is applied implicitly, meaning that it can be applied by mistake when a certain connection parameter is missing in the command line. +- The activated profile is applied implicitly, meaning that it can be applied by mistake when a typo is made in the name of an environment variable. +- The activated profile cannot be used in scripts, since it is saved in a file and its change in one terminal window will affect all other windows, possibly leading to an unexpected change of the DB in the middle of the loop being executed in the script. When you need to connect to any new database other than the initial one for the first time, we recommend that you deactivate the profile and always select it explicitly using the `--profile` option. @@ -20,7 +20,7 @@ Profile activation is performed by running the command {{ ydb-cli }} config profile activate [profile_name] ``` -where `[profile_name]` is an optional profile name. +, where `[profile_name]` is an optional profile name. If the profile name is specified, it is activated. If a profile with the specified name does not exist, an error is returned prompting you to view the list of available profiles: @@ -40,11 +40,11 @@ Please choose profile to activate: Please enter your numeric choice: ``` -* `1` terminates the command execution and keeps the currently activated profile activated. It's marked as `(active)` in the list of existing profiles starting from item 3. -* `2` deactivates the currently activated profile. If no profile has been activated before, nothing changes. -* `3` and so on activates the selected profile. The currently activated profile is marked as `(active)`. +- `1` terminates the command execution and keeps the currently activated profile activated. It's marked as `(active)` in the list of existing profiles starting from item 3. +- `2` deactivates the currently activated profile. If no profile has been activated before, nothing changes. +- `3` and so on activates the selected profile. The currently activated profile is marked as `(active)`. -If the profile is successfully activated, the execution ends with a message saying: +If the profile is successfully activated, the execution ends with a message saying ```text Profile "<profile_name>" was activated. @@ -80,3 +80,4 @@ echo 2 | {{ ydb-cli }} config profile activate ``` The efficiency of this method is not guaranteed in any way. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md index 8de06c3ac1..182df332f6 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/create.md @@ -49,9 +49,9 @@ The first step of the interactive scenario is different in the `init` and `profi Next, you'll be prompted to sequentially perform the following actions with each connection parameter that can be saved in the profile: -* Don't save. -* Set a new value or Use - Set a new value or Use <value>. -* Use current value (this option is available when updating an existing profile). +- Don't save +- Set a new value or Use <value> +- Use current value (this option is available when updating an existing profile) ## Example @@ -108,3 +108,4 @@ Creating a new `mydb1` profile: ```text Activate profile "mydb1" to use by default? (current active profile is not set) y/n: n ``` + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md index 2b6ccaafaa..0af6a67e77 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/delete.md @@ -35,3 +35,4 @@ echo y | {{ ydb-cli }} config profile delete my_profile ``` The efficiency of this method is not guaranteed in any way. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md index 11536c8d21..7c5f9e51fe 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/index.md @@ -2,7 +2,7 @@ A profile is a saved and locally named configuration of DB connection parameters. With profiles, you can reuse data about DB location and authentication parameters, making a CLI call much shorter: -* Calling the `scheme ls` command without a profile: +- Calling the `scheme ls` command without a profile: ```bash {{ ydb-cli }} \ @@ -12,7 +12,7 @@ A profile is a saved and locally named configuration of DB connection parameters scheme ls ``` -* Calling the same `scheme ls` command using a profile: +- Calling the same `scheme ls` command using a profile: ```bash {{ ydb-cli }} --profile db1 scheme ls @@ -20,14 +20,15 @@ A profile is a saved and locally named configuration of DB connection parameters ## Profile management commands {#commands} -* [Creating a profile](../create.md). -* [Using a profile](../use.md). -* [Getting a list of profiles and profile parameters](../list-and-get.md). -* [Deleting a profile](../delete.md). -* [Activating a profile and using the activated profile](../activate.md). +- [Creating a profile](../create.md) +- [Using a profile](../use.md) +- [Getting a list of profiles and profile parameters](../list-and-get.md) +- [Deleting a profile](../delete.md) +- [Activating a profile and using the activated profile](../activate.md) ## Where profiles are stored {#location} Profiles are stored locally in a file named `~/ydb/config/config.yaml`. {% include [location_overlay.md](location_overlay.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md index 9e98fdfce9..91966f6974 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/list-and-get.md @@ -42,3 +42,4 @@ Full information on all profiles and parameters stored in them: ``` The output of this command combines the output of the command to get a list of profiles (with the active profile marked) and the parameters of each profile in the lines following its name. + diff --git a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md index b6493ea46f..cda6440a88 100644 --- a/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md +++ b/ydb/docs/en/core/reference/ydb-cli/profile/_includes/use.md @@ -35,3 +35,4 @@ In this case, the connection parameters specified in the command line have prior If the `--profile` option is not specified in the command line, the {{ ydb-short-name }} CLI will try to take from the currently activated profile all the connection parameters that it couldn't define in other ways (from command-line options or environment variables, as described in [Connecting to and authenticating with a database](../../connect.md)). Implicit use of the activated profile may cause errors, so we recommend that you read the [Activated profile](../activate.md) article before using this mode. + diff --git a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml index c667db1dfa..0a8d0dabc3 100644 --- a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml @@ -17,6 +17,8 @@ items: href: commands/scheme-describe.md - name: Directories href: commands/dir.md + - name: Secondary indexes + href: commands/secondary_index.md - name: Renaming tables href: commands/tools/rename.md - name: Operations with data @@ -27,15 +29,15 @@ items: href: commands/explain-plan.md - name: Streaming table reads href: commands/readtable.md - - name: Working with secondary indexes - href: commands/index-ops.md - name: Scan queries href: commands/scan-query.md + - name: Importing and exporting data + include: { mode: link, path: export_import/toc_p.yaml } # - name: Utilities # items: # - name: Copy tables # href: commands/tools/copy.md - # - name: Dump + # - name: Dump backup # href: commands/tools/dump.md # - name: Restore backup # href: commands/tools/restore.md diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md index 48465c6b5a..b05d28f54c 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/auth.md @@ -24,7 +24,7 @@ You can click on any of the methods described below to go to the source code of | Anonymous | [`ydb.AnonymousCredentials()`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/anonymous-credentials) | | Access Token | [`ydb.AccessTokenCredentials( token )`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/access-token-credentials) | | Metadata | [`ydb.iam.MetadataUrlCredentials()`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/metadata-credentials) | - | Service Account Key | [`ydb.iam.ServiceAccountCredentials.from_file(`</br> `key_file, iam_endpoint=None, iam_channel_credentials=None )`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/service-account-credentials) | + | Service Account Key | [`ydb.iam.ServiceAccountCredentials.from_file(`</br> `key_file, iam_endpoint=None, iam_channel_credentials=None )`](https://github.com/yandex-cloud/ydb-python-sdk/tree/master/examples/service-account-credentials) | | Determined by environment variables | `ydb.construct_credentials_from_environ()` | - Go @@ -69,7 +69,7 @@ The following algorithm that is the same for all SDKs applies: 4. Otherwise, if the value of the `YDB_ACCESS_TOKEN_CREDENTIALS` environment variable is set, the **Access token** authentication mode is used, where the this variable value is passed. 5. Otherwise, the **Metadata** authentication mode is used. -If the last step of the algorithm is selecting the **Metadata** mode, you can deploy a working application on VMs and in Yandex Cloud Functions without setting any environment variables. +If the last step of the algorithm is selecting the **Metadata** mode, you can deploy a working application on VMs and in Cloud Functions in {{ yandex-cloud }} without setting any environment variables. ## Python SDK specifics @@ -80,8 +80,9 @@ The behavior of the Python SDK differs from the one described above. {% endnote %} 1. The algorithm for determining the authentication mode and the necessary parameters from the environment variables in the `construct_credentials_from_environ()` method differs from the one used in other SDKs: - * If the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. - * Otherwise, if the value of the `YDB_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. - * Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **System Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. - * Or else, no authentication information is added to requests. + - If the value of the `USE_METADATA_CREDENTIALS` environment variable is set to 1, the **Metadata** authentication mode is used. + - Otherwise, if the value of the `YDB_TOKEN` environment variable is set, the **Access Token** authentication mode is used, where this variable value is passed. + - Otherwise, if the value of the `SA_KEY_FILE` environment variable is set, the **System Account Key** authentication mode is used and the key is taken from the file whose name is specified in this variable. + - Or else, no authentication information is added to requests. 2. If no object responsible for generating tokens is passed when initializing the driver, the [general procedure](#env) for reading environment variable values applies. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md index 3c87c71abb..e0664c8244 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/index.md @@ -6,18 +6,19 @@ OpenSource SDKs in the following programming languages are available to work wit {% if oss %} -* C++ [https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp) +- C++ [https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp](https://github.com/ydb-platform/ydb/tree/main/ydb/public/sdk/cpp) {% endif %} -* C# (.NET) [https://github.com/ydb-platform/ydb-dotnet-sdk](https://github.com/ydb-platform/ydb-dotnet-sdk) -* Go [https://github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) (archived versions: [v1](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) and [v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2)) -* Java [https://github.com/yandex-cloud/ydb-java-sdk](https://github.com/yandex-cloud/ydb-java-sdk) -* Node.js [https://github.com/yandex-cloud/ydb-nodejs-sdk](https://github.com/yandex-cloud/ydb-nodejs-sdk) -* PHP [https://github.com/yandex-cloud/ydb-php-sdk](https://github.com/yandex-cloud/ydb-php-sdk) -* Python [https://github.com/yandex-cloud/ydb-python-sdk](https://github.com/yandex-cloud/ydb-python-sdk) +- C# (.NET) [https://github.com/ydb-platform/ydb-dotnet-sdk](https://github.com/ydb-platform/ydb-dotnet-sdk) +- Go [https://github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) (archived versions: [v1](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) and [v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2)) +- Java [https://github.com/yandex-cloud/ydb-java-sdk](https://github.com/yandex-cloud/ydb-java-sdk) +- Node.js [https://github.com/yandex-cloud/ydb-nodejs-sdk](https://github.com/yandex-cloud/ydb-nodejs-sdk) +- PHP [https://github.com/yandex-cloud/ydb-php-sdk](https://github.com/yandex-cloud/ydb-php-sdk) +- Python [https://github.com/yandex-cloud/ydb-python-sdk](https://github.com/yandex-cloud/ydb-python-sdk) The SDK documentation contains the following sections: -* [Installation](../install.md) -* [Authentication](../auth.md) -* [Test app](../example/index.md) -* [Code recipes](../recipes/index.md) +- [Installation](../install.md) +- [Authentication](../auth.md) +- [Test app](../example/index.md) +- [Code recipes](../recipes/index.md) + diff --git a/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md b/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md index 6368e1d32b..ec44d5c9b9 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md +++ b/ydb/docs/en/core/reference/ydb-sdk/_includes/install.md @@ -35,3 +35,4 @@ The build process using the source code is described in the source code reposito {% include [install/cmd_php.md](install/cmd_php.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md index 225e85e098..22977978e8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/addition.md @@ -3,3 +3,4 @@ The article is being updated. {% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md index bc1ad1a3a7..6760da6cf3 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/auxilary/pragmatablepathprefix.md @@ -6,3 +6,4 @@ SELECT * FROM episodes; ``` For more information about PRAGMA YQL, see the [YQL documentation](../../../../../yql/reference/index.md). + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md index 76fe582b48..a02c2d6082 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/example-dotnet.md @@ -1,6 +1,6 @@ # App in C# (.NET) -This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb-dotnet-examples), that is available as part of the [C# (.NET) SDK](https://github.com/ydb-platform/ydb-dotnet-sdk) {{ ydb-short-name }}. +This page contains a detailed description of the code of a [test app](https://github.com/ydb-platform/ydb-dotnet-examples), that uses the [C# (.NET) SDK](https://github.com/ydb-platform/ydb-dotnet-sdk) {{ ydb-short-name }}. {% include [addition.md](auxilary/addition.md) %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md index 1becb52a68..110c96faff 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/index.md @@ -4,13 +4,13 @@ This section describes the code of same-type test apps implemented using {{ ydb- {% if oss %} -* [C++](../example-cpp.md) +- [C++](../example-cpp.md) {% endif %} -* [C# (.NET)](../example-dotnet.md) -* [Go](../go/index.md) -* [Java](../example-java.md) -* [Node.js](../example-nodejs.md) -* [Python](../python/index.md) +- [C# (.NET)](../example-dotnet.md) +- [Go](../go/index.md) +- [Java](../example-java.md) +- [Node.js](../example-nodejs.md) +- [Python](../python/index.md) A test app performs the following steps: diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md index 97e4841525..b31f1357c8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/pars_from_profile_hint.md @@ -7,3 +7,4 @@ If you previously reviewed the articles of the "Getting started" section, you mu ``` {% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md index a4441fa42d..b49aab5618 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/01_init.md @@ -1,7 +1,8 @@ ## Initializing a database connection {#init} -To interact with {{ ydb-short-name }}, you have to create an instance of the driver, client, and session: +To interact with {{ ydb-short-name }}, create an instance of the driver, client, and session: * The {{ ydb-short-name }} driver lets the app and {{ ydb-short-name }} interact at the transport layer. The driver must exist throughout the {{ ydb-short-name }} access lifecycle and be initialized before creating a client or session. * The {{ ydb-short-name }} client runs on top of the {{ ydb-short-name }} driver and enables the handling of entities and transactions. * The {{ ydb-short-name }} session contains information about executed transactions and prepared queries, and is part of the {{ ydb-short-name }} client context. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md index b269efdf16..e8771af94d 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/02_create_table.md @@ -2,8 +2,9 @@ Creating tables to be used in operations on a test app. This step results in the creation of DB tables of the series directory data model: -* `Series` -* `Seasons` -* `Episodes` +- `Series` +- `Seasons` +- `Episodes` Once the tables are created, the method for getting information about data schema objects is called and the result of its execution is output. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md index fc04fcd5e8..4b2a1abeb5 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/07_param_prep_queries.md @@ -3,3 +3,4 @@ Parameterized prepared queries are saved as templates where specially formatted names are replaced by relevant parameter values each time you execute the query. Use parameterized queries to improve performance by reducing how often queries that only differ in parameter values are compiled and recompiled. The prepared query is stored in the session context. Code snippet for parameterized prepared queries: + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md index e32fd7b29a..4a2e6e219b 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/08_scan_query.md @@ -1,3 +1,4 @@ ## Scan queries {#scan-query} Making a [scan query](../../../../../concepts/scan_query.md) that results in a data stream. Streaming lets you read an unlimited number of rows and amount of data. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md index eb6c9370c3..41c92f22a4 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/_includes/steps/50_error_handling.md @@ -1,3 +1,4 @@ ## Handling errors {#error-handling} For more information about error handling, see [Error handling in the API](../../../error_handling.md). + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md index 67db33e50f..fb4181771b 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v1.md @@ -229,3 +229,4 @@ return nil ``` {% include [error_handling.md](../_includes/steps/50_error_handling.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md index 2a96fa2ff4..20dad36b27 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/archive/example-go-v2.md @@ -89,7 +89,7 @@ func describeTable(ctx context.Context, sp *table.SessionPool, path string) (err {% include [query_processing.md](../_includes/steps/04_query_processing.md) %} To execute YQL queries, use the `Session.Execute()` method. -The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the `TxControl` class. +The SDK lets you explicitly control the execution of transactions and configure the transaction execution mode using the ```TxControl``` class. ```go var ( @@ -211,3 +211,4 @@ Query results: ``` {% include [error_handling.md](../_includes/steps/50_error_handling.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md index 380059c389..9f7f6fba6e 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_custom.md @@ -9,12 +9,12 @@ Run the command as follows: go run ./basic -ydb="<endpoint>?database=<database>" ) ``` -Where: +, where -* `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint) -* `<database>` is the [DB location](../../../../../concepts/connect.md#database). -* `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. -* `<auth_mode_value>` is the authentication parameter value for the selected mode. +- `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint) +- `<database>` is the [DB location](../../../../../concepts/connect.md#database). +- `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. +- `<auth_mode_value>` is the authentication parameter value for the selected mode. For example: @@ -24,3 +24,4 @@ go run ./basic -ydb="grpcs://ydb.example.com:2135?database=/somepath/somelocatio ``` {% include [../../_includes/pars_from_profile_hint.md](../../_includes/pars_from_profile_hint.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md index 2d874b0c2d..db1ca73e9d 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_docker.md @@ -4,3 +4,4 @@ To connect to a locally deployed YDB database according to the [Docker](../../.. ( export YDB_ANONYMOUS_CREDENTIALS=1 && cd ydb-go-examples && \ go run ./basic -ydb="grpc://localhost:2136?database=/local" ) ``` + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md index 2a65ab4eca..dbb8af9b43 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/_includes/run_options.md @@ -9,3 +9,4 @@ {% include [run_custom.md](run_custom.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md index 9ab493f46a..24d1801c51 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/go/index.md @@ -51,7 +51,7 @@ db, err := ydb.New( ydb.WithConnectionString(dsn), // yc.WithInternalCA(), // using Yandex.Cloud certificates ydb.WithAccessTokenCredentials(token), // token-based authentication -// ydb.WithAnonimousCredentials(token), // anonymous authentication (for example, in docker ydb) +// ydb.WithAnonimousCredentials(token), // anonymous authentication (for example, in docker ydb) // yc.WithMetadataCredentials(token), // authentication from inside a VM in Yandex.Cloud or a function in Yandex Functions // yc.WithServiceAccountKeyFileCredentials("~/.ydb/sa.json"), // authentication in Yandex.Cloud using a service account file // environ.WithEnvironCredentials(ctx), // authentication using environment variables @@ -264,7 +264,8 @@ if err != nil { Sample code of a test app that uses archived of versions the Go SDK: -* [github.com/yandex-cloud/ydb-go-sdk](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) is available at this [link](../archive/example-go-v1.md), -* [github.com/yandex-cloud/ydb-go-sdk/v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2) is available at this [link](../archive/example-go-v2.md). +- [github.com/yandex-cloud/ydb-go-sdk](https://github.com/yandex-cloud/ydb-go-sdk/tree/v1.5.1) is available at this [link](../archive/example-go-v1.md), +- [github.com/yandex-cloud/ydb-go-sdk/v2](https://github.com/yandex-cloud/ydb-go-sdk/tree/v2.11.2) is available at this [link](../archive/example-go-v2.md). {% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md index df1f74b59c..4ba8a7a523 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_custom.md @@ -9,12 +9,12 @@ Run the command as follows: python3 ydb-python-sdk/examples/basic_example_v1/ -e <endpoint> -d <database> ``` -Where: +where -* `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint) -* `<database>` is the [DB location](../../../../../concepts/connect.md#database). -* `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. -* `<auth_mode_value>` is the authentication parameter value for the selected mode. +- `<endpoint>` is the [Endpoint](../../../../../concepts/connect.md#endpoint) +- `<database>` is the [DB location](../../../../../concepts/connect.md#database). +- `<auth_mode_var`> is the [Environment variable](../../../auth.md#env) that determines the authentication mode. +- `<auth_mode_value>` is the authentication parameter value for the selected mode. For example: @@ -24,3 +24,4 @@ python3 ydb-python-sdk/examples/basic_example_v1/ -e grpcs://ydb.example.com:213 ``` {% include [../../_includes/pars_from_profile_hint.md](../../_includes/pars_from_profile_hint.md) %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md index 1ae4ff757b..c855c9579c 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_docker.md @@ -4,3 +4,4 @@ To connect to a locally deployed YDB database according to the [Docker](../../.. YDB_ANONYMOUS_CREDENTIALS=1 \ python3 ydb-python-sdk/examples/basic_example_v1/ -e grpc://localhost:2136 -d /local ``` + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md index 2a65ab4eca..dbb8af9b43 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/_includes/run_options.md @@ -9,3 +9,4 @@ {% include [run_custom.md](run_custom.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md b/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md index a0d2f7135e..ab09fd5e37 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/example/python/index.md @@ -227,4 +227,5 @@ def explicit_tcl(session, path, series_id, season_id, episode_id): print("\n> explicit TCL call") tx.commit() -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md index 225e85e098..22977978e8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/addition.md @@ -3,3 +3,4 @@ The article is being updated. {% endnote %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md index a5f002ef52..43c59e5f93 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/_includes/wip.md @@ -1,6 +1,6 @@ {% note info %} -The feature is not implemented or the documentation is being developed +The feature is not implemented or the documentation is being developed. {% endnote %} diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md index c27f64e63f..e71825b7da 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/access_token.md @@ -11,3 +11,4 @@ Below are examples of the code for authentication using a token in different {{ {% include [go.md](access_token/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md index 463a17a206..d4b67f6595 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/anonymous.md @@ -2,7 +2,7 @@ {% include [work in progress message](../../_includes/addition.md) %} -Below are examples of the code for anonymous authentication in different {{ ydb-short-name }} SDKs. +Below are examples of the code for anonymous authentication in different {{ ydb-short-name }} SDKs {% list tabs %} @@ -11,3 +11,4 @@ Below are examples of the code for anonymous authentication in different {{ ydb- {% include [go.md](anonymous/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md index 4d20408aa2..e643f8857c 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/env.md @@ -20,3 +20,4 @@ Below are examples of the code for authentication using environment variables in {% include [go.md](env/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md index e78634eaf4..9de0e174fb 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/index.md @@ -2,7 +2,7 @@ {% include [work in progress message](../../_includes/addition.md) %} -{{ ydb-short-name }} supports multiple authentication methods when connecting to the server side. Each of them is usually specific to a particular environment pair: where the client application is located (in the trusted {{ ydb-short-name }} zone or outside it) and the {{ ydb-short-name }} server side (a Docker container, Yandex.Cloud, data cloud, or deployment on a separate cluster). +{{ ydb-short-name }} supports multiple authentication methods when connecting to the server side. Each of them is usually specific to a particular environment pair: where the client application is located (in the trusted {{ ydb-short-name }} zone or outside it) and the {{ ydb-short-name }} server side (a Docker container, {{ yandex-cloud }}, data cloud, or deployment on a separate cluster). This section contains code recipes with authentication settings in different {{ ydb-short-name }} SDKs. For a general description of the SDK authentication principles, see the [Authentication in an SDK](../../../auth.md) article. diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md index 7a38e58957..9241d3db47 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/metadata.md @@ -11,3 +11,4 @@ Below are examples of the code for authentication using environment variables in {% include [go.md](metadata/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md index 41876c2df8..1ffe2f6ab5 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/service_account.md @@ -11,3 +11,4 @@ Below are examples of the code for authentication using a service account file i {% include [go.md](service_account/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md index a0914b39fd..5d9a6a938d 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/auth/_includes/static.md @@ -7,3 +7,4 @@ Below are examples of the code for authentication based on a username and token {% list tabs %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md index 1c8c8b996e..be77aa2245 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/index.md @@ -2,7 +2,8 @@ {% include [work in progress message](../../_includes/addition.md) %} -{{ ydb-short-name }} uses client load balancing because it is more efficient when a lot of traffic from multiple client applications comes to a database. +{{ ydb-short-name }} uses client load balancing because it is more efficient when a lot of traffic from multiple client applications comes to a database. In most cases, it just works in the {{ ydb-short-name }} SDK. However, sometimes specific settings for client load balancing are required, for example, to reduce server hops and request time or to distribute the load across availability zones. -This section contains code recipes with client load balancing settings in different {{ ydb-short-name }} SDKs.
\ No newline at end of file +This section contains code recipes with client load balancing settings in different {{ ydb-short-name }} SDKs. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md index 7f0c23b1bc..9a50784323 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_local.md @@ -8,7 +8,7 @@ Below are examples of the code for setting the "prefer the nearest data center" - Go - {% include [go.md](prefer_local/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md index 706e357c63..b93efe00d8 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/prefer_location.md @@ -10,4 +10,5 @@ Below are examples of the code for setting the "prefer the availability zone" ba {% include [go.md](prefer_location/go.md) %} -{% endlist %}
\ No newline at end of file +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md index edd7e63554..a42b83814f 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/balancing/_includes/random_choice.md @@ -12,4 +12,5 @@ Below are examples of the code for forced setting of the "random choice" balanci {% include [go.md](random_choice/go.md) %} -{% endlist %}
\ No newline at end of file +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md index 335d3e1e5c..b59dd7a7e2 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/index.md @@ -5,3 +5,4 @@ When troubleshooting issues with {{ ydb-short-name }}, diagnostics tools such as logging, metrics, OpenTracing/Jaeger tracing are helpful. We strongly recommend that you enable them in advance before any problems occur. This will help see changes in the overall picture before, during, and after an issue when troubleshooting it. This greatly speeds up our investigation into incidents and lets us provide assistance much faster. This section contains code recipes for enabling diagnostics tools in different {{ ydb-short-name }} SDKs. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md index 82be14e6a6..45e762184e 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/jaeger.md @@ -10,4 +10,5 @@ Below are examples of the code enabling Jaeger tracing in different {{ ydb-short {% include [go.md](jaeger/go.md) %} -{% endlist %}
\ No newline at end of file +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md index b8227164a7..5703214250 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs.md @@ -11,3 +11,4 @@ Below are examples of code that enables logging in different {{ ydb-short-name } {% include [go.md](logs/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md index 23fbb07e64..4e78872988 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/logs/go.md @@ -1,8 +1,8 @@ There are several ways to enable logs in an application that uses `ydb-go-sdk`: * Set the environment variable `YDB_LOG_SEVERITY_LEVEL=info` (possible values: `trace`, `debug`, `info`, `warn`, `error`, `fatal`, and `quiet`, defaults to `quiet`). -This environment variable enables the built-in `ydb-go-sdk` logger (synchronous, non-block) with output to the standard output stream. -* {% cut "Connect a side logger `go.uber.org/zap`" %} + This environment variable enables the built-in `ydb-go-sdk` logger (synchronous, non-block) with output to the standard output stream. +* {% cut "Connect side logger `go.uber.org/zap`" %} ```go package main @@ -39,7 +39,7 @@ This environment variable enables the built-in `ydb-go-sdk` logger (synchronous, ``` {% endcut %} -* {% cut "Connect a side logger `github.com/rs/zerolog`" %} +* {% cut "Connect side logger `github.com/rs/zerolog`" %} ```go package main @@ -116,4 +116,5 @@ This environment variable enables the built-in `ydb-go-sdk` logger (synchronous, {% include [overlay](go_appendix.md) %} -* Implement your own logging package based on the `github.com/ydb-platform/ydb-go-sdk/v3/trace` tracing package.
\ No newline at end of file +* Implement your own logging package based on the `github.com/ydb-platform/ydb-go-sdk/v3/trace` tracing package. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md index d32a1bb00a..83f44762e0 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/debug/_includes/prometheus.md @@ -10,4 +10,5 @@ Below are examples of the code for enabling metrics in Prometheus in different { {% include [go.md](prometheus/go.md) %} -{% endlist %}
\ No newline at end of file +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md index 0b51dc067f..25f09db313 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/_includes/go.md @@ -1,16 +1,16 @@ In the {{ ydb-short-name }} Go SDK, correct error handling is implemented by several programming interfaces: -* The basic logic of error handling is implemented by the helper `retry.Retry` function - The details of the execution of repeated requests are hidden as much as possible. - The user can influence the logic of the function `retry.Retry` using two methods: +* The basic logic of error handling is implemented by the helper `retry.Retry` function. + The details of making request retries are hidden as much as possible. + The user can affect the logic of executing the `retry.Retry` function in two ways: * Via the context (where you can set the deadline and cancel) * Via the operation's idempotency flag `retry.WithIdempotent()`. By default, the operation is considered non-idempotent. The user passes a custom function to `retry.Retry` that returns an error by its signature. - If the custom function returns `nil`, then repeat queries stop. - If the custom function returns an error, the {{ ydb-short-name }} Go SDK tries to identify this error and executes retries depending on it. +If the custom function returns `nil`, then repeat queries stop. +If the custom function returns an error, the {{ ydb-short-name }} Go SDK tries to identify this error and executes retries depending on it. - {% cut "Example code using the function `retry.Retry`:" %} + {% cut "Example of code, using `retry.Retry` function:" %} ```go package main @@ -63,3 +63,4 @@ In the {{ ydb-short-name }} Go SDK, correct error handling is implemented by sev As in the previous case, the user can affect the logic of repeat queries using the context and the idempotence flag, while the {{ ydb-short-name }} Go SDK interprets errors returned by `op`. * Queries to other {{ ydb-short-name }} services (`db.Scripting()`, `db.Scheme()`, `db.Coordination()`, `db.Ratelimiter()`, and `db.Discovery()`) also use the `retry.Retry` function internally to make repeat queries. + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md index 926c59bef1..120aa1922c 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/retry/index.md @@ -17,3 +17,4 @@ Below are code examples showing the {{ ydb-short-name }} SDK built-in tools for {% include [go.md](_includes/go.md) %} {% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md index b9710de145..f6796047aa 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md +++ b/ydb/docs/en/core/reference/ydb-sdk/recipes/session_pool_limit/index.md @@ -16,4 +16,5 @@ Below are examples of the code for setting the session pool limit in different { {% include [go.md](_includes/go.md) %} -{% endlist %}
\ No newline at end of file +{% endlist %} + diff --git a/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml b/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml index b81d8bfccf..7a2fdca571 100644 --- a/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-sdk/toc_i.yaml @@ -11,3 +11,4 @@ items: href: error_handling.md - name: Code recipes include: { mode: link, path: recipes/toc_p.yaml } + diff --git a/ydb/docs/en/core/toc_i.yaml b/ydb/docs/en/core/toc_i.yaml index 22c194faa5..4e8d631d08 100644 --- a/ydb/docs/en/core/toc_i.yaml +++ b/ydb/docs/en/core/toc_i.yaml @@ -6,14 +6,17 @@ items: include: { mode: link, path: getting_started/toc_p.yaml } # Main -# - { name: Solutions, include: { mode: link, path: solutions/toc_i.yaml }, when: audience == "external" } +# - { name: Practical guidelines, include: { mode: link, path: solutions/toc_i.yaml }, when: audience == "external" } - { name: Concepts, include: { mode: link, path: concepts/toc_p.yaml } } - { name: Step-by-step instructions, include: { mode: link, path: operations/toc_p.yaml } } - { name: Recommendations, include: { mode: link, path: best_practices/toc_p.yaml } } -- { name: Maintenance, include: { mode: link, path: maintenance/toc_p.yaml } } -- { name: Diagnostics, include: { mode: link, path: troubleshooting/toc_p.yaml } } -- { name: Admin, include: { mode: link, path: deploy/toc_p.yaml } } +# - { name: Diagnostics, include: { mode: link, path: troubleshooting/toc_p.yaml } } moved into maintenance! +# - { name: Deploy, include: { mode: link, path: deploy/toc_p.yaml } } +# - { name: Maintenance, include: { mode: link, path: maintenance/toc_p.yaml } } + +- { name: Managing databases, include: { mode: link, path: db/toc_p.yaml } } +- { name: Managing a cluster, include: { mode: link, path: cluster/toc_p.yaml } } # References - { name: YQL, include: { mode: link, path: yql/toc_p.yaml } } diff --git a/ydb/docs/en/core/troubleshooting/_includes/index.md b/ydb/docs/en/core/troubleshooting/_includes/index.md new file mode 100644 index 0000000000..14dca3bb23 --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/_includes/index.md @@ -0,0 +1,7 @@ +# Diagnostics + +This section contains articles on YDB database diagnostics tools. + +- [System tables](../system_views_db.md) +- [Metrics](../monitoring.md) + diff --git a/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md b/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md index d1802e5721..673c56a318 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md +++ b/ydb/docs/en/core/troubleshooting/_includes/monitoring_sensors.md @@ -4,7 +4,7 @@ | Metric name<br/>Type, units of measurement | Description<br/>Labels | | ----- | ----- | -| `resources.storage.used_bytes`<br/>`IGAUGE`, bytes | The size of user and service data stored in distributed network storage. The service data includes the data of the primary and [secondary indexes](../../concepts/secondary_indexes.md). | +| `resources.storage.used_bytes`<br/>`IGAUGE`, bytes | The size of user and service data stored in distributed network storage. Housekeeping data include the data of the primary and [secondary indexes](../../concepts/secondary_indexes.md). | | `resources.storage.limit_bytes`<br/>`IGAUGE`, bytes | A limit on the size of user and service data that a database can store in distributed network storage. | ### API metrics {#api} @@ -16,9 +16,9 @@ | `api.grpc.request.inflight_count`<br/>`IGAUGE`, pieces | The number of requests that a database is simultaneously handling in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | | `api.grpc.request.inflight_bytes`<br/>`IGAUGE`, bytes | The size of requests that a database is simultaneously handling in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | | `api.grpc.response.bytes`<br/>`RATE`, bytes | The size of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | -| `api.grpc.response.count`<br/>`RATE`, pieces | The number of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`.<br/>- _status_: The query execution status. To learn more about statuses, see [Error handling](../../reference/ydb-sdk/error_handling.md). | +| `api.grpc.response.count`<br/>`RATE`, pieces | The number of responses sent by the database in a certain period of time.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`.<br/>- _status_ is the request execution status. See a more detailed description of statuses under [Error Handling](../../reference/ydb-sdk/error_handling.md). | | `api.grpc.response.dropped_count`<br/>`RATE`, pieces | The number of responses dropped at the transport (gRPC) layer due to an error.<br/>Labels:<br/>- _api_service_: The name of the gRPC API service, such as `table`.<br/>- _method_: The name of a gRPC API service method, such as `ExecuteDataQuery`. | -| `api.grpc.response.issues`<br/>`RATE`, pieces | The number of errors of a certain type, which occurred when executing queries during a certain period of time.<br/>Labels:<br/>- _issue_type_: The type of error, the only value is `optimistic_locks_invalidation`. For more information about locks invalidation, see [Transactions and queries to {{ ydb-short-name }}](../../concepts/transactions.md). | +| `api.grpc.response.issues`<br/>`RATE`, pieces | The number of errors of a certain type arising in the execution of a request over a certain period of time.<br/>Tags:<br/>- _issue_type_ is the error type wth the only value being `optimistic_locks_invalidation`. For more on lock invalidation, review [Transactions and requests to {{ ydb-short-name }}](../../concepts/transactions.md). | ### Session metrics {#sessions} @@ -29,12 +29,12 @@ ### Transaction processing metrics {#transactions} -You can analyze a transaction's execution time using a histogram counter. The intervals are set in milliseconds. The chart shows the number of transactions whose duration falls within a certain time interval. +You can analyze a transaction's execution time using a histogram counter. The intervals are set in milliseconds. The chart shows the number of transactions whose duration falls within a certain time interval. | Metric name<br/>Type, units of measurement | Description<br/>Labels | | ----- | ----- | | `table.transaction.total_duration_milliseconds`<br/>`HIST_RATE`, pieces | The number of transactions with a certain duration on the server and client. The duration of a transaction is counted from the point of its explicit or implicit start to committing changes or its rollback. Includes the transaction processing time on the server and the time on the client between sending different requests within the same transaction.<br/>Labels:<br/>- _tx_kind_: The transaction type, possible values are `read_only`, `read_write`, `write_only`, and `pure`. | -| `table.transaction.server_duration_milliseconds`<br/>`HIST_RATE`, pieces | The number of transactions with a certain duration on the server. The duration is the time of executing requests within a transaction on the server. Does not include the waiting time on the client between sending separate requests within a single transaction.<br/>Labels:<br/> -_tx_kind_: The transaction type, possible values are`read_only`, `read_write`, `write_only`, and `pure`. | +| `table.transaction.server_duration_milliseconds`<br/>`HIST_RATE`, pieces | The number of transactions with a certain duration on the server. The duration is the time of executing requests within a transaction on the server. Does not include the waiting time on the client between sending separate requests within a single transaction.<br/>Labels:<br/> -_tx_kind_: The transaction type, possible values are`read_only`, `read_write`, `write_only`, and `pure`. | | `table.transaction.client_duration_milliseconds`<br/>`HIST_RATE`, pieces | The number of transactions with a certain duration on the client. The duration is the waiting time on the client between sending individual requests within a single transaction. Does not include the time of executing requests on the server.<br/>Labels:<br/>- _tx_kind_: The transaction type, possible values are `read_only`, `read_write`, `write_only`, and `pure`. | ### Query processing metrics {#queries} @@ -86,3 +86,4 @@ You can analyze a transaction's execution time using a histogram counter. The in | `table.query.compilation.cache_evictions`<br/>`RATE`, pieces | The number of queries evicted from the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries) in a certain period of time. | | `table.query.compilation.cache_size_bytes`<br/>`IGAUGE`, bytes | The size of the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries). | | `table.query.compilation.cached_query_count`<br/>`IGAUGE`, pieces | The size of the cache of [prepared queries](../oss/public/reference/ydb-sdk/#param-prepared-queries). | + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md index 1eec417000..d531f9364c 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/distributed_storage.md @@ -100,3 +100,4 @@ Unlike other tables that show physical entities, the `ds_storage_stats` table sh | AvailableSizeToCreate | Uint64 | | Number of available bytes that will be obtained when creating all groups from AvailableGroupsToCreate. | It should be noted that AvailableGroupsToCreate shows the maximum number of groups that can be created if no other types of groups are created. So when extending a storage pool, the count of AvailableGroupsToCreate in several rows of statistics may change. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_cluster.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_cluster.md new file mode 100644 index 0000000000..e903ac40f7 --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_cluster.md @@ -0,0 +1,6 @@ +# Cluster system tables + +To enable internal introspection of the cluster state, the user can make queries to special service tables (system views). These tables are accessible from the cluster's root directory and use the `.sys` system path prefix. + +Hereinafter, in the descriptions of available fields, the **Key** column contains the corresponding table's primary key field index. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_db.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_db.md new file mode 100644 index 0000000000..18e8a4988d --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/intro_db.md @@ -0,0 +1,6 @@ +# Database system tables + +To enable internal introspection of the DB state, the user can make queries to special service tables (system views). These tables are accessible from the root of the database tree and use the `.sys` system path prefix. + +Hereinafter, in the descriptions of available fields, the **Key** column contains the corresponding table's primary key field index. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md index c36f003299..df3c85f771 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions.md @@ -3,3 +3,4 @@ Examples: {% include [example_yql](partitions_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md index 02891f4711..490f7ec11e 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_example_yql.md @@ -1,15 +1,17 @@ - Top 5 of most loaded partitions among all DB tables +Top 5 of most loaded partitions among all DB tables + ```sql SELECT Path, PartIdx, CPUCores - FROM `/cluster/path/to/database/.sys/partition_stats` + FROM `.sys/partition_stats` ORDER BY CPUCores DESC LIMIT 5 ``` List of DB tables with in-flight sizes and loads + ```sql SELECT Path, @@ -17,6 +19,7 @@ SUM(RowCount) as Rows, SUM(DataSize) as Size, SUM(CPUCores) as CPU - FROM `/cluster/path/to/database/.sys/partition_stats` + FROM `.sys/partition_stats` GROUP BY Path ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md index 4edb8c845c..6018aaac33 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/partitions_header.md @@ -8,8 +8,8 @@ Table structure: | **Field** | **Type** | **Key** | **Value** | | --- | --- | --- | --- | -| OwnerId | Uint64 | 0 | ID of the datashard serving the table | -| PathId | Uint64 | 1 | Path ID in the datashard | +| OwnerId | Uint64 | 0 | ID of the SchemeShard serving the table | +| PathId | Uint64 | 1 | Path ID in the SchemeShard | | PartIdx | Uint64 | 2 | Partition sequence number | | DataSize | Uint64 | | Approximate partition size in bytes | | RowCount | Uint64 | | Approximate number of rows | @@ -26,7 +26,7 @@ Table structure: | RowDeletes | Uint64 | | Number of rows deleted since the start | | RangeReads | Uint64 | | Number of row ranges read since the start | | RangeReadRows | Uint64 | | Number of rows read in the ranges since the start | -| InFlightTxCount | Uint64 | | Number of in-flight transactions on the partition | +| InFlightTxCount | Uint64 | | Number of in-flight transactions | | ImmediateTxCompleted | Uint64 | | Number of one-shard transactions completed since the start | | CoordinatedTxCompleted | Uint64 | | Number of coordinated transactions completed since the start | | TxRejectedByOverload | Uint64 | | Number of transactions rejected due to overload (since the start) | @@ -35,3 +35,4 @@ Table structure: Restrictions: * Cumulative fields (RowReads, RowUpdates, and so on) store the accumulated values since the last start of the tablet serving the partition + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md index 8548a2604c..e2fc18f715 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics.md @@ -3,3 +3,4 @@ Examples: {% include [example_yql](query_metrics_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md index 649def5eeb..c37a4fbb4b 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_example_yql.md @@ -1,4 +1,4 @@ - Top 10 queries for the last 6 hours by the total number of rows updated per minute +Top 10 queries for the last 6 hours by the total number of rows updated per minute ```sql SELECT @@ -6,7 +6,7 @@ Count, QueryText, IntervalEnd - FROM `/cluster/path/to/database/.sys/query_metrics_one_minute` + FROM `.sys/query_metrics_one_minute` ORDER BY SumUpdateRows DESC LIMIT 10 ``` @@ -20,8 +20,9 @@ SumReadBytes / Count as AvgReadBytes, MaxReadBytes, QueryText - FROM `/cluster/path/to/database/.sys/query_metrics_one_minute` + FROM `.sys/query_metrics_one_minute` WHERE SumReadBytes > 0 ORDER BY IntervalEnd DESC, SumReadBytes DESC LIMIT 100 ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md index b74f360d20..3734ec7cd6 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/query_metrics_header.md @@ -8,7 +8,7 @@ Table structure: | **Field** | **Type** | **Key** | **Value** | | --- | --- | --- | --- | -| IntervalEnd | Timestamp | 0 | Closing time of a minute or hour interval | +| IntervalEnd | Timestamp | 0 | Closing time of a minute interval | | Rank | Uint32 | 1 | Query rank per interval (by the SumCPUTime field) | | QueryText | Utf8 | | Query text | | Count | Uint64 | | Number of query runs | @@ -33,6 +33,9 @@ Table structure: | SumDeleteRows | Uint64 | | Total number of rows deleted | | MinDeleteRows | Uint64 | | Minimum number of rows deleted | | MaxDeleteRows | Uint64 | | Maximum number of rows deleted | +| SumRequestUnits | Uint64 | | Total number of [RequestUnits](../../../concepts/serverless_and_dedicated.md#serverless-options) used | +| MinRequestUnits | Uint64 | | Minimum number of [RequestUnits](../../../concepts/serverless_and_dedicated.md#serverless-options) used | +| MaxRequestUnits | Uint64 | | Maximum number of [RequestUnits](../../../concepts/serverless_and_dedicated.md#serverless-options) used | Restrictions: @@ -40,3 +43,4 @@ Restrictions: * The table contains the history for the last 6 hours. * Within the interval, information is provided for no more than 256 different queries. * Statistics may be incomplete if the database is under heavy load. + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md index 7c1cb50003..acb4488ba7 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops.md @@ -3,3 +3,4 @@ Examples: {% include [example_yql.md](tops_example_yql.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md index 796431ee9c..e21ca67b6f 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_example_yql.md @@ -1,17 +1,18 @@ - Top queries by execution time for the last minute when queries were made +Top queries by execution time for the last minute when queries were made ```sql + PRAGMA AnsiInForEmptyOrNullableItemsCollections; $last = ( SELECT MAX(IntervalEnd) - FROM `/cluster/path/to/database/.sys/top_queries_by_duration_one_minute` + FROM `.sys/top_queries_by_duration_one_minute` ); SELECT IntervalEnd, Rank, QueryText, Duration - FROM `/cluster/path/to/database/.sys/top_queries_by_duration_one_minute` + FROM `.sys/top_queries_by_duration_one_minute` WHERE IntervalEnd IN $last ``` @@ -24,6 +25,7 @@ ReadBytes, ReadRows, Partitions - FROM `/cluster/path/to/database/.sys/top_queries_by_read_bytes_one_minute` + FROM `.sys/top_queries_by_read_bytes_one_minute` WHERE Rank = 1 ``` + diff --git a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md index cdff4f948d..61de9b36ba 100644 --- a/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md +++ b/ydb/docs/en/core/troubleshooting/_includes/system_views/tops_header.md @@ -19,6 +19,7 @@ Table structure: | --- | --- | --- | --- | | IntervalEnd | Timestamp | 0 | Closing time of a minute or hour interval | | Rank | Uint32 | 1 | Rank of a top query | +| RequestUnits | Uint64 | | Number of [RequestUnits](../../../concepts/serverless_and_dedicated.md#serverless-options) used | | QueryText | Utf8 | | Query text | | Duration | Interval | | Total time of query execution | | EndTime | Timestamp | | Query execution end time | @@ -28,6 +29,7 @@ Table structure: | UpdateRows | Uint64 | | Number of rows updated | | UpdateBytes | Uint64 | | Number of bytes updated | | DeleteRows | Uint64 | | Number of rows deleted | +| DeleteBytes | Uint64 | | Number of bytes deleted | | Partitions | Uint64 | | Number of table partitions used during query execution | | UserSID | String | | User security ID | | ParametersSize | Uint64 | | Size of query parameters in bytes | @@ -50,3 +52,4 @@ Restrictions: * Query text limit is 4 KB. * Tables with minute intervals contain the history for the last 6 hours. * Tables with hourly intervals contain the history for the last 2 weeks. + diff --git a/ydb/docs/en/core/troubleshooting/index.md b/ydb/docs/en/core/troubleshooting/index.md new file mode 100644 index 0000000000..eb2590567d --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/index.md @@ -0,0 +1 @@ +{% include [_includes/index.md](_includes/index.md) %}
\ No newline at end of file diff --git a/ydb/docs/en/core/troubleshooting/monitoring.md b/ydb/docs/en/core/troubleshooting/monitoring.md index a181d1d03e..598b0fed12 100644 --- a/ydb/docs/en/core/troubleshooting/monitoring.md +++ b/ydb/docs/en/core/troubleshooting/monitoring.md @@ -1,7 +1,7 @@ --- editable: false --- - # Metric reference {% include notitle [ydb_monitoring_sensors.md](_includes/monitoring_sensors.md) %} + diff --git a/ydb/docs/en/core/troubleshooting/system_views_cluster.md b/ydb/docs/en/core/troubleshooting/system_views_cluster.md new file mode 100644 index 0000000000..697876861c --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/system_views_cluster.md @@ -0,0 +1,5 @@ +{% include [intro.md](_includes/system_views/intro_cluster.md) %} + +{% include [distributed_storage.md](_includes/system_views/distributed_storage.md) %} + +{% include [notes.md](_includes/system_views/notes.md) %} diff --git a/ydb/docs/en/core/troubleshooting/system_views_db.md b/ydb/docs/en/core/troubleshooting/system_views_db.md new file mode 100644 index 0000000000..68735978be --- /dev/null +++ b/ydb/docs/en/core/troubleshooting/system_views_db.md @@ -0,0 +1,9 @@ +{% include [intro.md](_includes/system_views/intro_db.md) %} + +{% include [partitions.md](_includes/system_views/partitions.md) %} + +{% include [tops.md](_includes/system_views/tops.md) %} + +{% include [query_metrics.md](_includes/system_views/query_metrics.md) %} + +{% include [notes.md](_includes/system_views/notes.md) %} diff --git a/ydb/docs/en/core/troubleshooting/toc_i.yaml b/ydb/docs/en/core/troubleshooting/toc_i.yaml index 67dc11dcc7..d01b7fbb1a 100644 --- a/ydb/docs/en/core/troubleshooting/toc_i.yaml +++ b/ydb/docs/en/core/troubleshooting/toc_i.yaml @@ -1,5 +1,5 @@ items: - name: System views - href: system_views.md + href: system_views_db.md - name: Monitoring href: monitoring.md
\ No newline at end of file diff --git a/ydb/docs/en/core/troubleshooting/toc_p.yaml b/ydb/docs/en/core/troubleshooting/toc_p.yaml index 5bfec4365d..94ce110868 100644 --- a/ydb/docs/en/core/troubleshooting/toc_p.yaml +++ b/ydb/docs/en/core/troubleshooting/toc_p.yaml @@ -1,2 +1,4 @@ items: +- name: Overview + href: index.md - include: { mode: link, path: toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/yql/reference/_includes/index/intro.md b/ydb/docs/en/core/yql/reference/_includes/index/intro.md index 72fef596cf..a1cd5dcef1 100644 --- a/ydb/docs/en/core/yql/reference/_includes/index/intro.md +++ b/ydb/docs/en/core/yql/reference/_includes/index/intro.md @@ -1,4 +1,4 @@ -# YQL overview +# YQL - Overview *YQL* (YDB Query Language) is a universal declarative query language for YDB, a dialect of SQL. YQL has been natively designed for large distributed databases, and therefore has a number of differences from the SQL standard. @@ -6,16 +6,16 @@ YDB tools support interfaces for sending YQL queries and receiving their executi {% include [yql/ui_prompt.md](yql/ui_prompt.md) %} -* [YDB CLI](../../../../reference/ydb-cli/index.md). -* [YDB SDK](../../../../reference/ydb-sdk/index.md). +- [YDB CLI](../../../../reference/ydb-cli/index.md) +- [YDB SDK](../../../../reference/ydb-sdk/index.md) This documentation section contains the YQL reference that includes the sections: -* [Data types](../../types/index.md) with a description of data types used in YQL. -* [Syntax](../../syntax/index.md) with a full list of YQL commands. -* [Built-in functions](../../builtins/index.md) with a description of the available built-in functions. +- [Data types](../../types/index.md) with a description of data types used in YQL +- [Syntax](../../syntax/index.md) with a full list of YQL commands +- [Built-in functions](../../builtins/index.md) with a description of the available built-in functions You can also take a tutorial to get familiar with the basic YQL commands, in the section -* [YQL tutorial](../../../tutorial/index.md) +- [YQL tutorial](../../../tutorial/index.md) diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md index 688e7712eb..467901871f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/_includes/cast_examples.md @@ -6,9 +6,9 @@ SELECT CAST("xyz" AS Uint64) IS NULL, -- true, because it failed CAST(-1 AS Uint16) IS NULL, -- true, a negative integer cast to an unsigned integer CAST([-1, 0, 1] AS List<Uint8?>), -- [null, 0, 1] - --The element type is optional: the failed element is cast to null. + --The item type is optional: the failed item is cast to null. CAST(["3.14", "bad", "42"] AS List<Float>), -- [3.14, 42] - --The element type is not optional: the failed element has been deleted. + --The item type is not optional: the failed item has been deleted. CAST(255 AS Uint8), -- 255 CAST(256 AS Uint8) IS NULL -- true, out of range diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md index 0f66ba39a2..74d3ef437c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/agg_list.md @@ -10,7 +10,7 @@ To return a list of multiple values from one line, *DO NOT* use the `AGGREGATE_L For example, you can combine it with `DISTINCT` and the function [String::JoinFromList](../../../udf/list/string.md) (it's an equivalent of `','.join(list)` in Python) to output to a string all the values found in the column after [GROUP BY](../../../syntax/group_by.md). -**Examples:** +**Examples** ```yql SELECT @@ -33,7 +33,7 @@ These functions also have a short notation: `AGG_LIST` and `AGG_LIST_DISTINCT`. {% note alert %} -Execution is **NOT** lazy, so when you use it, be sure that the list has a reasonable size (about a thousand elements or less). To stay on the safe side, better use a second optional numeric argument that limits the number of items in the list. +Execution is **NOT** lazy, so when you use it, be sure that the list has a reasonable size (about a thousand items or less). To stay on the safe side, better use a second optional numeric argument that limits the number of items in the list. {% endnote %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md index a3920fd582..4b3dea633c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/histogram.md @@ -70,7 +70,7 @@ While FastGreedyShrink is used most of the time, SlowShrink is mostly used for h When you use [aggregation factories](../../basic.md#aggregationfactory), a `Tuple` containing a value and a weight is passed as the first [AGGREGATE_BY](#aggregateby) argument. -**Examples:** +**Examples** ```yql SELECT @@ -95,7 +95,7 @@ SELECT FROM my_table; ``` -## LinearHistogram, LogarithmicHistogram and LogHistogram {#linearhistogram} +## LinearHistogram, LogarithmicHistogram, and LogHistogram {#linearhistogram} Plotting a histogram based on an explicitly specified fixed bucket scale. @@ -110,7 +110,7 @@ The format of the result is totally similar to [adaptive histograms](#histogram) If the spread of input values is uncontrollably large, we recommend that you specify the minimum and maximum values to prevent potential failures due to high memory consumption. -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/max_min_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/max_min_by.md index 94e7dbaf96..9fe2d008e9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/max_min_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/max_min_by.md @@ -11,7 +11,7 @@ When choosing N, we recommend that you don't exceed several hundreds or thousand If your task needs absolutely all values, and their number is measured in dozens of thousands or more, then instead of those aggregate functions better use `JOIN` on the source table with a subquery doing `GROUP BY + MIN/MAX` on the desired columns of this table. -{% note warning "Attention" %} +{% note warning %} If the second argument is always NULL, the aggregation result is NULL. @@ -19,7 +19,7 @@ If the second argument is always NULL, the aggregation result is NULL. When you use [aggregation factories](../../basic.md#aggregationfactory), a `Tuple` containing a value and a key is passed as the first [AGGREGATE_BY](#aggregateby) argument. -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md index 5bac2b8f60..77658864e6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/percentile_median.md @@ -2,7 +2,7 @@ Calculating percentiles using the amortized version of the [TDigest](https://github.com/tdunning/t-digest) algorithm. `MEDIAN`: An alias for `PERCENTILE(N, 0.5)`. -{% note info "Restriction" %} +{% note info %} The first argument (N) must be a table column name. If you need to bypass this restriction, use a subquery. The restriction is introduced to simplify calculations, since the implementation merges the calls with the same first argument (N) into a single pass. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md index 9b036f94b3..8a80dc8ac7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/aggregation/top_bottom.md @@ -1,8 +1,8 @@ ## TOP and BOTTOM {#top-bottom} -Return a list of the maximum/minimum values of an expression. The first argument is an expression, the second argument limits the number of elements. +Return a list of the maximum/minimum values of an expression. The first argument is an expression, the second argument limits the number of items. -**Examples:** +**Examples** ```yql SELECT @@ -27,7 +27,7 @@ Return a list of values of the first argument for the rows containing the maximu When you use [aggregation factories](../../basic.md#aggregationfactory), a `Tuple` containing a value and a key is passed as the first [AGGREGATE_BY](#aggregateby) argument. In this case, the limit for the number of items is passed by the second argument at factory creation. -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md index fca0f76ecd..1c563cba95 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/abs.md @@ -7,3 +7,4 @@ The absolute value of the number. ```yql SELECT Abs(-123); -- 123 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md index 7c93fbcf1a..b6721b8043 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/aggr_factory.md @@ -7,7 +7,8 @@ Arguments: 1. A string in double quotes with the name of an aggregate function, for example ["MIN"](../../aggregation.md#min). 2. Optional parameters of the aggregate function that are data-independent. For example, the percentile value in [PERCENTILE](../../aggregation.md#percentile). -The resulting factory can be used as the second parameter of the function [AGGREGATE_BY](../../aggregation.md#aggregateby). +The resulting factory can be used as the second parameter of the function [AGGREGATE_BY](.. +/../aggregation.md#aggregateby). If the aggregate function is applied to two columns instead of one, as, for example, [MIN_BY](../../aggregation.md#minby), then in [AGGREGATE_BY](../../aggregation.md#aggregateby), the first argument passes a `Tuple` of two values. See more details in the description of the applicable aggregate function. **Examples:** @@ -21,12 +22,12 @@ FROM my_table; ## AggregateTransform... {#aggregatetransform} -`AggregateTransformInput()` converts an [aggregation factory](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function, to other factory, in which the specified transformation of input elements is performed before starting aggregation. +`AggregateTransformInput()` converts an [aggregation factory](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function, to other factory, in which the specified transformation of input items is performed before starting aggregation. Arguments: 1. Aggregation factory. -2. A lambda function with one argument that converts an input element. +2. A lambda function with one argument that converts an input item. **Examples:** @@ -57,7 +58,7 @@ select ListAggregate([1,2,3], $g); -- 12 ## AggregateFlatten {#aggregateflatten} -Adapts a factory for [aggregation functions](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function in a way that allows aggregation of list input elements. This operation is similar to [FLATTEN LIST BY](../../../syntax/flatten.md): Each list element is aggregated. +Adapts a factory for [aggregation functions](../../aggregation.md), for example, obtained using the [AggregationFactory](#aggregationfactory) function in a way that allows aggregation of list input items. This operation is similar to [FLATTEN LIST BY](../../../syntax/flatten.md): Each list item is aggregated. Arguments: diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md index e1316a5f48..15adedeeda 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_container.md @@ -8,7 +8,7 @@ Specifics: * `AsTuple` and `AsStruct` can be called without arguments, and also the arguments can have different types. * The field names in `AsStruct` are set using `AsStruct(field_value AS field_name)`. * Creating a list requires at least one argument if you need to output the element types. To create an empty list with the given type of elements, use the function [ListCreate](../../list.md#listcreate). You can create an empty list as an `AsList()` call without arguments. In this case, this expression will have the `EmptyList` type. -* Creating a dictionary requires at least one argument if you need to output element types. To create an empty dictionary with the given type of elements, use the function [DictCreate](../../dict.md#dictcreate). You can create an empty dictionary as an `AsDict()` call without arguments, in this case, this expression will have the `EmptyDict` type. +* Creating a dictionary requires at least one argument if you need to output the element types. To create an empty dictionary with the given type of elements, use the function [DictCreate](../../dict.md#dictcreate). You can create an empty dictionary as an `AsDict()` call without arguments, in this case, this expression will have the `EmptyDict` type. * Creating a set requires at least one argument if you need to output element types. To create an empty set with the given type of elements, use the function [SetCreate](../../dict.md#setcreate). You can create an empty set as an `AsSet()` call without arguments, in this case, this expression will have the `EmptySet` type. * `AsList` outputs the common type of elements in the list. A type error is raised in the case of incompatible types. * `AsDict` separately outputs the common types for keys and values. A type error is raised in the case of incompatible types. @@ -17,7 +17,7 @@ Specifics: * `AsDict` and `AsDictStrict` expect `Tuple` of two elements as arguments (key and value, respectively). If the keys repeat, only the value for the first key remains in the dictionary. * `AsSet` and `AsSetStrict` expect keys as arguments. -**Examples:** +**Examples** ```yql SELECT @@ -35,3 +35,4 @@ SELECT ) AS `dict`, AsSet(1, 2, 3) AS `set` ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md index 981395a06c..326cdb9812 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/as_tagged.md @@ -1,6 +1,6 @@ ## AsTagged, Untag {#as-tagged} -Wraps the value in the [Tagged data type](../../../types/special.md) with the specified tag, preserving the physical data type. `Untag`: the reverse operation. +Wraps the value in the [Tagged data type](../../../types/special.md) with the specified tag, preserving the physical data type. `Untag`: The reverse operation. Required arguments: @@ -14,3 +14,4 @@ Examples of use cases: * Returns to the client's web interface the media files from BASE64-encoded strings{% if feature_webui %}. Tag support in the YQL Web UI [is described here](../../../interfaces/web_tagged.md){% endif %}. {% if feature_mapreduce %}* Prevent passing of invalid values at the boundaries of UDF calls.{% endif %} * Additional refinements at the level of returned columns types. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md index 92b33b2049..2ecf0818d4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/bitops.md @@ -16,3 +16,4 @@ SELECT TestBit(1u, 0), -- true SetBit(8u, 0); -- 9 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md index fc3b0d802d..ef9c9d0bde 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/byteat.md @@ -7,7 +7,7 @@ Arguments: 1. String: `String` or `Utf8`. 2. Index: `Uint32`. -**Examples:** +**Examples** ```yql SELECT @@ -15,3 +15,4 @@ SELECT ByteAt("foo", 1), -- 111 ByteAt("foo", 9); -- NULL ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md index 6fcec7de1e..30cd21dffb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/callable.md @@ -21,3 +21,4 @@ $callables = AsTuple( SELECT $callables.0(10), $callables.1(true); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md index 6ea6cf1e5a..09fe299a60 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/coalesce.md @@ -6,7 +6,7 @@ Lets you pass potentially empty values to functions that can't handle them by th A short format using the low-priority `??` operator is available (lower than the Boolean operations). You can use the `NVL` alias. -**Examples:** +**Examples** ```yql SELECT COALESCE( @@ -29,3 +29,4 @@ SELECT NVL( ``` <span style="color: gray;">(all three examples above are equivalent)</span> + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md index d80b891dc1..9ea09391d6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/container_literal.md @@ -13,7 +13,7 @@ For field names in the structure literal, you can use an expression that can be For nested lists, use [AsList](#aslist), for nested dictionaries, use [AsDict](#asdict), for nested sets, use [AsSet](#asset), for nested tuples, use [AsTuple](#astuple), for nested structures, use [AsStruct](#asstruct). -**Examples:** +**Examples** ```yql $name = "computed " || "member name"; @@ -33,3 +33,4 @@ SELECT } AS `dict`, {1, 2, 3} AS `set` ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md index afbcb053e4..c7748e2ff9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_tz.md @@ -4,7 +4,7 @@ The arguments that follow are optional and work same as [RANDOM](#random). -**Examples:** +**Examples** ```yql SELECT CurrentTzDate("Europe/Moscow"); @@ -25,7 +25,7 @@ Arguments: Result type: `TzDate`/`TzDatetime`/`TzTimestamp`, depending on the input data type. -**Examples:** +**Examples** ```yql SELECT AddTimezone(Datetime("2018-02-01T12:00:00Z"), "Europe/Moscow"); @@ -41,8 +41,9 @@ Arguments: Result type: `Date`/`Datetime`/`Timestamp`, depending on the input data type. -**Examples:** +**Examples** ```yql SELECT RemoveTimezone(TzDatetime("2018-02-01T12:00:00,Europe/Moscow")); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md index 28744179c6..36bb5eeb02 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/current_utc.md @@ -1,10 +1,10 @@ ## CurrentUtc... {#current-utc} -`CurrentUtcDate()`, `CurrentUtcDatetime()` and `CurrentUtcTimestamp()`: getting the current date and/or time in UTC. The result data type is specified at the end of the function name. +`CurrentUtcDate()`, `CurrentUtcDatetime()` and `CurrentUtcTimestamp()`: Getting the current date and/or time in UTC. The result data type is specified at the end of the function name. The arguments are optional and work same as [RANDOM](#random). -**Examples:** +**Examples** ```yql SELECT CurrentUtcDate(); @@ -13,3 +13,4 @@ SELECT CurrentUtcDate(); ```yql SELECT CurrentUtcTimestamp(TableRow()) FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md index e123553d29..1e201128e2 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/data-type-literals.md @@ -22,7 +22,7 @@ For the data types `TzDate`, `TzDatetime`, `TzTimestamp`, literals are also set {% include [decimal args](../../../_includes/decimal_args.md) %} -**Examples:** +**Examples** ```yql SELECT @@ -48,3 +48,4 @@ SELECT TzTimestamp("2017-11-27T13:24:00.123456,GMT"), Uuid("f9d5cc3f-f1dc-4d9c-b97e-766e57ca4ccb"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md index b7ea751287..d2509df357 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/ensure.md @@ -16,7 +16,7 @@ Arguments: To check the conditions based on the final calculation result, it's convenient to combine Ensure with [DISCARD SELECT](../../../syntax/discard.md). -**Examples:** +**Examples** ```yql SELECT Ensure( @@ -41,3 +41,4 @@ SELECT EnsureConvertibleTo( "expected value to be numeric" ) AS value FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md index 7bf8841e27..28729c4441 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/enum.md @@ -7,7 +7,7 @@ Arguments: * A string with the field name * Enumeration type -**Example:** +**Example** ```yql $enum_type = Enum<Foo, Bar>; @@ -20,11 +20,12 @@ SELECT Arguments: -* A string with the name of an enumeration element +* A string with the name of an enumeration item -**Example:** +**Example** ```yql SELECT AsEnum("Foo"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md index 1ec033e5ed..096709330d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/evaluate_expr_atom.md @@ -25,3 +25,4 @@ SELECT EvaluateExpr( ) ); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md index fdb3d4ce71..16e63c45e0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/files.md @@ -4,7 +4,7 @@ Both the [console](../../../interfaces/cli.md) and [web](../../../interfaces/web The `FileContent` and `FilePath` argument is a string with an alias. -**Examples:** +**Examples** ```yql SELECT "Content of " @@ -21,7 +21,7 @@ The argument is a string with a prefix among aliases. See also [PRAGMA File](../../../syntax/pragma.md#file) and [PRAGMA Folder](../../../syntax/pragma.md#folder). -**Examples:** +**Examples** ```yql PRAGMA File("foo/1.txt", "http://url/to/somewhere"); @@ -59,3 +59,4 @@ SELECT ListLength(ParseFile("String", "my_file.txt")); SELECT * FROM my_table WHERE int_column IN ParseFile("Int64", "my_file.txt")); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md index 0567e1ecf4..80fecee7ab 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/find.md @@ -13,7 +13,7 @@ Optional arguments: Returns the first substring position found or `NULL` (meaning that the desired substring hasn't been found starting from the specified position). -**Examples:** +**Examples** ```yql SELECT FIND("abcdefg_abcdefg", "abc"); -- 0 @@ -42,7 +42,7 @@ Optional arguments: Returns the first substring position found or `NULL` (meaning that the desired substring hasn't been found starting from the specified position). -**Examples:** +**Examples** ```yql SELECT RFIND("abcdefg_abcdefg", "bcd"); -- 9 @@ -55,3 +55,4 @@ SELECT RFIND("abcdefg_abcdefg", "bcd", 8); -- 1 ```yql SELECT RFIND("abcdefg_abcdefg", "bcd", 0); -- null ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md index c40179badc..dcd486afac 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/if.md @@ -6,7 +6,7 @@ It's a simplified alternative for [CASE WHEN ... THEN ... ELSE ... END](../../.. You may omit the `else_expression` argument. In this case, if the condition is false (`condition_expression` returned `false`), an empty value is returned with the type corresponding to `then_expression` and allowing for `NULL`. Hence, the result will have an [optional data type](../../../types/optional.md). -**Examples:** +**Examples** ```yql SELECT @@ -14,3 +14,4 @@ SELECT IF(foo > 0, foo) AS only_positive_foo FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md index 27b4b0e8c6..d8da1aff4f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/intro.md @@ -1,3 +1,4 @@ # Basic built-in functions -Below are the general-purpose functions. For specialized functions, see separate articles: [aggregate functions](../../aggregation.md){% if feature_window_functions %}, [window functions](../../window.md){% endif %}, and functions for [lists](../../list.md), [dictionaries](../../dict.md), [structures](../../struct.md), [data types](../../types.md){% if feature_codegen %}, and [code generation](../../codegen.md){% endif %}. +Below are the general-purpose functions. For specialized functions, there are separate articles: [aggregate functions](../../aggregation.md){% if feature_window_functions %}, [window functions](../../window.md){% endif %}, and functions for [lists](../../list.md), [dictionaries](../../dict.md), [structures](../../struct.md), [data types](../../types.md){% if feature_codegen %}, and [code generation](../../codegen.md){% endif %}. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md index dfcd3283d5..4dc75910f9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/length.md @@ -2,7 +2,7 @@ Returns the length of the string in bytes. This function is also available under the `LEN` name . -**Examples:** +**Examples** ```yql SELECT LENGTH("foo"); @@ -17,3 +17,4 @@ SELECT LEN("bar"); To calculate the length of a string in Unicode characters, you can use the function [Unicode::GetLength](../../../udf/list/unicode.md).<br><br>To get the number of elements in the list, use the function [ListLength](../../list.md#listlength). {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md index c722b800d4..d4f4afa916 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/max_min.md @@ -6,8 +6,9 @@ The argument types must be mutually castable and accept `NULL`. `GREATEST` is a synonym for `MAX_OF` and `LEAST` is a synonym for `MIN_OF`. -**Examples:** +**Examples** ```yql SELECT MIN_OF(1, 2, 3); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md index 5ab3345404..610f9bb0f2 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/metadata.md @@ -10,7 +10,7 @@ No arguments. If this data is missing, for example, when you run operations in the embedded mode, the functions return an empty string. -**Examples:** +**Examples** ```yql SELECT @@ -18,3 +18,4 @@ SELECT CurrentOperationSharedId(), CurrentAuthenticatedUser(); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md index 7b99ec0803..dc9b16d1b3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/nanvl.md @@ -9,10 +9,11 @@ Arguments: If one of the arguments is `Double`, the result is`Double`, otherwise, it's `Float`. If one of the arguments is `Optional`, then the result is `Optional`. -**Examples:** +**Examples** ```yql SELECT NANVL(double_column, 0.0) FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md index 20d0f0ea59..d0bc08c13e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/optional_ops.md @@ -4,7 +4,7 @@ The reverse operation is [Unwrap](#unwrap). -**Examples:** +**Examples** ```yql SELECT @@ -22,7 +22,7 @@ Arguments: Reverse operation is [Just](#just). -**Examples:** +**Examples** ```yql $value = Just("value"); @@ -32,7 +32,7 @@ SELECT Unwrap($value, "Unexpected NULL for $value"); `Nothing()`: Create an empty value for the specified [Optional](../../../types/optional.md) data type. -**Examples:** +**Examples** ```yql SELECT @@ -40,3 +40,4 @@ SELECT ``` [Learn more about ParseType and other functions for data types](../../types.md). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md index ad7e2b0335..1a852f530a 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/pickle.md @@ -4,7 +4,7 @@ `Unpickle()` is the inverse operation (deserialization), where with the first argument being the data type of the result and the second argument is the string with the result of `Pickle()` or `StablePickle()`. -**Examples:** +Examples: ```yql SELECT * @@ -16,3 +16,4 @@ WHERE Digest::MurMurHash32( $buf = Pickle(123); SELECT Unpickle(Int32, $buf); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md index 7862433886..1722cc6c9c 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/random.md @@ -20,7 +20,7 @@ Use cases: * `SELECT RANDOM(some_column), RANDOM(some_column) FROM table;`: Different random numbers for each row of the table, but two identical numbers within the same row. * `SELECT RANDOM(some_column), RANDOM(some_column + 1) FROM table;` or `SELECT RANDOM(some_column), RANDOM(other_column) FROM table;`: Two columns, with different numbers in both. -**Examples:** +**Examples** ```yql SELECT @@ -48,3 +48,4 @@ SELECT RANDOM(column, 2) AS randAnd2 -- different from randAnd1 FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md index 003d22b7e6..5a0de652b4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/starts_ends_with.md @@ -9,7 +9,7 @@ Required arguments: The arguments can be of the `String` or `Utf8` type and can be optional. -**Examples:** +**Examples** ```yql SELECT StartsWith("abc_efg", "abc") AND EndsWith("abc_efg", "efg"); -- true @@ -26,3 +26,4 @@ SELECT StartsWith("abcd", NULL); -- null ```yql SELECT EndsWith(NULL, Utf8("")); -- null ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md index 4756e6af17..6bb7b716ee 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticmap.md @@ -1,13 +1,13 @@ ## StaticMap -Transforms a structure or tuple by applying a lambda function to each element. +Transforms a structure or tuple by applying a lambda function to each item. Arguments: * Structure or tuple. -* Lambda for processing elements. +* Lambda for processing items. -Result: a structure or tuple with the same number and naming of elements as in the first argument, and with element data types determined by lambda results. +Result: a structure or tuple with the same number and naming of items as in the first argument, and with item data types determined by lambda results. **Examples:** diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md index e271646a1a..3643e19691 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/staticzip.md @@ -13,3 +13,4 @@ $two = <|k1:3.0, k2:4|>; -- Adding two structures item-by-item SELECT StaticMap(StaticZip($one, $two), ($tuple)->($tuple.0 + $tuple.1)) AS sum; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md index 3e64b79832..78f533ed6f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/substring.md @@ -11,10 +11,10 @@ Optional arguments: * Substring length: The number of bytes starting from the specified position (an integer, or the default `NULL` meaning "up to the end of the source string"). -Indexing starts from zero. If the specified position and length are beyond the string, it returns an empty string. +Indexing starts from zero. If the specified position and length are beyond the string, returns an empty string. If the input string is optional, the result is also optional. -**Examples:** +**Examples** ```yql SELECT SUBSTRING("abcdefg", 3, 1); -- d @@ -27,3 +27,4 @@ SELECT SUBSTRING("abcdefg", 3); -- defg ```yql SELECT SUBSTRING("abcdefg", NULL, 3); -- abc ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md index a31593f89c..24dbcfe745 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_path_name_recindex.md @@ -6,13 +6,13 @@ No arguments. Returns a string with the full path or an empty string and warning {% note info %} -The functions [TablePath](#tablepath), [TableName](#tablename), and [TableRecordIndex](#tablerecordindex) don't support temporary and anonymous tables (they return an empty row or 0 for [TableRecordIndex](#tablerecordindex)). +The [TablePath](#tablepath), [TableName](#tablename), and [TableRecordIndex](#tablerecordindex) functions don't support temporary and anonymous tables (they return an empty string or 0 for [TableRecordIndex](#tablerecordindex)). These functions are calculated when [executing](../../../syntax/select.md#selectexec) projections in `SELECT`, and by that time the current table may already be temporary. To avoid such a situation, create a subquery for calculating these functions, as shown in the second example below. {% endnote %} -**Examples:** +**Examples** ```yql SELECT TablePath() FROM CONCAT(table_a, table_b); @@ -32,7 +32,7 @@ Optional arguments: * Path to the table, `TablePath()` is used by default (see also its limitations). * Specifying the system ("yt") whose rules are used to determine the table name. You need to specify the system only if [USE](../../../syntax/select.md#use) doesn't specify the current cluster. -**Examples:** +**Examples** ```yql USE hahn; @@ -49,8 +49,9 @@ Access to the current sequence number of a row in the physical source table, **s No arguments. When used in combination with [CONCAT](../../../syntax/select.md#concat), [RANGE](../../../syntax/select.md#range) and other similar mechanisms, numbering restarts for each input table. If used in an incorrect context, it returns 0. -**Example:** +**Example** ```yql SELECT TableRecordIndex() FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md index 90cc5262ae..f6cedf2bf8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/table_row.md @@ -2,8 +2,9 @@ Getting the entire table row as a structure. No arguments{% if feature_join %}. `JoinTableRow` in case of `JOIN` always returns a structure with table prefixes{% endif %}. -**Example:** +**Example** ```yql SELECT TableRow() FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md index a4971a4605..75105178eb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/to_from_bytes.md @@ -2,7 +2,7 @@ Conversion of [primitive data types](../../../types/primitive.md) to a string with their binary representation and back. Numbers are represented in the [little endian](https://en.wikipedia.org/wiki/Endianness#Little-endian) format. -**Examples:** +**Examples** ```yql SELECT @@ -12,3 +12,4 @@ SELECT Uint64 ); -- 1234567890ul ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md index 2884f95ad6..7854ae7db0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/variant.md @@ -8,7 +8,7 @@ Arguments: * String with a field name or tuple index * Variant type -**Example:** +**Example** ```yql $var_type = Variant<foo: Int32, bar: Bool>; @@ -25,7 +25,7 @@ Arguments: * Value * A string with the field name -**Example:** +**Example** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md index bf40e1a1e3..e8a60bd796 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/basic/weakfield.md @@ -14,3 +14,4 @@ SELECT WeakField(my_table.other_column, Int64) FROM my_table; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md index 3dbbd75975..a2c0200033 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/codegen.md @@ -17,7 +17,7 @@ In the text representation, S-expressions have the following format: * Atom: ```'"foo"```. The apostrophe character (') denotes quoting of the next line that is usually enclosed in quotation marks. * List: ```'("foo" "bar")```. The apostrophe character (') denotes that there will be no function call in parentheses. -* Calling the built-in function: ```(foo "bar")```. The first element inside the brackets is the mandatory name of the function followed by the function arguments. +* Calling the built-in function: ```(foo "bar")```. The first item inside the brackets is the mandatory name of the function followed by the function arguments. * Declaring a lambda function: ```(lambda '(x y) (+ x y))```. The `lambda` keyword is followed by a list of argument names and then by the body of the lambda function. * The lambda function argument is ```x```. Unlike an atom, a string without an apostrophe character (') references a name in the current scope. When declaring a lambda function, the names of arguments are added to the body's visibility scope, and, if needed, the name is hidden from the global scope. * The ```world```. @@ -56,7 +56,7 @@ SELECT FormatCode(WorldCode()); ## AtomCode -Build a code node with the `atom` type from a string passed to the argument. +Build a code node with the `atom` type from a string passed to the argument. **Examples:** @@ -184,3 +184,4 @@ $makeClosure = ($y) -> { $closure = $makeClosure(2); SELECT $closure(1); -- 3 ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md index ba64e9d6b2..b2f126ee8b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/dict.md @@ -6,7 +6,7 @@ Construct an empty dictionary. Two arguments are passed: for a key and a value. [Documentation for the type definition format](../../types/type_string.md). -**Examples:** +**Examples** ```yql SELECT DictCreate(String, Tuple<String,Double?>); @@ -22,7 +22,7 @@ Construct an empty set. An argument is passed: the key type that can be built by [Documentation for the type definition format](../../types/type_string.md). -**Examples:** +**Examples** ```yql SELECT SetCreate(String); @@ -34,9 +34,9 @@ SELECT SetCreate(Tuple<Int32?,String>); ## DictLength {#dictlength} -The count of elements in the dictionary. +The count of items in the dictionary. -**Examples:** +**Examples** ```yql SELECT DictLength(AsDict(AsTuple(1, AsList("foo", "bar")))); @@ -52,9 +52,9 @@ SELECT DictLength(dict_column) FROM my_table; ## DictHasItems {#dicthasitems} -Check that the dictionary contains at least one element. +Check that the dictionary contains at least one item. -**Examples:** +**Examples** ```yql SELECT DictHasItems(AsDict(AsTuple(1, AsList("foo", "bar")))) FROM my_table; @@ -72,7 +72,7 @@ SELECT DictHasItems(dict_column) FROM my_table; Get dictionary contents as a list of tuples including key-value pairs (`List<Tuplekey_type,value_type>`). -**Examples:** +**Examples** ```yql SELECT DictItems(AsDict(AsTuple(1, AsList("foo", "bar")))); @@ -92,7 +92,7 @@ FROM my_table; Get a list of dictionary keys. -**Examples:** +**Examples** ```yql SELECT DictKeys(AsDict(AsTuple(1, AsList("foo", "bar")))); @@ -112,7 +112,7 @@ FROM my_table; Get a list of dictionary values. -**Examples:** +**Examples** ```yql SELECT DictPayloads(AsDict(AsTuple(1, AsList("foo", "bar")))); @@ -132,7 +132,7 @@ FROM my_table; Get a dictionary element by its key. -**Examples:** +**Examples** ```yql SELECT DictLookup(AsDict( @@ -155,7 +155,7 @@ FROM my_table; Checking if an element in the dictionary using its key. Returns true or false. -**Examples:** +**Examples** ```yql SELECT DictContains(AsDict( @@ -186,7 +186,7 @@ Arguments: 1. Dictionary. 2. [Aggregation factory](../basic.md#aggregationfactory). -**Examples:** +**Examples** ```sql SELECT DictAggregate(AsDict( @@ -205,7 +205,7 @@ So there are two options to make a call: * With the `Dict<K,V1>` and `List<K>` arguments. * With the `Dict<K,V1>` and `Dict<K,V2>` arguments. -**Examples:** +**Examples** ```sql SELECT SetIsDisjoint(ToSet(AsList(1, 2, 3)), AsList(7, 4)); -- true @@ -221,7 +221,7 @@ Arguments: * Two dictionaries: `Dict<K,V1>` and `Dict<K,V2>`. * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1,V2) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. -**Examples:** +**Examples** ```yql SELECT SetIntersection(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 3 } @@ -241,7 +241,7 @@ So there are two options to make a call: * With the `Dict<K,V1>` and `List<K>` arguments. * With the `Dict<K,V1>` and `Dict<K,V2>` arguments. -**Examples:** +**Examples** ```yql SELECT SetIncludes(ToSet(AsList(1, 2, 3)), AsList(3, 4)); -- false @@ -257,7 +257,7 @@ Arguments: * Two dictionaries: `Dict<K,V1>` and `Dict<K,V2>`. * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1?,V2?) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. -**Examples:** +**Examples** ```yql SELECT SetUnion(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2, 3, 4 } @@ -272,7 +272,7 @@ SELECT SetUnion( Construct a dictionary containing all the keys with their values in the first dictionary with no matching key in the second dictionary. -**Examples:** +**Examples** ```yql SELECT SetDifference(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2 } @@ -291,7 +291,7 @@ Arguments: * Two dictionaries: `Dict<K,V1>` and `Dict<K,V2>`. * An optional function that combines the values from the source dictionaries to construct the values of the output dictionary. If such a function has the `(K,V1?,V2?) -> U` type, the result type is `Dict<K,U>`. If the function is not specified, the result type is `Dict<K,Void>`, and the values from the source dictionaries are ignored. -**Examples:** +**Examples** ```yql SELECT SetSymmetricDifference(ToSet(AsList(1, 2, 3)), ToSet(AsList(3, 4))); -- { 1, 2, 4 } @@ -301,3 +301,4 @@ SELECT SetSymmetricDifference( ($k, $a, $b) -> { RETURN AsTuple($a, $b) }); -- { 2 : (null, "qwe"), 3 : ("bar", null) } ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md index 6a003a5b76..702cb639fd 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/index.md @@ -1,16 +1,17 @@ # Built-in YQL functions -* [Basic](../basic.md) -* [Aggregate](../aggregation.md) +- [Basic](../basic.md) +- [Aggregate](../aggregation.md) {% if feature_window_functions %} -* [Window](../window.md) +- [Window](../window.md) {% endif %} -* [For lists](../list.md) -* [For dictionaries](../dict.md) -* [For structures](../struct.md) -* [For types](../types.md) +- [For lists](../list.md) +- [For dictionaries](../dict.md) +- [For structures](../struct.md) +- [For types](../types.md) {% if feature_codegen %} -* [For code generation](../codegen.md) +- [For code generation](../codegen.md) {% endif %} -* [For JSON](../json.md) -* [C++ libraries](../../udf/list/index.md) +- [For JSON](../json.md) +- [C++ libraries](../../udf/list/index.md) + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md index 9ab60eb901..5bbd46a3f5 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/json.md @@ -85,9 +85,9 @@ In this query: The result of executing all JsonPath expressions is a sequence of JSON values. For example: -* The result of executing the `"Bobbie"` expression is a sequence with the only element `"Bobbie"`. Its length is 1. -* The result of executing the `$` expression (that takes the entire JSON object) in JSON `[1, 2, 3]` is `[1, 2, 3]`. A sequence of 1 element of the array `[1, 2, 3]`. -* The result of executing the `$[*]` expression (retrieving all array elements) in JSON `[1, 2, 3]` is `1, 2, 3`. A sequence of three items:`1`, `2`, and `3`. +- The result of executing the `"Bobbie"` expression is a sequence with the only element `"Bobbie"`. Its length is 1. +- The result of executing the `$` expression (that takes the entire JSON object) in JSON `[1, 2, 3]` is `[1, 2, 3]`. A sequence of 1 element of the array `[1, 2, 3]` +- The result of executing the `$[*]` expression (retrieving all array elements) in JSON `[1, 2, 3]` is `1, 2, 3`. A sequence of three items:`1`, `2`, and `3` If the input sequence consists of multiple values, some operations are performed for each element (for example, accessing a JSON object key). However, other operations require a sequence of one element as input (for example, binary arithmetic operations). @@ -159,7 +159,7 @@ JsonPath supports accessing JSON object keys, such as `$.session.user.name`. {% note info %} -Accessing keys without quotes is only supported for keys that start with an English letter or underscore and only contain English letters, underscores, numbers, and a dollar sign. Use quotes for all other keys. For example, `$.profile."this string has spaces"` or `$.user."42 is the answer"`. +Accessing keys without quotes is only supported for keys that start with an English letter or underscore and only contain English letters, underscores, numbers, and a dollar sign. Use quotes for all other keys. For example, `$.profile."this string has spaces"` or `$.user."42 is the answer"` {% endnote %} @@ -241,7 +241,7 @@ For each value from the input sequence: 6. If a segment is specified and its start index is greater than the end index (for example, `$[20 to 1]`), the query fails in `strict` mode. In `lax` mode, this segment is ignored. 7. All elements by the specified indexes are added to the result. Segments include **both ends**. -**Examples:** +**Examples**: ```json [ @@ -356,7 +356,7 @@ If each argument of a binary operation is not a single number or a number is div Unlike some other programming languages, Boolean values in JsonPath are not only `true` and `false`, but also `null` (uncertainty). -JsonPath considers any values received from a JSON document to be non-Boolean. For example, a query like `! $.is_valid_user` (a logical negation applied to the `is_valid_user`) field is syntactically invalid because the `is_valid_user` field value is not Boolean (even when it actually stores `true` or `false`). The correct way to write this kind of query is to explicitly use a comparison with a Boolean value, such as `$.is_valid_user == false`. +JsonPath considers any values received from a JSON document to be non-Boolean. For example, a query like `! $.is_valid_user` (a logical negation applied to the `is_valid_user`) field is syntactically invalid because the `is_valid_user` field value is not Boolean (even when it actually stores `true` or `false`). The correct way to write this kind of query is to explicitly use a comparison with a Boolean value, such as `$.is_valid_user == false`. ### Logical operations @@ -405,10 +405,10 @@ In the truth table, the first column is the left argument, the first row is the JsonPath implements comparison operators for values: -* Equality, `==` -* Inequality, `!=`and `<>` -* Less than and less than or equal to, `<` and `=` -* Greater than and greater than or equal to, `>` and `>=` +- Equality, `==` +- Inequality, `!=`and `<>` +- Less than and less than or equal to, `<` and `=` +- Greater than and greater than or equal to, `>` and `>=` All comparison operators return a Boolean value. Both operator arguments support multiple values. @@ -418,30 +418,30 @@ The arrays of each of the arguments are automatically unpacked. After that, for 1. The elements of the pair are compared 2. If an error occurs during the comparison, the `ERROR` flag is set. -3. If the comparison result is true, the flag set is `FOUND`. +3. If the comparison result is true, the flag set is `FOUND` 4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed. If the pair analysis results in: -1. The `ERROR` flag is set, the operator returns `null`. -2. The `FOUND` flag is set, the operator returns `true`. -3. Otherwise, it returns `false`. +1. The `ERROR` flag is set, the operator returns `null` +2. The `FOUND` flag is set, the operator returns `true` +3. Otherwise, it returns `false` We can say that this algorithm considers all pairs from the Cartesian product of the left and right arguments, trying to find the pair whose comparison returns true. Elements in a pair are compared according to the following rules: 1. If the left or right argument is an array or object, the comparison fails. -2. `null == null` returns true. +2. `null == null` returns true 3. In all other cases, if one of the arguments is `null`, false is returned. 4. If the left and right arguments are of different types, the comparison fails. 5. Strings are compared byte by byte. -6. `true` is considered greater than `false`. -7. Numbers are compared with the accuracy of `1e-20`. +6. `true` is considered greater than `false` +7. Numbers are compared with the accuracy of `1e-20` **Example:** -Let's take a JSON document as an example: +Let's take a JSON document as an example ```json { @@ -452,19 +452,19 @@ Let's take a JSON document as an example: and analyze the steps for executing the `lax $.left < $.right` query: -1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"`. -2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND`. +1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"` +2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND` 3. Since the query is executed in `lax` mode and the `FOUND` flag is set, we aren't analyzing any more pairs. 4. Since we have the `FOUND` flag set, the operator returns true. Let's take the same query in a different execution mode: `strict $.left < $.right`: -1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"`. -2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND`. -3. Let's take the pair `(2, 4)`. The comparison is successful as `2 < 4` is true. Set the flag `FOUND`. -4. Let's take the pair `(1, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR`. -5. Let's take the pair `(2, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR`. -6. Since we have the `ERROR` flag set, the operator returns `null`. +1. Auto unpacking of arrays in the left and right arguments. As a result, the left argument is the sequence `1, 2` and the right argument is `4, "Iranos"` +2. Let's take the pair `(1, 4)`. The comparison is successful as `1 < 4` is true. Set the flag `FOUND` +3. Let's take the pair `(2, 4)`. The comparison is successful as `2 < 4` is true. Set the flag `FOUND` +4. Let's take the pair `(1, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR` +5. Let's take the pair `(2, "Iranos")`. The comparison fails as a number can't be compared with a string. Set the flag `ERROR` +6. Since we have the `ERROR` flag set, the operator returns `null` ### Predicates @@ -507,7 +507,7 @@ If the pair analysis results in: 2. Setting the `FOUND` flag, the predicate returns `true` 3. Otherwise, the predicate returns `false` -**Examples:** +**Examples** 1. `"123456" like_regex "^[0-9]+$"` returns `true` 2. `"123abcd456" like_regex "^[0-9]+$"` returns `false` @@ -539,21 +539,21 @@ The second argument of the predicate must be a sequence of (possibly, multiple) For each element in a sequence of prefix strings: -1. A check is made for whether "an element is a prefix of an input string". +1. A check is made for whether "an element is a prefix of an input string" 2. If the element isn't a string, the `ERROR` flag is set. 3. If the check result is true, the `FOUND` flag is set. 4. If either the `ERROR` or `FOUND` flag is set and the query is executed in `lax` mode, no more pairs are analyzed. If the pair analysis results in: -1. Setting the `ERROR` flag, the predicate returns `null`. -2. Setting the `FOUND` flag, the predicate returns `true`. -3. Otherwise, the predicate returns `false`. +1. Setting the `ERROR` flag, the predicate returns `null` +2. Setting the `FOUND` flag, the predicate returns `true` +3. Otherwise, the predicate returns `false` -**Examples:** +**Examples** -1. `"James Holden" starts with "James"` returns `true`. -2. `"James Holden" starts with "Amos"` returns `false`. +1. `"James Holden" starts with "James"` returns `true` +2. `"James Holden" starts with "Amos"` returns `false` #### `exists` @@ -570,11 +570,11 @@ Where `<expression>` is the JsonPath expression to be checked. Parentheses aroun **Execution** 1. The passed JsonPath expression is executed. -2. If an error occurs, the predicate returns `null`. -3. If an empty sequence is obtained as a result of the execution, the predicate returns `false`. -4. Otherwise, the predicate returns `true`. +2. If an error occurs, the predicate returns `null` +3. If an empty sequence is obtained as a result of the execution, the predicate returns `false` +4. Otherwise, the predicate returns `true` -**Examples:** +**Examples** Let's take a JSON document: @@ -605,12 +605,12 @@ Where `<expression>` is the JsonPath expression to be checked. Only expressions **Execution** -1. If the passed expression returns `null`, the predicate returns `true`. -2. Otherwise, the predicate returns `false`. +1. If the passed expression returns `null`, the predicate returns `true` +2. Otherwise, the predicate returns `false` -**Examples:** +**Examples** -1. `(1 == 2) is unknown` returns `false`. The `1 == 2` expression returned `false`, which is not `null`. +1. `(1 == 2) is unknown` returns `false`. The `1 == 2` expression returned `false`, which is not `null` 2. `(1 == "string") is unknown` returns `true`. The `1 == "string"` expression returned `null`, because strings and numbers can't be compared in JsonPath. ### Filters @@ -623,13 +623,13 @@ Before filtering, the input sequence arrays are automatically unpacked. For each element of the input sequence: 1. The value of the current filtered `@` object becomes equal to the current element of the input sequence. -2. Executing the expression in the filter. +2. Executing the expression in the filter 3. If an error occurs during the expression execution, the current element of the input sequence is skipped. 4. If the expression execution result is the only `true` value, the current element is added to the filter result. **Example:** -Suppose we have a JSON document describing the user's friends: +Suppose we have a JSON document describing the user's friends ```json { @@ -710,10 +710,10 @@ For each element of the input sequence, the method adds this string to the outpu | Array | `"array"` | | Object | `"object"` | -**Examples:** +**Examples** -1. `"Naomi".type()` returns `"string"`. -2. `false.type()` returns `"boolean"`. +1. `"Naomi".type()` returns `"string"` +2. `false.type()` returns `"boolean"` #### `size` @@ -722,9 +722,9 @@ The `size` method returns the size of the array. For each element of the input sequence, the method adds the following to the output sequence: 1. The size of the array if the element type is an array. -2. For all other types (including objects), it adds `1`. +2. For all other types (including objects), it adds `1` -**Examples:** +**Examples** Let's take a JSON document: @@ -741,9 +741,9 @@ Let's take a JSON document: And queries to it: -1. `$.array.size()` returns `3`. -2. `$.object.size()` returns `1`. -3. `$.scalar.size()` returns `1`. +1. `$.array.size()` returns `3` +2. `$.object.size()` returns `1` +3. `$.scalar.size()` returns `1` #### `Double` @@ -753,11 +753,11 @@ Before its execution, the input sequence arrays are automatically unpacked. All elements in the input sequence must be strings that contain decimal numbers. It's allowed to specify the fractional part and exponent. -**Examples:** +**Examples** -1. `"125".double()` returns `125`. -2. `"125.456".double()` returns `125.456`. -3. `"125.456e-3".double()` returns `0.125456`. +1. `"125".double()` returns `125` +2. `"125.456".double()` returns `125.456` +3. `"125.456e-3".double()` returns `0.125456` #### `ceiling` @@ -767,12 +767,12 @@ Before its execution, the input sequence arrays are automatically unpacked. All elements in the input sequence must be numbers. -**Examples:** +**Examples** -1. `(1.3).ceiling()` returns `2`. -2. `(1.8).ceiling()` returns `2`. -3. `(1.5).ceiling()` returns `2`. -4. `(1.0).ceiling()` returns `1`. +1. `(1.3).ceiling()` returns `2` +2. `(1.8).ceiling()` returns `2` +3. `(1.5).ceiling()` returns `2` +4. `(1.0).ceiling()` returns `1` #### `floor` @@ -782,12 +782,12 @@ Before its execution, the input sequence arrays are automatically unpacked. All elements in the input sequence must be numbers. -**Examples:** +**Examples** -1. `(1.3).floor()` returns `1`. -2. `(1.8).floor()` returns `1`. -3. `(1.5).floor()` returns `1`. -4. `(1.0).floor()` returns `1`. +1. `(1.3).floor()` returns `1` +2. `(1.8).floor()` returns `1` +3. `(1.5).floor()` returns `1` +4. `(1.0).floor()` returns `1` #### `abs` @@ -797,11 +797,11 @@ Before its execution, the input sequence arrays are automatically unpacked. All elements in the input sequence must be numbers. -**Examples:** +**Examples** -1. `(0.0).abs()` returns `0`. -2. `(1.0).abs()` returns `1`. -3. `(-1.0).abs()` returns `1`. +1. `(0.0).abs()` returns `0` +2. `(1.0).abs()` returns `1` +3. `(-1.0).abs()` returns `1` #### `keyvalue` @@ -814,12 +814,12 @@ All elements in the input sequence must be objects. For each element of the input sequence: 1. Each key-value pair in the element is analyzed. -2. For each key-value pair, an object is generated with the keys `name` and `value`. +2. For each key-value pair, an object is generated with the keys `name` and `value` 3. `name` stores a string with the name of the key from the pair. 4. `value` stores the value from the pair. 5. All objects for this element are added to the output sequence. -**Examples:** +**Examples** Let's take a JSON document: @@ -871,9 +871,9 @@ Unlike many programming languages, JsonPath doesn't support creating new variabl All functions for JSON accept: -1. A JSON value (can be an arbitrary `Json` or `Json?` expression). -2. A JsonPath query (must be explicitly specified with a string literal). -3. **(Optional)** `PASSING` section. +1. A JSON value (can be an arbitrary `Json` or `Json?` expression) +2. A JsonPath query (must be explicitly specified with a string literal) +3. **(Optional)** `PASSING` section ### PASSING section @@ -890,13 +890,13 @@ PASSING `<expression>` can have the following types: -* Numbers, `Date`, `DateTime`, and `Timestamp` (a `CAST` into `Double` will be made before passing a value to JsonPath) -* `Utf8`, `Bool`, and `Json` +- Numbers, `Date`, `DateTime`, and `Timestamp` (a `CAST` into `Double` will be made before passing a value to JsonPath) +- `Utf8`, `Bool`, and `Json` You can set a `<variable name>` in several ways: -* As an SQL name like `variable` -* In quotes, for example, `"variable"` +- As an SQL name like `variable` +- In quotes, for example, `"variable"` **Example:** @@ -988,37 +988,37 @@ JSON_VALUE( **Default values:** -1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY`. -2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR`. -3. If the `RETURNING` section isn't specified, then for `<type>`, the type used is `Utf8`. +1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY` +2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR` +3. If the `RETURNING` section isn't specified, then for `<type>`, the type used is `Utf8` **Behavior:** -1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `<type>?`. +1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `<type>?` 2. If an error occurs, the returned value depends on the `ON ERROR` section: - * `NULL`: Return an empty `<type>?`. - * `ERROR`: Abort the entire query. - * `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the entire query fails, too. + - `NULL`: Return an empty `<type>?` + - `ERROR`: Abort the entire query + - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the entire query fails, too. 3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section: - * `NULL`: Return an empty `<type>?`. - * `ERROR`: Abort the entire query. - * `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the behavior matches the `ON ERROR` section. + - `NULL`: Return an empty `<type>?` + - `ERROR`: Abort the entire query + - `DEFAULT <expr>`: Return `<expr>` after running the `CAST` function to convert the data type to `<type>?`. If the `CAST` fails, the behavior matches the `ON ERROR` section. 4. If the result of JsonPath execution is a single value, then: - * If the `RETURNING` section isn't specified, the value is converted to `Utf8`. - * Otherwise, the `CAST` function is run to convert the value to `<type>`. If the `CAST` fails, the behavior matches the `ON ERROR` section. In this case, the value from JSON must match the `<type>` type. -5. Return the result. + - If the `RETURNING` section isn't specified, the value is converted to `Utf8`. + - Otherwise, the `CAST` function is run to convert the value to `<type>`. If the `CAST` fails, the behavior matches the `ON ERROR` section. In this case, the value from JSON must match the `<type>` type. +5. Return the result Correlation between JSON and YQL types: -* JSON Number: Numeric types, `Date`, `DateTime`, and `Timestamp`. -* JSON Bool: `Bool`. -* JSON String: `Utf8` and `String`. +- JSON Number: Numeric types, `Date`, `DateTime`, and `Timestamp` +- JSON Bool: `Bool` +- JSON String: `Utf8` and `String` Errors executing `JSON_VALUE` are as follows: -* Errors evaluating JsonPath. -* The result of JsonPath execution is a number of values or a non-scalar value. -* The type of value returned by JSON doesn't match the expected one. +- Errors evaluating JsonPath +- The result of JsonPath execution is a number of values or a non-scalar value. +- The type of value returned by JSON doesn't match the expected one. `The RETURNING` section supports such types as numbers, `Date`, `DateTime`, `Timestamp`, `Utf8`, `String`, and `Bool`. @@ -1086,10 +1086,10 @@ JSON_QUERY( **Default values:** -1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY`. -2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR`. -3. If the `WRAPPER` section isn't specified, the section used is `WITHOUT WRAPPER`. -4. If the `WITH WRAPPER` section is specified but `CONDITIONAL` or `UNCONDITIONAL` is omitted, then the section used is `UNCONDITIONAL`. +1. If the `ON EMPTY` section isn't specified, the section used is `NULL ON EMPTY` +2. If the `ON ERROR` section isn't specified, the section used is `NULL ON ERROR` +3. If the `WRAPPER` section isn't specified, the section used is `WITHOUT WRAPPER` +4. If the `WITH WRAPPER` section is specified but `CONDITIONAL` or `UNCONDITIONAL` is omitted, then the section used is `UNCONDITIONAL` **Behavior:** @@ -1099,27 +1099,27 @@ You can't specify the `WITH ... WRAPPER` and `ON EMPTY` sections at the same tim {% endnote %} -1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Json?`. +1. If `<JSON expression>` is `NULL` or an empty `Json?`, it returns an empty `Json?` 2. If the `WRAPPER` section is specified, then: - * `WITHOUT WRAPPER` or `WITHOUT ARRAY WRAPPER`: Don't convert the result of JsonPath execution in any way. - * `WITH UNCONDITIONAL WRAPPER` or `WITH UNCONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array. - * `WITH CONDITIONAL WRAPPER` or `WITH CONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array if it isn't the only array or object. + - `WITHOUT WRAPPER` or `WITHOUT ARRAY WRAPPER`: Don't convert the result of JsonPath execution in any way. + - `WITH UNCONDITIONAL WRAPPER` or `WITH UNCONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array. + - `WITH CONDITIONAL WRAPPER` or `WITH CONDITIONAL ARRAY WRAPPER`: Wrap the result of JsonPath execution in an array if it isn't the only array or object. 3. If the JsonPath execution result is empty, the returned value depends on the `ON EMPTY` section: - * `NULL`: Return an empty `Json?`. - * `ERROR`: Abort the entire query. - * `EMPTY ARRAY`: Return an empty JSON array, `[]`. - * `EMPTY OBJECT`: Return an empty JSON object, `{}`. + - `NULL`: Return an empty `Json?` + - `ERROR`: Abort the entire query + - `EMPTY ARRAY`: Return an empty JSON array, `[]` + - `EMPTY OBJECT`: Return an empty JSON object, `{}` 4. If an error occurs, the returned value depends on the `ON ERROR` section: - * `NULL`: Return an empty `Json?`. - * `ERROR`: Abort the entire query. - * `EMPTY ARRAY`: Return an empty JSON array, `[]`. - * `EMPTY OBJECT`: Return an empty JSON object, `{}`. -5. Return the result. + - `NULL`: Return an empty `Json?` + - `ERROR`: Abort the entire query + - `EMPTY ARRAY`: Return an empty JSON array, `[]` + - `EMPTY OBJECT`: Return an empty JSON object, `{}` +5. Return the result Errors running a `JSON_QUERY`: -* Errors evaluating JsonPath. -* The result of JsonPath execution is a number of values (even after using the `WRAPPER` section) or a scalar value. +- Errors evaluating JsonPath +- The result of JsonPath execution is a number of values (even after using the `WRAPPER` section) or a scalar value. **Examples:** @@ -1147,3 +1147,4 @@ SELECT JSON_QUERY($json, "$.friends[0]" WITH CONDITIONAL WRAPPER), -- {"name": "James Holden", "age": 35} JSON_QUERY($json, "$.friends.name" WITH CONDITIONAL WRAPPER); -- ["James Holden", "Naomi Nagata"] ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md index f15c465422..c139971689 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/list.md @@ -6,7 +6,7 @@ Construct an empty list. The only argument specifies a string describing the dat [Documentation for the type definition format](../../types/type_string.md). -**Examples:** +**Examples** ```yql SELECT ListCreate(Tuple<String,Double?>); @@ -20,7 +20,7 @@ SELECT ListCreate(OptionalType(DataType("String"))); Construct a list based on one or more arguments. The argument types must be compatible in the case of `AsList` and strictly match in the case of `AsListStrict`. -**Examples:** +**Examples** ```yql SELECT AsList(1, 2, 3, 4, 5); @@ -30,7 +30,7 @@ SELECT AsList(1, 2, 3, 4, 5); The count of items in the list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -43,7 +43,7 @@ SELECT ListLength(list_column) FROM my_table; Check that the list contains at least one item. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -54,9 +54,9 @@ SELECT ListHasItems(list_column) FROM my_table; ## ListCollect {#listcollect} -Convert a lazy list (which can be built by such functions as [ListFilter](#listfilter), [ListMap](#listmap), and [ListFlatMap](#listflatmap)) to an eager list. In contrast to a lazy list, where each new pass re-calculates the list contents, in an eager list the content is built at once by consuming more memory. +Convert a lazy list (it can be built by such functions as [ListFilter](#listfilter), [ListMap](#listmap), [ListFlatMap](#listflatmap)) to an eager list. In contrast to a lazy list, where each new pass re-calculates the list contents, in an eager list the content is built at once by consuming more memory. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -72,9 +72,9 @@ Sort the list. By default, the ascending sorting order is applied (`ListSort` is Arguments: 1. List. -2. An optional expression to get the sort key from a list element (the element itself by default). +2. An optional expression to get the sort key from a list element (it's the element itself by default). -**Examples:** +**Examples** {% if feature_column_container_type %} @@ -105,11 +105,11 @@ The example used a [lambda function](../../syntax/expressions.md#lambda). ## ListExtend and ListExtendStrict {#listextend} Sequentially join lists (concatenation of lists). The arguments can be lists, optional lists, and `NULL`. -The types of list elements must be compatible in the case of `ListExtend` and strictly match in the case of `ListExtendStrict`. +The types of list items must be compatible in the case of `ListExtend` and strictly match in the case of `ListExtendStrict`. If at least one of the lists is optional, then the result is also optional. If at least one argument is `NULL`, then the result type is `NULL`. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -128,7 +128,7 @@ Sequentially join lists of structures (concatenation of lists). A field is added If at least one of the lists is optional, then the result is also optional. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -143,12 +143,12 @@ SELECT ListUnionAll( ## ListZip and ListZipAll {#listzip} -Based on the input lists, build a list of pairs containing the list elements with matching indexes (`List<Tuplefirst_list_element_type,second_list_element_type>`). +Based on the input lists, build a list of pairs containing the list items with matching indexes (`List<Tuplefirst_list_element_type,second_list_element_type>`). The length of the returned list is determined by the shortest list for ListZip and the longest list for ListZipAll. When the shorter list is exhausted, a `NULL` value of a relevant [optional type](../../types/optional.md) is paired with the elements of the longer list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -164,7 +164,7 @@ FROM my_table; Build a list of pairs (Tuple) containing the element number and the element itself (`List<TupleUint64,list_element_type>`). -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -177,7 +177,7 @@ SELECT ListEnumerate(list_column) FROM my_table; Reverse the list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -192,7 +192,7 @@ Returns a copy of the list, skipping the specified number of its first elements. The first argument specifies the source list and the second argument specifies how many elements to skip. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -209,7 +209,7 @@ Returns a copy of the list containing a limited number of elements from the seco The first argument specifies the source list and the second argument specifies the maximum number of elements to be taken from the beginning of the list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -222,7 +222,7 @@ SELECT ListTake(list_column, 3) FROM my_table; Searches the list for an element with the specified value and returns its index at the first occurrence. Indexes count from 0. If such element is missing, it returns `NULL`. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -233,7 +233,7 @@ FROM my_table; {% endif %} -## ListMap, ListFilter and ListFlatMap {#listmap} +## ListMap, ListFilter, and ListFlatMap {#listmap} Apply the function specified as the second argument to each list element. The functions differ in their returned result: @@ -252,13 +252,13 @@ Arguments: 1. Source list. 2. Functions for processing list elements, such as: * [Lambda function](../../syntax/expressions.md#lambda). - * `Module::Function` - С++ UDF; + * `Module::Function` - C++ UDF; {% if feature_udf_noncpp %} - * [Python UDF](../../udf/python.md), [JavaScript UDF](../../udf/javascript.md) or any other callable value. + * [Python UDF](../../udf/python.md), [JavaScript UDF](../../udf/javascript.md) or any other called value. If the source list is optional, then the output list is also optional. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -279,7 +279,7 @@ Applies transformation to the source list, skipping empty optional items and str If the source list is optional, then the output list is also optional. -**Examples:** +**Examples** ```yql SELECT ListNotNull([1,2]), -- [1,2] @@ -288,11 +288,11 @@ SELECT ListNotNull([1,2]), -- [1,2] ## ListFlatten {#listflatten} -Expands the list of lists into a flat list, preserving the order of items. As the top-level list item, you can use an optional list that is interpreted as an empty list in the case of `NULL`. +Expands the list of lists into a flat list, preserving the order of items. As the top-level list item you can use an optional list that is interpreted as an empty list in the case of `NULL`. If the source list is optional, then the output list is also optional. -**Examples:** +**Examples** ```yql SELECT ListFlatten([[1,2],[3,4]]), -- [1,2,3,4] @@ -303,7 +303,7 @@ SELECT ListFlatten([[1,2],[3,4]]), -- [1,2,3,4] Returns a copy of the list containing only distinct elements. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -316,14 +316,14 @@ FROM my_table; ## ListAny and ListAll {#listany} -Returns `true` for a list of Boolean values if: +Returns `true` for a list of Boolean values, if: * `ListAny`: At least one element is `true`. * `ListAll`: All elements are `true`. Otherwise, it returns false. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -339,7 +339,7 @@ FROM my_table; Show whether the list contains the specified element. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -354,7 +354,7 @@ FROM my_table; Returns the first and last item of the list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -370,7 +370,7 @@ FROM my_table; Apply the appropriate aggregate function to all elements of the numeric list. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -386,7 +386,7 @@ FROM my_table; ## ListFromRange {#listfromrange} -Generate a sequence of numbers with the specified step. It's similar to `xrange` in Python 2, but additionally supports floating points. +Generate a sequence of numbers with the specified step. It's similar to `xrange` in Python 2, but additionally supports floats. Arguments: @@ -398,12 +398,12 @@ Specifics: * The end is not included, i.e. `ListFromRange(1,3) == AsList(1,2)`. * The type for the resulting elements is selected as the broadest from the argument types. For example, `ListFromRange(1, 2, 0.5)` results in a `Double` list. -* The list is "lazy", but if used improperly, it can still consume lots of RAM. +* The list is "lazy", but if it's used incorrectly, it can still consume a lot of RAM. * If the step is positive and the end is less than or equal to the start, the result list is empty. * If the step is negative and the end is greater than or equal to the start, the result list is empty. * If the step is neither positive nor negative (0 or NaN), the result list is empty. -**Examples:** +**Examples** ```yql SELECT @@ -420,7 +420,7 @@ Required arguments: 1. Value. 2. Number of copies. -**Examples:** +**Examples** ```yql SELECT ListReplicate(true, 3); -- [true, true, true] @@ -431,7 +431,7 @@ SELECT ListReplicate(true, 3); -- [true, true, true] Concatenates a list of strings into a single string. You can set a separator as the second parameter. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -447,7 +447,7 @@ FROM my_table; For a list of structures, it returns a list of contained fields having the specified name. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -462,8 +462,8 @@ FROM my_table; `ListTakeWhile` returns a list from the beginning while the predicate is true, then the list ends. -`ListSkipWhile` skips the list segment from the beginning while the predicate is true, then returns the rest of the list disregarding the predicate. -`ListTakeWhileInclusive` returns a list from the beginning while the predicate is true. Then the list ends, but it also includes the element that triggered the stopping predicate. +`ListSkipWhile` skips the list segment from the beginning while the predicate is true, then returns the rest of the list ignoring the predicate. +`ListTakeWhileInclusive` returns a list from the beginning while the predicate is true. Then the list ends, but it also includes the item on which the stopping predicate triggered. `ListSkipWhileInclusive` skips a list segment from the beginning while the predicate is true, then returns the rest of the list disregarding the predicate, but excluding the element that matched the predicate and starting with the next element after it. Required arguments: @@ -473,7 +473,7 @@ Required arguments: If the input list is optional, then the result is also optional. -**Examples:** +**Examples** ```yql $data = AsList(1, 2, 5, 1, 2, 7); @@ -496,7 +496,7 @@ Arguments: 1. List. 2. [Aggregation factory](../basic.md#aggregationfactory). -**Examples:** +**Examples** ```yql SELECT ListAggregate(AsList(1, 2, 3), AggregationFactory("Sum")); -- 6 @@ -504,7 +504,7 @@ SELECT ListAggregate(AsList(1, 2, 3), AggregationFactory("Sum")); -- 6 ## ToDict and ToMultiDict {#todict} -Convert a list of tuples containing key-value pairs to a dictionary. If there are conflicting keys in the input list, `ToDict` leaves the first value and `ToMultiDict` builds a list of all the values. +Convert a list of tuples containing key-value pairs to a dictionary. In case of conflicting keys in the input list, `ToDict` leaves the first value and `ToMultiDict` builds a list of all the values. It means that: @@ -513,7 +513,7 @@ It means that: Optional lists are also supported, resulting in an optional dictionary. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -531,7 +531,7 @@ An optional list is also supported, resulting in an optional dictionary. Inverse function: get a list of keys for the [DictKeys](../dict.md#dictkeys) dictionary. -**Examples:** +**Examples** {% if feature_column_container_type %} ```yql @@ -539,4 +539,6 @@ SELECT ToSet(list_column) FROM my_table; ``` + {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md index f727c1ba4b..1b824272b7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/struct.md @@ -36,7 +36,7 @@ Arguments: * The first argument passes the source structure to be expanded. * All the other arguments must be named, each argument adds a new field and the argument's name is used as the field's name (as in [AsStruct](../basic.md#asstruct)). -**Examples:** +**Examples** ```yql $struct = <|a:1|>; @@ -60,7 +60,7 @@ Arguments: 2. Name of the new field. 3. Value of the new field. -**Examples:** +**Examples** ```yql $struct = <|a:1|>; @@ -83,7 +83,7 @@ Arguments: 1. Source structure. 2. Field name. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -105,7 +105,7 @@ Arguments: 1. Source structure. 2. Field name. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -127,7 +127,7 @@ Arguments: 1. Source structure. 2. List of field names. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2, c:3|>; @@ -149,7 +149,7 @@ Arguments: 1. Source structure. 2. List of field names. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2, c:3|>; @@ -171,7 +171,7 @@ Arguments: 1. Source structure. 2. List of field names. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2, c:3|>; @@ -190,7 +190,7 @@ If the resulting field set contains duplicate values, an error is returned. Arguments: two or more structures. -**Examples:** +**Examples** ```yql $struct1 = <|a:1, b:2|>; @@ -208,9 +208,9 @@ Combining the fields from multiple new structures into another new structure wit If the resulting field set contains duplicate values, an error is returned. -Arguments: two or more tuples of two elements: prefix and structure. +Arguments: two or more tuples of two items: prefix and structure. -**Examples:** +**Examples** ```yql $struct1 = <|a:1, b:2|>; @@ -228,7 +228,7 @@ Returns an unordered list of field names (possibly removing one Optional level) Argument: structure -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -245,7 +245,7 @@ Arguments: 1. Source structure. 2. A tuple of field names: the original name, the new name. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -262,7 +262,7 @@ Arguments: 1. Source structure. 2. A tuple of field names: the original name, the new name. -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -272,11 +272,11 @@ SELECT ## GatherMembers {#gathermembers} -Returns an unordered list of tuples including the field name and value. For the `NULL` argument, `EmptyList` is returned. It can be used only in the cases when the types of elements in the structure are the same or compatible. Returns an optional list for an optional structure. +Returns an unordered list of tuples including the field name and value. For the `NULL` argument, `EmptyList` is returned. It can be used only in the cases when the types of items in the structure are the same or compatible. Returns an optional list for an optional structure. Argument: structure -**Examples:** +**Examples** ```yql $struct = <|a:1, b:2|>; @@ -286,14 +286,14 @@ SELECT ## SpreadMembers {#spreadmembers} -Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of edits includes a field that is not in the list of expected fields, an error is returned. +Creates a structure with a specified list of fields and applies a specified list of edits to it in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of edits includes a field that is not in the list of expected fields, an error is returned. Arguments: 1. List of tuples: field name, field value. 2. A list of all possible field names in the structure. -**Examples:** +**Examples** ```yql @@ -303,17 +303,18 @@ SELECT ## ForceSpreadMembers {#forcespreadmembers} -Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is Optional or `NULL`, the result has the same type. If the list of updates includes a field that is not in the list of expected fields, this edit is ignored. +Creates a structure with a specified list of fields and applies to it the specified list of updates in the format (field name, field value). All types of fields in the resulting structure are the same and equal to the type of values in the update list with added Optional (unless they are optional already). If the field wasn't mentioned among the list of updated fields, it's returned as `NULL`. Among all updates for a field, the latest one is written. If the update list is optional or equal to `NULL`, the result has the same type. If the list of updates includes a field that is not in the list of expected fields, this edit is ignored. Arguments: 1. List of tuples: field name, field value. 2. A list of all possible field names in the structure. -**Examples:** +**Examples** ```yql SELECT ForceSpreadMembers([('a',1),('a',2),('c',100)],['a','b']); -- (a: 2, b: null) ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md index 4dcf44d0f0..152d4ba455 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/types.md @@ -8,7 +8,7 @@ Serializing a type {% if feature_codegen %} or a handle type{% endif %} to a hum Building a type from a string with description. [Documentation for its format](../../types/type_string.md). -**Examples:** +**Examples** ```yql SELECT FormatType(ParseType("List<Int32>")); -- List<int32> @@ -18,7 +18,7 @@ SELECT FormatType(ParseType("List<Int32>")); -- List<int32> Getting the type of value passed to the argument. -**Examples:** +**Examples** ```yql SELECT FormatType(TypeOf("foo")); -- String @@ -34,7 +34,7 @@ Returns an instance of the specified type that can only be used to get the type If this instance remains in the computation graph by the end of optimization, the operation fails. -**Examples:** +**Examples** ```yql SELECT FormatType(TypeOf( @@ -47,7 +47,7 @@ SELECT FormatType(TypeOf( Returns a type for [primitive data types](../../types/primitive.md) based on type name. -**Examples:** +**Examples** ```yql SELECT FormatType(DataType("Bool")); -- Bool @@ -58,7 +58,7 @@ SELECT FormatType(DataType("Decimal","5","1")); -- Decimal(5,1) Adds the option to assign `NULL` to the passed type. -**Examples:** +**Examples** ```yql SELECT FormatType(OptionalType(DataType("Bool"))); -- Bool? @@ -68,7 +68,7 @@ SELECT FormatType(OptionalType(DataType("Bool"))); -- Bool? Builds a list type or stream type based on the passed element type. -**Examples:** +**Examples** ```yql SELECT FormatType(ListType(DataType("Bool"))); -- List<Bool> @@ -78,7 +78,7 @@ SELECT FormatType(ListType(DataType("Bool"))); -- List<Bool> Builds a dictionary type based on the passed key types (first argument) and value types (second argument). -**Examples:** +**Examples** ```yql SELECT FormatType(DictType( @@ -91,7 +91,7 @@ SELECT FormatType(DictType( Builds the tuple type from the passed element types. -**Examples:** +**Examples** ```yql SELECT FormatType(TupleType( @@ -105,7 +105,7 @@ SELECT FormatType(TupleType( Builds the structure type based on the passed element types. The standard syntax of named arguments is used to specify the element names. -**Examples:** +**Examples** ```yql SELECT FormatType(StructType( @@ -118,7 +118,7 @@ SELECT FormatType(StructType( Returns the type of a variant based on the underlying type (structure or tuple). -**Examples:** +**Examples** ```yql SELECT FormatType(VariantType( @@ -130,7 +130,7 @@ SELECT FormatType(VariantType( Returns the type of the [resource](../../types/special.md) based on the passed string label. -**Examples:** +**Examples** ```yql SELECT FormatType(ResourceType("Foo")); -- Resource<'Foo'> @@ -144,7 +144,7 @@ Constructs the type of the called value using the following arguments: 2. Result type. 3. All the next arguments of CallableType are treated as types of arguments of the callable value, but with a shift for two required arguments (for example, the third argument of the CallableType describes the type of the first argument in the callable value). -**Examples:** +**Examples** ```yql SELECT FormatType(CallableType( @@ -155,11 +155,11 @@ SELECT FormatType(CallableType( )); -- Callable<(String,[Int64?])->Double> ``` -## GenericType, UnitType and VoidType {#generictype} +## GenericType, UnitType, and VoidType {#generictype} Return the same-name [special data types](../../types/special.md). They have no arguments because they are not parameterized. -**Examples:** +**Examples** ```yql SELECT FormatType(VoidType()); -- Void @@ -167,31 +167,33 @@ SELECT FormatType(VoidType()); -- Void ## OptionalItemType, ListItemType and StreamItemType {#optionalitemtype} -{% if feature_codegen %} If a type is passed to these functions, then they {% else %}Do{% endif %} the action reverse to [OptionalType](#optionaltype), [ListType](#listtype), and [StreamType](#streamtype): return the element type based on its container type. +{% if feature_codegen %} If a type is passed to these functions, then they perform {% else %}Perform{% endif %} the action reverse to [OptionalType](#optionaltype), [ListType](#listtype), and [StreamType](#streamtype): return the item type based on its container type. {% if feature_codegen %} -If a type handle is passed to these functions, then they do the action reverse to [OptionalTypeHandle](#optionaltypehandle), [ListTypeHandle](#listtypehandle), and [StreamTypeHandle](#streamtypehandle): they return the handle of the element type based on the type handle of its container. -{% endif %} +If a type handle is passed to these functions, then they perform the action reverse to [OptionalTypeHandle](#optionaltypehandle), [ListTypeHandle](#listtypehandle), and [StreamTypeHandle](#streamtypehandle): they return the handle of the element type based on the type handle of its container.{% endif %} -**Examples:** +**Examples** ```yql SELECT FormatType(ListItemType( ParseType("List<Int32>") )); -- Int32 ``` + {% if feature_codegen %} + ```yql SELECT FormatType(ListItemType( ParseTypeHandle("List<Int32>") )); -- Int32 ``` + {% endif %} ## DictKeyType and DictPayloadType {#dictkeytype} Returns the type of the key or value based on the dictionary type. -**Examples:** +**Examples** ```yql SELECT FormatType(DictKeyType( @@ -203,7 +205,7 @@ SELECT FormatType(DictKeyType( Returns the tuple's element type based on the tuple type and the element index (index starts from zero). -**Examples:** +**Examples** ```yql SELECT FormatType(TupleElementType( @@ -215,7 +217,7 @@ SELECT FormatType(TupleElementType( Returns the type of the structure element based on the structure type and element name. -**Examples:** +**Examples** ```yql SELECT FormatType(StructMemberType( @@ -227,7 +229,7 @@ SELECT FormatType(StructMemberType( `CallableResultType` returns the result type based on the type of the called value. `CallableArgumentType` returns the argument type based on the called value type and its index (index starts from zero). -**Examples:** +**Examples** ```yql $callable_type = ParseType("(String,Bool)->Double"); @@ -242,12 +244,11 @@ FormatType(CallableArgumentType( ## VariantUnderlyingType {#variantunderlyingtype} -{% if feature_codegen %}If a type is passed to this function, then it {% else %}Does{% endif %} an action reverse to [VariantType](#varianttype): it returns the underlying type based on the variant type. +{% if feature_codegen %}If a type is passed to this function, then it {% else %}Performs{% endif %} an action reverse to [VariantType](#varianttype): it returns the underlying type based on the variant type. {% if feature_codegen %} -If a type handle is passed to this function, it does the action reverse to[VariantTypeHandle](#varianttypehandle): returns the handle of the underlying type based on the handle of the variant type. -{% endif %} +If a type handle is passed to this function, it performs the action reverse to [VariantTypeHandle](#varianttypehandle): returns the handle of the underlying type based on the handle of the variant type.{% endif %} -**Examples:** +**Examples** ```yql SELECT FormatType(VariantUnderlyingType( @@ -257,7 +258,9 @@ FormatType(VariantUnderlyingType( ParseType("Variant<Int32,Double>") )); -- Tuple<Int32,Double> ``` + {% if feature_codegen %} + ```yql SELECT FormatType(VariantUnderlyingType( ParseTypeHandle("Variant<foo:Int32,bar:Double>") @@ -266,9 +269,11 @@ FormatType(VariantUnderlyingType( ParseTypeHandle("Variant<Int32,Double>") )); -- Tuple<Int32,Double> ``` + {% endif %} {% if feature_codegen %} + # Functions for data types during calculations To work with data types during calculations, use handle types: these are [resources](../../types/special.md) that contain an opaque type definition. After constructing the type handle, you can revert to the regular type using the [EvaluateType](#evaluatetype) function. For debug purposes, you can convert a handle type to a string using the [FormatType](#formattype) function. @@ -622,3 +627,4 @@ SELECT LambdaArgumentsCount(($x, $y)->($x+$y)) ``` {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md index 95d4b45b43..6f5f78cdfb 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/aggregate.md @@ -14,3 +14,4 @@ WINDOW w1 AS (ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW), w2 AS (); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md index 3556ffc0f1..a3dc0c052b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/first_last_value.md @@ -4,7 +4,7 @@ Access values from the first and last rows of the [window frame](../../../syntax Optionally, `OVER` can be preceded by the additional modifier `IGNORE NULLS`. It changes the behavior of functions to the first or last __non-empty__ (i.e., non-`NULL`) value among the window frame rows. The antonym of this modifier is `RESPECT NULLS`: it's the default behavior that can be omitted. -**Examples:** +**Examples** ```yql SELECT @@ -19,3 +19,4 @@ SELECT FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md index 3ece1f89f5..334d448fed 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/lag_lead.md @@ -2,7 +2,7 @@ Accessing a value from a row in the [section](../../../syntax/window.md#partition) that lags behind (`LAG`) or leads (`LEAD`) the current row by a fixed number. The first argument specifies the expression to be accessed, and the second argument specifies the offset in rows. You may omit the offset. By default, the neighbor row is used: the previous or next, respectively (hence, 1 is assumed by default). For the rows having no neighbors at a given distance (for example `LAG(expr, 3)` `NULL` is returned in the first and second rows of the section. -**Examples:** +**Examples** ```yql SELECT @@ -10,3 +10,4 @@ SELECT FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md index 46a570d50b..88e66e32d0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/rank_dense.md @@ -11,7 +11,7 @@ Passing an argument to `RANK`/`DENSE_RANK` is a non-standard extension in YQL. {% endnote %} -**Examples:** +**Examples** ```yql SELECT @@ -25,4 +25,5 @@ SELECT RANK() OVER w FROM my_table WINDOW w AS (ORDER BY my_column); -``` + + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md index 7f1b758b67..49c93db7fc 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/row_number.md @@ -2,7 +2,7 @@ Row number within a [partition](../../../syntax/window.md#partition). No arguments. -**Examples:** +**Examples** ```yql SELECT @@ -10,3 +10,4 @@ SELECT FROM my_table WINDOW w AS (ORDER BY key); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md index 6993f977d4..241d78b744 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/_includes/window/session_state.md @@ -2,3 +2,4 @@ A non-standard window function `SessionState()` (without arguments) lets you get the session calculation status from [SessionWindow](../../../syntax/group_by.md#session-window) for the current row. It's allowed only if `SessionWindow()` is present in the `PARTITION BY` section in the window definition. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml index 99ee69f245..9481cd1ccc 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/builtins/toc_i.yaml @@ -1,13 +1,13 @@ items: -- { name: Overview, href: index.md } -- { name: Basic, href: basic.md } -- { name: Aggregate, href: aggregation.md } -- { name: Window, href: window.md, when: feature_window_functions } -- { name: For lists, href: list.md } -- { name: For dictionaries, href: dict.md } -- { name: For structures, href: struct.md } -- { name: For types, href: types.md } -- { name: For code generation, href: codegen.md, when: feature_codegen } +- { name: Overview, href: index.md } +- { name: Basic, href: basic.md } +- { name: Aggregate, href: aggregation.md } +- { name: Window, href: window.md, when: feature_window_functions } +- { name: For lists, href: list.md } +- { name: For dictionaries, href: dict.md } +- { name: For structures, href: struct.md } +- { name: For types, href: types.md } +- { name: For code generation, href: codegen.md, when: feature_codegen } - { name: For JSON, href: json.md } - name: C++ libraries include: { mode: link, path: ../udf/list/toc_i.yaml }
\ No newline at end of file diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md index 9e19a988c5..0ee45fa2f3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/begin.md @@ -18,3 +18,4 @@ DO BEGIN SELECT 2 -- here and in the previous example, you might omit ';' before END END DO ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md index 3f6c920d43..97ef941ee7 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/define_do.md @@ -4,7 +4,7 @@ Specifies a named action that is a parameterizable block of multiple top-level e **Syntax** -1. `DEFINE ACTION`: Action definition. +1. `DEFINE ACTION`: action definition. 1. [Action name](../../expressions.md#named-nodes) that will be used to access the defined action further in the query. 1. The values of parameter names are listed in parentheses. 1. `AS` keyword. @@ -48,3 +48,4 @@ DO $hello_world(NULL); DO $hello_world("John"); DO $hello_world(NULL, "Earth"); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md index f19b7e9d8e..1a69668908 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/action/evaluate.md @@ -3,18 +3,18 @@ `EVALUATE IF`: Executing an action depending on the condition. It's followed by: 1. Condition. -1. [DO](#do) with the name and parameters of the action or an anonymous action. -1. An optional `ELSE` followed by the second `DO` for a situation where the condition is not met. +2. [DO](#do) with the name and parameters of the action or an anonymous action. +3. An optional `ELSE` followed by the second `DO` for a situation where the condition is not met. ## EVALUATE FOR {#evaluate-for} -`EVALUATE FOR`: Executing an action for each element in the list. It's followed by: +`EVALUATE FOR`: Executing an action for each item in the list. It's followed by: 1. [A named expression](../../expressions.md#named-nodes) applied to each next element in the list. -1. `IN` keyword. -1. The above-declared named expression applied to the list the action is executed on. -1. [DO](#do) with the name and parameters of an action or an anonymous action. In the parameters, you can use both the current element from the first paragraph and any named expressions declared above, including the list itself. -1. An optional `ELSE` followed by the second `DO` for the situation when the list is empty. +2. `IN` keyword. +3. The above-declared named expression applied to the list the action is executed on. +4. [DO](#do) with the name and parameters of an action or an anonymous action. In the parameters, you can use both the current element from the first paragraph and any named expressions declared above, including the list itself. +5. An optional `ELSE` followed by the second `DO` for the situation when the list is empty. **Examples** @@ -73,3 +73,4 @@ ELSE Note that `EVALUATE` is run before the operation starts. Please also note that in `EVALUATE` you can't use [anonymous tables](../../select.md#temporary-tables). {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md index 4e63c175fa..f57d319135 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/alter_table.md @@ -91,7 +91,7 @@ ALTER TABLE series_with_families ALTER COLUMN release_date SET FAMILY family_small; ``` -Using the ```ALTER FAMILY``` command, you can change the parameters of the column group. The code below changes the storage type to ```hdd``` for the ```default``` column group in the ```series_with_families``` table: +Using the ```ALTER FAMILY``` command, you can change the parameters of the column group. The code below changes the storage type to ```hdd``` for the ```default``` column group in the ```series_with_families``` table: ```sql ALTER TABLE series_with_families ALTER FAMILY default SET DATA "hdd"; diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/create_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/create_table.md index b4c5eb5f53..e6c108b61b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/create_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/create_table.md @@ -112,7 +112,7 @@ WITH ( Here, key is the name of the parameter and value is its value. -For a list of possible parameter names and their values, see [{{ backend_name }} table description]({{ concept_table }}). +For a list of possible parameter names and their values, see [table description {{ backend_name }}]({{ concept_table }}). For example, this code will create a table with enabled automatic partitioning by partition size and the preferred size of each partition is 512 MB: diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md index 3d9d15c4df..003af35c54 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/declare/general.md @@ -15,7 +15,7 @@ DECLARE $named-node AS data_type; 1. `DECLARE` keyword. 1. `$named-node`: The name by which you can access the passed value. It must start with `$`. 1. `AS` keyword. -1. `data_type`: The data type [represented as a string in the accepted format](../../../types/type_string.md). +1. `data_type` is the data type [represented as a string in the accepted format](../../../types/type_string.md). Only serializable data types are allowed: @@ -33,3 +33,4 @@ DECLARE $z AS List<String>; SELECT $x, $y, $z; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md index 9f51cbd49a..e1f3edfeb5 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/between.md @@ -2,7 +2,7 @@ Checking whether a value is in a range. It's equivalent to two conditions with `>=` and `<=` (range boundaries are included). Can be used with the `NOT` prefix to support inversion. -**Examples:** +**Examples** ```yql SELECT * FROM my_table diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md index 00535fe777..064bcb9a32 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/bitcast.md @@ -2,7 +2,7 @@ Performs a bitwise conversion of an integer value to the specified integer type. The conversion is always successful, but may lose precision or high-order bits. -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md index 199c9e4990..50ee8fd8cd 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/case.md @@ -6,7 +6,7 @@ The `ELSE` branch is mandatory in the `CASE` expression. Expressions in `WHEN` a Since its syntax is quite sophisticated, it's often more convenient to use the built-in function [IF](../../../builtins/basic.md#if). -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md index bd10a5fbf9..9820614f91 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/cast.md @@ -9,7 +9,7 @@ For more information about casting rules, see [here](../../../types/cast.md). {% include [decimal_args](../../../_includes/decimal_args.md) %} -**Examples:** +**Examples** {% include [cast_examples](../../../_includes/cast_examples.md) %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md index 4e2f29e95b..864acf6890 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/check-match.md @@ -15,7 +15,7 @@ All other characters are literals that represent themselves. The most popular way to use the `LIKE` and `REGEXP` keywords is to filter a table using the statements with the `WHERE` clause. However, there are no restrictions on using templates in this context: you can use them in most of contexts involving strings, for example, with concatenation by using `||`. -**Examples:** +**Examples** ```yql SELECT * FROM my_table diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md index 7fee75d903..bf77137627 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/concatenation.md @@ -6,9 +6,9 @@ As with other binary operators, if the data on either side is `NULL`, the result Don't confuse this operator with a logical "or": in SQL, it's denoted by the `OR` keyword. It's also not worth doing concatenation using `+`. -**Examples:** +**Examples** -``` sql +```sql SELECT "fo" || "o"; ``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md index a341d3ab54..58f67d5175 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/in.md @@ -17,7 +17,7 @@ The `COMPACT` hint must be used with care. Since the hash table is built in-memo {% if feature_mapreduce %} Since YQL imposes a limit on the query size in bytes (it's about 1Mb), add large lists of values to your query by URLs and use the [ParseFile](../../../builtins/basic.md#parsefile) function. {% endif %} -**Examples:** +**Examples** ```yql SELECT column IN (1, 2, 3) diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md index cf3d836419..f8242bf2f4 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/is-null.md @@ -2,7 +2,7 @@ Matching an empty value (`NULL`). Since `NULL` is a special value [equal to nothing](../../../types/optional.md#null_expr), the ordinary [comparison operators](#comparison-operators) can't be used to match it. -**Examples:** +**Examples** ```yql SELECT key FROM my_table diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md index c3a2ad7029..6f4c1d2ad3 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/items-access.md @@ -9,7 +9,7 @@ For accessing the values inside containers: When using this syntax to access containers within table columns, be sure to specify the full column name, including the table name or table alias separated by a dot (see the first example below). -**Examples:** +**Examples** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md index ca32ff8582..bb4d836725 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/lambda.md @@ -10,7 +10,7 @@ Only use pure expressions inside the lambda body (those might also be other lamb One or more of the last lambda parameters can be marked with a question mark as optional: if they haven't been specified when calling lambda, they are assigned the `NULL` value. -**Examples:** +**Examples** ```yql $f = ($y) -> { diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md index 1a5afb359e..3d60d2f123 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/named-nodes.md @@ -38,13 +38,13 @@ select $_; --- error: Unable to reference anonymous name $_ export $_; --- An error: Can not export anonymous name $_ ``` -{% if feature_mapreduce %} Moreover, you can't import a named expression with an anonymous alias: +{% if feature_mapreduce %}Moreover, you can't import a named expression with an anonymous alias: ```yql import utils symbols $sqrt as $_; --- error: Can not import anonymous name $_ ``` -{% endif %} Anonymous argument names are also supported for [lambda functions](#lambda), [ACTION](../../action.md#define-action){% if feature_subquery %}, [SUBQUERY](../../subquery.md#define-subquery){% endif %}{% if feature_mapreduce %}, and in [EVALUATE FOR](../../action.md#evaluate-for){% endif %}. +{% endif %}Anonymous argument names are also supported for [lambda functions](#lambda), [ACTION](../../action.md#define-action){% if feature_subquery %}, [SUBQUERY](../../subquery.md#define-subquery){% endif %}{% if feature_mapreduce %}, and in [EVALUATE FOR](../../action.md#evaluate-for){% endif %}. {% note info %} @@ -52,7 +52,7 @@ If named expression substitution results in completely identical subgraphs in th {% endnote %} -**Examples:** +**Examples** ```yql $multiplier = 712; diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md index bec469f869..36f0685055 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/expressions/operators.md @@ -6,7 +6,7 @@ The operators `+`, `-`, `*`, `/`, `%` are defined for [primitive data types](../ For the Decimal data type, bankers rounding is used (to the nearest even integer). -**Examples:** +**Examples** ```yql SELECT 2 + 2; @@ -23,7 +23,7 @@ The operators `=`, `==`, `!=`, `<>`, `>`, `<` are defined for: * Primitive data types except Yson and Json. * Tuples and structures with the same set of fields. No order is defined for structures, but you can check for (non-)equality. Tuples are compared element-by-element left to right. -**Examples:** +**Examples** ```yql SELECT 2 > 1; @@ -33,7 +33,7 @@ SELECT 2 > 1; Use the operators `AND`, `OR`, `XOR` for logical operations on Boolean values (`Bool`). -**Examples:** +**Examples** ```yql SELECT 3 > 0 AND false; @@ -45,10 +45,10 @@ Bitwise operations on numbers: * `&`, `|`, `^`: AND, OR, and XOR, respectively. Don't confuse bitwise operations with the related keywords. The keywords `AND`, `OR`, and `XOR` are used * for Boolean values only*, but not for numbers. * ` ~ `: A negation. -* `<<`, `>>`: Left or right shifts. -* `|<<`, `>>|`: Circular left or right shifts. +* `<`, `>`: Left or right shifts. +* `|<`, `>|`: Circular left or right shifts. -**Examples:** +**Examples** ```yql SELECT @@ -71,7 +71,7 @@ The operators in the table are listed in descending order of precedence. | Priority | Operator | Description | Associativity | | --- | --- | --- | --- | -| 1 | <code>a[], a.foo, a()</code> | Accessing a container element, calling a function | Left | +| 1 | <code>a[], a.foo, a()</code> | Accessing a container item, calling a function | Left | | 2 | <code>+a, -a, ~a, NOT a</code> | Unary operators: plus, minus, bitwise and logical negation | Right | | 3 | <code>a || b</code> | [String concatenation](#concatenation) | Left | | 4 | <code>a*b, a/b, a%b</code> | Multiplication, division, remainder of division | Left | diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md index 4cf9c91d79..62163ace66 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_by.md @@ -65,7 +65,7 @@ This conversion can be convenient in the following cases: `FLATTEN BY` interprets [optional data types](../../../types/optional.md) as lists of length 0 or 1. The table rows with `NULL` are skipped, and the column type changes to a similar non-optional type. -`FLATTEN BY` makes only one conversion at a time, so use `FLATTEN LIST BY` or `FLATTEN OPTIONAL BY` on optional containers, for example, `Optional<List<String>>`. +`FLATTEN BY` makes only one conversion at a time, so use, `FLATTEN LIST BY` or `FLATTEN OPTIONAL BY` on optional containers, for example,`Optional<List<String>>`. {% endnote %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md index 1a229f14d5..3692fe89ef 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/flatten/flatten_other_db.md @@ -1,7 +1,8 @@ ### Analogs of FLATTEN BY in other DBMS {#flatten-other-dmb} -* PostgreSQL: `unnest`. -* Hive: `LATERAL VIEW`. -* MongoDB: `unwind`. -* Google BigQuery: `FLATTEN`. -* ClickHouse: `ARRAY JOIN / arrayJoin`. +* PostgreSQL: `unnest` +* Hive: `LATERAL VIEW` +* MongoDB: `unwind` +* Google BigQuery: `FLATTEN` +* ClickHouse: `ARRAY JOIN / arrayJoin` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md index 7634b231a3..9070a7cb22 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/distinct.md @@ -8,9 +8,9 @@ Applying `DISTINCT` to calculated values is not currently implemented. For this {% endnote %} -**Example:** +**Example** -``` sql +```sql SELECT key, COUNT (DISTINCT value) AS count -- top 3 keys by the number of unique values diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md index 25b44dbf56..b951a44300 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/general.md @@ -31,7 +31,7 @@ Aggregate functions ignore `NULL` in their arguments, except for `COUNT`. YQL also provides aggregation factories implemented by the functions [`AGGREGATION_FACTORY`](../../../builtins/basic.md#aggregationfactory) and [`AGGREGATE_BY`](../../../builtins/aggregation.md#aggregateby). -**Examples:** +**Examples** ```sql SELECT key, COUNT(*) FROM my_table diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md index 93582e00f2..5eca8c3171 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/having.md @@ -2,7 +2,7 @@ Filtering a `SELECT` based on the calculation results of [aggregate functions](../../../builtins/aggregation.md). The syntax is similar to [WHERE](../../select.md#where). -**Example:** +**Example** ```yql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md index 036701b67a..d2c73fa0db 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/rollup_cube_sets.md @@ -31,7 +31,7 @@ The values of columns not used in calculations are replaced with `NULL` in the s * `0`: If `NULL` is used for the original empty value. * `1`: If `NULL` is added for a subtotal or overall total. -**Example:** +**Example** ```sql SELECT diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md index 902009abcd..fb07666454 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/group_by/session_window.md @@ -16,10 +16,10 @@ GROUP BY user, SessionWindow(<time_expr>, <timeout_expr>) AS session_start The following happens in this case: 1) The input table is partitioned by the grouping keys specified in `GROUP BY`, ignoring SessionWindow (in this case, it's based on `user`). - If `GROUP BY` includes nothing more than SessionWindow, then the input table gets into one partition. + If `GROUP BY` includes nothing more than SessionWindow, then the input table gets into one partition 2) Each partition is split into disjoint subsets of rows (sessions). For this, the partition is sorted in the ascending order of the `time_expr` expression. - The session limits are drawn between neighboring elements of the partition, that differ in their `time_expr` values by more than `timeout_expr`. + The session limits are drawn between neighboring items of the partition, that differ in their `time_expr` values by more than `timeout_expr` 3) The sessions obtained in this way are the final partitions on which aggregate functions are calculated. The SessionWindow() key column (in the example, it's `session_start`) has the value "the minimum `time_expr` in the session". @@ -31,16 +31,17 @@ An extended version of SessionWindow with four arguments is also supported: `SessionWindow(<order_expr>, <init_lambda>, <update_lambda>, <calculate_lambda>)` Where: -* `<order_expr>`: An expression used to sort the source partition. -* `<init_lambda>`: A lambda function to initialize the state of session calculation. It has the signature `(TableRow())->State`. It's called once for the first (following the sorting order) element of the source partition. + +* `<order_expr>`: An expression used to sort the source partition +* `<init_lambda>`: A lambda function to initialize the state of session calculation. It has the signature `(TableRow())->State`. It's called once for the first (following the sorting order) element of the source partition * `<update_lambda>`: A lambda function to update the status of session calculation and define the session limits. It has the signature `(TableRow(), State)->Tuple<Bool, State>`. It's called for every item of the source partition, except the first one. The new value of state is calculated based on the current row of the table and the previous state. If the first item in the return tuple is `True`, then a new session starts from the _current_ row. The key of the new session is obtained by applying `<calculate_lambda>` to the second item in the tuple. * `<calculate_lambda>`: A lambda function for calculating the session key (the "value" of SessionWindow() that is also accessible via SessionStart()). The function has the signature `(TableRow(), State)->SessionKey`. It's called for the first item in the partition (after `<init_lambda>`) and those items for which `<update_lambda>` has returned `True` in the first item in the tuple. Please note that to start a new session, you should make sure that `<calculate_lambda>` has returned a value different from the previous session key. Sessions having the same keys are not merged. For example, if `<calculate_lambda>` returns the sequence `0, 1, 0, 1`, then there will be four different sessions. Using the extended version of SessionWindow, you can, for example, do the following: divide a partition into sessions, as in the SessionWindow use case with two arguments, but with the maximum session length limited by a certain constant: -**Example:** +**Example** -``` sql +```sql $max_len = 1000; is the maximum session length. $timeout = 100; is the timeout (timeout_expr in a simplified version of SessionWindow). @@ -62,3 +63,4 @@ GROUP BY user, SessionWindow(ts, $init, $update, $calculate) AS session_start ``` You can use SessionWindow in GROUP BY only once. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/join.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/join.md index cb962c4cc0..b7de8f91da 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/join.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/join.md @@ -16,7 +16,7 @@ SELECT ... FROM table_1 -- The right-hand selection are entries in table_n -- JOIN can include the following steps ... -WHERE ... +WHERE ... ``` At each JOIN step, rules are used to establish correspondences between rows in the left-hand and right-hand data selections, creating a new selection that includes every combination of rows that meet the JOIN conditions. @@ -38,7 +38,7 @@ Since columns in YQL are identified by their names, and you can't have two colum * `CROSS`: A full cartesian product of two tables without specifying key columns and no explicit `ON/USING`. * `EXCLUSION`: Both sides minus the intersection. - + {% note info %} @@ -79,7 +79,7 @@ LEFT JOIN c_table AS c ON c.ref = a.key and c.column1 = b.value; {% if feature_mapreduce %} If the statement filters data in addition to `JOIN`, we recommend that you wrap the criteria that would return `true` for most of the rows in the `LIKELY(...)` function call. If your assumption that positive values prevail for the criteria is correct, such a hint might speed up your query. `LIKELY` can be useful when the predicate calculation is a resource-intensive operation and JOIN significantly reduces the number of rows. -In front of any data source for `JOIN`, you can add the `ANY` keyword to suppress duplicate `JOIN` keys on the given side. In this case, only one row is left from the set of rows with the same `JOIN` key value (no matter which one, that's why the keyword is called `ANY`). +In front of any data source for `JOIN`, you can add the `ANY` keyword to suppress duplicate `JOIN` keys on the given side. In this case, only one row is left from the set of rows with the same `JOIN` key value (no matter which one, that's why the keyword is called `ANY`). This syntax differs from the one used in [ClickHouse]{% if lang == "en" %}(https://clickhouse.com/docs/en/sql-reference/statements/select/join/){% endif %}{% if lang == "ru" %}(https://clickhouse.tech/docs/ru/sql-reference/statements/select/join/){% endif %}where `ANY` is placed before the `JOIN` type and applies to the right side only. Request diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md index 7bc2c9941e..8b7cb79647 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/debug.md @@ -3,37 +3,39 @@ ## Debugging and auxiliary settings {#debug} {% if feature_webui %} + ### `DirectRead` | Value type | Default | -| ---------- | ------- | -| Flag | false | +| --- | --- | +| Flag | false | An auxiliary setting for previewing tables in the [HTTP API](../../../interfaces/http.md) (both for the web interface and console client). {% endif %} ### `config.flags("ValidateUdf", "Lazy")` -| Value type | Default | -| ------------------------ | ------- | -| String: None/Lazy/Greedy | None | +| Value type | Default | +| --- | --- | +| String: None/Lazy/Greedy | None | Validating whether UDF results match the declared signature. The Greedy mode enforces materialization of lazy containers, although the Lazy mode doesn't. ### `{{ backend_name_lower }}.DefaultCluster` -| Value type | Default | -| ------------------------------ | ------- | -| A string with the cluster name | hahn | +| Value type | Default | +| --- | --- | +| A string with the cluster name | hahn | Selecting a cluster for calculations that don't access tables. ### `config.flags("Diagnostics")` | Value type | Default | -| ---------- | ------- | -| Flag | false | +| --- | --- | +| Flag | false | Getting diagnostic information from YQL as an additional result of a query. {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md index 02ad9b8ff8..dcc3cbe888 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/definition.md @@ -12,7 +12,7 @@ Redefinition of settings. * `Kb`, `Mb`, `Gb`: For the data amounts. * `sec`, `min`, `h`, `d`: For the time values. -**Examples:** +**Examples** ```yql PRAGMA AutoCommit; @@ -36,3 +36,4 @@ Unless otherwise specified, a pragma affects all the subsequent expressions up t If necessary and logically possible, you can change the value of this setting several times within a given query to make it different at different execution steps. There are also special scoped pragmas with the scope defined by the same rules as the scope of [named expressions](../../expressions.md#named-nodes). Unlike scoped pragmas, regular pragmas can only be used in the global scope (not inside lambda functions, ACTION{% if feature_subquery %}, SUBQUERY{% endif %}, etc.). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md index 6579feef5f..835e96ce40 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/files.md @@ -27,6 +27,7 @@ Attach a set of files to the query by URL. Works similarly to adding multiple fi | One or two arguments: the file name and an optional URL | — | Static | Treat the specified attached file as a library from which you can do [IMPORT](../../export_import.md). The syntax type for the library is determined from the file extension: + * `.sql`: For the YQL dialect of SQL <span style="color: green;">(recommended)</span>. * `.yql`: For [s-expressions](/docs/s_expressions). @@ -56,3 +57,4 @@ SELECT $x; ``` {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md index 5a812939ea..68da312c59 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/global.md @@ -65,9 +65,9 @@ When set to "auto", it enables a new compute engine. Computing is made, whenever | --- | --- | | Flag | true | -When you use `SELECT foo.* FROM ... AS foo`, remove the `foo.` prefix from the names of the result columns. +When you use `SELECT foo.* FROM ... AS foo`, remove the `foo.` prefix from the names of the result columns. -It can be also used with a [JOIN](../../join.md), but in this case it may fail in the case of a name conflict (that can be resolved by using [WITHOUT](../../select.md#without) and renaming columns). For JOIN in SimpleColumns mode, an implicit Coalesce is made for key columns: the query `SELECT * FROM T1 AS a JOIN T2 AS b USING(key)` in the SimpleColumns mode works same as `SELECT a.key ?? b.key AS key, ... FROM T1 AS a JOIN T2 AS b USING(key)`. +It can be also used with a [JOIN](../../join.md), but in this case it may fail in the case of a name conflict (that can be resolved by using [WITHOUT](../../select.md#without) and renaming columns). For JOIN in SimpleColumns mode, an implicit Coalesce is made for key columns: the query `SELECT * FROM T1 AS a JOIN T2 AS b USING(key)` in the SimpleColumns mode works same as `SELECT a.key ?? b.key AS key, ... FROM T1 AS a JOIN T2 AS b USING(key)` ### CoalesceJoinKeysOnQualifiedAll @@ -77,7 +77,7 @@ It can be also used with a [JOIN](../../join.md), but in this case it may fail i | --- | --- | | Flag | true | -Controls implicit Coalesce for the key `JOIN` columns in the SimpleColumns mode. If the flag is set, the Coalesce is made for key columns if there is at least one expression in the format `foo.*` or `*` in SELECT: for example, `SELECT a.* FROM T1 AS a JOIN T2 AS b USING(key)`. If the flag is not set, then Coalesce for JOIN keys is made only if there is an asterisk '*' after `SELECT`. +Controls implicit Coalesce for the key `JOIN` columns in the SimpleColumns mode. If the flag is set, the Coalesce is made for key columns if there is at least one expression in the format `foo.*` or `*` in SELECT: for example, `SELECT a.* FROM T1 AS a JOIN T2 AS b USING(key)`. If the flag is not set, then Coalesce for JOIN keys is made only if there is an asterisk '*' after `SELECT` ### StrictJoinKeyTypes @@ -116,7 +116,7 @@ For more information about the `IN` behavior when operands include `NULL`s, see Aligns the RANK/DENSE_RANK behavior with the standard if there are optional types in the window sort keys or in the argument of such window functions. It means that: -* The result type is always Uint64 rather than Uint64?. +* The result type is always Uint64 rather than Uint64?; * NULLs in keys are treated as equal to each other (the current implementation returns NULL). You can explicitly select the old behavior by using the `DisableAnsiRankForNullableKeys` pragma. If no pragma is set, then a warning is issued and the old version works. @@ -201,3 +201,4 @@ Increasing the limit on the number of dimensions in [GROUP BY](../../group_by.md Use this option with care, because the computational complexity of the query grows exponentially with the number of dimensions. {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md index 1a3405eae0..1cca5678d8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/ydb.md @@ -11,3 +11,4 @@ An experimental pragma that allows you to reduce the isolation level of the current YDB transaction. {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md index 1a6ae1786a..50058c132f 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/pragma/yson.md @@ -25,3 +25,4 @@ Strict mode control in all Yson UDF calls, including implicit calls. If the valu | Flag | false | An inverted version of `yson.Strict`. If the value is omitted or is `"true"`, it disables the strict mode. If the value is `"false"`, it enables the strict mode. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md index 1bf5b933a5..4a4cf40498 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/assume_order_by.md @@ -11,3 +11,4 @@ SELECT key || "suffix" as key, -CAST(subkey as Int32) as subkey FROM my_table ASSUME ORDER BY key, subkey DESC; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md index 7f4a575419..09bc26c7f5 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/calc.md @@ -13,3 +13,4 @@ SELECT "Hello, world!"; ```yql SELECT 2 + 2; ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md index 62ab9ee3d7..77d31d1a1b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/column_order.md @@ -11,14 +11,14 @@ The column order is ignored in YQL by default: If you enable `PRAGMA OrderedColumns;`, the order of columns is preserved in the query results and is derived from the order of columns in the input tables using the following rules: * `SELECT`: an explicit column enumeration dictates the result order. -* `SELECT` with an asterisk (`SELECT * FROM ...`) inherits the order from its input. -{% if feature_join %}* The order of columns after [JOIN](../../join.md): First output the left-hand columns, then the right-hand ones. If the column order in any of the sides in the `JOIN` output is undefined, the column order in the result is also undefined.{% endif %} +* `SELECT` with an asterisk (`SELECT * FROM ...`) inherits the order from its input. {% if feature_join %}* The order of columns after [JOIN](../../join.md): First output the left-hand columns, then the right-hand ones. If the column order in any of the sides in the `JOIN` output is undefined, the column order in the result is also undefined.{% endif %} * The order in `UNION ALL` depends on the [UNION ALL](#unionall) execution mode. * The column order for [AS_TABLE](#as_table) is undefined. -{% note warning %} +{% note warning %} In the YT table schema, key columns always precede non-key columns. The order of key columns is determined by the order of the composite key. When `PRAGMA OrderedColumns;` is enabled, non-key columns preserve their output order. {% endnote %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md index 1bb572b18f..0f42606613 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/commit.md @@ -13,4 +13,5 @@ INSERT INTO result2 SELECT * FROM my_table; COMMIT; -- result2 will already include the SELECT contents from the second line: INSERT INTO result3 SELECT * FROM result2; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md index e3fa933e29..4c1a31cc9b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/distinct.md @@ -8,11 +8,12 @@ Applying `DISTINCT` to calculated values is not currently implemented. For this {% endnote %} -**Example:** +**Example** ```yql SELECT DISTINCT value -- only unique values from the table FROM my_table; ``` -The `DISTINCT` keyword can also be used to apply [aggregate functions](../../../builtins/aggregation.md) only to distinct values. For more information, see the documentation for [GROUP BY](../../group_by.md).
\ No newline at end of file +The `DISTINCT` keyword can also be used to apply [aggregate functions](../../../builtins/aggregation.md) only to distinct values. For more information, see the documentation for [GROUP BY](../../group_by.md). + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md index 6419d5108a..fdd742f90b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/execution.md @@ -4,16 +4,17 @@ The `SELECT` query result is calculated as follows: * Determine the set of input tables by evaluating the [FROM](#from) clauses. * Apply [SAMPLE](#sample)/[TABLESAMPLE](#sample) to input tables. -* Execute [FLATTEN COLUMNS](../../flatten.md#flatten-columns) or [FLATTEN BY](../../flatten.md); the aliases set in `FLATTEN BY` become visible after this point. +* Execute [FLATTEN COLUMNS](../../flatten.md#flatten-columns) or [FLATTEN BY](../../flatten.md); aliases set in `FLATTEN BY` become visible after this point; {% if feature_join %}* Execute every [JOIN](../../join.md).{% endif %} -* Add to (or replace in) the data the columns listed in [GROUP BY ... AS ...](../../group_by.md). +* Add to (or replace in) the data the columns listed in [GROUP BY ... AS ...](../../group_by.md); * Execute [WHERE](#where) — Discard all the data mismatching the predicate. * Execute [GROUP BY](../../group_by.md), evaluate aggregate functions. -* Apply the filter [HAVING](../../group_by.md#having). -{% if feature_window_functions %} * Evaluate [window functions](../../window.md).{% endif %} -* Evaluate expressions in `SELECT`. +* Apply the filter [HAVING](../../group_by.md#having); +{% if feature_window_functions %} * Evaluate [window functions](../../window.md);{% endif %} +* Evaluate expressions in `SELECT` * Assign names set by aliases to expressions in `SELECT`. * Apply top-level [DISTINCT](#distinct) to the resulting columns. -* Execute similarly every subquery inside [UNION ALL](#unionall), combine them (see [PRAGMA AnsiOrderByLimitInUnionAll](../../pragma.md#pragmas)). +* Execute similarly every subquery inside [UNION ALL](#unionall), combine them (see [PRAGMA AnsiOrderByLimitInUnionAll](../../pragma.md#pragmas)); * Perform sorting with [ORDER BY](#order-by). -* Apply [OFFSET and LIMIT](#limit-offset) to the result.
\ No newline at end of file +* Apply [OFFSET and LIMIT](#limit-offset) to the result. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md index 2dda348156..436de60509 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/folder.md @@ -40,4 +40,5 @@ $table_paths = ( ); SELECT COUNT(*) FROM EACH($table_paths); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md index 9560572640..65b9356c6e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from.md @@ -4,7 +4,7 @@ Data source for `SELECT`. The argument can accept the table name, the result of {% if feature_mapreduce %}The table is searched by name in the database specified by the operator [USE](../../use.md).{% endif %} -**Examples:** +**Examples** ```yql SELECT key FROM my_table; @@ -18,4 +18,5 @@ SELECT * FROM ```yql $table_name = "my_table"; SELECT * FROM $table_name; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md index f5d90c4c67..951b52805b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_as_table.md @@ -4,7 +4,7 @@ Accessing named expressions as tables using the `AS_TABLE` function. `AS_TABLE($variable)` lets you use the value of `$variable` as the data source for the query. In this case, the variable `$variable` must have the type `List<Struct<...>>`. -**Example:** +**Example** ```yql $data = AsList( @@ -13,4 +13,5 @@ $data = AsList( AsStruct(3u AS Key, "v3" AS Value)); SELECT Key, Value FROM AS_TABLE($data); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md index 5e8121a416..386e5ca429 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/from_select.md @@ -2,7 +2,7 @@ An inverted format, first specifying the data source and then the operation. -**Examples:** +**Examples** ```yql FROM my_table SELECT key, value; @@ -13,4 +13,5 @@ FROM a_table AS a JOIN b_table AS b USING (key) SELECT *; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md index 2219b99023..33db1d9a0e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/functional_tables.md @@ -110,4 +110,5 @@ SELECT * FROM FILTER( `my_folder`, $callable ); -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md index d55c2ecf0a..17de5253c0 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/limit_offset.md @@ -2,9 +2,9 @@ `LIMIT` limits the output to the specified number of rows. By default, the output is not restricted. -`OFFSET` specifies the offset from the beginning (in rows). By default, it's zero. +`OFFSET`: specifies the offset from the beginning (in rows). By default, it's zero. -**Examples:** +**Examples** ```yql SELECT key FROM my_table @@ -19,4 +19,5 @@ LIMIT 7 OFFSET 3; ```yql SELECT key FROM my_table LIMIT 3, 7; -- equivalent to the previous example -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md index bb43b3186f..2390f02238 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/order_by.md @@ -3,18 +3,21 @@ Sorting the `SELECT` result using a comma-separated list of sorting criteria. As a criteria, you can use a column value or an expression on columns. Ordering by column sequence number is not supported (`ORDER BY N`where `N` is a number). Each criteria can be followed by the sorting direction: -* `ASC`: Sorting in the ascending order. Applied by default. -* `DESC`: Sorting in the descending order. + +- `ASC`: Sorting in the ascending order. Applied by default. +- `DESC`: Sorting in the descending order. Multiple sorting criteria will be applied left-to-right. -**Example:** +**Example** ```yql SELECT key, string_column FROM my_table ORDER BY key DESC, LENGTH(string_column) ASC; ``` + {% if feature_window_functions %} -You can use `ORDER BY` also for [window functions](../../window.md). -{% endif %}
\ No newline at end of file +You can also use `ORDER BY` for [window functions](../../window.md). +{% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md index 4574b8a2ac..acb8988b0e 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/sample.md @@ -16,7 +16,7 @@ The data is split into sufficiently large blocks, and the whole data blocks are {% note info %} -In the `BERNOULLI` mode, if the `REPEATABLE` keyword is added, the seed is mixed with the chunk ID for each chunk in the table. That's why sampling from different tables with the same content might produce different results. +In the `BERNOULLI` mode, if the `REPEATABLE` keyword is added, the seed is mixed with the chunk ID for each chunk in the table. That's why sampling from different tables with the same content might produce different results. {% endnote %} @@ -38,4 +38,5 @@ TABLESAMPLE SYSTEM(1.0); -- about one percent of the table SELECT * FROM my_table SAMPLE 1.0 / 3; -- one-third of the table -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md index 2d5e58989e..30da15826d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/secondary_index.md @@ -8,7 +8,7 @@ SELECT * WHERE … ``` -**Examples:** +**Examples** * Select all the fields from the `series` table using the `views_index` index with the `views >=someValue` criteria: @@ -26,4 +26,5 @@ SELECT * INNER JOIN users VIEW name_index AS t2 ON t1.uploaded_user_id == t2.user_id WHERE t2.name == userName; - ```
\ No newline at end of file + ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md index f4de7a5d54..0117bfd058 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/temporary_table.md @@ -30,4 +30,5 @@ SELECT 1 AS one, 2 AS two, 3 AS three; COMMIT; SELECT * FROM @$tmp_name; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md index 5d93b113d1..505dc1f182 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all.md @@ -12,6 +12,7 @@ The order of output columns in this mode is equal to the largest common prefix o If the largest common prefix is empty (for example, if the order isn't specified for one of the inputs), then the output order is undefined. In the "by position" mode, the output of the resulting data schema uses the following rules: + * All inputs must have equal number of columns * The order of columns must be defined for all inputs * The names of the resulting columns must match the names of columns in the first table @@ -19,7 +20,7 @@ In the "by position" mode, the output of the resulting data schema uses the foll The order of the output columns in this mode is the same as the order of columns in the first input. -**Examples:** +**Examples** ```yql SELECT 1 AS x @@ -29,7 +30,7 @@ UNION ALL SELECT 3 AS z; ``` -In the default mode, this query returns a selection with three columns x, y, and z. When `PRAGMA PositionalUnionAll;` is enabled, the selection includes only the x column. +In the default mode, this query returns a selection with three columns x, y, and z. When `PRAGMA PositionalUnionAll;` is enabled, the selection only includes the x column. ```yql PRAGMA PositionalUnionAll; @@ -37,4 +38,5 @@ PRAGMA PositionalUnionAll; SELECT 1 AS x, 2 as y UNION ALL SELECT * FROM AS_TABLE([<|x:3, y:4|>]); -- error: the order of columns in AS_TABLE is undefined -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md index 7fdbe70a31..7c41e08bf8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/union_all_rules.md @@ -1,4 +1,5 @@ * The resulting table includes all columns that were found in at least one of the input tables. * If a column wasn't present in all the input tables, then it's automatically assigned the [optional data type](../../../types/optional.md) (that can accept `NULL`). * If a column in different input tables had different types, then the shared type (the broadest one) is output. -* If a column in different input tables had a heterogeneous type, for example, string and numeric, an error is raised.
\ No newline at end of file +* If a column in different input tables had a heterogeneous type, for example, string and numeric, an error is raised. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md index 4286c475f1..193e6d526b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/view.md @@ -19,4 +19,5 @@ If the meta attributes of the table specify an automatic UDF call to convert raw USE some_cluster; SELECT * FROM my_table VIEW my_view; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md index ac2aabe77b..3c5e5bf18b 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/where.md @@ -2,9 +2,10 @@ Filtering rows in the `SELECT` result based on a condition. -**Example:** +**Example** ```yql SELECT key FROM my_table WHERE value > 0; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md index 1d6be48d85..b99e65fd32 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/with.md @@ -39,4 +39,5 @@ SELECT key, value FROM my_table WITH COLUMNS Struct<value:Int32?>; ```yql SELECT key, value FROM EACH($my_tables) WITH SCHEMA Struct<key:String, value:List<Int32>>; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md index b3c5306469..f61e5e9625 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/select/without.md @@ -2,7 +2,7 @@ Excluding columns from the result of `SELECT *`. -**Examples:** +**Examples** ```yql SELECT * WITHOUT foo, bar FROM my_table; @@ -12,4 +12,5 @@ SELECT * WITHOUT foo, bar FROM my_table; PRAGMA simplecolumns; SELECT * WITHOUT t.foo FROM my_table AS t CROSS JOIN (SELECT 1 AS foo) AS v; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md index 4bf1e4e3ee..f9e2f51a7a 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/subquery.md @@ -29,7 +29,7 @@ Even if the list of parameters in the subquery template definition is empty, whe {% if feature_mapreduce %} In some cases, instead of `DEFINE SUBQUERY` it's more convenient to use an equivalent [lambda function](../expressions.md#lambda). -In this case, the lambda function must accept, as the first argument, the special object called `world` that passes dependencies to make certain PRAGMA or COMMIT statements visible at the query template's point of use. Make also sure to pass this object as the first argument along with the other arguments (if any) to other query templates, if you use them in your lambda function. +In this case, the lambda function must accept, as the first argument, the special object called `world` that passes dependencies to make certain PRAGMA or COMMIT statements visible at the query template's point of use. Also, make sure to pass this object as the first argument along with the other arguments (if any) to other query templates, if you use them in your lambda function. The return value of the lambda function must have the structure list type (output table) or a list of variants over a tuple of structures (multiple output tables). In the latter case, the following unpacking is usually used at the query template's point of use: ```yql @@ -199,3 +199,4 @@ $sub2 = SubqueryOrderBy($sub, [('x',false), ('y',true)]); PROCESS $sub2(); ``` + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md index 775f009a08..bde3ca2aea 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/_includes/upsert_into.md @@ -5,7 +5,7 @@ Adds or updates multiple rows in a table based on primary key matching. Missing {% if feature_mapreduce %} The table is searched by name in the database specified by the [USE](../use.md) operator.{% endif %} {% if feature_replace %} -`UPSERT` and [`REPLACE`](../replace_into.md) are data modification operations that don't require a pre-fetch and run faster and cheaper than other operations because of that. +`UPSERT` and [`REPLACE`](../replace_into.md) are data modification operations that don't require a prefetch and run faster and cheaper than other operations because of that. {% else %} `UPSERT` is the only data modification operation that doesn't require prefetching and runs faster and cheaper than other operations because of that. {% endif %} @@ -16,7 +16,7 @@ Column mapping when using `UPSERT INTO ... SELECT` is done by names. Use `AS` to ```yql UPSERT INTO my_table -SELECT pk_column, data_column1, col24 as data_column3 FROM other_table +SELECT pk_column, data_column1, col24 as data_column3 FROM other_table ``` ```yql diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md index a381efbc22..e3bce9d7dd 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/expressions.md @@ -29,3 +29,4 @@ {% include [x](_includes/expressions/lambda.md) %} {% include [x](_includes/expressions/items-access.md) %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md index af923dcb90..3594e5b469 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/index.md @@ -53,3 +53,4 @@ {% if feature_mapreduce %} * [EXPORT and IMPORT](export_import.md) {% endif %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md index fda95879dd..ed23eefd90 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/select.md @@ -47,3 +47,4 @@ {% include [x](_includes/select/temporary_table.md) %} {% include [x](_includes/select/from_as_table.md) %} + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml index a551690344..db6c5392e8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/syntax/toc_i.yaml @@ -1,7 +1,7 @@ items: -- { name: Overview, href: index.md } -- { name: Lexical structure, href: lexer.md } -- { name: Expressions, href: expressions.md } +- { name: Overview, href: index.md } +- { name: Lexical structure, href: lexer.md } +- { name: Expressions, href: expressions.md } - { name: ACTION, href: action.md } - { name: ALTER TABLE, href: alter_table.md, when: feature_map_tables } - { name: CREATE TABLE, href: create_table.md } @@ -10,11 +10,11 @@ items: - { name: DISCARD, href: discard.md } - { name: DROP TABLE, href: drop_table.md } - { name: GROUP BY, href: group_by.md } -- { name: EXPORT and IMPORT, href: export_import.md, when: feature_mapreduce } +- { name: EXPORT and IMPORT, href: export_import.md, when: feature_mapreduce } - { name: FLATTEN, href: flatten.md } - { name: INSERT, href: insert_into.md } - { name: INTO RESULT, href: into_result.md } -- { name: JOIN, href: join.md, when: feature_join } +- { name: JOIN, href: join.md, when: feature_join } - { name: PRAGMA, href: pragma.md } - { name: PROCESS STREAM, href: process.md, when: feature_mapreduce and process_command == "PROCESS STREAM" } - { name: PROCESS, href: process.md, when: feature_mapreduce and process_command == "PROCESS" } diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md index cd112ee587..b89466dfc8 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/cast.md @@ -27,8 +27,8 @@ SELECT ### Rules for List/Dict -* To create a list, `CAST` is applied to each element in the source list to cast it to the target type. -* If the target element type is non-optional and `CAST` on the element might fail, then such casting is discarded. In this case, the resulting list might be shorter or even empty if every casting failed. +* To create a list, `CAST` is applied to each item in the source list to cast it to the target type. +* If the target item type is non-optional and `CAST` on the item might fail, then such casting is discarded. In this case, the resulting list might be shorter or even empty if every casting failed. * For dictionaries, the casting is totally similar to lists, with `CAST` being applied to keys and values. ```yql @@ -42,7 +42,7 @@ SELECT ### Rules for Struct/Tuple -* A structure or tuple is created by applying `CAST` to each element of the source type to cast it to an element with the same name or target type index. +* A structure or tuple is created by applying `CAST` to each item of the source type to cast it to an item with the same name or target type index. * If some field is missing in the target type, it's simply discarded. * If some field is missing in the source value type, then it can be added only if it's optional and accepts the `NULL` value. * If some field is non-optional in the target type, but its casting might fail, then `CAST` adds Optional to the structure or tuple level and might return `NULL` for the entire result. @@ -53,7 +53,7 @@ SELECT CAST((-2, 0) AS Tuple<Uint16, Utf8>), -- null CAST((3, 4) AS Tuple<Uint16, String>), -- (3, "4"): the type is Tuple<Uint16, String>? CAST(("4",) AS Tuple<Uint16, String?>), -- (4, null) - CAST((5, 6, null) AS Tuple<Uint8?>); -- (5,): the elements were removed. + CAST((5, 6, null) AS Tuple<Uint8?>); -- (5,): the items were removed. SELECT -- One field was removed and one field was added: ("three":null, "two": "42") CAST(<|one:"8912", two:42|> AS Struct<two:Utf8, three:Date?>); diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md index 69dc03e12a..b9146a071d 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/containers.md @@ -4,7 +4,7 @@ | ------------ | ---------------- | ------------- | | List | `List<Type>`,</br>`List<Int32>` | A variable-length sequence consisting of same-type elements. | | Dictionary | `Dict<KeyType, ValueType>`,</br>`Dict<String,Int32>` | Set of key-value pairs with a fixed type of keys and values. | -| Set | `Set<KeyType>`,</br>`Set<String>` | A set of elements with a fixed type is a special case of a dictionary with the `Void` type of values. | +| Set | `Set<KeyType>`,</br>`Set<String>` | A set of elements with a fixed type is a special case of a dictionary with the `Void` value type. | | Tuple | `Tuple<Type1, ..., TypeN>`,</br>`Tuple<Int32,Double>` | Set of unnamed fixed-length elements with types specified for all elements. | | Structure | `Struct<Name1:Type1, ..., NameN:TypeN>`,</br> `Struct<Name:String,Age:Int32>` | A set of named fields with specified value types, fixed at query start (must be data-independent). | | Stream | `Stream<Type>`,</br> `Stream<Int32>` | Single-pass iterator by same-type values, not serializable | diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md index e2d00e1d57..75a55b38e1 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_number.md @@ -14,6 +14,6 @@ | `Decimal` | A real number with the specified precision, up to 35 decimal digits | {% if feature_map_tables %}When used in table columns, precision is fixed: Decimal (22,9).</br>Can't be used in the primary key{% endif %} | {% if feature_map_tables %} -| `DyNumber` | A binary representation of a real number with an accuracy of up to 38 digits.<br/>Acceptable values: positive numbers from 1×10<sup>-130</sup> up to 1×10<sup>126</sup>–1, negative numbers from -1×10<sup>126</sup>–1 up to -1×10<sup>-130</sup>, and 0.<br/>Compatible with the `Number` type in AWS DynamoDB. It's not recommended for {{ backend_name_lower }}-native applications. | +`DyNumber` | A binary representation of a real number with an accuracy of up to 38 digits.<br/>Acceptable values: positive numbers from 1×10<sup>-130</sup> up to 1×10<sup>126</sup>–1, negative numbers from -1×10<sup>126</sup>–1 to -1×10<sup>-130</sup>, and 0.<br/>Compatible with the `Number` type in AWS DynamoDB. It's not recommended for {{ backend_name_lower }}-native applications. | {% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_string.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_string.md index 0fbfd8e255..95d59aafc6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_string.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/datatypes_primitive_string.md @@ -7,7 +7,7 @@ | `Yson` | [YSON](https://yt.yandex-team.ru/docs/description/common/yson.html) in a textual or binary representation. | Doesn't support matching{% if feature_map_tables %}, can't be used in the primary key{% endif %} | | `Uuid` | Universally unique identifier [UUID](https://tools.ietf.org/html/rfc4122) | Not supported for table columns | -{% note info %} +{% note info "Cell size restrictions" %} The maximum value size for a {% if feature_map_tables %}non-key {% endif %} column cell with any string data type is 8 MB. diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md index 0560c4a57a..17791bc2f9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/types/_includes/primitive.md @@ -42,7 +42,7 @@ Explicit casting using [CAST](../../syntax/expressions.md#cast): | **Timestamp** | No | Yes | Yes | Yes | Yes | No | | **Interval** | No | Yes | Yes | Yes | Yes | No | -<sup>1</sup> `True` is converted to `1`, `False` is converted to `0`. +<sup>1</sup> `True` is converted to `1` and `False` to `0`. <sup>2</sup> Any value other than `0` is converted to `True`, `0` is converted to `False`. <sup>3</sup> Possible only in the case of a non-negative value. <sup>4</sup> Using the built-in function [Yson::ConvertTo](../../udf/list/yson.md#ysonconvertto). diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md index 4553f311e9..89b44b2246 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/unicode.md @@ -69,7 +69,7 @@ Returns a substring starting with ```from``` with the length of ```len``` charac The third argument includes the following parameters: - DelimeterString:Bool? — treating a delimiter as a string (true, by default) or a set of characters "any of" (false) - SkipEmpty:Bool? - whether to skip empty strings in the result, is false by default - - Limit:Uint64? - Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last element + - Limit:Uint64? - Limits the number of fetched components (unlimited by default); if the limit is exceeded, the raw suffix of the source string is returned in the last item * ```Unicode::JoinFromList(List<Utf8>{Flags:AutoMap}, Utf8) -> Utf8``` diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md index 91bd4a5b12..1c21c062a9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/url.md @@ -4,8 +4,8 @@ * ```Url::Normalize(String) -> String?``` -Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, etc. -The normalization result depends only on the URL itself. The normalization **DOES NOT** include operations depending on the external data: transformation based on duplicates, mirrors, etc. +Normalizes the URL in a robot-friendly way: converts the hostname into lowercase, strips out certain fragments, and so on. +The normalization result only depends on the URL itself. The normalization **DOES NOT** include operations depending on the external data: transformation based on duplicates, mirrors, etc. Returned value: diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_footer.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_footer.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_footer.md diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_header.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_header.md new file mode 100644 index 0000000000..f2b39dec3b --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/intro_header.md @@ -0,0 +1,2 @@ +YSON is a JSON-like data format developed at Yandex. + diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/ypath_overlay.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/ypath_overlay.md new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/_includes/yson/ypath_overlay.md diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md index 6c778270d8..726201e1c9 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/re2.md @@ -90,7 +90,7 @@ Notes on Re2::Options from the official [repository](https://github.com/google/r | Literal:Bool? | false | interpret string as literal, not regexp | | LogErrors:Bool? | true | log syntax and execution errors to ERROR | | LongestMatch:Bool? | false | search for longest match, not first match | -| MaxMem:Uint64? | - | (see below) approx. max memory footprint of RE2 | +| MaxMem:Uint64? | - | (see below) approx. max memory footprint of RE2 | | NeverCapture:Bool? | false | parse all parents as non-capturing | | NeverNl:Bool? | false | never match \n, even if it is in regexp | | PosixSyntax:Bool? | false | restrict regexps to POSIX egrep syntax | diff --git a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md index c355b9e21e..7d7f925cb6 100644 --- a/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md +++ b/ydb/docs/en/core/yql/reference/yql-docs-core-2/udf/list/yson.md @@ -1,6 +1,6 @@ # Yson -[YSON](https://yt.yandex-team.ru/docs/description/common/yson.html) is a JSON-like data format developed at Yandex: +{% include [_includes/yson/intro_header.md](_includes/yson/intro_header.md) %} * Similarities with JSON: * Does not have a strict scheme. @@ -18,25 +18,24 @@ Implementation specifics and functionality of the module: * `Yson::Parse***`: Getting a resource with a DOM object from serialized data, with all further operations performed on the obtained resource. * `Yson::From`: Getting a resource with a DOM object from simple YQL data types or containers (lists or dictionaries). * `Yson::ConvertTo***`: Converting a resource to [primitive data types](../../types/primitive.md) or [containers](../../types/containers.md). - * `Yson::Lookup***`: Getting a single list element or a dictionary with optional conversion to the relevant data type. + * `Yson::Lookup***`: Getting a single list item or a dictionary with optional conversion to the relevant data type. * `Yson::YPath***`: Getting one element from the document tree based on the relative path specified, optionally converting it to the relevant data type. * `Yson::Serialize***`: Getting a copy of data from the resource and serializing the data in one of the formats. -* For convenience, when serialized Yson and Json are passed to functions expecting a resource with a DOM object, implicit conversion using `Yson::Parse` or `Yson::ParseJson` is done automatically. In SQL syntax, the dot or square brackets operator automatically adds a `Yson::Lookup` call. To serialize a resource, you still need to call `Yson::ConvertTo***` ([here's the ticket about CAST syntax support](https://st.yandex-team.ru/YQL-2610)) or `Yson::Serialize***`. It means that, for example, to get the "foo" element as a string from the Yson column named mycolumn and serialized as a dictionary, you can write: `SELECT Yson::ConvertToString(mycolumn["foo"]) FROM mytable;` or `SELECT Yson::ConvertToString(mycolumn.foo) FROM mytable;`. In the variant with a dot, special characters can be escaped by [general rules for IDs](../../syntax/expressions.md#escape). -* Technically, the module is based on a snapshot of the [YT](https://wiki.yandex-team.ru/yt/) code base in Arcadia. +* For convenience, when serialized Yson and Json are passed to functions expecting a resource with a DOM object, implicit conversion using `Yson::Parse` or `Yson::ParseJson` is done automatically. In SQL syntax, the dot or square brackets operator automatically adds a `Yson::Lookup` call. To serialize a resource, you still need to call `Yson::ConvertTo***` or `Yson::Serialize***`. It means that, for example, to get the "foo" element as a string from the Yson column named mycolumn and serialized as a dictionary, you can write: `SELECT Yson::ConvertToString(mycolumn["foo"]) FROM mytable;` or `SELECT Yson::ConvertToString(mycolumn.foo) FROM mytable;`. In the variant with a dot, special characters can be escaped by [general rules for IDs](../../syntax/expressions.md#escape). The module's functions must be considered as "building blocks" from which you can assemble different structures, for example: * `Yson::Parse*** -> Yson::Serialize***`: Converting from one format to other. * `Yson::Parse*** -> Yson::Lookup -> Yson::Serialize***`: Extracting the value of the specified subtree in the source YSON tree. -* `Yson::Parse*** -> Yson::ConvertToList -> ListMap -> Yson::Lookup***`: Extracting elements by a key from the YSON list. +* `Yson::Parse*** -> Yson::ConvertToList -> ListMap -> Yson::Lookup***`: Extracting items by a key from the YSON list. -See also examples of combining YSON functions in the [tutorial](https://yql.yandex-team.ru/Tutorial/yt_17_Yson_and_Json). +{% include [_includes/yson/intro_footer.md](_includes/yson/intro_footer.md) %} **Examples** ```yql $node = Json(@@ - {"abc": {"def": 123, "ghi": "привет"}} + {"abc": {"def": 123, "ghi": "hello"}} @@); SELECT Yson::SerializeText($node.abc) AS `yson`; -- {"def"=123;"ghi"="\xD0\xBF\xD1\x80\xD0\xB8\xD0\xB2\xD0\xB5\xD1\x82"} @@ -79,7 +78,7 @@ The result of all three functions is non-serializable: it can only be passed as {% note info %} -The `Yson::ParseJsonDecodeUtf8` expects that characters outside the ASCII range must be additionally escaped. For detailed escaping rules, see the [YT code](https://a.yandex-team.ru/arc/trunk/arcadia/yt/yt/core/misc/utf8_decoder.cpp). +The `Yson::ParseJsonDecodeUtf8` expects that characters outside the ASCII range must be additionally escaped. {% endnote %} @@ -89,7 +88,13 @@ The `Yson::ParseJsonDecodeUtf8` expects that characters outside the ASCII range Yson::From(T) -> Resource<'Yson2.Node'> ``` -`Yson::From` is a polymorphic function that converts most primitive data types and containers (lists, dictionaries, tuples, structures, and so on) into a Yson resource, see the [example](https://yql.yandex-team.ru/Operations/X5sdMpdg8tLNyOczX6J8qtBTJv7iItZt01ExReYM0o0=). The source object type must be Yson-compatible. For example, in dictionary keys, you can only use the `String` or `Utf8` data types, but not `String?` or `Utf8?` . +`Yson::From` is a polymorphic function that converts most primitive data types and containers (lists, dictionaries, tuples, structures, and so on) into a Yson resource. The source object type must be Yson-compatible. For example, in dictionary keys, you can only use the `String` or `Utf8` data types, but not `String?` or `Utf8?` . + +**Example** + +```sql +SELECT Yson::Serialize(Yson::From(TableRow())) FROM table1; +``` ## Yson::WithAttributes @@ -161,13 +166,35 @@ Yson::ConvertToDoubleDict(Resource<'Yson2.Node'>{Flags:AutoMap}) -> Dict<String, Yson::ConvertToStringDict(Resource<'Yson2.Node'>{Flags:AutoMap}) -> Dict<String,String> ``` -{% note warning "Attention" %} +{% note warning %} These functions do not do implicit type casting by default, that is, the value in the argument must exactly match the function called. {% endnote %} -`Yson::ConvertTo` is a polymorphic function that converts the data type that is specified in the second argument and supports containers (lists, dictionaries, tuples, structures, and so on) into a Yson resource, see the [example](https://yql.yandex-team.ru/Operations/X5AsHC--PI3cxBmdA89P5XVtX5m6tn9P2l31MAFsqNo=). +`Yson::ConvertTo` is a polymorphic function that converts the data type that is specified in the second argument and supports containers (lists, dictionaries, tuples, structures, and so on) into a Yson resource. + +**Example** + +```sql +$data = Yson(@@{ + "name" = "Anya"; + "age" = 15u; + "params" = { + "ip" = "95.106.17.32"; + "last_time_on_site" = 0.5; + "region" = 213; + "user_agent" = "Mozilla/5.0" + } +}@@); +SELECT Yson::ConvertTo($data, + Struct< + name: String, + age: Uint32, + params: Dict<String,Yson> + > +); +``` ## Yson::Contains {#ysoncontains} @@ -206,7 +233,9 @@ Yson::YPathDict(Resource<'Yson2.Node'>{Flags:AutoMap}, String) -> Dict<String,Re Yson::YPathList(Resource<'Yson2.Node'>{Flags:AutoMap}, String) -> List<Resource<'Yson2.Node'>>? ``` -Lets you get a part of the resource based on the source resource and the part's path in [YPath](https://yt.yandex-team.ru/docs/description/common/ypath.html) format. +Lets you get a part of the resource based on the source resource and the part's path in YPath format. + +{% include [_includes/yson/ypath_overlay.md](_includes/yson/ypath_overlay.md) %} ## Yson::Attributes {#ysonattributes} @@ -231,7 +260,7 @@ Yson::SerializeJson(Resource<'Yson2.Node'>{Flags:AutoMap}, [Resource<'Yson2.Opti ``` * `SkipMapEntity` serializes `#` values in dictionaries. The value of attributes is not affected by the flag. By default, `false`. -* `EncodeUtf8` responsible for escaping non-ASCII characters. For detailed escaping rules, see the [YT code](https://a.yandex-team.ru/arc/trunk/arcadia/yt/yt/core/misc/utf8_decoder.cpp). By default, `false`. +* `EncodeUtf8` responsible for escaping non-ASCII characters. By default, `false`. The `Yson` and `Json` data types returned by serialization functions are special cases of a string that is known to contain data in the given format (Yson/Json). diff --git a/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md b/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md index d6c6b823f1..9134bdbbb6 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/index/intro.md @@ -6,6 +6,7 @@ keywords: - yql tutorial - yql basic operations --- + # YQL Tutorial - Overview From this tutorial, you will learn how to perform basic operations with data in {{ ydb-short-name }} and get familiar with the YQL syntax. diff --git a/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md b/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md index 6d5f5dabac..850bdb1e72 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/index/steps.md @@ -14,4 +14,5 @@ The tutorial consists of 15 steps: 1. [{#T}](../../update.md) 1. [{#T}](../../delete.md) 1. [{#T}](../../alter_table.md) -1. [{#T}](../../delete_table.md)
\ No newline at end of file +1. [{#T}](../../delete_table.md) + diff --git a/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md b/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md index d0a127faff..d4eda537bc 100644 --- a/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md +++ b/ydb/docs/en/core/yql/tutorial/_includes/yql_tutorial_prerequisites.md @@ -2,4 +2,5 @@ We assume that you already created tables in step [{#T}](../create_demo_tables.md) and populated them with data in step [{#T}](../fill_tables_with_data.md). -{% endnote %}
\ No newline at end of file +{% endnote %} + diff --git a/ydb/docs/en/core/yql/tutorial/alter_table.md b/ydb/docs/en/core/yql/tutorial/alter_table.md index cf95093f0f..fe79e0b8a2 100644 --- a/ydb/docs/en/core/yql/tutorial/alter_table.md +++ b/ydb/docs/en/core/yql/tutorial/alter_table.md @@ -18,4 +18,5 @@ Delete the column you added from the table: ```sql ALTER TABLE episodes DROP COLUMN viewers; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/basic_aggregation.md b/ydb/docs/en/core/yql/tutorial/basic_aggregation.md index 80bf81e70b..51b1e2d8f3 100644 --- a/ydb/docs/en/core/yql/tutorial/basic_aggregation.md +++ b/ydb/docs/en/core/yql/tutorial/basic_aggregation.md @@ -29,4 +29,5 @@ ORDER BY ; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md b/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md index b962e99dc1..fa0dba4ca3 100644 --- a/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md +++ b/ydb/docs/en/core/yql/tutorial/basic_filter_and_sort.md @@ -27,4 +27,5 @@ LIMIT 3 -- LIMIT N after ORDER BY means ; -- depending on sort order. COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/conditional_values.md b/ydb/docs/en/core/yql/tutorial/conditional_values.md index c7da120dd8..89a1912be2 100644 --- a/ydb/docs/en/core/yql/tutorial/conditional_values.md +++ b/ydb/docs/en/core/yql/tutorial/conditional_values.md @@ -34,8 +34,8 @@ WHERE series_id IN (1,2) -- IN defines the set of values in the WHERE cla -- IN or NOT IN may lead to undesirable outcomes. AND season_id = 1 GROUP BY - CASE -- CASE evaluates a list of conditions - -- and returns one of multiple possible resulting + CASE -- CASE evaluates a list of conditions and + -- returns one of multiple possible resulting -- expressions. CASE can be used in any -- statement or with any clause -- that supports a given statement. For example, you can use CASE in @@ -52,3 +52,4 @@ GROUP BY COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/create_demo_tables.md b/ydb/docs/en/core/yql/tutorial/create_demo_tables.md index 1463ccb4ae..c7d5f2c0ed 100644 --- a/ydb/docs/en/core/yql/tutorial/create_demo_tables.md +++ b/ydb/docs/en/core/yql/tutorial/create_demo_tables.md @@ -45,3 +45,4 @@ CREATE TABLE episodes COMMIT; ``` + diff --git a/ydb/docs/en/core/yql/tutorial/delete.md b/ydb/docs/en/core/yql/tutorial/delete.md index 7fcd5f9d6d..d9d726a626 100644 --- a/ydb/docs/en/core/yql/tutorial/delete.md +++ b/ydb/docs/en/core/yql/tutorial/delete.md @@ -41,4 +41,5 @@ COMMIT; SELECT * FROM episodes WHERE series_id = 1 AND season_id = 1; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/delete_table.md b/ydb/docs/en/core/yql/tutorial/delete_table.md index 5d008a40bc..0fa9da6de9 100644 --- a/ydb/docs/en/core/yql/tutorial/delete_table.md +++ b/ydb/docs/en/core/yql/tutorial/delete_table.md @@ -6,4 +6,5 @@ Delete the [created](create_demo_tables.md) tables using the [DROP TABLE](../ref DROP TABLE episodes; DROP TABLE seasons; DROP TABLE series; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md b/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md index 90f78abae8..c46585333f 100644 --- a/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md +++ b/ydb/docs/en/core/yql/tutorial/fill_tables_with_data.md @@ -115,4 +115,5 @@ VALUES (2, 5, 8, "Fifty-One Percent", CAST(Date("2018-05-13") AS Uint64)); COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/insert_into.md b/ydb/docs/en/core/yql/tutorial/insert_into.md index d4c63fed72..ee4c778227 100644 --- a/ydb/docs/en/core/yql/tutorial/insert_into.md +++ b/ydb/docs/en/core/yql/tutorial/insert_into.md @@ -1,6 +1,6 @@ # Inserting data with INSERT -Add data to the table using [INSERT INTO](../reference/syntax/insert_into.md). +Add data to the table using [INSERT INTO](../reference/syntax/insert_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -36,4 +36,5 @@ COMMIT; SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/join_tables.md b/ydb/docs/en/core/yql/tutorial/join_tables.md index 3b101d2175..7e5f905d8b 100644 --- a/ydb/docs/en/core/yql/tutorial/join_tables.md +++ b/ydb/docs/en/core/yql/tutorial/join_tables.md @@ -24,4 +24,5 @@ ORDER BY -- Sorting of the results. -- Columns are separated by commas. COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/replace_into.md b/ydb/docs/en/core/yql/tutorial/replace_into.md index 221a127b7a..2e791d3eb4 100644 --- a/ydb/docs/en/core/yql/tutorial/replace_into.md +++ b/ydb/docs/en/core/yql/tutorial/replace_into.md @@ -1,6 +1,6 @@ # Inserting and updating data with REPLACE -Add data to the table using [REPLACE INTO](../reference/syntax/replace_into.md). +Add data to the table using [REPLACE INTO](../reference/syntax/replace_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -31,4 +31,5 @@ COMMIT; SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/select_all_columns.md b/ydb/docs/en/core/yql/tutorial/select_all_columns.md index 02163fb936..a0d55bbff6 100644 --- a/ydb/docs/en/core/yql/tutorial/select_all_columns.md +++ b/ydb/docs/en/core/yql/tutorial/select_all_columns.md @@ -1,6 +1,6 @@ # Selecting data from all columns -Select all columns from the table using [SELECT](../reference/syntax/select.md). +Select all columns from the table using [SELECT](../reference/syntax/select.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -12,4 +12,5 @@ SELECT -- Data selection operator. FROM episodes; -- The table to select the data from. COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/update.md b/ydb/docs/en/core/yql/tutorial/update.md index ba5b33613d..134d9d004e 100644 --- a/ydb/docs/en/core/yql/tutorial/update.md +++ b/ydb/docs/en/core/yql/tutorial/update.md @@ -1,6 +1,6 @@ # Updating data with UPDATE -Update data in the table using the [UPDATE](../reference/syntax/update.md) operator. +Update data in the table using the [UPDATE](../reference/syntax/update.md) operator: {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -44,4 +44,5 @@ COMMIT; SELECT * FROM episodes WHERE series_id = 1 AND season_id = 1; COMMIT; -```
\ No newline at end of file +``` + diff --git a/ydb/docs/en/core/yql/tutorial/upsert_into.md b/ydb/docs/en/core/yql/tutorial/upsert_into.md index 8532613d42..75f2137f48 100644 --- a/ydb/docs/en/core/yql/tutorial/upsert_into.md +++ b/ydb/docs/en/core/yql/tutorial/upsert_into.md @@ -1,6 +1,6 @@ # Inserting and updating data with UPSERT -Add data to the table using [UPSERT INTO](../reference/syntax/upsert_into.md). +Add data to the table using [UPSERT INTO](../reference/syntax/upsert_into.md): {% include [yql-reference-prerequisites](_includes/yql_tutorial_prerequisites.md) %} @@ -29,4 +29,5 @@ COMMIT; SELECT * FROM episodes WHERE series_id = 2 AND season_id = 5; COMMIT; -```
\ No newline at end of file +``` + |
