aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authoralexv-smirnov <alex@ydb.tech>2023-05-16 09:44:13 +0300
committeralexv-smirnov <alex@ydb.tech>2023-05-16 09:44:13 +0300
commit70134716182107a07f3f5d70e0dcc1bf63964859 (patch)
tree3601209f2dc10ffd47d41d4f1557ea9fa4e410eb
parent809153ed08e63d3a0ced63eb09123e9fd9702a0e (diff)
downloadydb-70134716182107a07f3f5d70e0dcc1bf63964859.tar.gz
Tuning"parameterized-queries-cli.md"
-rw-r--r--ydb/docs/en/core/reference/ydb-cli/parameterized-queries-cli.md72
1 files changed, 36 insertions, 36 deletions
diff --git a/ydb/docs/en/core/reference/ydb-cli/parameterized-queries-cli.md b/ydb/docs/en/core/reference/ydb-cli/parameterized-queries-cli.md
index eabbe699feb..70428230ea9 100644
--- a/ydb/docs/en/core/reference/ydb-cli/parameterized-queries-cli.md
+++ b/ydb/docs/en/core/reference/ydb-cli/parameterized-queries-cli.md
@@ -2,39 +2,39 @@
## Overview
-{{ ydb-short-name }} CLI can execute [parameterized YQL queries](https://en.wikipedia.org/wiki/Prepared_statement). To use parameters in your YQL query text, you need to declare them first [by the YQL `DECLARE`](../../yql/reference/syntax/declare.md) command.
+{{ ydb-short-name }} CLI can execute [parameterized YQL queries](https://en.wikipedia.org/wiki/Prepared_statement). To use parameters you need to declare them using [the YQL `DECLARE`](../../yql/reference/syntax/declare.md) command in your YQL query text.
-You can use the following {{ ydb-short-name }} CLI commands to execute your YQL queries:
+To run parameterized YQL queries you can use the following {{ ydb-short-name }} CLI commands:
* [ydb yql](yql.md).
* [ydb scripting yql](scripting-yql.md).
* [ydb table query execute](table-query-execute.md).
-These commands support the same syntax and options to transmit query parameters. You can set parameter values on the command line, upload them in [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} files, and read from `stdin` in binary or [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format. When transmitting parameters via `stdin`, you can execute YQL queries in-stream with different parameter values and batch mode supported.
+These commands support the same query parametrization options. Parameter values can be set on the command line, uploaded from [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} files, and read from `stdin` in binary or [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format. On `stdin` you can stream multiple parameter values triggering multiple YQL query executions with batching options.
{% note warning %}
-Out of all the commands used for YQL query execution, only the `table query execute` command applies retry policies. Such policies ensure high query reliability and continuity in the event of changes in partition sets for your tables and other typical situations in your distributed database that make part of your data unavailable to operations for a short time.
+Among the above commands, only the `table query execute` applies retry policies. Such policies ensure reliable query execution and continuity when certain data ranges are unavailable for a short time because of partition changes or other regular processes in a distributed database.
{% endnote %}
## Executing a single YQL query {#one-request}
-To pass parameters for YQL query execution, you can use the command line, JSON files, and `stdin`. The commands of the {{ ydb-short-name }} CLI provide the following parameters for this purpose:
+To provide parameters for a YQL query execution, you can use command line, JSON files, and `stdin`, using the following {{ ydb-short-name }} CLI options:
| Name | Description |
---|---
-| `-p, --param` | An expression in the format `$name=value`, where `$name` is the name of the YQL query parameter and `value` is its value (a correct [JSON value](https://www.json.org/json-ru.html)). It can be specified multiple times.<br><br>All the specified parameters must be declared in the YQL query by the [DECLARE operator](../../yql/reference/syntax/declare.md); otherwise, the query will result in the error: "Query does not contain parameter". If you specify a parameter several times, you will see the error: "Parameter value found in more than one source".<br><br>Depending on your operating system, you might need to escape the `$` character or enclose your expression in single quotes (`'`). |
-| `--param-file` | Name of the file in [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format and in [UTF-8]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/UTF-8){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/UTF-8){% endif %} encoding that specifies values of the parameters matched against the YQL query parameters by key names. It can be specified several times.<br><br>However, if values of the parameter declared in the YQL query are found in multiple files or set by the `--param` command-line option, you'll see the error: "Parameter value found in more than one source".<br><br>Names of keys in the JSON file are specified without the leading `$`. Keys that are present in the file but aren't declared in the YQL query, will be ignored without an error message. |
-| `--input-format` | Format of parameter values. Applies to all the methods of parameter transmission (among command parameters, in a file or using `stdin`).<br>Acceptable values:<ul><li>`json-unicode` (default):[JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %}.</li><li>`json-base64`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format in which values of binary string parameters (`DECLARE $par AS String`) are [Base64]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Base64){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/Base64){% endif %}-encoded. This feature enables you to transmit binary data, and it will be decoded from Base64 by the {{ ydb-short-name }} CLI.</li></ul> |
-| `--stdin-format` | Format of parameter values encoding for `stdin`.<br>{{ ydb-short-name }} The CLI automatically detects that a file or an output of another shell command has been sent to the standard input device, `stdin`. In this case, the CLI interprets the resulting data based on the following acceptable values:<ul><li>`json-unicode`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %}.</li><li>`json-base64`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format in which values of binary string parameters (`DECLARE $par AS String`) are [Base64]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Base64){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/Base64){% endif %}-encoded.</li><li>`raw`: Binary data.</li></ul>If the format of parameter encoding for `stdin` isn't specified, the format set in `--input-format` is used. |
-| `--stdin-par` | Name of the parameter whose value is sent over `stdin`. It's specified without a `$`. The name is required when you use the `raw` format in `--stdin-format`.<br><br>When used with JSON formats, `stdin` isn't interpreted as a JSON document but as a JSON value passed to the parameter with the specified name. |
+| `-p, --param` | An expression in the format `$name=value`, where `$name` is the name of the YQL query parameter and `value` is its value (a correct [JSON value](https://www.json.org/json-ru.html)). The option can be specified repeatedly.<br><br>All the specified parameters must be declared in the YQL query by the [DECLARE operator](../../yql/reference/syntax/declare.md); otherwise, you will get an error "Query does not contain parameter". If you specify the same parameter several times, you will get an error "Parameter value found in more than one source".<br><br>Depending on your operating system, you might need to escape the `$` character or enclose your expression in single quotes (`'`). |
+| `--param-file` | Name of a file in [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} format in [UTF-8]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/UTF-8){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/UTF-8){% endif %} encoding that contains parameter values matched against the YQL query parameters by key names. The option can be specified repeatedly.<br><br>If values of the same parameter are found in multiple files or set by the `--param` command line option, you'll get an error "Parameter value found in more than one source".<br><br>Names of keys in the JSON file are expected without the leading `$` sign. Keys that are present in the file but aren't declared in the YQL query will be ignored without an error message. |
+| `--input-format` | Format of parameter values, applied to all sources of parameters (command line, file, or `stdin`).<br>Available options:<ul><li>`json-unicode` (default):[JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %}.</li><li>`json-base64`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} with values of binary string parameters (`DECLARE $par AS String`) are [Base64]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Base64){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/Base64){% endif %}-encoded. This feature enables you to process binary data, being decoded from Base64 by the {{ ydb-short-name }} CLI.</li></ul> |
+| `--stdin-format` | Format of parameter values for `stdin`.<br>The {{ ydb-short-name }} CLI automatically detects that a file or an output of another shell command has been redirected to the standard input device `stdin`. In this case, the CLI interprets the incoming data based on the following available options:<ul><li>`json-unicode`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %}.</li><li>`json-base64`: [JSON]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/JSON){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/JSON){% endif %} with values of binary string parameters (`DECLARE $par AS String`) are [Base64]{% if lang == "ru" %}(https://ru.wikipedia.org/wiki/Base64){% endif %}{% if lang == "en" %}(https://en.wikipedia.org/wiki/Base64){% endif %}-encoded.</li><li>`raw`: Binary data.</li></ul>If format of parameter values for `stdin` isn't specified, the `--input-format` is used. |
+| `--stdin-par` | Name of a parameter whose value is provided on `stdin`, without a `$` sign. This name is required when you use the `raw` format in `--stdin-format`.<br><br>When used with JSON formats, `stdin` is interpreted not as a JSON document but as a JSON value passed to the parameter with the specified name. |
-The query will be sent for execution on the server only once, provided that values are specified for all the parameters [in the `DECLARE` clause](../../yql/reference/syntax/declare.md). If a value is omitted for at least one parameter, the query will fail with the "Missing value for parameter" message.
+The query will be executed on the server once, provided that values are specified for all the parameters [in the `DECLARE` clause](../../yql/reference/syntax/declare.md). If a value is absent for at least one parameter, the command fails with the "Missing value for parameter" message.
### Examples {#examples-one-request}
-In our examples, we use the `table query execute` command, but you can also run them by the `yql` and `scripting yql` commands.
+In our examples, we use the `table query execute` command, but you can also run them using the `yql` and `scripting yql` commands.
{% include [ydb-cli-profile](../../_includes/ydb-cli-profile.md) %}
@@ -130,35 +130,35 @@ Command output (exact number of bytes may vary):
## Iterative streaming processing {#streaming-iterate}
-{{ ydb-short-name }} CLI supports execution of multiple YQL queries including different sets of parameter values, provided that they are sent through `stdin`. In this case, the database connection is established once and the query execution plan is cached. This substantially increases the performance of such an approach compared to individual CLI calls.
+{{ ydb-short-name }} CLI supports execution of a YQL query multiple times with different sets of parameter values provided on `stdin`. In this case, the database connection is established once and the query execution plan is cached. This substantially increases the performance of such an approach compared to separate CLI calls.
-To use this feature, you need to pass different sets of the same parameters to `stdin` one-by-one, providing to the {{ ydb-short-name }} CLI a rule to separate the sets from one another.
+To use this feature, you need to stream different sets of the same parameters to `stdin` one after another, specifying a rule for the {{ ydb-short-name }} CLI on how to separate the sets from each other.
-The YQL query runs as many times as the number of parameter value sets received through `stdin`. Each set received through `stdin` is joined with the parameter values defined through other sources (`--param`, `--param-file`). The command will complete once the `stdin` stream is closed. Each query is executed within a dedicated transaction.
+The YQL query runs as many times as many parameter value sets received on `stdin`. Each set received on `stdin` is joined with the parameter values defined on other sources (`--param`, `--param-file`). The command will complete once the `stdin` stream is closed. Each query is executed within a dedicated transaction.
-The rule for separating parameter sets from one another (the framing rule) is added to the `stdin`'s parameter encoding format defined in `--stdin-format`:
+A rule for separating parameter sets from one another (framing) complements the `stdin` format specified by the `--stdin-format` option:
| Name | Description |
---|---
-| `--stdin-format` | It defines the `stdin` framing. <br>Possible values:<ul><li>`no-framing` (default): `stdin` expects a single set of parameters, and the YQL query is executed only once when the `stdin` read completes.</li><li>`newline-delimited`: The newline character is used in `stdin` to end a given parameter set, separating it from the next one. The YQL query is executed whenever `stdin` receives a newline character.</li></ul> |
+| `--stdin-format` | Defines the `stdin` framing. <br>Available options:<ul><li>`no-framing` (default): No framing, `stdin` expects a single set of parameters, and the YQL query is executed only once when the `stdin` stream is closed.</li><li>`newline-delimited`: A newline character marks the end of one set of parameter values on `stdin`, separating it from the next one. The YQL query is executed each time a newline character is read from `stdin`.</li></ul> |
{% note warning %}
-When using a newline character as a separator between the parameter sets, make sure that it isn't used within the parameter sets. You cannot use newlines within such text, even if you enclose the text in double quotes. Multiline JSON documents are not acceptable.
+When using a newline character as a separator between the parameter sets, make sure that it isn't used inside the parameter sets. Putting some text value in quotes does not enable newlines within the text. Multiline JSON documents are not allowed.
{% endnote %}
### Example {#example-streaming-iterate}
-#### Streamed processing of multiple parameter sets {#example-iterate}
+#### Streaming processing of multiple parameter sets {#example-iterate}
-For example, you need to run your query thrice, with the following sets of values for the`a` and `b` parameters:
+Suppose you need to run your query thrice, with the following sets of values for the `a` and `b` parameters:
1. `a` = 10, `b` = 20
2. `a` = 15, `b` = 25
3. `a` = 35, `b` = 48
-Let's create a file that will include lines with JSON representations of these sets:
+Let's create a file that includes lines with JSON representations of these sets:
```bash
echo -e '{"a":10,"b":20}\n{"a":15,"b":25}\n{"a":35,"b":48}' > par1.txt
@@ -173,7 +173,7 @@ Command output:
{"a":35,"b":48}
```
-Let's execute the query by passing to `stdin` the content of this file, formatting the output as JSON:
+Let's execute the query by passing the content of this file to `stdin`, formatting the output as JSON:
```bash
cat par1.txt | \
@@ -193,11 +193,11 @@ Command output:
{"column0":83}
```
-This output format can be passed as input to the next YQL query command.
+This output can be passed as input to the next YQL query command.
#### Streaming processing with joining parameter values from different sources {#example-iterate-union}
-For example, you need to run your query thrice, with the following sets of values for the`a` and `b` parameters:
+For example, you need to run your query thrice, with the following sets of values for the `a` and `b` parameters:
1. `a` = 10, `b` = 100
2. `a` = 15, `b` = 100
@@ -225,7 +225,7 @@ Command output:
## Batched streaming processing {#streaming-batch}
-The {{ ydb-short-name }} CLI supports automatic conversion of parameter sets to a `List<>`. This way, you can use a single server request to process multiple parameter sets per transaction. As a result, you can have a substantial performance gain compared to one-by-one query processing.
+The {{ ydb-short-name }} CLI supports automatic conversion of multiple consecutive parameter sets to a `List<>`, enabling you to process them in a single request and transaction. As a result, you can have a substantial performance gain compared to one-by-one query processing.
Two batch modes are supported:
@@ -240,29 +240,29 @@ Use this batch mode when you want to ensure transaction atomicity by applying al
### Adaptive batch mode {#batch-adaptive}
-In the `adaptive` mode, the input stream is split into multiple transactions, with the batch size selected for each of them.
+In the `adaptive` mode, the input stream is split into multiple transactions, with the batch size automatically determined for each of them.
-In this mode, you can process a broad range of inputs with unpredictable or infinite amounts of data, as well as unpredictable or intensely varying input rates. For example, such a profile is used when sending the output of another command to `stdin` using the `|` operator.
+In this mode, you can process a broad range of dynamic workloads with unpredictable or infinite amounts of data, as well as with unpredictable or significantly varying rate of new sets appearance at the input. For example, such a profile is typical when sending the output of another command to `stdin` using the `|` operator.
-The adaptive mode solves two basic issues of the dynamic stream processing:
+The adaptive mode solves two basic issues of dynamic stream processing:
1. Limiting the maximum batch size.
2. Limiting the maximum data processing delay.
### Syntax {#batch-syntax}
-To use the batch mode, define the `List<...>` or `List<Struct<...>>` parameter in the YQL query's DECLARE clause:
+To use the batching capbilities, define the `List<...>` or `List<Struct<...>>` parameter in the YQL query's DECLARE clause, and use the following options:
| Name | Description |
---|---
-| `--batch` | The batch mode of transmitting parameter sets received via `stdin`.<br>Acceptable values:<ul><li>`iterative` (default): The batch mode is [disabled](#streaming-iterate).</li><li>`full`: Full batch mode. The YQL query runs only once when `stdin` is closed, with all the resulting sets of parameters wrapped into a `List<>`, and the parameter name set in `--stdin-par`.</li><li>`adaptive`: Adaptive packaging. The YQL query runs on exceeding the limits on the number of parameter sets per query (`--batch-limit`) or on the batch processing delay (`--batch-max-delay`). All the sets of parameters received by that time are wrapped into a `List<>`, and the parameter name is set by the `--stdin-par` option. |
+| `--batch` | The batch mode applied to parameter sets on `stdin`.<br>Available options:<ul><li>`iterative` (default): Batching is [disabled](#streaming-iterate).</li><li>`full`: Full batch mode. The YQL query runs only once when `stdin` is closed, with all the received sets of parameters wrapped into `List<>`, the parameter name is set by the `--stdin-par` option.</li><li>`adaptive`: Adaptive batch mode. The YQL query runs every time when limits are exceeded either on the number of parameter sets per query (`--batch-limit`) or on the batch processing delay (`--batch-max-delay`). All the sets of parameters received by that moment are wrapped into a `List<>`, the parameter name is set by the `--stdin-par` option. |
In the adaptive batch mode, you can use the following additional parameters:
| Name | Description |
---|---
-| `--batch-limit` | A maximum number of sets of parameters per batch in the adaptive batch mode. The next batch will be sent to the SQL query if the number of datasets in it reaches the specified limit. When it's `0`, there's no limit.<br><br>The default value is `1000`.<br><br>Parameters are transmitted to YQL without streaming, and the total size per GRPC request that includes the parameter values has the upper limit of 5 MB. |
-| `--batch-max-delay` | The maximum delay related to processing the resulting parameter set in the adaptive batch mode. It's set as a number of `s`, `ms`, `m`.<br><br>Default value: `1s` (1 second).<br><br>{{ ydb-short-name }} The CLI will trigger a delay timer when it receives the first set of parameters for the batch, and will send the accumulated batch for execution once the timer expires. With this parameter, you can improve batch performance when new parameter sets arrive unpredictably to `stdin`. |
+| `--batch-limit` | The maximum number of sets of parameters per batch in the adaptive batch mode. The next batch will be sent to the YQL query if the number of parameter sets in it reaches the specified limit. When it's `0`, there's no limit.<br><br>Default value: `1000`.<br><br>Parameter values are sent to each YQL execution without streaming, so the total size per GRPC request that includes the parameter values has the upper limit of about 5 MB. |
+| `--batch-max-delay` | The maximum delay to submit a received parameter set for processing in the adaptive batch mode. It's set as a number with a time unit - `s`, `ms`, `m`.<br><br>Default value: `1s` (1 second).<br><br>The {{ ydb-short-name }} CLI starts a timer when it receives a first set of parameters for the batch on `stdin`, and sends the whole accumulated batch for execution once the timer expires. With this parameter, you can batch efficiently when new parameter sets arrival rate on `stdin` is unpredictable. |
### Examples: Full batch processing {#example-batch-full}
@@ -289,7 +289,7 @@ Command output:
#### Limiting the maximum data processing delay {#example-adaptive-delay}
-To demonstrate the adaptive batch mode with triggering of a delay limit, in the first line of the command below, we generate 1,000 rows at a delay of 0.2 seconds on `stdout` and send them through `stdin` to the YQL query execution command. The YQL query execution command shows the parameter batches in each subsequent YQL query call.
+This example demonstrates the adaptive batching triggered by a processing delay. In the first line of the command below, we generate 1,000 rows at a delay of 0.2 seconds on `stdout` and pipe them to `stdin` to the YQL query execution command. The YQL query execution command shows the parameter batches in each subsequent YQL query call.
```bash
for i in $(seq 1 1000);do echo "Line$i";sleep 0.2;done | \
@@ -323,13 +323,13 @@ Command output (actual values may differ):
^C
```
-The first batch includes all the rows accumulated at the input since the DB connection has been established, that's why it's larger than the next ones.
+The first batch includes all the rows accumulated at the input while the database connection has had been establishing, that's why it's larger than the next ones.
-You can terminate the command by Ctrl+C or wait for the 200 seconds allocated for input generation.
+You can terminate the command by Ctrl+C or wait 200 seconds until the input generation is finished.
#### Limit on the number of records {#example-adaptive-limit}
-To demonstrate the adaptive batch mode with a trigger set on the number of parameter sets, in the first line of the command below, we generate 200 rows. The command will show parameter batches in each subsequent YQL query call, applying the given limit `--batch-limit` of 20 (the default limit is 1,000).
+This example demonstrates the adaptive batching triggered by a number of parameter sets. In the first line of the command below, we generate 200 rows. The command will show parameter batches in each subsequent YQL query call, applying the given limit `--batch-limit` of 20 (the default limit is 1,000).
In this example, we also demonstrate the option to join parameters from different sources and generate JSON at the output.