diff options
author | alextarazanov <alextarazanov@yandex-team.com> | 2023-07-26 13:50:48 +0300 |
---|---|---|
committer | root <root@qavm-2ed34686.qemu> | 2023-07-26 13:50:48 +0300 |
commit | 3f972bd6eb355d1ed5981b9115d43727ee25b522 (patch) | |
tree | 49b293eed1f4b9ec8c27f670e645b34d98b85fb6 | |
parent | f9c18a4383b6afaede9f2145460f7a82be4a28c6 (diff) | |
download | ydb-3f972bd6eb355d1ed5981b9115d43727ee25b522.tar.gz |
Translation for,,,
*[YDB] Описать TPC-H workload
*[YDB] Новые параметры workload topic
*[YDB] Описание параметра metering_mode в Topic DDL
*[YDB] Обновить документацию для релиза YDB CLI 2.5.0
*Обрабатываем перевод нескольких задач YDB (Аркадия)
10 files changed, 347 insertions, 45 deletions
diff --git a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md index ac2313e308..bdb564dd5d 100644 --- a/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md +++ b/ydb/docs/en/core/reference/ydb-cli/commands/workload/_includes/index.md @@ -24,4 +24,5 @@ The following types of load tests are supported at the moment: * [Stock](../stock.md): An online store warehouse simulator. * [Key-value](../../../workload-kv.md): Key-Value load. * [ClickBench](../../../workload-click-bench.md): ClickBench analytical benchmark (https://github.com/ClickHouse/ClickBench). -<!-- * [Topic](../../../workload-topic.md): Topic load. --> +* [TPC-H](../../../workload-tpch.md): TPC-H benchmark (https://www.tpc.org/tpch/). +* [Topic](../../../workload-topic.md): Topic load.
\ No newline at end of file diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/import-file.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/import-file.md index df2fa39811..9765edd8c3 100644 --- a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/import-file.md +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/import-file.md @@ -28,14 +28,14 @@ General format of the command: ### Additional options {#optional} -* `--timeout VAL`: The time within which the operation should be completed on the server. The default value is `300s`. +* `--timeout VAL`: Time within which the operation should be completed on the server. Default: `300s`. * `--skip-rows NUM`: A number of rows from the beginning of the file that will be skipped at import. The default value is `0`. * `--header`: Use this option if the first row (excluding the rows skipped by `--skip-rows`) includes names of data columns to be mapped to table columns. If the header row is missing, the data is mapped according to the order in the table schema. * `--delimiter STRING`: The data column delimiter character. You can't use the tabulation character as a delimiter in this option. For tab-delimited import, use the `import file tsv` subcommand. Default value: `,`. * `--null-value STRING`: The value to be imported as `NULL`. Default value: `""`. * `--batch-bytes VAL`: Split the imported file into batches of specified sizes. If a row fails to fit into a batch completely, it's discarded and added to the next batch. Whatever the batch size is, the batch must include at least one row. Default value: `1 MiB`. * `--max-in-flight VAL`: The number of data batches imported in parallel. You can increase this option value to import large files faster. The default value is `100`. -* `--threads VAL`: The maximum number of threads that load data. Default value: number of logical processors. +* `--threads VAL`: Maximum number of threads used to import data. Default value: Number of logical processors. * `--columns`: List of data columns in the file delimited by a `comma` (for `csv` format) or by a tab character (for `tsv` format). If you use the `--header` option, the column names in it will be replaced by column names from the list. If the number of columns in the list mismatches the number of data columns, you will get an error. * `--newline-delimited`: This flag guarantees that there will be no line breaks in records. If this flag is set, and the data is loaded from a file, then different upload streams will process different parts of the source file. This way you can distribute the workload across all partitions, ensuring the maximum performance when uploading sorted datasets to partitioned tables. diff --git a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md index 7b4f03bf89..cffb5cf0af 100644 --- a/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md +++ b/ydb/docs/en/core/reference/ydb-cli/export_import/_includes/tools_dump.md @@ -26,6 +26,8 @@ The `tools dump` command dumps the database data and objects schema to the clien `--save-partial-result`: Don't delete the result of partial dumping. Without this option, the dumps that terminated with an error are deleted. +`--ordered`: Rows in the exported tables will be sorted by the primary key. + ## Examples {% include [ydb-cli-profile.md](../../../../_includes/ydb-cli-profile.md) %} diff --git a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml index 471da1eb40..1efedeb110 100644 --- a/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-cli/toc_i.yaml @@ -116,4 +116,6 @@ items: - name: Key-Value load href: workload-kv.md - name: Topic load - href: workload-topic.md + href: workload-topic.md + - name: TPC-H load + href: workload-tpch.md diff --git a/ydb/docs/en/core/reference/ydb-cli/workload-click-bench.md b/ydb/docs/en/core/reference/ydb-cli/workload-click-bench.md index a9c35c4b60..e364ccf8a3 100644 --- a/ydb/docs/en/core/reference/ydb-cli/workload-click-bench.md +++ b/ydb/docs/en/core/reference/ydb-cli/workload-click-bench.md @@ -25,6 +25,7 @@ See the description of the command to init the data load: | Parameter name | Parameter description | ---|--- | `--table <value>` | The table name. The default value is `clickbench/hits`. | +| `--store <value>` | Type of table storage. Acceptable values: `row`, `column`. Default value: `row`. | ## Uploading data to the table {#load} diff --git a/ydb/docs/en/core/reference/ydb-cli/workload-topic.md b/ydb/docs/en/core/reference/ydb-cli/workload-topic.md index c388ce1d33..f72c14459c 100644 --- a/ydb/docs/en/core/reference/ydb-cli/workload-topic.md +++ b/ydb/docs/en/core/reference/ydb-cli/workload-topic.md @@ -29,6 +29,7 @@ Subcommand options: | Option name | Option description | ---|--- +| `--topic` | Topic name.<br>Default value: `workload-topic`. | | `--partitions`, `-p` | Number of topic partitions.<br>Default value: `128`. | | `--consumers`, `-c` | Number of topic consumers.<br>Default value: `1`. | @@ -61,10 +62,13 @@ Subcommand options: | Option name | Option description | ---|--- -| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `10`. | +| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `60`. | | `--window`, `-w` | Statistics window in seconds.<br>Default value: `1`. | | `--quiet`, `-q` | Output only the final test result. | | `--print-timestamp` | Print the time together with the statistics of each time window. | +| `--warmup-time` | Test warm-up period (in seconds).<br>Within the period, no statistics are calculated. It's needed to eliminate the effect of transition processes at startup.<br>Default value: `5`. | +| `--percentile` | Percentile that is output in statistics.<br>Default value: `50`. | +| `--topic` | Topic name.<br>Default value: `workload-topic`. | | `--threads`, `-t` | Number of producer threads.<br>Default value: `1`. | | `--message-size`, `-m` | Message size in bytes. Use the `K`, `M`, or `G` suffix to set the size in KB, MB, or GB, respectively.<br>Default value: `10K`. | | `--message-rate` | Total target write rate in messages per second. Can't be used together with the `--byte-rate` option.<br>Default value: `0` (no limit). | @@ -81,7 +85,7 @@ Subcommand options: > > ```text > Window Write speed Write time Inflight -> # msg/s MB/s P99(ms) max msg +> # msg/s MB/s percentile,ms percentile,msg > 1 20 0 1079 72 > 2 8025 78 1415 78 > 3 7987 78 1431 79 @@ -93,13 +97,13 @@ Subcommand options: > 9 7062 69 1455 79 > 10 9912 96 3679 250 > Window Write speed Write time Inflight -> # msg/s MB/s P99(ms) max msg +> # msg/s MB/s percentile,ms percentile,msg > Total 7203 70 3023 250 > ``` > > * `Window`: Sequence number of the statistics window. > * `Write speed`: Message write rate in messages per second and MB/s. -> * `Write time`: 99th percentile of the message write time, in milliseconds. +> * `Write time`: Percentile of the message write time, in milliseconds. > * `Inflight`: Maximum number of messages awaiting commit across all partitions. ## Read load {#run-read} @@ -125,10 +129,13 @@ Subcommand options: | Option name | Option description | ---|--- -| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `10`. | +| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `60`. | | `--window`, `-w` | Statistics window in seconds.<br>Default value: `1`. | | `--quiet`, `-q` | Output only the final test result. | | `--print-timestamp` | Print the time together with the statistics of each time window. | +| `--warmup-time` | Test warm-up period (in seconds).<br>Within the period, no statistics are calculated. It's needed to eliminate the effect of transition processes at startup.<br>Default value: `5`. | +| `--percentile` | Percentile that is output in statistics.<br>Default value: `50`. | +| `--topic` | Topic name.<br>Default value: `workload-topic`. | | `--consumers`, `-c` | Number of consumers.<br>Default value: `1`. | | `--threads`, `-t` | Number of consumer threads.<br>Default value: `1`. | @@ -141,28 +148,28 @@ Subcommand options: > You will see statistics for in-progress time windows and final statistics when the test is complete: > > ```text -> Window Lag Lag time Read speed Full time -> # max msg P99(ms) msg/s MB/s P99(ms) -> 1 0 0 48193 471 0 -> 2 30176 0 66578 650 0 -> 3 30176 0 68999 674 0 -> 4 30176 0 66907 653 0 -> 5 27835 0 67628 661 0 -> 6 30176 0 67938 664 0 -> 7 30176 0 71628 700 0 -> 8 20338 0 61367 599 0 -> 9 30176 0 61770 603 0 -> 10 30176 0 58291 569 0 -> Window Lag Lag time Read speed Full time -> # max msg P99(ms) msg/s MB/s P99(ms) -> Total 30176 0 80267 784 0 +> Window Lag Lag time Read speed Full time +> # percentile,msg percentile,ms msg/s MB/s percentile,ms +> 1 0 0 0 0 0 +> 2 30176 0 66578 650 0 +> 3 30176 0 68999 674 0 +> 4 30176 0 66907 653 0 +> 5 27835 0 67628 661 0 +> 6 30176 0 67938 664 0 +> 7 30176 0 71628 700 0 +> 8 20338 0 61367 599 0 +> 9 30176 0 61770 603 0 +> 10 30176 0 58291 569 0 +> Window Lag Lag time Read speed Full time +> # percentile,msg percentile,ms msg/s MB/s percentile,ms +> Total 30176 0 80267 784 0 > ``` > > * `Window`: Sequence number of the statistics window. > * `Lag`: Maximum consumer lag in the statistics window. Messages across all partitions are included. -> * `Lag time`: 99th percentile of the message lag time in milliseconds. +> * `Lag time`: Percentile of the message lag time in milliseconds. > * `Read`: Message read rate for the consumer (in messages per second and MB/s). -> * `Full time`: 99th percentile of the full message processing time (from writing by the producer to reading by the consumer), in milliseconds. +> * `Full time`: Percentile of the full message processing time (from writing by the producer to reading by the consumer), in milliseconds. ## Read and write load {#run-full} @@ -187,10 +194,13 @@ Subcommand options: | Option name | Option description | ---|--- -| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `10`. | +| `--seconds`, `-s` | Test duration in seconds.<br>Default value: `60`. | | `--window`, `-w` | Statistics window in seconds.<br>Default value: `1`. | | `--quiet`, `-q` | Output only the final test result. | | `--print-timestamp` | Print the time together with the statistics of each time window. | +| `--warmup-time` | Test warm-up period (in seconds).<br>Within the period, no statistics are calculated. It's needed to eliminate the effect of transition processes at startup.<br>Default value: `5`. | +| `--percentile` | Percentile that is output in statistics.<br>Default value: `50`. | +| `--topic` | Topic name.<br>Default value: `workload-topic`. | | `--producer-threads`, `-p` | Number of producer threads.<br>Default value: `1`. | | `--message-size`, `-m` | Message size in bytes. Use the `K`, `M`, or `G` suffix to set the size in KB, MB, or GB, respectively.<br>Default value: `10K`. | | `--message-rate` | Total target write rate in messages per second. Can't be used together with the `--message-rate` option.<br>Default value: `0` (no limit). | @@ -208,31 +218,31 @@ Subcommand options: > You will see statistics for in-progress time windows and final statistics when the test is complete: > > ```text -> Window Write speed Write time Inflight Lag Lag time Read speed Full time -> # msg/s MB/s P99(ms) max msg max msg P99(ms) msg/s MB/s P99(ms) -> 1 39 0 1215 4 0 0 30703 300 29716 -> 2 1091 10 2143 8 2076 20607 40156 392 30941 -> 3 1552 15 2991 12 7224 21887 41040 401 31886 -> 4 1733 16 3711 15 10036 22783 38488 376 32577 -> 5 1900 18 4319 15 10668 23551 34784 340 33372 -> 6 2793 27 5247 21 9461 24575 33267 325 34893 -> 7 2904 28 6015 22 12150 25727 34423 336 35507 -> 8 2191 21 5087 21 12150 26623 29393 287 36407 -> 9 1952 19 2543 10 7627 27391 33284 325 37814 -> 10 1992 19 2655 9 10104 28671 29101 284 38797 -> Window Write speed Write time Inflight Lag Lag time Read speed Full time -> # msg/s MB/s P99(ms) max msg max msg P99(ms) msg/s MB/s P99(ms) -> Total 1814 17 5247 22 12150 28671 44827 438 40252 +> Window Write speed Write time Inflight Lag Lag time Read speed Full time +> # msg/s MB/s percentile,ms percentile,msg percentile,msg percentile,ms msg/s MB/s percentile,ms +> 1 0 0 0 0 0 0 0 0 0 +> 2 1091 10 2143 8 2076 20607 40156 392 30941 +> 3 1552 15 2991 12 7224 21887 41040 401 31886 +> 4 1733 16 3711 15 10036 22783 38488 376 32577 +> 5 1900 18 4319 15 10668 23551 34784 340 33372 +> 6 2793 27 5247 21 9461 24575 33267 325 34893 +> 7 2904 28 6015 22 12150 25727 34423 336 35507 +> 8 2191 21 5087 21 12150 26623 29393 287 36407 +> 9 1952 19 2543 10 7627 27391 33284 325 37814 +> 10 1992 19 2655 9 10104 28671 29101 284 38797 +> Window Write speed Write time Inflight Lag Lag time Read speed Full time +> # msg/s MB/s percentile,ms percentile,msg percentile,msg percentile,ms msg/s MB/s percentile,ms +> Total 1814 17 5247 22 12150 28671 44827 438 40252 > ``` > > * `Window`: Sequence number of the statistics window. > * `Write speed`: Message write rate in messages per second and MB/s. -> * `Write time`: 99th percentile of the message write time in milliseconds. +> * `Write time`: Percentile of the message write time, in milliseconds. > * `Inflight`: Maximum number of messages awaiting commit across all partitions. > * `Lag`: Maximum number of messages awaiting reading, in the statistics window. Messages across all partitions are included. -> * `Lag time`: 99th percentile of the message lag time in milliseconds. +> * `Lag time`: Percentile of the message lag time in milliseconds. > * `Read`: Message read rate for the consumer (in messages per second and MB/s). -> * `Full time`: 99th percentile of the full message processing time, from writing by the producer to reading by the consumer, in milliseconds. +> * `Full time`: Percentile of the full message processing time, from writing by the producer to reading by the consumer, in milliseconds. ## Deleting a topic {#clean} @@ -245,6 +255,12 @@ When the work is complete, you can delete the test topic: General format of the * `global options`: [Global options](commands/global-options.md). * `options`: Subcommand options. +Subcommand options: + +| Option name | Option description | +---|--- +| `--topic` | Topic name.<br>Default value: `workload-topic`. | + > To delete the `workload-topic` test topic, run the following command: > > ```bash diff --git a/ydb/docs/en/core/reference/ydb-cli/workload-tpch.md b/ydb/docs/en/core/reference/ydb-cli/workload-tpch.md new file mode 100644 index 0000000000..c79b14988d --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/workload-tpch.md @@ -0,0 +1,70 @@ +# TPC-H workload + +The workload is based on the TPC-H [documentation](https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf), and the queries and table schema are adapted for {{ ydb-short-name }}. + +The test generates a typical decision support workload. + +## Initializing a load test {#init} + +Before running the benchmark, create a table: + +```bash +{{ ydb-cli }} workload tpch init +``` + +See the description of the command to run the data load: + +```bash +{{ ydb-cli }} workload tpch init --help +``` + +### Available parameters {#init-options} + +| Option name | Option description | +---|--- +| `--path <value>` | Directory to create tables in. Default value: `""`. | +| `--store <value>` | Type of table storage. Acceptable values: `row`, `column`. Default value: `row`. | + +## Uploading data to the table {#load} + +You can download the dataset generator for the TPC-H benchmark by the [link](http://tpc.org/tpc_documents_current_versions/current_specifications5.asp). +Then follow the instructions from README. +In the `dss.h` file, you can specify a field separator: Default: `#define SEPARATOR '|'`. +In the sample data upload script, `'\t'` is used as a separator. + +```bash +for table in region nation supplier customer part partsupp orders lineitem; do + echo "Start data load to $table" + {{ ydb-cli }} import file tsv --header --path "$table" --input-file $table.tsv --newline-delimited + echo "Finish data load to $table" +done +``` + +## Running a load test {#run} + +Run the load: + +```bash +{{ ydb-cli }} workload tpch run +``` + +During this test, workload statistics for each query are displayed on the screen. + +See the description of the command to run the data load: + +```bash +{{ ydb-cli }} workload tpch run --help +``` + +### Global parameters for all types of load {#run-options} + +| Option name | Option description | +---|--- +| `--output <value>` | The name of the file in which the query execution results will be saved. The default value is `results.out`. | +| `--iterations <value>` | The number of executions of each load generating query. The default value is `1`. | +| `--json` | The name of the file in which the query execution statistics will be saved in `json` format. By default, the file is not saved. | +| `--ministat` | The name of the file in which the query execution statistics will be saved in `ministat` format. By default, the file is not saved. | +| `--query-settings` | Query execution settings. By default, not specified. | +| `--ext-queries-dir` | Name of the directory with external queries used to apply the workload. | +| `--include` | The numbers or number sections of the queries to be executed as part of the load. By default, all queries are executed. Separated by commas, for example, `1,2,4-6`. | +| `--exclude` | The numbers or number sections of the queries to be excluded as part of the load. By default, all queries are executed. Separated by commas, for example, `1,2,4-6`. | diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/alter_topic.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/alter_topic.md new file mode 100644 index 0000000000..713c846990 --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/alter_topic.md @@ -0,0 +1,140 @@ +# ALTER TOPIC + +You can use the `ALTER TOPIC` command to change the [topic](../../../../concepts/topic) settings, as well as add, update, or delete its consumers. + +{% include [trunk](../../../_includes/trunk.md) %} + +Here is the general format of the `ALTER TOPIC` command: + +```sql +ALTER TOPIC topic_path action1, action2, ..., actionN; +``` + +`action` is one of the alter actions described below. + +## Updating a set of consumers + +`ADD CONSUMER`: Adds a [consumer](../../../../concepts/topic#consumer) to a topic. + +The following example will add a consumer with default settings to the topic. + +```sql +ALTER TOPIC `my_topic` ADD CONSUMER new_consumer; +``` + +When adding consumers, you can specify their settings, for example: + +```sql +ALTER TOPIC `my_topic` ADD CONSUMER new_consumer2 WITH (important = false); +``` + +### Full list of available topic consumer settings + +* `important`: Defines an important consumer. No data will be deleted from the topic until all the important consumers read them. Value type: `boolean`, default value: `false`. +* `read_from`: Sets up the message write time starting from which the consumer will receive data. Data written before this time will not be read. Value type: `Datetime` OR `Timestamp` OR `integer` (unix-timestamp in the numeric format). Default value: `now`. + +{% if feature_topic_codecs %} +* `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the consumer. + +{% endif %} + +`DROP CONSUMER`: Deletes the consumer from the topic. + +```sql +ALTER TOPIC `my_topic` DROP CONSUMER old_consumer; +``` + +## Updating consumer settings + +`ALTER CONSUMER`: Adds a consumer for a topic. + +Here is the general syntax for `ALTER CONSUMER`: + +```sql +ALTER TOPIC `topic_name` ALTER CONSUMER consumer_name consumer_action; +``` + +Supports the following types of `consumer_action`: +* `SET`: Sets consumer settings + +{% if feature_topic_settings_reset %} +* `RESET`: Resets consumer settings to defaults. + +{% endif %} + +The following example will assign the `important` parameter to the consumer. + +```sql +ALTER TOPIC `my_topic` ALTER CONSUMER my_consumer SET (important = true); +``` + +{% if feature_topic_settings_reset %} +This example will reset `read_from` to default. + +```sql +ALTER TOPIC `my_topic` ALTER CONSUMER my_consumer RESET (read_from); +``` + +{% endif %} + +You can specify several `ALTER CONSUMER` statements for a consumer. However, the settings applied by them shouldn't +repeat. + +This is a valid statement: + +```sql +ALTER TOPIC `my_topic` + ALTER CONSUMER my_consumer SET (important = true) + ALTER CONSUMER my_consumer SET (read_from = 0); +``` + +But this statement will raise an error. + +```sql +ALTER TOPIC `my_topic` + ALTER CONSUMER my_consumer SET (important = true) + ALTER CONSUMER my_consumer SET (important = false); +``` + +## Updating topic settings + +Using the `SET (option = value[, ...])` action, you can update your topic settings. + +The example below will change the retention period for the topic and the writing quota per partition: + +```sql +ALTER TOPIC `my_topic` SET ( + retention_period = Interval('PT36H'), + partition_write_speed_bytes_per_second = 3000000 +); +``` + +{% if feature_topic_settings_reset %} + +The `RESET (option[, option2, ...])` action enables you to reset the topic settings to defaults. + +**Example** + +```sql +ALTER TOPIC `my_topic` RESET ( + min_active_partitions, + partition_count_limit +); +``` + +{% endif %} + +### Full list of available topic settings + +* `min_active_partitions`: Minimum number of topic partitions. During automatic load balancing, the number of active partitions will not decrease below this value. Value type: `integer`, default value: `1`. +* `partition_count_limit`: Maximum number of active partitions in the topic. `0` is interpreted as unlimited. Value type: `integer`, default value: `0`. +* `retention_period`: Data retention period in the topic. Value type: `Interval`, default value: `18h`. +* `retention_storage_mb`: Limit on the maximum disk space occupied by the topic data. When this value is exceeded, the older data is cleared, like under a retention policy. `0` is interpreted as unlimited. Value type: `integer`, default value: `0` +* `partition_write_speed_bytes_per_second`: Maximum allowed write speed per partition. If a write speed for a given partition exceeds this value, the write speed will be capped. Value type: `integer`, default value: `2097152` (2MB). +* `partition_write_burst_bytes`: Write quota allocated for write bursts. When set to zero, the actual write_burst value is equalled to the quota value (this allows write bursts of up to one second). Value type: `integer`, default value: `0`. +* `metering_mode`: Resource metering mode (`RESERVED_CAPACITY` - based on the allocated resources or `REQUEST_UNITS` - based on actual usage). This option applies to topics in serverless databases. Value type: `String`. + +{% if feature_topic_codecs %} +* `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the topic. Value type: `String`. + +{% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/create_topic.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/create_topic.md new file mode 100644 index 0000000000..c186138c8d --- /dev/null +++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/create_topic.md @@ -0,0 +1,68 @@ +# CREATE TOPIC + +The `CREATE TOPIC` call creates a [topic](../../../../concepts/topic). + +{% include [trunk](../../../_includes/trunk.md) %} + +When creating a topic, you can add topic [consumers](../../../../concepts/topic#consumer) to it and topic settings. + +``` + CREATE TOPIC topic_path ( + CONSUMER consumer1, + CONSUMER consumer2 WITH (setting1 = value1) + ) WITH ( + topic_setting2 = value2 + ); +``` + +All the parameters except the topic name are optional. By default, a topic is created without consumers. All +the omitted settings are also set by default (both for the topic and its consumers). + +## Examples + +Creating a topic without consumers with default settings: + +```sql +CREATE TOPIC `my_topic`; +``` + +Creating a topic with a single consumer and the important option enabled: + +```sql +CREATE TOPIC `my_topic` ( + CONSUMER my_consumer WITH (important = true) +); +``` + +### Full list of available topic consumer settings + +* `important`: Defines an important consumer. No data will be deleted from the topic until all the important consumers read them. Value type: `boolean`, default value: `false`. +* `read_from`: Sets up the message write time starting from which the consumer will receive data. Data written before this time will not be read. Value type: `Datetime` OR `Timestamp` OR `integer` (unix-timestamp in the numeric format). Default value: `now`. + +{% if feature_topic_codecs %} +* `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the consumer. + +{% endif %} + +Creating a topic with the retention period of one day: + +```sql +CREATE TOPIC `my_topic` WITH( + retention_period = Interval('P1D') +); +``` + +### Full list of available topic settings + +* `min_active_partitions`: Minimum number of topic partitions. During automatic load balancing, the number of active partitions will not decrease below this value. Value type: `integer`, default value: `1`. +* `partition_count_limit`: Maximum number of active partitions in the topic. `0` is interpreted as unlimited. Value type: `integer`, default value: `0`. +* `retention_period`: Data retention period in the topic. Value type: `Interval`, default value: `18h`. +* `retention_storage_mb`: Limit on the maximum disk space occupied by the topic data. When this value is exceeded, the older data is cleared, like under a retention policy. `0` is interpreted as unlimited. Value type: `integer`, default value: `0` +* `partition_write_speed_bytes_per_second`: Maximum allowed write speed per partition. If a write speed for a given partition exceeds this value, the write speed will be capped. Value type: `integer`, default value: `2097152` (2MB). +* `partition_write_burst_bytes`: Write quota allocated for write bursts. When set to zero, the actual write_burst value is equalled to the quota value (this allows write bursts of up to one second). Value type: `integer`, default value: `0`. +* `metering_mode`: Resource metering mode (`RESERVED_CAPACITY` - based on the allocated resources or `REQUEST_UNITS` - based on actual usage). This option applies to topics in serverless databases. Value type: `String`. + +{% if feature_topic_codecs %} +* `supported_codecs`: List of [codecs](../../../../concepts/topic#message-codec) supported by the topic. Value type: `String`. + +{% endif %} diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml index d12c52bec1..4a84dc14f8 100644 --- a/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml +++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml @@ -5,9 +5,11 @@ items: - { name: ACTION, href: action.md } - { name: ALTER GROUP, href: alter-group.md, when: feature_user_and_group } - { name: ALTER TABLE, href: alter_table.md, when: feature_map_tables } +- { name: ALTER TOPIC, href: alter_topic.md, when: feature_topic_control_plane } - { name: ALTER USER, href: alter-user.md, when: feature_user_and_group } - { name: CREATE GROUP, href: create-group.md, when: feature_user_and_group } - { name: CREATE TABLE, href: create_table.md } +- { name: CREATE TOPIC, href: create_topic.md, when: feature_topic_control_plane } - { name: CREATE USER, href: create-user.md, when: feature_user_and_group } - { name: DECLARE, href: declare.md } - { name: DELETE, href: delete.md, when: feature_map_tables } |