diff options
| author | github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> | 2026-07-01 06:36:14 +0000 |
|---|---|---|
| committer | GitHub <[email protected]> | 2026-07-01 06:36:14 +0000 |
| commit | 013d0f2d43328d5cd91c81a281fef74e922a03b2 (patch) | |
| tree | 2d29a4f77a9dff317fa0d9731ca662ce0ed57483 | |
| parent | d667c177a8b4ad5ed55c4fc77ec361ae33bae2a8 (diff) | |
Auto-translate docs from PR #44880 (#45101)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
27 files changed, 940 insertions, 0 deletions
diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-additional-params.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-additional-params.md new file mode 100644 index 00000000000..75c04e8f841 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-additional-params.md @@ -0,0 +1,12 @@ +- +- `--retries NUM`: Number of retry attempts for export that the server will make. Default value: `10`. +- `--compression STRING`: Compress exported data. With the default compression level for the [Zstandard](https://en.wikipedia.org/wiki/Zstd) algorithm, data can be compressed by 5-10 times. Data compression uses CPU resources and may affect the speed of other database operations. Valid values: + + - `zstd` — compression using the Zstandard algorithm with the default compression level (`3`). + - `zstd-N` — compression using the Zstandard algorithm, `N` — compression level (`1` — `22`). +- `--encryption-algorithm ALGORITHM`: Encrypt exported data using the specified algorithm. Supported values: `AES-128-GCM`, `AES-256-GCM`, `ChaCha20-Poly1305`. +- `--encryption-key-file PATH`: Path to the file containing the encryption key (only for encrypted exports). This file is binary and must contain the exact number of bytes corresponding to the key length in the selected encryption algorithm (16 bytes for `AES-128-GCM`, 32 bytes for `AES-256-GCM` and `ChaCha20-Poly1305`). The key can also be passed via the `YDB_ENCRYPTION_KEY` environment variable, in hexadecimal string representation. +- `--format STRING`: Output result format. Valid values: + + - `pretty` — human-readable format (default). + - `proto-json-base64` — [Protocol Buffers](https://en.wikipedia.org/wiki/Protocol_Buffers) in [JSON](https://en.wikipedia.org/wiki/JSON) format, binary strings encoded in [Base64](https://en.wikipedia.org/wiki/Base64). diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-alternative-syntax-warning.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-alternative-syntax-warning.md new file mode 100644 index 00000000000..a0ebd21a6bc --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-alternative-syntax-warning.md @@ -0,0 +1,5 @@ +{% note warning %} + +Exports made using alternative syntax will not contain a list of objects in the backup, so some features may be unavailable for them (in particular, encrypted backups), and import is only possible using the corresponding alternative import syntax. + +{% endnote %} diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-forget-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-forget-intro.md new file mode 100644 index 00000000000..0506edc18ba --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-forget-intro.md @@ -0,0 +1 @@ +After the export is complete, use the `operation forget` command to mark the export as finished (removing it from the list of operations): diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-list-tail.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-list-tail.md new file mode 100644 index 00000000000..5cb787c84a1 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-list-tail.md @@ -0,0 +1 @@ +The output format `operation list` is also set by the option `--format`. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-json-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-json-intro.md new file mode 100644 index 00000000000..c2dbb30bca8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-json-intro.md @@ -0,0 +1 @@ +- In the `proto-json-base64` output mode, the identifier is in the "id" attribute: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-pretty-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-pretty-intro.md new file mode 100644 index 00000000000..84a96455b09 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-result-pretty-intro.md @@ -0,0 +1 @@ +- In the `pretty` output mode (default), the operation ID is shown in the id field highlighted with pseudographics: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-after-get.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-after-get.md new file mode 100644 index 00000000000..3ec83b29814 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-after-get.md @@ -0,0 +1,24 @@ +The output format `operation get` is also set by the option `--format`. + +Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string. + +The completion of the export is tracked by the change of the "progress" attribute: + +- In the `pretty` output mode (default), a successfully completed operation is reflected by the value "Done" in the `progress` field highlighted with pseudographics: + + + ```text + ┌───── ... ──┬───────┬─────────┬──────────┬─... + | id | ready | status | progress | ... + ├──────... ──┼───────┼─────────┼──────────┼─... + | ydb://... | true | SUCCESS | Done | ... + ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴... + ... + ``` + +- In the `proto-json-base64` output mode, a completed operation is reflected by the value `PROGRESS_DONE` of the `progress` attribute: + + + ```json + {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... } + ``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-intro.md new file mode 100644 index 00000000000..373fb28fdf3 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-operation-status-intro.md @@ -0,0 +1 @@ +Data export runs in the background. You can get information about the status and progress of the export by calling the `operation get` command, which must be passed a **quoted** operation ID, for example: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-root-include-exclude-params.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-root-include-exclude-params.md new file mode 100644 index 00000000000..4e413ad21cb --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-root-include-exclude-params.md @@ -0,0 +1,5 @@ +`--root-path PATH`: Root directory for exported objects. If not specified, the root directory of the database is used. + +`--include PATH`: Data schema objects to include in the export. Directories are traversed recursively. Paths are specified relative to `root-path`. This parameter can be specified multiple times to include multiple objects. If not specified, all non-system objects in `root-path` are exported. + +`--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) to exclude paths from the export. Paths are specified relative to `root-path`. This parameter can be specified multiple times for different patterns. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-s3.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-s3.md new file mode 100644 index 00000000000..e388576a674 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-s3.md @@ -0,0 +1,221 @@ +# Export to S3-compatible storage + +The `export s3` command starts a server-side process of exporting data and schema object information to an S3-compatible storage in the format described in the [File structure](../file-structure.md) article: + + +```bash +{{ ydb-cli }} [connection options] export s3 [options] +``` + + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +{% note warning %} + +{% include [export-supported-object-types.md](export-supported-object-types.md) %} + +For a simpler export of single row and column tables to an S3-compatible data storage, you can use [external data sources](../../../../concepts/datamodel/external_data_source.md). For more details, see the [{#T}](../../../../concepts/query_execution/federated_query/s3/write_data.md#export-to-s3) article. + +{% endnote %} + +## Command-line parameters {#pars} + +`[options]` — command parameters: + +### S3 parameters {#s3-params} + +The export to S3 command requires specifying [S3 connection parameters](../auth-s3.md). Since the export is performed asynchronously by the {{ ydb-short-name }} server, the specified endpoint must be accessible for establishing a connection from the server side. + +`--destination-prefix PREFIX`: Key prefix in the S3 bucket. + +### List of exported objects {#items} + +{% include [export-root-include-exclude-params.md](export-root-include-exclude-params.md) %} + +{% cut "Alternative method" %} + +An alternative way to specify the list of objects is supported: + +`--item STRING`: Description of the export object. The `--item` parameter can be specified multiple times if you need to export several objects. `STRING` is specified in the `<property>=<value>,...` format, with the following required properties: + +- `source`, `src`, or `s` — path to the exported directory or table; `.` points to the root directory of the database. When specifying a directory, all non-system objects in it are exported, as well as all non-system subdirectories recursively. +- `destination`, `dst`, or `d` — path (key prefix) in S3 for placing the exported objects. + +`--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) to exclude paths from the export. This parameter can be specified multiple times for different patterns. + +{% include [export-alternative-syntax-warning.md](export-alternative-syntax-warning.md) %} + +{% endcut %} + +### Additional parameters {#aux} + +{% include [export-additional-params.md](export-additional-params.md) %} + +## Running the export {#exec} + +{% include [server-export-workflow.md](server-export-workflow.md) %} + +### Launch result {#result} + +Upon successful execution, the `export s3` command outputs summary information about the queued export operation to S3, in the format specified by the `--format` option. The actual export is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation: + +{% include [export-operation-result-pretty-intro.md](export-operation-result-pretty-intro.md) %} + + +```text +┌───────────────────────────────────────────┬───────┬─────... +| id | ready | stat... +├───────────────────────────────────────────┼───────┼─────... +| ydb://export/6?id=281474976788395&kind=s3 | true | SUCC... +├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... +| StorageClass: NOT_SET +| Items: +... +``` + + +{% include [export-operation-result-json-intro.md](export-operation-result-json-intro.md) %} + + +```json +{"id":"ydb://export/6?id=281474976788395&kind=s3","ready":true, ... } +``` + + +### Export status {#status} + +{% include [export-operation-status-intro.md](export-operation-status-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation get "ydb://export/6?id=281474976788395&kind=s3" +``` + + +{% include [export-operation-status-after-get.md](export-operation-status-after-get.md) %} + +### Completing the export operation {#forget} + +{% include [export-operation-forget-intro.md](export-operation-forget-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation forget "ydb://export/6?id=281474976788395&kind=s3" +``` + + +### List of export operations {#list} + +To get the list of export operations, use the `operation list export/s3` command: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/s3 +``` + + +{% include [export-operation-list-tail.md](export-operation-list-tail.md) %} + +## Examples {#examples} + +{% include [ydb-cli-profile.md](../../../../_includes/ydb-cli-profile.md) %} + +### Exporting a database {#example-full-db} + +Export all non-system database objects to the `export1` directory in the `mybucket` bucket using S3 authentication parameters from environment variables or the `~/.aws/credentials` file: + + +```bash +{{ ydb-cli }} -p quickstart export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --destination-prefix export1 +``` + + +### Exporting multiple directories {#example-specific-dirs} + +Export objects from the `dir1` and `dir2` directories of the database to the `export1` directory in the `mybucket` bucket, using explicitly specified S3 authentication parameters: + + +```bash +{{ ydb-cli }} -p quickstart export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key <access-key> --secret-key <secret-key> \ + --destination-prefix export1 --include dir1 --include dir2 +``` + + +Or using an alternative method: + + +```bash +{{ ydb-cli }} -p quickstart export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key <access-key> --secret-key <secret-key> \ + --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2 +``` + + +### Export with encryption {#example-encryption} + +Export the entire database with encryption: + +- Using the `AES-128-GCM` encryption algorithm +- Generating a random key with the `openssl` utility to the `~/my_secret_key` file +- Reading the generated key from the `~/my_secret_key` file +- To the path prefix `export1` in the S3 bucket `mybucket` +- Using S3 authentication parameters from environment variables or the `~/.aws/credentials` file + + +```bash +openssl rand -out ~/my_secret_key 16 +{{ ydb-cli }} -p quickstart export s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \ + --encryption-algorithm AES-128-GCM --encryption-key-file ~/my_secret_key +``` + + +Export the `dir1` directory of the database with encryption: + +- Using the `AES-256-GCM` encryption algorithm +- Generating a random key with the `openssl` utility to the `YDB_ENCRYPTION_KEY` environment variable +- Reading the generated key from the `YDB_ENCRYPTION_KEY` environment variable +- To the path prefix `export1` in the S3 bucket `mybucket` +- Using S3 authentication parameters from environment variables or the `~/.aws/credentials` file + + +```bash +export YDB_ENCRYPTION_KEY=$(openssl rand -hex 32) +{{ ydb-cli }} -p quickstart export s3 \ + --root-path dir1 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket --destination-prefix export1 \ + --encryption-algorithm AES-256-GCM +``` + + +### Getting operation IDs {#example-list-oneline} + +To get a list of export operation IDs in a format convenient for processing in bash scripts, you can use the [jq](https://stedolan.github.io/jq/download/) utility: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" +``` + + +You will get output where each new line contains an operation ID, for example: + + +```text +ydb://export/6?id=281474976789577&kind=s3 +ydb://export/6?id=281474976789526&kind=s3 +ydb://export/6?id=281474976788779&kind=s3 +``` + + +These IDs can be used, for example, to run a loop to complete all current operations: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p quickstart operation forget $line;done +``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-supported-object-types.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-supported-object-types.md new file mode 100644 index 00000000000..91444501cb6 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/export-supported-object-types.md @@ -0,0 +1,13 @@ +Export is available only for the following object types: + +- [directory](../../../../concepts/datamodel/dir.md). +- [row table](../../../../concepts/datamodel/table.md#row-oriented-tables). +- [secondary index](../../../../concepts/glossary.md#secondary-index). +- [vector index](../../../../concepts/glossary.md#vector-index). +- [full-text index](../../../../concepts/glossary.md#fulltext-index). +- [topic](../../../../concepts/datamodel/topic.md) (schema only). +- [view](../../../../concepts/datamodel/view.md). +- [asynchronous replication](../../../../concepts/async-replication.md). +- [transfer](../../../../concepts/transfer.md). +- [external data source](../../../../concepts/datamodel/external_data_source.md). +- [external table](../../../../concepts/datamodel/external_table.md). diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-additional-params.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-additional-params.md new file mode 100644 index 00000000000..ecc4eccfacc --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-additional-params.md @@ -0,0 +1,8 @@ +- +- `--retries NUM`: Number of upload retries the server will attempt. Default value: `10`. +- `--skip-checksum-validation`: Skip the validation stage of [checksums](../file-structure.md#checksums) of the uploaded objects. +- `--encryption-key-file PATH`: Path to the file containing the encryption key (only for encrypted exports). This file is binary and must contain the exact number of bytes corresponding to the key length in the selected encryption algorithm (16 bytes for `AES-128-GCM`, 32 bytes for `AES-256-GCM` and `ChaCha20-Poly1305`). The key can also be passed via the `YDB_ENCRYPTION_KEY` environment variable, in hexadecimal string representation. +- `--format STRING`: Output result format. Valid values: + + - `pretty` — human-readable format (default). + - `proto-json-base64` — [Protocol Buffers](https://en.wikipedia.org/wiki/Protocol_Buffers) in [JSON](https://en.wikipedia.org/wiki/JSON) format, binary strings encoded in [Base64](https://en.wikipedia.org/wiki/Base64). diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax-warning.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax-warning.md new file mode 100644 index 00000000000..61ea6e6479e --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax-warning.md @@ -0,0 +1 @@ +Some features may be unavailable when using alternative syntax (in particular, encrypted backups or listing of export objects). diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax.md new file mode 100644 index 00000000000..98f36d506b3 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-alternative-syntax.md @@ -0,0 +1,3 @@ +For backward compatibility, an alternative way to specify the list of objects is supported: + +`--item STRING`: Description of the object to import. The `--item` parameter can be specified multiple times if you need to import several objects. If the `--item` or `--include` parameters are not specified, all objects present in the specified export will be imported. `STRING` is specified in the `<property>=<value>,...` format with the following required properties: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-objects-params.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-objects-params.md new file mode 100644 index 00000000000..bf071181135 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-objects-params.md @@ -0,0 +1,5 @@ +`--destination-path PATH`: Target directory for imported objects; the default value is the database root. + +`--include PATH`: Data schema objects to include in the import. Directories are traversed recursively. To include multiple objects, the parameter can be specified multiple times. If not specified, all exported objects are loaded. + +`--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) for excluding paths from the import. This parameter can be specified multiple times for different patterns. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-forget-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-forget-intro.md new file mode 100644 index 00000000000..5428470a6af --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-forget-intro.md @@ -0,0 +1 @@ +After the import is complete, use the `operation forget` command to remove it from the list of operations: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-list-tail.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-list-tail.md new file mode 100644 index 00000000000..5cb787c84a1 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-list-tail.md @@ -0,0 +1 @@ +The output format `operation list` is also set by the option `--format`. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-json-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-json-intro.md new file mode 100644 index 00000000000..c2dbb30bca8 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-json-intro.md @@ -0,0 +1 @@ +- In the `proto-json-base64` output mode, the identifier is in the "id" attribute: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-pretty-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-pretty-intro.md new file mode 100644 index 00000000000..84a96455b09 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-result-pretty-intro.md @@ -0,0 +1 @@ +- In the `pretty` output mode (default), the operation ID is shown in the id field highlighted with pseudographics: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-after-get.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-after-get.md new file mode 100644 index 00000000000..562407ecef6 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-after-get.md @@ -0,0 +1,24 @@ +The output format `operation get` is also set by the option `--format`. + +Although the operation ID is in URL format, it is not guaranteed to be preserved in the future. It should be interpreted only as a string. + +Completion of the upload is tracked by the change of the "progress" attribute: + +- In the `pretty` output mode (default), a successfully completed operation is reflected by the value "Done" in the `progress` field highlighted with pseudographics: + + + ```text + ┌───── ... ──┬───────┬─────────┬──────────┬─... + | id | ready | status | progress | ... + ├──────... ──┼───────┼─────────┼──────────┼─... + | ydb://... | true | SUCCESS | Done | ... + ├╴╴╴╴╴ ... ╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴╴╴╴┴╴... + ... + ``` + +- In the `proto-json-base64` output mode, a completed operation is reflected by the value `PROGRESS_DONE` of the `progress` attribute: + + + ```json + {"id":"ydb://...", ...,"progress":"PROGRESS_DONE",... } + ``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-intro.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-intro.md new file mode 100644 index 00000000000..f1805b883c7 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-operation-status-intro.md @@ -0,0 +1 @@ +Data loading is performed in the background. You can get information about the status and progress of the loading by calling the `operation get` command, which must be passed a **quoted** operation ID, for example: diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-s3.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-s3.md new file mode 100644 index 00000000000..3e23f0b741b --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/import-s3.md @@ -0,0 +1,203 @@ +# Importing from S3-compatible storage + +The `import s3` command starts the process of importing data and schema object information from an S3-compatible storage on the server side, in the format described in the article [File structure](../file-structure.md): + + +```bash +{{ ydb-cli }} [connection options] import s3 [options] +``` + + +{% note info %} + +Importing tables from an S3-compatible storage in other formats is possible using [external tables](../../../../concepts/query_execution/federated_query/s3/external_table.md); for more details, see the article [{#T}](../../../../concepts/query_execution/federated_query/import_and_export.md#import). + +{% endnote %} + +{% include [conn_options_ref.md](../../commands/_includes/conn_options_ref.md) %} + +Unlike the [`tools restore` command](../tools-restore.md), the `import s3` command always creates objects entirely, so for it to succeed, none of the imported objects (neither directories nor tables) should exist. + +If you need to load additional data into existing tables from S3, you can copy the S3 contents to the file system (for example, using [S3cmd](https://s3tools.org/s3cmd)) and use the [`tools restore` command](../tools-restore.md). + +## Command-line parameters {#pars} + +`[options]` — command parameters: + +### S3 parameters {#s3-params} + +The S3 import command requires specifying [S3 connection parameters](../auth-s3.md). Since the import is performed asynchronously by the {{ ydb-short-name }} server, the specified endpoint must be accessible for establishing a connection from the server side. + +`--source-prefix PREFIX`: Import prefix in the S3 bucket. + +### Imported database schema objects {#objects} + +{% include [import-objects-params.md](./import-objects-params.md) %} + +{% cut "Alternative method" %} + +{% include [import-alternative-syntax.md](./import-alternative-syntax.md) %} + +- `source`, `src`, or `s` — the S3 key prefix with the imported directory or table. +- `destination`, `dst`, or `d` — the path in the database for placing the imported directory or table. The final path element must not exist. All directories along the path will be created if they do not exist. + +{% include [import-alternative-syntax-warning.md](./import-alternative-syntax-warning.md) %} + +{% endcut %} + +### Additional parameters {#aux} + +{% include [import-additional-params.md](import-additional-params.md) %} + +- `--list`: List objects in an existing export. + +## Running the import {#exec} + +{% include [server-import-workflow.md](server-import-workflow.md) %} + +### Launch result {#result} + +Upon successful execution, the `import s3` command outputs summary information about the queued S3 import operation, in the format specified by the `--format` option. The actual import is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation: + +{% include [import-operation-result-pretty-intro.md](import-operation-result-pretty-intro.md) %} + + +```text +┌───────────────────────────────────────────┬───────┬─────... +| id | ready | stat... +├───────────────────────────────────────────┼───────┼─────... +| ydb://import/8?id=281474976788395&kind=s3 | true | SUCC... +├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... +| Items: +... +``` + + +{% include [import-operation-result-json-intro.md](import-operation-result-json-intro.md) %} + + +```json +{"id":"ydb://import/8?id=281474976788395&kind=s3","ready":true, ... } +``` + + +### Import status {#status} + +{% include [import-operation-status-intro.md](import-operation-status-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation get "ydb://import/8?id=281474976788395&kind=s3" +``` + + +{% include [import-operation-status-after-get.md](import-operation-status-after-get.md) %} + +### Completing the import operation {#forget} + +{% include [import-operation-forget-intro.md](import-operation-forget-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation forget "ydb://import/8?id=281474976788395&kind=s3" +``` + + +### List of import operations {#list} + +To get a list of import operations, use the `operation list import/s3` command: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/s3 +``` + + +{% include [import-operation-list-tail.md](import-operation-list-tail.md) %} + +## Examples {#examples} + +{% include [ydb-cli-profile.md](../../../../_includes/ydb-cli-profile.md) %} + +### Importing to the database root {#example-full-db} + +Importing the contents of the `export1` directory in the `mybucket` bucket to the database root, using S3 authentication parameters from environment variables or the `~/.aws/credentials` file: + + +```bash +{{ ydb-cli }} -p quickstart import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --source-prefix export1 +``` + + +### Importing multiple directories {#example-specific-dirs} + +Importing objects from the `dir1` and `dir2` directories of the export located in the `export1` directory in the `mybucket` bucket, into the identically named directories of the database, using explicitly specified S3 authentication parameters: + + +```bash +{{ ydb-cli }} -p quickstart import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key <access-key> --secret-key <secret-key> \ + --source-prefix export1 + --include dir1 --include dir2 +``` + + +### Listing objects in an existing encrypted export {#example-list} + +Listing the paths of all objects in an existing encrypted export located in the `export1` directory in the `mybucket` bucket, using the secret key from the `~/my_secret_key` file. + + +```bash +{{ ydb-cli }} -p quickstart import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key <access-key> --secret-key <secret-key> \ + --source-prefix export1 + --encryption-key-file ~/my_secret_key + --list +``` + + +### Importing an encrypted export {#example-encryption} + +Importing a single table that was exported along the `dir/my_table` path, to the `dir1/dir/my_table` path, from an encrypted export located under the `export1` prefix in the `mybucket` bucket, using the secret key from the `~/my_secret_key` file. + + +```bash +{{ ydb-cli }} -p quickstart import s3 \ + --s3-endpoint storage.yandexcloud.net --bucket mybucket \ + --access-key <access-key> --secret-key <secret-key> \ + --source-prefix export1 --destination-path dir1 \ + --include dir/my_table \ + --encryption-key-file ~/my_secret_key +``` + + +### Getting operation IDs {#example-list-oneline} + +To get a list of import operation IDs in a format convenient for processing in bash scripts, you can use the [jq](https://stedolan.github.io/jq/download/) utility: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/s3 --format proto-json-base64 | jq -r ".operations[].id" +``` + + +You will get output where each new line contains an operation ID, for example: + + +```text +ydb://import/8?id=281474976789577&kind=s3 +ydb://import/8?id=281474976789526&kind=s3 +ydb://import/8?id=281474976788779&kind=s3 +``` + + +These IDs can be used, for example, to run a loop to complete all current operations: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/s3 --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p quickstart operation forget $line;done +``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-export-workflow.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-export-workflow.md new file mode 100644 index 00000000000..24c0f3f4d50 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-export-workflow.md @@ -0,0 +1,7 @@ +### Server export operation flow {#server-export-workflow} + +1. A server-side asynchronous export operation is created. +2. A service directory `export-{id}` is created in the root directory of the database, where `{id}` is the numeric operation ID. +3. A consistent copy of the tables is created in this directory using the `CopyTables` mechanism. +4. The data of each table is written from the copy to the storage in the [export file structure](../file-structure.md) format, with parallel writing from different nodes of the cluster. +5. After successful completion, the service directory `export-{id}` and the table copies are deleted. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-import-workflow.md b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-import-workflow.md new file mode 100644 index 00000000000..4f56a4736a9 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/_includes/server-import-workflow.md @@ -0,0 +1,6 @@ +### Progress of server import operation {#server-import-workflow} + +1. A server asynchronous import operation is created. +2. The server reads object metadata from files (`scheme.pb`, `metadata.json`, etc.) from the storage. +3. For each object, a new table is created in the database. Target paths **must not exist** — import cannot overwrite existing tables. +4. Data is loaded from files in parallel on all nodes of the cluster. diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/export-nfs.md b/ydb/docs/en/core/reference/ydb-cli/export-import/export-nfs.md new file mode 100644 index 00000000000..1c731160a41 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/export-nfs.md @@ -0,0 +1,214 @@ +# Export to NFS + +The `export nfs` command starts a server-side process of exporting data and schema object information to the network file system ( [Network File System](https://en.wikipedia.org/wiki/Network_File_System), NFS) of the {{ ydb-short-name }} cluster hosts, in the format described in the [File structure](./file-structure.md) article: + + +```bash +{{ ydb-cli }} [connection options] export nfs [options] +``` + + +{% include [conn_options_ref.md](../commands/_includes/conn_options_ref.md) %} + +{% note warning %} + +{% include [export-supported-object-types.md](_includes/export-supported-object-types.md) %} + +{% endnote %} + +## Command-line parameters {#pars} + +`[options]` — command parameters: + +### NFS parameters {#nfs-params} + +The NFS export command requires specifying a mounted directory (or subdirectory) common to all objects involved in the export. Since the export is performed asynchronously on all {{ ydb-short-name }} hosts, the specified directory must exist on each {{ ydb-short-name }} host and be mounted in NFS. + +`--fs-path PATH`: path to the mounted directory (or subdirectory). + +### List of exported objects {#items} + +{% include [export-root-include-exclude-params.md](_includes/export-root-include-exclude-params.md) %} + +{% cut "Alternative method" %} + +An alternative way to specify the list of objects is supported: + +`--item STRING`: Description of the export object. The `--item` parameter can be specified multiple times if you need to export several objects. `STRING` is specified in the `<property>=<value>,...` format, with the following required properties: + +- `source`, `src`, or `s` — path to the exported directory or table; `.` points to the root directory of the database. When specifying a directory, all non-system objects in it are exported, as well as all non-system subdirectories recursively. +- `destination`, `dst`, or `d` — path in NFS (relative to `--fs-path`). + +`--exclude STRING`: Pattern ( [PCRE](https://www.pcre.org/original/doc/html/pcrepattern.html)) to exclude paths from the export. This parameter can be specified multiple times for different patterns. + +{% include [export-alternative-syntax-warning.md](_includes/export-alternative-syntax-warning.md) %} + +{% endcut %} + +### Additional parameters {#aux} + +{% include [export-additional-params.md](_includes/export-additional-params.md) %} + +## Running the export {#exec} + +{% include [server-export-workflow.md](_includes/server-export-workflow.md) %} + +### Launch result {#result} + +On successful execution, the `export nfs` command outputs summary information about the queued NFS export operation, in the format specified by the `--format` option. The actual export is performed asynchronously by the server. The summary information includes the operation ID, which can be used later to check the status and perform actions on the operation: + +{% include [export-operation-result-pretty-intro.md](_includes/export-operation-result-pretty-intro.md) %} + + +```text +┌───────────────────────────────────────────┬───────┬─────... +| id | ready | stat... +├───────────────────────────────────────────┼───────┼─────... +| ydb://export/6?id=281474976788395&kind=fs | true | SUCC... +├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... +| Include index data: false +| Items: +... +``` + + +{% include [export-operation-result-json-intro.md](_includes/export-operation-result-json-intro.md) %} + + +```json +{"id":"ydb://export/6?id=281474976788395&kind=fs","ready":true, ... } +``` + + +### Export status {#status} + +{% include [export-operation-status-intro.md](_includes/export-operation-status-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation get "ydb://export/6?id=281474976788395&kind=fs" +``` + + +{% include [export-operation-status-after-get.md](_includes/export-operation-status-after-get.md) %} + +### Completing the export operation {#forget} + +{% include [export-operation-forget-intro.md](_includes/export-operation-forget-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation forget "ydb://export/6?id=281474976788395&kind=fs" +``` + + +### List of export operations {#list} + +To get the list of export operations, use the `operation list export/nfs` command: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/nfs +``` + + +{% include [export-operation-list-tail.md](_includes/export-operation-list-tail.md) %} + +## Examples {#examples} + +{% include [ydb-cli-profile.md](../../../_includes/ydb-cli-profile.md) %} + +### Exporting a database {#example-full-db} + +Export all non-system database objects to the `/mnt/nfs/backups/export1` directory on the file system: + + +```bash +{{ ydb-cli }} -p quickstart export nfs \ + --fs-path /mnt/nfs/backups/export1 +``` + + +### Exporting multiple directories {#example-specific-dirs} + +Export objects from the `dir1` and `dir2` directories of the database to the `/mnt/nfs/backups/export1` directory on the file system: + + +```bash +{{ ydb-cli }} -p quickstart export nfs \ + --fs-path /mnt/nfs/backups/export1 \ + --include dir1 --include dir2 +``` + + +Or using an alternative method: + + +```bash +{{ ydb-cli }} -p quickstart export nfs \ + --fs-path /mnt/nfs/backups \ + --item src=dir1,dst=export1/dir1 --item src=dir2,dst=export1/dir2 +``` + + +### Export with encryption {#example-encryption} + +Export the entire database with encryption: + +- Using the `AES-128-GCM` encryption algorithm +- Generating a random key with the `openssl` utility to the `~/my_secret_key` file +- Reading the generated key from the `~/my_secret_key` file +- To the `/mnt/nfs/backups/export1` directory on the file system + + +```bash +openssl rand -out ~/my_secret_key 16 +{{ ydb-cli }} -p quickstart export nfs \ + --fs-path /mnt/nfs/backups/export1 \ + --encryption-algorithm AES-128-GCM --encryption-key-file ~/my_secret_key +``` + + +Export the `dir1` directory of the database with encryption: + +- Using the `AES-256-GCM` encryption algorithm +- Generating a random key with the `openssl` utility to the `YDB_ENCRYPTION_KEY` environment variable +- Reading the generated key from the `YDB_ENCRYPTION_KEY` environment variable +- To the `/mnt/nfs/backups/export1` directory on the file system + + +```bash +export YDB_ENCRYPTION_KEY=$(openssl rand -hex 32) +{{ ydb-cli }} -p quickstart export nfs \ + --root-path dir1 \ + --fs-path /mnt/nfs/backups/export1 \ + --encryption-algorithm AES-256-GCM +``` + + +### Getting operation IDs {#example-list-oneline} + +To get a list of export operation IDs in a format convenient for processing in bash scripts, you can use the [jq](https://stedolan.github.io/jq/download/) utility: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/nfs --format proto-json-base64 | jq -r ".operations[].id" +``` + + +You will get output where each new line contains an operation ID, for example: + + +```text +ydb://export/6?id=281474976789577&kind=fs +ydb://export/6?id=281474976789526&kind=fs +ydb://export/6?id=281474976788779&kind=fs +``` + + +These IDs can be used, for example, to run a loop to complete all current operations: + + +```bash +{{ ydb-cli }} -p quickstart operation list export/nfs --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p quickstart operation forget $line;done +``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/import-nfs.md b/ydb/docs/en/core/reference/ydb-cli/export-import/import-nfs.md new file mode 100644 index 00000000000..dcb7db47893 --- /dev/null +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/import-nfs.md @@ -0,0 +1,175 @@ +# Import from NFS + +The `import nfs` command starts a server-side process of importing data and schema object information from the network file system ( [Network File System](https://en.wikipedia.org/wiki/Network_File_System), NFS) of the {{ ydb-short-name }} cluster hosts, in the format described in the [File structure](./file-structure.md) article: + + +```bash +{{ ydb-cli }} [connection options] import nfs [options] +``` + + +{% include [conn_options_ref.md](../commands/_includes/conn_options_ref.md) %} + +Unlike the [`tools restore` command](./tools-restore.md), the `import nfs` command always creates objects entirely, so for it to succeed, none of the imported objects (neither directories nor tables) should exist. + +If you need to additionally load data into existing tables, use the [`tools restore` command](./tools-restore.md) directly on the mounted NFS directory. + +## Command-line parameters {#pars} + +`[options]` — command parameters: + +### NFS parameters {#nfs-params} + +The import from NFS command requires specifying a mounted directory (or subdirectory) common to all objects involved in the import. Since the import is performed asynchronously on all {{ ydb-short-name }} hosts, the specified directory must exist on each {{ ydb-short-name }} host and be mounted in NFS. + +`--fs-path PATH`: path to the mounted directory (or subdirectory). + +### Imported database schema objects {#objects} + +{% include [import-objects-params.md](_includes/import-objects-params.md) %} + +{% cut "Alternative method" %} + +{% include [import-alternative-syntax.md](_includes/import-alternative-syntax.md) %} + +- `source`, `src`, or `s` — path in NFS (relative to `fs-path`) with the imported directory or table. +- `destination`, `dst`, or `d` — the path in the database for placing the imported directory or table. The final path element must not exist. All directories along the path will be created if they do not exist. + +{% include [import-alternative-syntax-warning.md](_includes/import-alternative-syntax-warning.md) %} + +{% endcut %} + +### Additional parameters {#aux} + +{% include [import-additional-params.md](_includes/import-additional-params.md) %} + +## Running the import {#exec} + +{% include [server-import-workflow.md](_includes/server-import-workflow.md) %} + +### Launch result {#result} + +Upon successful execution, the `import nfs` command outputs summary information about the queued import from NFS operation, in the format specified by the `--format` option. The actual import is performed asynchronously by the server. The summary information includes the operation ID, which can later be used to check the status and perform actions on the operation: + +{% include [import-operation-result-pretty-intro.md](_includes/import-operation-result-pretty-intro.md) %} + + +```text +┌───────────────────────────────────────────┬───────┬─────... +| id | ready | stat... +├───────────────────────────────────────────┼───────┼─────... +| ydb://import/8?id=281474976788395&kind=fs | true | SUCC... +├╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴╴┴╴╴╴╴╴╴╴┴╴╴╴╴╴... +| Items: +... +``` + + +{% include [import-operation-result-json-intro.md](_includes/import-operation-result-json-intro.md) %} + + +```json +{"id":"ydb://import/8?id=281474976788395&kind=fs","ready":true, ... } +``` + + +### Import status {#status} + +{% include [import-operation-status-intro.md](_includes/import-operation-status-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation get "ydb://import/8?id=281474976788395&kind=fs" +``` + + +{% include [import-operation-status-after-get.md](_includes/import-operation-status-after-get.md) %} + +### Completing the import operation {#forget} + +{% include [import-operation-forget-intro.md](_includes/import-operation-forget-intro.md) %} + + +```bash +{{ ydb-cli }} -p quickstart operation forget "ydb://import/8?id=281474976788395&kind=fs" +``` + + +### List of import operations {#list} + +To get a list of import operations, use the `operation list import/nfs` command: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/nfs +``` + + +{% include [import-operation-list-tail.md](_includes/import-operation-list-tail.md) %} + +## Examples {#examples} + +{% include [ydb-cli-profile.md](../../../_includes/ydb-cli-profile.md) %} + +### Importing to the database root {#example-full-db} + +Importing the contents of the `/mnt/nfs/backups/export1` directory on the file system into the database root: + + +```bash +{{ ydb-cli }} -p quickstart import nfs \ + --fs-path /mnt/nfs/backups/export1 +``` + + +### Importing multiple directories {#example-specific-dirs} + +Importing objects from the `dir1` and `dir2` directories of the export located in `/mnt/nfs/backups/export1` on the file system into the identically named database directories: + + +```bash +{{ ydb-cli }} -p quickstart import nfs \ + --fs-path /mnt/nfs/backups/export1 \ + --include dir1 --include dir2 +``` + + +### Importing an encrypted export {#example-encryption} + +Importing a single table that was exported at path `dir/my_table` into the path `dir1/dir/my_table` from an encrypted export located in `/mnt/nfs/backups/export1` on the file system, using a secret key from the `~/my_secret_key` file. + + +```bash +{{ ydb-cli }} -p quickstart import nfs \ + --fs-path /mnt/nfs/backups/export1 --destination-path dir1 \ + --include dir/my_table \ + --encryption-key-file ~/my_secret_key +``` + + +### Getting operation IDs {#example-list-oneline} + +To get a list of import operation IDs in a format convenient for processing in bash scripts, you can use the [jq](https://stedolan.github.io/jq/download/) utility: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id" +``` + + +You will get output where each new line contains an operation ID, for example: + + +```text +ydb://import/8?id=281474976789577&kind=fs +ydb://import/8?id=281474976789526&kind=fs +ydb://import/8?id=281474976788779&kind=fs +``` + + +These IDs can be used, for example, to run a loop to complete all current operations: + + +```bash +{{ ydb-cli }} -p quickstart operation list import/nfs --format proto-json-base64 | jq -r ".operations[].id" | while read line; do {{ ydb-cli }} -p quickstart operation forget $line;done +``` diff --git a/ydb/docs/en/core/reference/ydb-cli/export-import/toc_i.yaml b/ydb/docs/en/core/reference/ydb-cli/export-import/toc_i.yaml index 5a911f313c3..04c63ed2e30 100644 --- a/ydb/docs/en/core/reference/ydb-cli/export-import/toc_i.yaml +++ b/ydb/docs/en/core/reference/ydb-cli/export-import/toc_i.yaml @@ -11,6 +11,10 @@ items: href: export-s3.md - name: Importing data from S3 href: import-s3.md +- name: Export to NFS + href: export-nfs.md +- name: Import from NFS + href: import-nfs.md - name: Importing data from a file to an existing table href: import-file.md - name: Backup collections |
