Added the English version of the docs for asynchronous replication (#9609)

Co-authored-by: Ilnaz Nizametdinov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

11 files changed, 376 insertions, 5 deletions

diff --git a/ydb/docs/en/core/concepts/async-replication.md b/ydb/docs/en/core/concepts/async-replication.md
new file mode 100644
index 00000000000..512ef03b743
--- /dev/null
+++ b/ydb/docs/en/core/concepts/async-replication.md
@@ -0,0 +1,142 @@
+# Asynchronous replication
+
+Asynchronous replication allows for synchronizing data between {{ ydb-short-name }} [databases](glossary.md#database) in near real time. It can also be used for data migration between databases with minimal downtime for applications interacting with these databases. Such databases can be located in the same {{ ydb-short-name }} [cluster](glossary.md#cluster) as well as in different clusters.
+
+## Overview {#how-it-works}
+
+Asynchronous replication is based on [Change Data Capture](cdc.md) and operates on logical data. The following diagram illustrates the replication process:
+
+```mermaid
+sequenceDiagram
+  participant dst as Source
+  participant src as Target
+
+  dst-->src: Initialization
+  dst->>dst: Creating an asynchronous replication instance
+  dst->>src: Creating changefeeds
+  dst->>dst: Creating replica objects
+
+  dst-->src: Initial table scan
+  loop
+    dst->>src: Request to get data
+    src->>dst: Data
+  end
+
+  dst-->src: Change data replication
+  loop
+    dst->>src: Request to get data
+    src->>dst: Data
+  end
+```
+
+As shown in the diagram above, asynchronous replication involves two databases:
+
+1. **Source**. A database with [replicated objects](glossary.md#replicated-object).
+2. **Target**. A database where an [asynchronous replication instance](glossary.md#async-replication-instance) and [replica objects](glossary.md#replica-object) will be created.
+
+Asynchronous replication consists of the following stages:
+
+* [Initialization](#init)
+* [Initial table scan](#initial-scan)
+* [Change data replication](#replication-of-changes)
+
+### Initialization {#init}
+
+Initialization of asynchronous replication includes the following steps:
+
+* Creating an asynchronous replication instance on the target database using the [CREATE ASYNC REPLICATION](../yql/reference/syntax/create-async-replication.md) YQL expression.
+* Establishing a connection with the source database. The target database connects to the source using the [connection parameters](../yql/reference/syntax/create-async-replication.md#params) specified during the creation of the asynchronous replication instance.
+
+{% note info %}
+
+The user account that is used to connect to the source database must have the following [permissions](../security/short-access-control-notation.md#access-rights):
+
+* Read permissions for schema objects and directory objects
+* Create, update, delete, and read permissions for changefeeds
+
+{% endnote %}
+
+* The following objects are created for replicated objects on the source:
+  * [changefeeds](glossary.md#changefeed) on the source
+  * [replica objects](glossary.md#replica-object) on the target
+
+{% note info %}
+
+Replicas are created under the user account that was used to create the asynchronous replication instance.
+
+{% endnote %}
+
+### Initial table scan {#initial-scan}
+
+During the [initial table scan](cdc.md#initial-scan) the source data is exported to changefeeds. The target runs [consumers](topic.md#consumer) that read the source data from the changefeeds and write it to replicas.
+
+You can get the progress of the initial table scan from the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance.
+
+### Change data replication {#replication-of-changes}
+
+After the initial table scan is completed, the consumers read the change data and write it to replicas.
+
+Each change data block has its *creation time* ($created\_at$). Consumers track the *reception time* of the change data ($received\_at$). Thus, you can use the following formula to calculate the *replication lag*:
+
+$$
+replication\_lag = received\_at - created\_at
+$$
+
+You can also get the replication lag from the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance.
+
+## Restrictions {#restrictions}
+
+* The set of replicated objects is immutable and is generated when {{ ydb-short-name }} creates an asynchronous replication instance.
+* {{ ydb-short-name }} supports the following types of replicated objects:
+
+  * [row-based tables](datamodel/table.md#row-oriented-tables)
+  * [directories](datamodel/dir.md)
+
+    {{ ydb-short-name }} will replicate all row-based tables that are located in the given directories and subdirectories at the time the asynchronous replication instance is created.
+
+* During asynchronous replication, you cannot [add or delete columns](../yql/reference/syntax/alter_table/columns.md) in the source tables.
+* During asynchronous replication, replicas are available only for reading.
+
+## Error handling during asynchronous replication {#error-handling}
+
+Possible errors during asynchronous replication can be grouped into the following classes:
+
+* **Temporary failures**, such as transport errors, system overload, etc. Requests will be resent until they are processed successfully.
+* **Critical errors**, such as access violation errors, schema errors, etc. Replication will be aborted, and the [description](../reference/ydb-cli/commands/scheme-describe.md) of the asynchronous replication instance will include the text of the error.
+
+{% note warning %}
+
+Currently, asynchronous replication that is aborted due to a critical error cannot be resumed. In this case, you must [drop](../yql/reference/syntax/drop-async-replication.md) and [create](../yql/reference/syntax/create-async-replication.md) a new asynchronous replication instance.
+
+{% endnote %}
+
+For more information about error classes and how to address them, refer to [Error handling](../reference/ydb-sdk/error_handling.md).
+
+## Asynchronous replication completion {#done}
+
+Completion of asynchronous replication might be an end goal of data migration from one database to another. In this case the client stops writing data to the source, waits for the zero replication lag, and completes replication. After the replication process is completed, replicas become available both for reading and writing. Then you can switch the load from the source database to the target database and complete data migration.
+
+{% note info %}
+
+You cannot resume completed asynchronous replication.
+
+{% endnote %}
+
+{% note warning %}
+
+{{ ydb-short-name }} currently supports only **forced** completion of asynchronous replication, when no additional checks are performed for data consistency, replication lag, etc.
+
+{% endnote %}
+
+To complete asynchronous replication, use the [ALTER ASYNC REPLICATION](../yql/reference/syntax/alter-async-replication.md) YQL expression.
+
+## Dropping an asynchronous replication instance {#drop}
+
+When you drop an asynchronous replication instance:
+
+* Changefeeds are deleted in the source tables.
+* The source tables are unlocked, and you can add and delete columns again.
+* Optionally, all replicas are deleted.
+* Asynchronous replication instance is deleted.
+
+To drop an asynchronous replication instance, use the [DROP ASYNC REPLICATION](../yql/reference/syntax/drop-async-replication.md) YQL expression.
diff --git a/ydb/docs/en/core/concepts/glossary.md b/ydb/docs/en/core/concepts/glossary.md
index f2d9ccdb525..1ab084c018c 100644
--- a/ydb/docs/en/core/concepts/glossary.md
+++ b/ydb/docs/en/core/concepts/glossary.md
@@ -181,7 +181,23 @@ A **consumer** is an entity that reads messages from a topic.
 
 ### Change data capture {#cdc}
 
-**Change data capture** or **CDC** is a mechanism that allows subscribing to a stream of changes to a given [table](#table). Technically, it is implemented on top of [topics](#topic). It is described in more detail in a separate article [{#T}](cdc.md).
+**Change data capture** or **CDC** is a mechanism that allows subscribing to a **stream of changes** to a given [table](#table). Technically, it is implemented on top of [topics](#topic). It is described in more detail in a separate article [{#T}](cdc.md).
+
+#### Changefeed {#changefeed}
+
+**Changefeed** or **stream of changes** is an ordered list of changes in a given [table](#table) published via a [topic](#topic).
+
+### Asynchronous replication instance {#async-replication-instance}
+
+**Asynchronous replication instance** is a named entity that stores [asynchronous replication](async-replication.md) settings (connection properties, a list of replicated objects, etc.) It can also be used to retrieve the status of asynchronous replication, such as the [initial synchronization process](async-replication.md#initial-scan), [replication lag](async-replication.md#replication-of-changes), [errors](async-replication.md#error-handling), and more.
+
+#### Replicated object {#replicated-object}
+
+**Replicated object** is an object, for example, a table, that is asynchronously replicated to the target database.
+
+#### Replica object {#replica-object}
+
+**Replica object** is a mirror copy of the replicated object, automatically created by an [asynchronous replication instance](#async-replication-instance). Replica objects are typically read-only.
 
 ### YQL {#yql}
 
diff --git a/ydb/docs/en/core/concepts/toc_i.yaml b/ydb/docs/en/core/concepts/toc_i.yaml
index 842c6d07bf6..21a6c3282b3 100644
--- a/ydb/docs/en/core/concepts/toc_i.yaml
+++ b/ydb/docs/en/core/concepts/toc_i.yaml
@@ -22,6 +22,7 @@ items:
     href: cluster/common_scheme_ydb.md
   - name: Disk subsystem of a cluster
     href: cluster/distributed_storage.md
+- { name: Asynchronous replication,               href: async-replication.md, when: feature_async_replication }
 - name: Federated query
   include: { path: federated_query/toc_p.yaml, mode: link }
 - name: Query optimizer
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/alter-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/alter-async-replication.md
new file mode 100644
index 00000000000..5820bd515f4
--- /dev/null
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/alter-async-replication.md
@@ -0,0 +1,35 @@
+# ALTER ASYNC REPLICATION
+
+The `ALTER ASYNC REPLICATION` statement modifies the status and parameters of an [asynchronous replication instance](../../../concepts/async-replication.md).
+
+## Syntax {#syntax}
+
+```yql
+ALTER ASYNC REPLICATION <name> SET (option = value [, ...])
+```
+
+### Parameters {#params}
+
+* `name` — a name of the asynchronous replication instance.
+* `SET (option = value [, ...])` — asynchronous replication parameters:
+
+    * `STATE` — the state of asynchronous replication. This parameter can only be used in combination with the `FAILOVER_MODE` parameter (see below). Valid values are:
+
+        * `DONE` — [completion of the asynchronous replication process](../../../concepts/async-replication.md#done).
+
+    * `FAILOVER_MODE` — the mode for changing the replication state. This parameter can only be used in combination with the `STATE` parameter. Valid values are:
+
+        * `FORCE` — forced failover.
+
+## Examples {#examples}
+
+The following statement forces the asynchronous replication process to complete:
+
+```yql
+ALTER ASYNC REPLICATION my_replication SET (STATE = "DONE", FAILOVER_MODE = "FORCE");
+```
+
+## See also
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/create-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-async-replication.md
new file mode 100644
index 00000000000..4d50b7769d7
--- /dev/null
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-async-replication.md
@@ -0,0 +1,90 @@
+# CREATE ASYNC REPLICATION
+
+The `CREATE ASYNC REPLICATION` statement creates an [asynchronous replication instance](../../../concepts/async-replication.md).
+
+## Syntax {#syntax}
+
+```yql
+CREATE ASYNC REPLICATION <name>
+FOR <remote_path> AS <local_path> [, <another_remote_path> AS <another_local_path>]
+WITH (option = value [, ...])
+```
+
+### Parameters {#params}
+
+* `name` — a name of the asynchronous replication instance.
+* `remote_path` — a relative or absolute path to a table or directory in the source database.
+* `local_path` — a relative or absolute path to a target table or directory in the local database.
+* `WITH (option = value [, ...])` — asynchronous replication parameters:
+
+
+    * `CONNECTION_STRING` — a [connection string](../../../concepts/connect.md#connection_string) for the source database (mandatory).
+    * Authentication details for the source database (mandatory) depending on the authentication method:
+
+        * [Access token](../../../recipes/ydb-sdk/auth-access-token.md):
+
+            * `TOKEN_SECRET_NAME` — the name of the [secret](../../../concepts/datamodel/secrets.md) that contains the token.
+
+        * [Login and password](../../../recipes/ydb-sdk/auth-static.md):
+
+            * `USER` — a database user name.
+            * `PASSWORD_SECRET_NAME` — the name of the [secret](../../../concepts/datamodel/secrets.md) that contains the password for the source database user.
+
+## Examples {#examples}
+
+{% note tip %}
+
+Before creating an asynchronous replication instance, you must [create](create-object-type-secret.md) a secret with authentication credentials for the source database or ensure that you have access to an existing secret.
+
+{% endnote %}
+
+The following statement creates an asynchronous replication instance to synchronize the `original_table` source table in the `/Root/another_database` database to the `replica_table` target table in the local database:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_single_table
+FOR original_table AS replica_table
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The statement above uses the token from the `my_secret` secret for authentication and the `grpcs://example.com:2135` [endpoint](../../../concepts/connect.md#endpoint) to connect to the `/Root/another_database` database.
+
+The following statement creates an asynchronous replication instance to replicate the source tables `original_table_1` and `original_table_2` to the target tables `replica_table_1` and `replica_table_2`:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_multiple_tables
+FOR original_table_1 AS replica_table_1, original_table_2 AS replica_table_2
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement creates an asynchronous replication instance for the objects in the `original_dir` directory:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_dir
+FOR original_dir AS replica_dir
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement creates an asynchronous replication instance for the objects in the `/Root/another_database` database:
+
+```yql
+CREATE ASYNC REPLICATION my_replication_for_database
+FOR `/Root/another_database` AS `/Root/my_database`
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+## See also
+
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/create-object-type-secret.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-object-type-secret.md
new file mode 100644
index 00000000000..6f2a26c1932
--- /dev/null
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/create-object-type-secret.md
@@ -0,0 +1,28 @@
+# CREATE OBJECT (TYPE SECRET)
+
+{% note warning %}
+
+The syntax for managing secrets will change in future {{ ydb-full-name }} releases.
+
+{% endnote %}
+
+The `CREATE OBJECT (TYPE SECRET)` statement creates a [secret](../../../concepts/datamodel/secrets.md).
+
+## Syntax {#syntax}
+
+```yql
+CREATE OBJECT <secret_name> (TYPE SECRET) WITH value="<secret_value>";
+```
+
+### Parameters
+
+* `secret_name` - the name of the secret.
+* `secret_value` - the contents of the secret.
+
+## Example {#examples}
+
+The following statement creates a secret named `MySecretName` with `MySecretData` as a value.
+
+```yql
+CREATE OBJECT MySecretName (TYPE SECRET) WITH value="MySecretData";
+```
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/drop-async-replication.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/drop-async-replication.md
new file mode 100644
index 00000000000..6ec774ce69a
--- /dev/null
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/drop-async-replication.md
@@ -0,0 +1,47 @@
+# DROP ASYNC REPLICATION
+
+The `DROP ASYNC REPLICATION` statement deletes an [asynchronous replication](../../../concepts/async-replication.md) instance. When an asynchronous replication instance is [deleted](../../../concepts/async-replication.md#drop), the following objects are also deleted:
+
+* automatically created [streams of changes](../../../concepts/glossary.md#changefeed)
+* [replicas](../../../concepts/glossary.md#replica-object) (optionally)
+
+## Syntax {#syntax}
+
+```yql
+DROP ASYNC REPLICATION <name> [CASCADE]
+```
+
+### Parameters
+
+* `name` — the name of the asynchronous replication instance.
+* `CASCADE` — cascaded deletion of the replicas that were created for a given asynchronous replication instance.
+
+## Examples {#examples}
+
+This section contains examples of YQL statements that drop the asynchronous replication instance created with the following expression:
+
+```yql
+CREATE ASYNC REPLICATION my_replication
+FOR original_table AS replica_table
+WITH (
+    CONNECTION_STRING = 'grpcs://example.com:2135/?database=/Root/another_database',
+    TOKEN_SECRET_NAME = 'my_secret'
+);
+```
+
+The following statement drops an asynchronous replication instance and the automatically created stream of changes for the `original_table` table, but the `replica_table` table is not deleted:
+
+```yql
+DROP ASYNC REPLICATION my_replication;
+```
+
+The following statement drops an asynchronous replication instance, the automatically created stream of changes for the `original_table` table, and the `replica_table` table:
+
+```yql
+DROP ASYNC REPLICATION my_replication CASCADE;
+```
+
+## See also
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md b/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md
index c29cec6e1fd..34b4c12c4a1 100644
--- a/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/index.md
@@ -97,3 +97,11 @@
 
 {% endif %}
 
+{% if feature_async_replication %}
+
+* [CREATE ASYNC REPLICATION](create-async-replication.md)
+* [ALTER ASYNC REPLICATION](alter-async-replication.md)
+* [DROP ASYNC REPLICATION](drop-async-replication.md)
+
+{% endif %}
+
diff --git a/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml b/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml
index 739c55a0eab..87aad7dfc8f 100644
--- a/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml
+++ b/ydb/docs/en/core/yql/reference/yql-core/syntax/toc_i.yaml
@@ -3,13 +3,16 @@ items:
 - { name: Lexical structure,            href: lexer.md }
 - { name: Expressions,                  href: expressions.md }
 - { name: ACTION,                       href: action.md }
+- { name: ALTER ASYNC REPLICATION,      href: alter-async-replication.md,    when: feature_async_replication }
 - { name: ALTER GROUP,                  href: alter-group.md,                when: feature_user_and_group }
 - { name: ALTER TABLE, include: { mode: link, path: alter_table/toc_p.yaml }, when: feature_map_tables }
 - { name: ALTER VIEW,                   href: alter-view.md,                 when: feature_view }
 - { name: ALTER TOPIC,                  href: alter_topic.md,                when: feature_topic_control_plane }
 - { name: ALTER USER,                   href: alter-user.md,                 when: feature_user_and_group }
 - { name: ANALYZE,                      href: analyze.md,                    when: backend_name == "YDB" }
+- { name: CREATE ASYNC REPLICATION,     href: create-async-replication.md,   when: feature_async_replication }
 - { name: CREATE GROUP,                 href: create-group.md,               when: feature_user_and_group }
+- { name: CREATE OBJECT TYPE SECRET,    href: create-object-type-secret.md,  when: feature_federated_queries}
 - { name: CREATE TABLE,                 include: { mode: link,               path: create_table/toc_p.yaml } }
 - { name: CREATE VIEW,                  href: create-view.md,                when: feature_view }
 - { name: CREATE TOPIC,                 href: create_topic.md,               when: feature_topic_control_plane }
@@ -18,6 +21,7 @@ items:
 - { name: DECLARE,                      href: declare.md }
 - { name: DELETE,                       href: delete.md,                     when: feature_map_tables }
 - { name: DISCARD,                      href: discard.md }
+- { name: DROP ASYNC REPLICATION,       href: drop-async-replication.md,     when: feature_async_replication }
 - { name: DROP GROUP,                   href: drop-group.md,                 when: feature_user_and_group }
 - { name: DROP TABLE,                   href: drop_table.md }
 - { name: DROP VIEW,                    href: drop-view.md,                  when: feature_view }
diff --git a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md
index 6033fc13a60..03ff06e82fb 100644
--- a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md
+++ b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-async-replication.md
@@ -6,7 +6,7 @@
 
 ```yql
 CREATE ASYNC REPLICATION <name>
-FOR remote_path AS local_path [, another_remote_path AS another_local_path]
+FOR <remote_path> AS <local_path> [, <another_remote_path> AS <another_local_path>]
 WITH (option = value [, ...])
 ```
 
diff --git a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md
index a4a6d90cdaf..a1b95aea66e 100644
--- a/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md
+++ b/ydb/docs/ru/core/yql/reference/yql-core/syntax/create-object-type-secret.md
@@ -2,14 +2,14 @@
 
 {% note warning %}
 
-Текущий систаксис работы с секретами является временным, в будущих релизах {{ydb-full-name}} он будет изменен.
+Текущий синтаксис работы с секретами является временным, в будущих релизах {{ydb-full-name}} он будет изменен.
 
 {% endnote %}
 
 Для создания [секрета](../../../concepts/datamodel/secrets.md) используется следующий SQL-запрос:
 
 ```yql
-CREATE OBJECT `secret_name` (TYPE SECRET) WITH value=`secret_value`;
+CREATE OBJECT <secret_name> (TYPE SECRET) WITH value="<secret_value>";
 ```
 
 Где:
@@ -22,5 +22,5 @@ CREATE OBJECT `secret_name` (TYPE SECRET) WITH value=`secret_value`;
 Следующий запрос создает секрет с именем `MySecretName` и значением `MySecretData`.
 
 ```yql
-CREATE OBJECT `MySecretName` (TYPE SECRET) WITH value=`MySecretData`;
+CREATE OBJECT MySecretName (TYPE SECRET) WITH value="MySecretData";
 ```