aboutsummaryrefslogtreecommitdiffstats
diff options
context:
space:
mode:
authoralextarazanov <alextarazanov@yandex-team.com>2023-05-05 15:54:49 +0300
committeralextarazanov <alextarazanov@yandex-team.com>2023-05-05 15:54:49 +0300
commitfa138a9ac750b5a8717395ca0b76e06366b864ec (patch)
tree72df7e670cf35383021eb5f37b7e5af724dce95b
parent2dc4553f10c81e38f864d4452ae30a2d00f57dbc (diff)
downloadydb-fa138a9ac750b5a8717395ca0b76e06366b864ec.tar.gz
translation
Лог локальной сборки без ошибок.
-rw-r--r--ydb/docs/en/core/reference/ydb-sdk/feature-parity.md10
-rw-r--r--ydb/docs/en/core/reference/ydb-sdk/topic.md234
2 files changed, 210 insertions, 34 deletions
diff --git a/ydb/docs/en/core/reference/ydb-sdk/feature-parity.md b/ydb/docs/en/core/reference/ydb-sdk/feature-parity.md
index 3bb8f23d54..f72c8ac9e7 100644
--- a/ydb/docs/en/core/reference/ydb-sdk/feature-parity.md
+++ b/ydb/docs/en/core/reference/ydb-sdk/feature-parity.md
@@ -102,12 +102,12 @@
| DescribeNode | \+ | ? | \+ | \- | \- | \- |
| Session (leader election, distributed lock) | \+ | ? | \- | \- | \- | \- |
| **Topic service** |
-| CreateTopic | \+ | \- | \+ | \- | \- | \- | \- |
-| DescribeTopic | \+ | \- | \+ | \- | \- | \- | \- |
+| CreateTopic | \+ | \+ | \+ | \- | \- | \- | \- |
+| DescribeTopic | \+ | \+ | \+ | \- | \- | \- | \- |
| AlterTopic | \+ | \- | \+ | \- | \- | \- | \- |
-| DropTopic | \+ | \- | \+ | \- | \- | \- | \- |
-| StreamWrite | \+ | \- | \+ | \- | \- | \- | \- |
-| StreamRead | \+ | \- | \+ | \- | \- | \- | \- |
+| DropTopic | \+ | \+ | \+ | \- | \- | \- | \- |
+| StreamWrite | \+ | \+ | \+ | \- | \- | \- | \- |
+| StreamRead | \+ | \+ | \+ | \- | \- | \- | \- |
| **Ratelimiter service** |
| CreateResource | \+ | ? | \+ | \- | \- | \- | \- |
| AlterResource | \+ | ? | \+ | \- | \- | \- | \- |
diff --git a/ydb/docs/en/core/reference/ydb-sdk/topic.md b/ydb/docs/en/core/reference/ydb-sdk/topic.md
index 7f5d642820..7f5e678ff7 100644
--- a/ydb/docs/en/core/reference/ydb-sdk/topic.md
+++ b/ydb/docs/en/core/reference/ydb-sdk/topic.md
@@ -4,8 +4,9 @@ This article provides examples of how to use the {{ ydb-short-name }} SDK to wor
Before performing the examples, [create a topic](../ydb-cli/topic-create.md) and [add a consumer](../ydb-cli/topic-consumer-add.md).
-## Managing topics
-### Creating a topic
+## Managing topics {#manage}
+
+### Creating a topic {#create-topic}
{% list tabs %}
@@ -16,23 +17,34 @@ The only mandatory parameter for creating a topic is its path, other parameters
For a full list of supported parameters, see the [SDK documentation](https://pkg.go.dev/github.com/ydb-platform/ydb-go-sdk/v3/topic/topicoptions#CreateOption).
Example of creating a topic with a list of supported codecs and a minimum number of partitions
+
```go
err := db.Topic().Create(ctx, "topic-path",
// optional
- topicoptions.CreateWithSupportedCodecs(topictypes.CodecRaw, topictypes.CodecGzip),
+ topicoptions.CreateWithSupportedCodecs(topictypes.CodecRaw, topictypes.CodecGzip),
+
+ // optional
+ topicoptions.CreateWithMinActivePartitions(3),
+ )
+ ```
- // optional
- topicoptions.CreateWithMinActivePartitions(3),
+- Python
+
+ ```python
+ driver.topic_client.create_topic(topic_path,
+ supported_codecs=[ydb.TopicCodec.RAW, ydb.TopicCodec.GZIP], # optional
+ min_active_partitions=3, # optional
)
```
{% endlist %}
-### Updating a topic
+### Updating a topic {#alter-topic}
When you update a topic, you must specify the topic path and the parameters to be changed.
{% list tabs %}
+
- Go
For a full list of supported parameters, see the [SDK documentation](https://pkg.go.dev/github.com/ydb-platform/ydb-go-sdk/v3/topic/topicoptions#AlterOption).
@@ -40,16 +52,21 @@ When you update a topic, you must specify the topic path and the parameters to b
Example of adding a consumer to a topic
```go
- err := db.Topic().Alter(ctx, "topic-path",
- topicoptions.AlterWithAddConsumers(topictypes.Consumer{
- Name: "new-consumer",
- SupportedCodecs: []topictypes.Codec{topictypes.CodecRaw, topictypes.CodecGzip}, // optional
- }),
- )
+ err := db.Topic().Alter(ctx, "topic-path",
+ topicoptions.AlterWithAddConsumers(topictypes.Consumer{
+ Name: "new-consumer",
+ SupportedCodecs: []topictypes.Codec{topictypes.CodecRaw, topictypes.CodecGzip}, // optional
+ }),
+ )
```
+
+- Python
+
+ This feature is under development.
+
{% endlist %}
-### Getting topic information
+### Getting topic information {#describe-topic}
{% list tabs %}
@@ -57,16 +74,23 @@ When you update a topic, you must specify the topic path and the parameters to b
```go
descResult, err := db.Topic().Describe(ctx, "topic-path")
- if err != nil {
- log.Fatalf("failed drop topic: %v", err)
- return
- }
- fmt.Printf("describe: %#v\n", descResult)
+ if err != nil {
+ log.Fatalf("failed drop topic: %v", err)
+ return
+ }
+ fmt.Printf("describe: %#v\n", descResult)
+ ```
+
+- Python
+
+ ```python
+ info = driver.topic_client.describe_topic(topic_path)
+ print(info)
```
{% endlist %}
-### Deleting a topic
+### Deleting a topic {#drop-topic}
To delete a topic, just specify the path to it.
@@ -78,10 +102,16 @@ To delete a topic, just specify the path to it.
err := db.Topic().Drop(ctx, "topic-path")
```
+- Python
+
+ ```python
+ driver.topic_client.drop_topic(topic_path)
+ ```
+
{% endlist %}
+## Message writes {#write}
-## Message writes
### Connecting to a topic for message writes {#start-writer}
Only connections with matching producer_id and message_group_id are currently supported. This restriction will be removed in the future.
@@ -100,10 +130,15 @@ Only connections with matching producer_id and message_group_id are currently su
}
```
-{% endlist %}
+- Python
+ ```python
+ writer = driver.topic_client.writer(topic_path)
+ ```
+
+{% endlist %}
-### Asynchronous message writes
+### Asynchronous message writes {#async-write}
{% list tabs %}
@@ -126,8 +161,39 @@ Only connections with matching producer_id and message_group_id are currently su
}
```
-{% endlist %}
+- Python
+
+ To deliver messages, you can either simply transmit message content (bytes, str) or set certain properties manually. You can send objects one-by-one or as a list. The `write` method is asynchronous. The method returns immediately once messages are put to the client's internal buffer; this is usually a fast process. If the internal buffer is filled up, you might need to wait until part of the data is sent to the server.
+ ```python
+ # Simple delivery of messages, without explicit metadata.
+ # Easy to get started, easy to use if everything you need is the message content.
+ writer = driver.topic_client.writer(topic_path)
+ writer.write("mess") # Rows will be transmitted in UTF-8; this is the easiest way to send
+ # text messages.
+ writer.write(bytes([1, 2, 3])) # These bytes will be transmitted as they are, this is the easiest way to send
+ # binary data.
+ writer.write(["mess-1", "mess-2"]) # This line multiple messages per call
+ # to decrease overheads on internal SDK processes.
+ # This makes sense when the message stream is high.
+
+ # This is the full form; it is used when except the message content you need to manually specify its properties.
+ writer = driver.topic_client.writer(topic="topic-path", auto_seqno=False, auto_created_at=False)
+
+ writer.write(ydb.TopicWriterMessage("asd", seqno=123, created_at=datetime.datetime.now()))
+ writer.write(ydb.TopicWriterMessage(bytes([1, 2, 3]), seqno=124, created_at=datetime.datetime.now()))
+
+ # In the full form, you can also send multiple messages per function call.
+ # This approach is useful when the message stream is high, and you want to
+ # reduce overheads on SDK internal calls.
+ writer.write([
+ ydb.TopicWriterMessage("asd", seqno=123, created_at=datetime.datetime.now()),
+ ydb.TopicWriterMessage(bytes([1, 2, 3]), seqno=124, created_at=datetime.datetime.now(),
+ ])
+
+ ```
+
+{% endlist %}
### Message writes with storage confirmation on the server
@@ -155,17 +221,44 @@ Only connections with matching producer_id and message_group_id are currently su
}
```
+- Python
+
+ There are two ways to get a message write acknowledgement from the server:
+
+ * `flush()`: Waits until all the messages previously written to the internal buffer are acknowledged.
+ * `write_with_ack(...)`: Sends a message and waits for the acknowledgement of its delivery from the server. This method is slow when you are sending multiple messages in a row.
+
+ ```python
+ # Put multiple messages to the internal buffer and then wait
+ # until all of them are delivered to the server.
+ for mess in messages:
+ writer.write(mess)
+
+ writer.flush()
+
+ # You can send multiple messages and wait for an acknowledgment for the entire group.
+ writer.write_with_ack(["mess-1", "mess-2"])
+
+ # Waiting on sending each message: this method will return the result only after an
+ # acknowledgment from the server.
+ # This is the slowest message delivery option; use it when this mode is
+ # absolutely needed.
+ writer.write_with_ack("message")
+
+ ```
+
{% endlist %}
-### Selecting a codec for message compression
+### Selecting a codec for message compression {#codec}
+
+By default, the SDK selects the codec automatically (subject to topic settings). In automatic mode, the SDK first sends one group of messages with each of the allowed codecs, then it sometimes tries to compress messages with all the available codecs, and then selects the codec that yields the smallest message size. If the list of allowed codecs for the topic is empty, the SDK makes automatic selection between Raw and Gzip codecs.
+
+If necessary, a fixed codec can be set in the connection options. It will then be used and no measurements will be taken.
{% list tabs %}
- Go
- By default, the SDK selects the codec automatically (subject to topic settings). In automatic mode, the SDK will first send one group of messages with each of the allowed codecs, then sometimes try to compress messages with all available codecs, and select the codec that provides the smallest message size. If the list of allowed codecs for the topic is empty, auto-select is made between Raw and Gzip codecs.
- If necessary, a fixed codec can be set in the connection options. It will then be used and no measurements will be taken.
-
```go
producerAndGroupID := "group-id"
writer, _ := db.Topic().StartWriter(producerAndGroupID, "topicName",
@@ -174,11 +267,18 @@ Only connections with matching producer_id and message_group_id are currently su
)
```
-{% endlist %}
+- Python
+ ```python
+ writer = driver.topic_client.writer(topic_path,
+ codec=ydb.TopicCodec.GZIP,
+ )
+ ```
+
+{% endlist %}
+## Reading messages {#reading}
-## Message reads
### Connecting to a topic for message reads {#start-reader}
To create a connection to the existing `my-topic` topic via the added `my-consumer` consumer, use the following code:
@@ -194,6 +294,12 @@ To create a connection to the existing `my-topic` topic via the added `my-consum
}
```
+- Python
+
+ ```python
+ reader = driver.topic_client.reader(topic="topic-path", consumer="consumer_name")
+ ```
+
{% endlist %}
You can also use the advanced connection creation option to specify multiple topics and set read parameters. The following code will create a connection to the `my-topic` and `my-specific-topic` topics via the `my-consumer` consumer and also set the time to start reading messages:
@@ -218,6 +324,10 @@ You can also use the advanced connection creation option to specify multiple top
}
```
+- Python
+
+ This feature is under development.
+
{% endlist %}
### Reading messages {#reading-messages}
@@ -250,6 +360,14 @@ To read messages one by one, use the following code:
}
```
+- Python
+
+ ```python
+ while True:
+ message = reader.receive_message()
+ process(message)
+ ```
+
{% endlist %}
To read message batches, use the following code:
@@ -270,6 +388,14 @@ To read message batches, use the following code:
}
```
+- Python
+
+ ```python
+ while True:
+ batch = reader.receive_batch()
+ process(batch)
+ ```
+
{% endlist %}
#### Reading with a commit {#commit}
@@ -293,6 +419,15 @@ To commit messages one by one, use the following code:
}
```
+- Python
+
+ ```python
+ while True:
+ message = reader.receive_message()
+ process(message)
+ reader.commit(message)
+ ```
+
{% endlist %}
To commit message batches, use the following code:
@@ -314,6 +449,15 @@ To commit message batches, use the following code:
}
```
+- Python
+
+ ```python
+ while True:
+ batch = reader.receive_batch()
+ process(batch)
+ reader.commit(batch)
+ ```
+
{% endlist %}
#### Reading with consumer offset storage on the client side {#client-commit}
@@ -364,6 +508,10 @@ When reading starts, the client code must transmit the starting consumer offset
}
```
+- Python
+
+ This feature is under development.
+
{% endlist %}
### Processing a server read interrupt {#stop}
@@ -395,6 +543,17 @@ In case of a _hard interruption_, the client receives a notification that it is
```
+- Python
+
+ No special processing is required.
+
+ ```python
+ while True:
+ batch = reader.receive_batch()
+ process(batch)
+ reader.commit(batch)
+ ```
+
{% endlist %}
#### Hard reading interruption {#hard-stop}
@@ -420,4 +579,21 @@ In case of a _hard interruption_, the client receives a notification that it is
}
```
+- Python
+
+ In this example, processing of messages within the batch will stop if the partition is reassigned during operation. This kind of optimization requires that you run extra code on the client side. In simple cases when processing of reassigned partitions is not a problem, you may skip this optimization.
+
+ ```python
+ def process_batch(batch):
+ for message in batch.messages:
+ if not batch.alive:
+ return False
+ process(message)
+ return True
+
+ batch = reader.receive_batch()
+ if process_batch(batch):
+ reader.commit(batch)
+ ```
+
{% endlist %}