special_tokens.h « tokenizer « cpp « library - ydb - Mirror of YDB github repos

about summary refs log tree commit diff stats

path: root/library/cpp/tokenizer/special_tokens.h

blob: 1a6dc686c7bb5aca8ecb88d392952600e02e47de (plain) (blame)

#pragma once

#include <util/charset/wide.h>

size_t GetSpecialTokenLength(const wchar16* text, size_t maxLen);